Commit graph

3313 commits

Author SHA1 Message Date
David Anderson
8542f3e4bb Merge "libsnapshot: Remove ISnapshotWriter." 2023-06-21 22:41:18 +00:00
Jiyong Park
e09de2a5ca libdm: don't expect uevent for devices that have never been activated
Previously, libdm ensured that uevent will be generated when a dm device
is deleted. However, there actually are cases where uevent can't be
expected; for example, if an empty dm device is created, but then gets
deleted without being activated, then there should be no uevent upon
deleting because there actually is nothing to remove from /dev. (dm
device gets added to /dev only if it is activated).

Actually such a case exists in apexd. As a performance optimization, it
proactively creates empty dm devices for all APEXes it scanns. But
some of them don't ever get activated if the APEX is already in a
dm-verity protected block device (i.e. the pre-instaled APEX is the
latest one). In that cases, the empty dm devices are deleted at the end
of the boot process as a clean up.

The libdm triggered error during the clean up, because there was no
uevent generated for the empty dm devices. This has triggered a lot of
false alarms to the apex team.

This CL fixes this by ensuring uevent only for activate dm devices. If
the dm device doesn't show up in /dev, we just delete the in-kernel
object and don't expect it to generate uevent for it - which actually is
the kernel's behavior.

Bug: 286176029
Test: build and run aosp_cf_x86_64_phone. observe dmesg.

Before the change:
I apexd   : Deleting unused dm device com.android.hardware.core_permissions
E apexd   : Didn't generate uevent for [com.android.hardware.core_permissions] removal
W apexd   : Failed to delete dm-device com.android.hardware.core_permissions

After the change:
I apexd   : Deleting unused dm device com.android.hardware.core_permissions
<no error or warning logs>

Change-Id: I52042de7d4d9ab62e6a13428c32a67e13395d1b5
2023-06-22 00:43:32 +09:00
David Anderson
5eae2cb450 libsnapshot: Remove ISnapshotWriter.
This only added one real method to ICowWriter, so let's just fold it
into ICowWriter. CompressedSnapshotWriter goes away as well.
CompressedSnapshotReader is now part of libsnapshot_cow.

Bug: 280529365
Test: m otapackage
      apply ota
Change-Id: I55358c3c9be111d5aee1a0c22c29cb1539d05494
2023-06-14 12:03:58 -07:00
David Anderson
4ac7d0e82e snapuserd: Create snapuserd_verify.h.
Bug: N/A
Test: builds
Change-Id: I380bf705520858dbec8053432c0a7c3fa8bed26d
2023-06-14 08:43:49 -07:00
David Anderson
1779dcca32 snapuserd: Create snapuserd_readahead.h.
This moves ReadAhead out of snapuserd_core.h so it's easier to find.

Bug: N/A
Test: builds
Change-Id: I888b87921950f2f7582b94c078e9e136f8fdd09e
2023-06-14 08:43:49 -07:00
David Anderson
874fdaed43 libsnapshot: Add CowWriterBase, clean up CowWriter.
To support multiple implementations of CowWriter, we will need to move
direct usage of CowWriter to ICowWriter. This CL does this while also
adding some small cleanups:

- Move ICowWriter implementation methods to a new CowWriterBase class.
  This keeps ICowWriter as a clean interface.
- Make the "Add" methods pure virtual, move the "Emit" methods to
  CowWriterBase as an implementation detail.
- Simplify Initialize/InitializeAppend so they can be shared.
- Rename CowWriter to CowWriterV2.
- Rename cow_writer.cpp to writer_v2.cpp.
- Rename cow_api_test.cpp to test_v2.cpp.
- Remove ICowWriter::options, replace with GetBlockSize.
- Add a CreateCowWriter helper to avoid implementation details in
  update_engine.

Bug: 280529365
Test: builds
Change-Id: If50faf03b292c6c8b23a6170e3f37329fb759ff6
2023-06-07 20:42:14 -07:00
David Anderson
aa45319453 Merge changes Ie2f531bd,I7e418ffe
* changes:
  libsnapshot: Remove OnlineKernelSnapshotWriter.
  libsnapshot: Remove unused SupportsCopyOperation.
2023-06-07 15:20:11 +00:00
David Anderson
8aee6d4a41 Merge "vts_fs_test: Relax filesystem constraints for fixed partitions." 2023-06-07 04:10:26 +00:00
David Anderson
084c94e43e vts_fs_test: Relax filesystem constraints for fixed partitions.
Adjust this test to align with VSR. First, skip the test if kernel is
< 5.10 or < Android 13. Second, allow non-dynamic partitions to be vfat.

Bug: 286038059
Test: vts_fs_test on CF
Change-Id: I5bd874f68296f7155d8c4366ebc13fe3d59c3ee6
2023-06-06 16:46:21 -07:00
David Anderson
82804e5dd0 Merge changes Ibc0f9818,I29b5617e
* changes:
  libsnapshot: Reject bad cow versions.
  libsnapshot: Split CowReader into CowParserV2.
2023-06-06 22:59:01 +00:00
Eric Biggers
b9be43ae5c Merge "fs_mgr_fstab: allow fileencryption without equals sign" 2023-06-06 19:43:49 +00:00
David Anderson
6cef6905d7 Move ENOSPC tests to libfiemap.
These tests are still giving us trouble. Move them to libfiemap, which
(1) is closer to the source of implementation, and (2) allows us to
re-use the temporary filesystem code. This won't perturb the state of
the actual device.

The new test creates a 64MB ext4 or f2fs mount point as a sandbox, which
should be much safer.

Bug: 285197715
Test: fiemap_writer_test
Change-Id: I33502d49613be4f269a80e5c632514fc56a0246a
2023-06-05 15:06:29 -07:00
Eric Biggers
cc4a15b97a fs_mgr_fstab: allow fileencryption without equals sign
A point of confusion that has been encountered recently is that the
fileencryption argument is optional (since Android 11), yet the fstab
parser requires the equals sign in "fileencryption=".  Thus, someone
wanting to use the default options must use "...,fileencryption=,...".

Make "fileencryption" by itself mean the same thing so that it works as
expected.

Test: atest CtsFsMgrTestCases
Change-Id: I65ce6b9513942bec2107838396835e7aafb3bf37
2023-06-05 14:35:11 +00:00
Ray Chin
702a779edb Fix overflow issue when computing ideal size of scratch partition
The type of f_frsize is `unsigned long` which is 32 bit for some system so it will overflow after multiplied by f_bfree. This solution adds one more type casting to ensure the type is 64 bit unsigned integer during the computing.

Bug: 281599020
Test: adb root; adb shell disable-verity
Change-Id: I377ed722d5e245c235c3ae12ff66ac7e91d1d6e8
2023-05-23 09:49:19 +00:00
David Anderson
95b26e1d0c libsnapshot: Remove OnlineKernelSnapshotWriter.
This was never used or tested. The idea was to unify update_engine's
write paths on ISnapshotWriter. But given that legacy VAB is "legacy",
it doesn't make sense to begin refactoring that code and potentially
introducing bugs. Let's just remove this instead.

Bug: 280529365
Test: builds
Change-Id: Ie2f531bd140e183dfde4b65a144f3b4acc28e78a
2023-05-22 11:00:11 -07:00
David Anderson
0ec7115779 libsnapshot: Reject bad cow versions.
Remove CowWriter::GetCowVersion. Instead, reject any update that asks
for VABC but has an unsupported version field. Do not fallback to legacy
VAB as the update was likely built improperly.

Bug: 280529365
Test: vts_libsnapshot_test
Change-Id: Ibc0f981801fd47bf39d7a19944134e4b3c66e5bf
2023-05-22 11:00:11 -07:00
David Anderson
85c65e98cd libsnapshot: Remove unused SupportsCopyOperation.
This function is never used. It was intended to allow update_engine to
unify all writes through ISnapshotWriter, but that never came to pass.

Bug: 280529365
Test: builds
Change-Id: I7e418ffee404cd63faa5a5659d8971988f8d0e03
2023-05-22 11:00:11 -07:00
David Anderson
d70a174e95 libsnapshot: Split CowReader into CowParserV2.
Remove format-specific logic from CowReader and split it out into a new
class called CowParserV2. To make reading the header easier, the
version and size bits are now in a separate CowHeaderPrefix struct.

Bug: 280529365
Test: apply OTA on CF
      inspect_cow
Change-Id: I29b5617ec094d4fb0c284485883d2e921a5bdbf8
2023-05-22 11:00:09 -07:00
David Anderson
bf11ce74ff Merge "libfiemap: Handle EAGAIN in fallocate()." 2023-05-22 17:39:07 +00:00
David Anderson
2fb1c671d5 Merge "ueventd: Fix a race condition in handling device-mapper events." 2023-05-19 19:43:21 +00:00
David Anderson
0fa371076a libfiemap: Handle EAGAIN in fallocate().
When we changed our ENOSPC tests, it exposed a path in F2FS were
fallocate can return EAGAIN. This is expected if F2FS attempts GC to
acquire more chunks, and it can leave the file in a partially allocated
state.

As a fix, keep attempting fallocate() as long as (1) it returns EAGAIN,
and (2) the allocated size keeps growing. If (2) fails we return ENOSPC.

Bug: N/A
Test: treehugger
Change-Id: I5f867b5a200b9260e486985f203f9872a949b3f9
2023-05-17 15:52:33 -07:00
David Anderson
59abbfe647 ueventd: Fix a race condition in handling device-mapper events.
We've had flake in libdm_test for a long time, with no clear cause.
Lately however it has become particularly reproducible when running
the UeventAfterLoadTable test in isolation, and thus we've identified
the root cause.

uevents for device-mapper are fired when the sysfs node is added, but at
that time, the "dm" subnode has not yet been added. The root node and dm
node are added very close together, so usually it works, but sometimes
ueventd is too fast.

Instead of relying on sysfs, query the uuid/name node directly from
device-mapper.

Bug: 270183812
Test: libdm_test
Change-Id: I258de5de05d813c3cb7f129e82e56dbfe8bf3117
2023-05-17 15:52:16 -07:00
Treehugger Robot
9f6e88567b Merge "Support ZSTD in userspace COW" 2023-05-16 19:02:57 +00:00
Kelvin Zhang
32dcac7851 Support ZSTD in userspace COW
Perf:
Lz4:
Update took 429 seconds
merge time 35.4s
cow size: 3.18GB

ZSTD:
Update took 676 seconds
merge time 49.4s
cow size: 2.62GB

Gz:
Update took 1057 seconds
merge time: 50.0s
cow size: 2.55GB

In summary, ZSTD is a middle point between LZ4 and GZ.

Speed: LZ4 > ZSTD > GZ
Space: LZ4 > ZSTD > GZ

Bug: 274129758

Change-Id: I203bf088b7c2a9ce429f75478799da0e7126febf
2023-05-16 10:13:52 -07:00
David Anderson
25613816d8 Merge "libsnapshot: Fix flaky low-space tests." 2023-05-10 18:29:56 +00:00
David Anderson
d90545bc33 libsnapshot: Fix flaky low-space tests.
These tests have been a plague of flakiness forever, and while we've
made improvements, the test is fundamentally flaky, and keeps showing up
in presubmit.

This change removes the code that attempts to fill /data until it's
almost full, which is unreliable and destructive. Instead, attempt to
allocate a file that is one block smaller than the maximum file size.
This should always fail with ENOSPC, as opposed to the maximum file size
which would fail with EFBIG.

Bug: N/A
Test: vts_libsnapshot_test
Change-Id: I4652b83e31c50975ddb50b6cbe846e102ba0f322
2023-05-09 10:12:14 -07:00
Mitchell Wills
2da7808bad Fix the return values of DmTable::valid to match the expected type
Test: make libdm
Change-Id: Id2be81eafe5168be9bfb37a5bcabfbdba3de8fa7
2023-05-08 13:26:35 -07:00
Dmitrii Merkurev
82899e4190 Merge "remount: Increase scratch size from 50% to 85% of available data" 2023-05-04 04:29:04 +00:00
David Anderson
c274088e5c Merge changes I7ec850b5,I62aa8c7b
* changes:
  libsnapshot: Use pointers instead of references in ICowOpIter.
  libsnapshot: Move BLOCK_SZ to snapuserd.
2023-05-04 02:51:29 +00:00
David Anderson
f16b2f7988 Merge "snapuserd: Remove dead code from legacy snapuserd." 2023-05-03 22:53:17 +00:00
David Anderson
f9bdc146ff Merge changes Ie8ac02ad,I876f858a,Iccfd6e72
* changes:
  libsnapshot: Improve inspect_cow output.
  libsnapshot: Rename and clarify some methods in ICowOpIter.
  libsnapshot: Remove fallible ICowReader::GetHeader.
2023-05-03 16:52:11 +00:00
Dmitrii Merkurev
9292c09c06 remount: Increase scratch size from 50% to 85% of available data
adb remount clearly indicates that the user will use
a scratch partition to modify RO content. In this CL we're
trying to be more aggressive and take more space from
data to scratch to improve the developers experience.

Bug: 228945443
Test: launch cvd with updated value and check it reflects
Change-Id: Ied9cee6e98d3b2dd594babcf213071a80bd3cf14
Signed-off-by: Dmitrii Merkurev <dimorinny@google.com>
2023-05-03 05:39:51 +01:00
David Anderson
1318e76655 libsnapshot: Use pointers instead of references in ICowOpIter.
Most callers already converted the reference to a pointer, and pointers
will make it much easier to move around the op list.

Bug: 280529365
Test: builds
Change-Id: I7ec850b58a79c33e57b0e2602e1be953484daa46
2023-05-02 19:45:31 -07:00
David Anderson
4485a6dfa4 libsnapshot: Move BLOCK_SZ to snapuserd.
Block sizes should not be hardcoded, since the value is set in the
cow header. Move this to snapuserd to isolate use.

Bug: 280529365
Test: builds
Change-Id: I62aa8c7b795f9d258d4b6deb0779a536cba65d52
2023-05-02 19:45:25 -07:00
David Anderson
de20e57f08 libsnapshot: Improve inspect_cow output.
Clean up the formatting and unit display, and add a line displaying how
long it took to parse the COW.

Example of new output:

    Version: 2.0
    Header size: 38
    Footer size: 84
    Block size: 4096
    Merge ops: 0
    Readahead buffer: 2097152 bytes
    Footer: ops usage: 1005680 bytes
    Footer: op count: 50284
    Parse time: 6.77921ms
    Data ops: 50015
    Replace ops: 50015
    Zero ops: 0
    Copy ops: 0
    Xor ops: 0

Bug: 280529365
Test: inspect_cow /dev/block/mapper/product_b-cow
Change-Id: Ie8ac02adc004289cc3a08ef44aa3048280a2407f
2023-05-02 19:45:19 -07:00
David Anderson
82356c3f6a libsnapshot: Rename and clarify some methods in ICowOpIter.
Done and RDone are not exactly symmetrical, so rename them to match
their underlying STL behavior. Done becomes "AtEnd" and RDone becomes
"AtBegin".

Bug: 280529365
Test: builds
Change-Id: I876f858a7bef84d8b37c1c822fb42941fbcded25
2023-05-02 19:45:16 -07:00
David Anderson
359350250b libsnapshot: Remove fallible ICowReader::GetHeader.
This function cannot fail, so replace it with an infallible version.
This is also slightly more efficient since there's no copying involved.

Bug: 280529365
Test: builds
Change-Id: Iccfd6e72bc2192f4e1efda50cf544e1e956a9263
2023-05-02 19:45:12 -07:00
Akilesh Kailash
defe8381aa libsnapshot: Turn off vabc_legacy_tests on presubmit
Temporarily turn off these tests until root cause is found.

Bug: 279009697
Test: presubmit
Change-Id: I90f695fac318b71871ff60c1f74c90265437687d
Signed-off-by: Akilesh Kailash <akailash@google.com>
2023-05-02 20:40:25 +00:00
David Anderson
467af3d67b libsnapshot: Remove the IByteSink API.
Bug: 278637212
Test: builds
Change-Id: I355aa1e7b1f67352848f2659489a7b831ac89cf8
2023-05-02 08:21:06 -07:00
David Anderson
3baad9e820 snapuserd: Remove IByteSink usage from snapuserd.
This added some new failure paths, so the patch also includes an RAII
helper to assist with handling those.

Bug: 278637212
Test: vts_libsnapshot_test
      apply ota on CF
Change-Id: Ide59c83cb34b6d448fe9afdc1d15c6139a0c99fa
2023-04-28 14:53:41 -07:00
David Anderson
78d8d5d9de snapuserd: Remove IByteSink usage from legacy snapuserd.
Bug: 278637212
Test: vabc_legacy_tests
      apply ota on CF w/ userspace snapshots disabled
Change-Id: Ia17bcc862ed9a7d7b78732139f457cac58fa79b9
2023-04-28 14:53:41 -07:00
David Anderson
ed4c853d14 libsnapshot: Disable another legacy merge path in tests.
Bug: 279009697
Test: presubmit
Change-Id: I1954b38f497bb8e4f7aa288e80541ac9ad5c075c
2023-04-28 14:53:41 -07:00
David Anderson
112dfd1754 libsnapshot: Replace IByteSink usage in snapshot_reader.
Bug: 278637212
Test: vts_libsnapshot_test
Change-Id: I2e684c91ed4a81c984d4bd6a3a9f1e0221aaee01
2023-04-27 08:03:41 -07:00
David Anderson
e280e2bea9 libsnapshot: Replace IByteSink usage in inspect_cow.
Bug: 278637212
Test: inspect_cow
Change-Id: I69f41b23a5bf66bcbaed10a64aba171ad4bbc987
2023-04-25 17:07:43 -07:00
David Anderson
1ea351b699 libsnapshot: Deprecate the IByteSink API.
This introduces a replacement for the IByteSink API, which was never
really used in any advantageous way, and is not expected to be useful.
The new API attempts to fill an entire buffer rather than request
individual slices of a buffer.

This patch simply introduces the API and refactors tests. Subsequent
patches will replace IByteSink callers and remove the API.

Bug: 278637212
Test: cow_api_test
Change-Id: Ib740de5e65fee8d61f603b106752338cc8e95967
2023-04-25 17:07:43 -07:00
David Anderson
405e74f9ce libsnapshot: Disable legacy merge tests.
These are failing on 6.1 kernels. Disabling them since legacy merges +
6.1 is not a supported combination.

Bug: 279009697
Test: vabc_legacy_tests on 6.1 CF
Change-Id: Iddc86f858e1a0101a7823b79fd0e81e221797f33
2023-04-24 15:40:29 -07:00
David Anderson
5c4a265514 snapuserd: Remove dead code from legacy snapuserd.
This code is now dead post xor removal.

Bug: N/A
Test: builds
Change-Id: I16e8357b9ee5c731441c57d7a345a4186d5269c2
2023-04-24 19:26:34 +00:00
David Anderson
657960396c Merge "snapuserd: Remove legacy xor code." 2023-04-14 23:03:39 +00:00
David Anderson
2812fd841e Merge "libsnapshot: Remove various unused components." 2023-04-14 22:42:42 +00:00
David Anderson
3933cbeba0 libsnapshot: Remove various unused components.
PowerTest is no longer relevant with snapuserd, and was not particularly
good at rooting out dm issues anyway.

make_cow_from_ab_ota and estimate_cow_from_nonab_ota are no longer
relevant as they were mainly for estimation during VABC prototyping.

The update_metadata proto was part of the fuzzer which has since been
removed.

Bug: N/A
Test: th builds
Change-Id: Ie341e360f1824ea16794c4625817cfc8f8b8644b
2023-04-14 13:39:14 -07:00