Commit graph

3332 commits

Author SHA1 Message Date
Kelvin Zhang
786dac3d50 Update some fs_mgr/debuggerd to use getpagesize() instead of PAGE_SIZE
Test: th
Bug: 279808236
Change-Id: I9d30cfe19d2b1a7d624cc5425e4315dc6e3b2ad2
2023-06-27 10:50:07 -07:00
Treehugger Robot
8ec7d3da58 Merge "Revert "snapuserd: Make header_response a state variable."" 2023-06-27 06:55:22 +00:00
David Anderson
e2e8f55b81 Revert "snapuserd: Make header_response a state variable."
This reverts commit d4e035ebc3.

Reason for revert: Breaks incremental OTAs

Change-Id: Ib9703a66b83e08114ca4d11370d669b8bcdf4789
2023-06-27 05:29:20 +00:00
David Anderson
53bb327f29 Merge "snapuserd: Make header_response a state variable." 2023-06-26 22:51:41 +00:00
David Anderson
b41cd681aa Merge changes I07031e89,I1ba276e1
* changes:
  libsnapshot: Remove direct accesses of CowOperation::source and compression.
  libsnapshot: Add helpers for accessing CowOperation offsets.
2023-06-26 18:39:57 +00:00
Treehugger Robot
c5d1978666 Merge "Make libfstab available to APEXes." 2023-06-26 16:29:26 +00:00
Jiakai Zhang
cf16f4d794 Make libfstab available to APEXes.
The ART module needs this library to determine whether to put dexopt
artifacts in dalvik-cache.

Bug: 287958783
Test: m
Change-Id: Idf338702d4f54e9c40c0692ea29e7d83e91aca38
2023-06-23 22:28:49 +01:00
David Anderson
d4e035ebc3 snapuserd: Make header_response a state variable.
header_response is meant to only be true for the first call to
WriteDmUserPayload. Codify this by making it a member variable and
resetting it on each request.

Bug: 288273605
Test: snapuserd_test
Change-Id: Ia125f86b7f22f4801be1e0796e8f85540ed5f31f
2023-06-23 13:14:50 -07:00
David Anderson
cf311bd00f libsnapshot: Remove direct accesses of CowOperation::source and compression.
For the remaining use cases of accessing CowOperation::source and
compression, directly, introduce helper functions, since these fields
will be removed in the v3 format.

Bug: 280529365
Test: cow_api_test
Change-Id: I07031e8968525a7c1314ca45c284e476b51d8104
2023-06-23 13:14:50 -07:00
David Anderson
09ccef4961 libsnapshot: Add helpers for accessing CowOperation offsets.
These helpers avoid manual inspection of CowOperation::source, and
fiddling with block sizes.

Bug: 280529365
Test: cow_api_test
      vts_libsnapshot_test
      apply OTA
Change-Id: I1ba276e155eb8232a3115066caaa11030c171e11
2023-06-23 13:14:50 -07:00
Treehugger Robot
8790a71bc4 Merge "Remove dead code from fs_mgr" 2023-06-23 20:07:09 +00:00
Paul Lawrence
c7998f2abb Remove dead code from fs_mgr
Test: Compiles
Change-Id: I76402758396b658fc1539f81541b162723b709a2
2023-06-23 09:27:55 -07:00
David Anderson
27fb5200aa libsnapshot: Add a helper around CowReader::GetRawBytes.
This avoids having to use op->source which will require more work after
the new format lands.

Bug: 280529365
Test: inspect_cow
Change-Id: Iffa18974bb0152dad73d69bb03c683f1aa549031
2023-06-22 20:52:10 -07:00
Treehugger Robot
fe4066520e Merge "Only skip checkpointing for freshly formatted ext4" 2023-06-22 23:28:01 +00:00
David Anderson
9fd1147017 Merge changes Icaeba861,I19007a78,I11863dcb
* changes:
  inspect_cow: Add --show_raw_ops.
  inspect_cow: Switch to gflags.
  libsnapshot: Remove dependence between CowWriterV2 and CowReader.
2023-06-22 20:10:41 +00:00
Daniel Rosenberg
2127b87b70 Only skip checkpointing for freshly formatted ext4
"Do not reboot with commit failure when ext4 userdata is wiped" has
caused a regression in cuttlefish on f2fs. Overlay.img rapidly grows to
the full userdata size. This updates that CL to only affect partitions
marked for block based checkpointing.

Test: Boot Cuttlefish, monitor overlay.img for 5 minutes
Bug: 285019395
Change-Id: I7d8634e6535ee1531a0df9fb51c6f9c410cbfe92
2023-06-21 18:56:28 -07:00
David Anderson
8542f3e4bb Merge "libsnapshot: Remove ISnapshotWriter." 2023-06-21 22:41:18 +00:00
Jiyong Park
e09de2a5ca libdm: don't expect uevent for devices that have never been activated
Previously, libdm ensured that uevent will be generated when a dm device
is deleted. However, there actually are cases where uevent can't be
expected; for example, if an empty dm device is created, but then gets
deleted without being activated, then there should be no uevent upon
deleting because there actually is nothing to remove from /dev. (dm
device gets added to /dev only if it is activated).

Actually such a case exists in apexd. As a performance optimization, it
proactively creates empty dm devices for all APEXes it scanns. But
some of them don't ever get activated if the APEX is already in a
dm-verity protected block device (i.e. the pre-instaled APEX is the
latest one). In that cases, the empty dm devices are deleted at the end
of the boot process as a clean up.

The libdm triggered error during the clean up, because there was no
uevent generated for the empty dm devices. This has triggered a lot of
false alarms to the apex team.

This CL fixes this by ensuring uevent only for activate dm devices. If
the dm device doesn't show up in /dev, we just delete the in-kernel
object and don't expect it to generate uevent for it - which actually is
the kernel's behavior.

Bug: 286176029
Test: build and run aosp_cf_x86_64_phone. observe dmesg.

Before the change:
I apexd   : Deleting unused dm device com.android.hardware.core_permissions
E apexd   : Didn't generate uevent for [com.android.hardware.core_permissions] removal
W apexd   : Failed to delete dm-device com.android.hardware.core_permissions

After the change:
I apexd   : Deleting unused dm device com.android.hardware.core_permissions
<no error or warning logs>

Change-Id: I52042de7d4d9ab62e6a13428c32a67e13395d1b5
2023-06-22 00:43:32 +09:00
David Anderson
17421a93ba inspect_cow: Add --show_raw_ops.
This bypasses CowReader and shows ops directly from CowParserV2. This
will come in handy once CowOperation is versioned and CowReader will be
translating from older versions to new versions.

Bug: 280529365
Test: inspect_cow
Change-Id: Icaeba86105c31c4dfe2812309a4028dd1d0248c0
2023-06-15 11:18:25 -07:00
David Anderson
691b78a0fd inspect_cow: Switch to gflags.
gflags is a lot cleaner to use than getopt, and we don't really need any
advanced argument parsing.

Test: inspect_cow
Bug: N/A
Change-Id: I19007a787a2e5cd3a67486c7949b34ba76f762e4
2023-06-15 11:01:56 -07:00
David Anderson
76622d444d libsnapshot: Remove dependence between CowWriterV2 and CowReader.
The plan of record for the new CowOperation layout is to automatically
translate from CowOperationV2. However, CowWriterV2 currently imports
directly from CowReader, which means we'd need two translation steps.

Luckily we now have CowParserV2, which means we can directly import into
CowWriterV2.

Bug: 280529365
Test: vts_libsnapshot_test
Test: cow_api_test
Change-Id: I11863dcb483f0448686d6c492d8eb128840ce18c
2023-06-14 12:03:59 -07:00
David Anderson
5eae2cb450 libsnapshot: Remove ISnapshotWriter.
This only added one real method to ICowWriter, so let's just fold it
into ICowWriter. CompressedSnapshotWriter goes away as well.
CompressedSnapshotReader is now part of libsnapshot_cow.

Bug: 280529365
Test: m otapackage
      apply ota
Change-Id: I55358c3c9be111d5aee1a0c22c29cb1539d05494
2023-06-14 12:03:58 -07:00
David Anderson
4ac7d0e82e snapuserd: Create snapuserd_verify.h.
Bug: N/A
Test: builds
Change-Id: I380bf705520858dbec8053432c0a7c3fa8bed26d
2023-06-14 08:43:49 -07:00
David Anderson
1779dcca32 snapuserd: Create snapuserd_readahead.h.
This moves ReadAhead out of snapuserd_core.h so it's easier to find.

Bug: N/A
Test: builds
Change-Id: I888b87921950f2f7582b94c078e9e136f8fdd09e
2023-06-14 08:43:49 -07:00
David Anderson
874fdaed43 libsnapshot: Add CowWriterBase, clean up CowWriter.
To support multiple implementations of CowWriter, we will need to move
direct usage of CowWriter to ICowWriter. This CL does this while also
adding some small cleanups:

- Move ICowWriter implementation methods to a new CowWriterBase class.
  This keeps ICowWriter as a clean interface.
- Make the "Add" methods pure virtual, move the "Emit" methods to
  CowWriterBase as an implementation detail.
- Simplify Initialize/InitializeAppend so they can be shared.
- Rename CowWriter to CowWriterV2.
- Rename cow_writer.cpp to writer_v2.cpp.
- Rename cow_api_test.cpp to test_v2.cpp.
- Remove ICowWriter::options, replace with GetBlockSize.
- Add a CreateCowWriter helper to avoid implementation details in
  update_engine.

Bug: 280529365
Test: builds
Change-Id: If50faf03b292c6c8b23a6170e3f37329fb759ff6
2023-06-07 20:42:14 -07:00
David Anderson
aa45319453 Merge changes Ie2f531bd,I7e418ffe
* changes:
  libsnapshot: Remove OnlineKernelSnapshotWriter.
  libsnapshot: Remove unused SupportsCopyOperation.
2023-06-07 15:20:11 +00:00
David Anderson
8aee6d4a41 Merge "vts_fs_test: Relax filesystem constraints for fixed partitions." 2023-06-07 04:10:26 +00:00
David Anderson
084c94e43e vts_fs_test: Relax filesystem constraints for fixed partitions.
Adjust this test to align with VSR. First, skip the test if kernel is
< 5.10 or < Android 13. Second, allow non-dynamic partitions to be vfat.

Bug: 286038059
Test: vts_fs_test on CF
Change-Id: I5bd874f68296f7155d8c4366ebc13fe3d59c3ee6
2023-06-06 16:46:21 -07:00
David Anderson
82804e5dd0 Merge changes Ibc0f9818,I29b5617e
* changes:
  libsnapshot: Reject bad cow versions.
  libsnapshot: Split CowReader into CowParserV2.
2023-06-06 22:59:01 +00:00
Eric Biggers
b9be43ae5c Merge "fs_mgr_fstab: allow fileencryption without equals sign" 2023-06-06 19:43:49 +00:00
David Anderson
6cef6905d7 Move ENOSPC tests to libfiemap.
These tests are still giving us trouble. Move them to libfiemap, which
(1) is closer to the source of implementation, and (2) allows us to
re-use the temporary filesystem code. This won't perturb the state of
the actual device.

The new test creates a 64MB ext4 or f2fs mount point as a sandbox, which
should be much safer.

Bug: 285197715
Test: fiemap_writer_test
Change-Id: I33502d49613be4f269a80e5c632514fc56a0246a
2023-06-05 15:06:29 -07:00
Eric Biggers
cc4a15b97a fs_mgr_fstab: allow fileencryption without equals sign
A point of confusion that has been encountered recently is that the
fileencryption argument is optional (since Android 11), yet the fstab
parser requires the equals sign in "fileencryption=".  Thus, someone
wanting to use the default options must use "...,fileencryption=,...".

Make "fileencryption" by itself mean the same thing so that it works as
expected.

Test: atest CtsFsMgrTestCases
Change-Id: I65ce6b9513942bec2107838396835e7aafb3bf37
2023-06-05 14:35:11 +00:00
Ray Chin
702a779edb Fix overflow issue when computing ideal size of scratch partition
The type of f_frsize is `unsigned long` which is 32 bit for some system so it will overflow after multiplied by f_bfree. This solution adds one more type casting to ensure the type is 64 bit unsigned integer during the computing.

Bug: 281599020
Test: adb root; adb shell disable-verity
Change-Id: I377ed722d5e245c235c3ae12ff66ac7e91d1d6e8
2023-05-23 09:49:19 +00:00
David Anderson
95b26e1d0c libsnapshot: Remove OnlineKernelSnapshotWriter.
This was never used or tested. The idea was to unify update_engine's
write paths on ISnapshotWriter. But given that legacy VAB is "legacy",
it doesn't make sense to begin refactoring that code and potentially
introducing bugs. Let's just remove this instead.

Bug: 280529365
Test: builds
Change-Id: Ie2f531bd140e183dfde4b65a144f3b4acc28e78a
2023-05-22 11:00:11 -07:00
David Anderson
0ec7115779 libsnapshot: Reject bad cow versions.
Remove CowWriter::GetCowVersion. Instead, reject any update that asks
for VABC but has an unsupported version field. Do not fallback to legacy
VAB as the update was likely built improperly.

Bug: 280529365
Test: vts_libsnapshot_test
Change-Id: Ibc0f981801fd47bf39d7a19944134e4b3c66e5bf
2023-05-22 11:00:11 -07:00
David Anderson
85c65e98cd libsnapshot: Remove unused SupportsCopyOperation.
This function is never used. It was intended to allow update_engine to
unify all writes through ISnapshotWriter, but that never came to pass.

Bug: 280529365
Test: builds
Change-Id: I7e418ffee404cd63faa5a5659d8971988f8d0e03
2023-05-22 11:00:11 -07:00
David Anderson
d70a174e95 libsnapshot: Split CowReader into CowParserV2.
Remove format-specific logic from CowReader and split it out into a new
class called CowParserV2. To make reading the header easier, the
version and size bits are now in a separate CowHeaderPrefix struct.

Bug: 280529365
Test: apply OTA on CF
      inspect_cow
Change-Id: I29b5617ec094d4fb0c284485883d2e921a5bdbf8
2023-05-22 11:00:09 -07:00
David Anderson
bf11ce74ff Merge "libfiemap: Handle EAGAIN in fallocate()." 2023-05-22 17:39:07 +00:00
David Anderson
2fb1c671d5 Merge "ueventd: Fix a race condition in handling device-mapper events." 2023-05-19 19:43:21 +00:00
David Anderson
0fa371076a libfiemap: Handle EAGAIN in fallocate().
When we changed our ENOSPC tests, it exposed a path in F2FS were
fallocate can return EAGAIN. This is expected if F2FS attempts GC to
acquire more chunks, and it can leave the file in a partially allocated
state.

As a fix, keep attempting fallocate() as long as (1) it returns EAGAIN,
and (2) the allocated size keeps growing. If (2) fails we return ENOSPC.

Bug: N/A
Test: treehugger
Change-Id: I5f867b5a200b9260e486985f203f9872a949b3f9
2023-05-17 15:52:33 -07:00
David Anderson
59abbfe647 ueventd: Fix a race condition in handling device-mapper events.
We've had flake in libdm_test for a long time, with no clear cause.
Lately however it has become particularly reproducible when running
the UeventAfterLoadTable test in isolation, and thus we've identified
the root cause.

uevents for device-mapper are fired when the sysfs node is added, but at
that time, the "dm" subnode has not yet been added. The root node and dm
node are added very close together, so usually it works, but sometimes
ueventd is too fast.

Instead of relying on sysfs, query the uuid/name node directly from
device-mapper.

Bug: 270183812
Test: libdm_test
Change-Id: I258de5de05d813c3cb7f129e82e56dbfe8bf3117
2023-05-17 15:52:16 -07:00
Treehugger Robot
9f6e88567b Merge "Support ZSTD in userspace COW" 2023-05-16 19:02:57 +00:00
Kelvin Zhang
32dcac7851 Support ZSTD in userspace COW
Perf:
Lz4:
Update took 429 seconds
merge time 35.4s
cow size: 3.18GB

ZSTD:
Update took 676 seconds
merge time 49.4s
cow size: 2.62GB

Gz:
Update took 1057 seconds
merge time: 50.0s
cow size: 2.55GB

In summary, ZSTD is a middle point between LZ4 and GZ.

Speed: LZ4 > ZSTD > GZ
Space: LZ4 > ZSTD > GZ

Bug: 274129758

Change-Id: I203bf088b7c2a9ce429f75478799da0e7126febf
2023-05-16 10:13:52 -07:00
David Anderson
25613816d8 Merge "libsnapshot: Fix flaky low-space tests." 2023-05-10 18:29:56 +00:00
David Anderson
d90545bc33 libsnapshot: Fix flaky low-space tests.
These tests have been a plague of flakiness forever, and while we've
made improvements, the test is fundamentally flaky, and keeps showing up
in presubmit.

This change removes the code that attempts to fill /data until it's
almost full, which is unreliable and destructive. Instead, attempt to
allocate a file that is one block smaller than the maximum file size.
This should always fail with ENOSPC, as opposed to the maximum file size
which would fail with EFBIG.

Bug: N/A
Test: vts_libsnapshot_test
Change-Id: I4652b83e31c50975ddb50b6cbe846e102ba0f322
2023-05-09 10:12:14 -07:00
Mitchell Wills
2da7808bad Fix the return values of DmTable::valid to match the expected type
Test: make libdm
Change-Id: Id2be81eafe5168be9bfb37a5bcabfbdba3de8fa7
2023-05-08 13:26:35 -07:00
Dmitrii Merkurev
82899e4190 Merge "remount: Increase scratch size from 50% to 85% of available data" 2023-05-04 04:29:04 +00:00
David Anderson
c274088e5c Merge changes I7ec850b5,I62aa8c7b
* changes:
  libsnapshot: Use pointers instead of references in ICowOpIter.
  libsnapshot: Move BLOCK_SZ to snapuserd.
2023-05-04 02:51:29 +00:00
David Anderson
f16b2f7988 Merge "snapuserd: Remove dead code from legacy snapuserd." 2023-05-03 22:53:17 +00:00
David Anderson
f9bdc146ff Merge changes Ie8ac02ad,I876f858a,Iccfd6e72
* changes:
  libsnapshot: Improve inspect_cow output.
  libsnapshot: Rename and clarify some methods in ICowOpIter.
  libsnapshot: Remove fallible ICowReader::GetHeader.
2023-05-03 16:52:11 +00:00