Commit graph

3296 commits

Author SHA1 Message Date
Ray Chin
702a779edb Fix overflow issue when computing ideal size of scratch partition
The type of f_frsize is `unsigned long` which is 32 bit for some system so it will overflow after multiplied by f_bfree. This solution adds one more type casting to ensure the type is 64 bit unsigned integer during the computing.

Bug: 281599020
Test: adb root; adb shell disable-verity
Change-Id: I377ed722d5e245c235c3ae12ff66ac7e91d1d6e8
2023-05-23 09:49:19 +00:00
David Anderson
bf11ce74ff Merge "libfiemap: Handle EAGAIN in fallocate()." 2023-05-22 17:39:07 +00:00
David Anderson
2fb1c671d5 Merge "ueventd: Fix a race condition in handling device-mapper events." 2023-05-19 19:43:21 +00:00
David Anderson
0fa371076a libfiemap: Handle EAGAIN in fallocate().
When we changed our ENOSPC tests, it exposed a path in F2FS were
fallocate can return EAGAIN. This is expected if F2FS attempts GC to
acquire more chunks, and it can leave the file in a partially allocated
state.

As a fix, keep attempting fallocate() as long as (1) it returns EAGAIN,
and (2) the allocated size keeps growing. If (2) fails we return ENOSPC.

Bug: N/A
Test: treehugger
Change-Id: I5f867b5a200b9260e486985f203f9872a949b3f9
2023-05-17 15:52:33 -07:00
David Anderson
59abbfe647 ueventd: Fix a race condition in handling device-mapper events.
We've had flake in libdm_test for a long time, with no clear cause.
Lately however it has become particularly reproducible when running
the UeventAfterLoadTable test in isolation, and thus we've identified
the root cause.

uevents for device-mapper are fired when the sysfs node is added, but at
that time, the "dm" subnode has not yet been added. The root node and dm
node are added very close together, so usually it works, but sometimes
ueventd is too fast.

Instead of relying on sysfs, query the uuid/name node directly from
device-mapper.

Bug: 270183812
Test: libdm_test
Change-Id: I258de5de05d813c3cb7f129e82e56dbfe8bf3117
2023-05-17 15:52:16 -07:00
Treehugger Robot
9f6e88567b Merge "Support ZSTD in userspace COW" 2023-05-16 19:02:57 +00:00
Kelvin Zhang
32dcac7851 Support ZSTD in userspace COW
Perf:
Lz4:
Update took 429 seconds
merge time 35.4s
cow size: 3.18GB

ZSTD:
Update took 676 seconds
merge time 49.4s
cow size: 2.62GB

Gz:
Update took 1057 seconds
merge time: 50.0s
cow size: 2.55GB

In summary, ZSTD is a middle point between LZ4 and GZ.

Speed: LZ4 > ZSTD > GZ
Space: LZ4 > ZSTD > GZ

Bug: 274129758

Change-Id: I203bf088b7c2a9ce429f75478799da0e7126febf
2023-05-16 10:13:52 -07:00
David Anderson
25613816d8 Merge "libsnapshot: Fix flaky low-space tests." 2023-05-10 18:29:56 +00:00
David Anderson
d90545bc33 libsnapshot: Fix flaky low-space tests.
These tests have been a plague of flakiness forever, and while we've
made improvements, the test is fundamentally flaky, and keeps showing up
in presubmit.

This change removes the code that attempts to fill /data until it's
almost full, which is unreliable and destructive. Instead, attempt to
allocate a file that is one block smaller than the maximum file size.
This should always fail with ENOSPC, as opposed to the maximum file size
which would fail with EFBIG.

Bug: N/A
Test: vts_libsnapshot_test
Change-Id: I4652b83e31c50975ddb50b6cbe846e102ba0f322
2023-05-09 10:12:14 -07:00
Mitchell Wills
2da7808bad Fix the return values of DmTable::valid to match the expected type
Test: make libdm
Change-Id: Id2be81eafe5168be9bfb37a5bcabfbdba3de8fa7
2023-05-08 13:26:35 -07:00
Dmitrii Merkurev
82899e4190 Merge "remount: Increase scratch size from 50% to 85% of available data" 2023-05-04 04:29:04 +00:00
David Anderson
c274088e5c Merge changes I7ec850b5,I62aa8c7b
* changes:
  libsnapshot: Use pointers instead of references in ICowOpIter.
  libsnapshot: Move BLOCK_SZ to snapuserd.
2023-05-04 02:51:29 +00:00
David Anderson
f16b2f7988 Merge "snapuserd: Remove dead code from legacy snapuserd." 2023-05-03 22:53:17 +00:00
David Anderson
f9bdc146ff Merge changes Ie8ac02ad,I876f858a,Iccfd6e72
* changes:
  libsnapshot: Improve inspect_cow output.
  libsnapshot: Rename and clarify some methods in ICowOpIter.
  libsnapshot: Remove fallible ICowReader::GetHeader.
2023-05-03 16:52:11 +00:00
Dmitrii Merkurev
9292c09c06 remount: Increase scratch size from 50% to 85% of available data
adb remount clearly indicates that the user will use
a scratch partition to modify RO content. In this CL we're
trying to be more aggressive and take more space from
data to scratch to improve the developers experience.

Bug: 228945443
Test: launch cvd with updated value and check it reflects
Change-Id: Ied9cee6e98d3b2dd594babcf213071a80bd3cf14
Signed-off-by: Dmitrii Merkurev <dimorinny@google.com>
2023-05-03 05:39:51 +01:00
David Anderson
1318e76655 libsnapshot: Use pointers instead of references in ICowOpIter.
Most callers already converted the reference to a pointer, and pointers
will make it much easier to move around the op list.

Bug: 280529365
Test: builds
Change-Id: I7ec850b58a79c33e57b0e2602e1be953484daa46
2023-05-02 19:45:31 -07:00
David Anderson
4485a6dfa4 libsnapshot: Move BLOCK_SZ to snapuserd.
Block sizes should not be hardcoded, since the value is set in the
cow header. Move this to snapuserd to isolate use.

Bug: 280529365
Test: builds
Change-Id: I62aa8c7b795f9d258d4b6deb0779a536cba65d52
2023-05-02 19:45:25 -07:00
David Anderson
de20e57f08 libsnapshot: Improve inspect_cow output.
Clean up the formatting and unit display, and add a line displaying how
long it took to parse the COW.

Example of new output:

    Version: 2.0
    Header size: 38
    Footer size: 84
    Block size: 4096
    Merge ops: 0
    Readahead buffer: 2097152 bytes
    Footer: ops usage: 1005680 bytes
    Footer: op count: 50284
    Parse time: 6.77921ms
    Data ops: 50015
    Replace ops: 50015
    Zero ops: 0
    Copy ops: 0
    Xor ops: 0

Bug: 280529365
Test: inspect_cow /dev/block/mapper/product_b-cow
Change-Id: Ie8ac02adc004289cc3a08ef44aa3048280a2407f
2023-05-02 19:45:19 -07:00
David Anderson
82356c3f6a libsnapshot: Rename and clarify some methods in ICowOpIter.
Done and RDone are not exactly symmetrical, so rename them to match
their underlying STL behavior. Done becomes "AtEnd" and RDone becomes
"AtBegin".

Bug: 280529365
Test: builds
Change-Id: I876f858a7bef84d8b37c1c822fb42941fbcded25
2023-05-02 19:45:16 -07:00
David Anderson
359350250b libsnapshot: Remove fallible ICowReader::GetHeader.
This function cannot fail, so replace it with an infallible version.
This is also slightly more efficient since there's no copying involved.

Bug: 280529365
Test: builds
Change-Id: Iccfd6e72bc2192f4e1efda50cf544e1e956a9263
2023-05-02 19:45:12 -07:00
Akilesh Kailash
defe8381aa libsnapshot: Turn off vabc_legacy_tests on presubmit
Temporarily turn off these tests until root cause is found.

Bug: 279009697
Test: presubmit
Change-Id: I90f695fac318b71871ff60c1f74c90265437687d
Signed-off-by: Akilesh Kailash <akailash@google.com>
2023-05-02 20:40:25 +00:00
David Anderson
467af3d67b libsnapshot: Remove the IByteSink API.
Bug: 278637212
Test: builds
Change-Id: I355aa1e7b1f67352848f2659489a7b831ac89cf8
2023-05-02 08:21:06 -07:00
David Anderson
3baad9e820 snapuserd: Remove IByteSink usage from snapuserd.
This added some new failure paths, so the patch also includes an RAII
helper to assist with handling those.

Bug: 278637212
Test: vts_libsnapshot_test
      apply ota on CF
Change-Id: Ide59c83cb34b6d448fe9afdc1d15c6139a0c99fa
2023-04-28 14:53:41 -07:00
David Anderson
78d8d5d9de snapuserd: Remove IByteSink usage from legacy snapuserd.
Bug: 278637212
Test: vabc_legacy_tests
      apply ota on CF w/ userspace snapshots disabled
Change-Id: Ia17bcc862ed9a7d7b78732139f457cac58fa79b9
2023-04-28 14:53:41 -07:00
David Anderson
ed4c853d14 libsnapshot: Disable another legacy merge path in tests.
Bug: 279009697
Test: presubmit
Change-Id: I1954b38f497bb8e4f7aa288e80541ac9ad5c075c
2023-04-28 14:53:41 -07:00
David Anderson
112dfd1754 libsnapshot: Replace IByteSink usage in snapshot_reader.
Bug: 278637212
Test: vts_libsnapshot_test
Change-Id: I2e684c91ed4a81c984d4bd6a3a9f1e0221aaee01
2023-04-27 08:03:41 -07:00
David Anderson
e280e2bea9 libsnapshot: Replace IByteSink usage in inspect_cow.
Bug: 278637212
Test: inspect_cow
Change-Id: I69f41b23a5bf66bcbaed10a64aba171ad4bbc987
2023-04-25 17:07:43 -07:00
David Anderson
1ea351b699 libsnapshot: Deprecate the IByteSink API.
This introduces a replacement for the IByteSink API, which was never
really used in any advantageous way, and is not expected to be useful.
The new API attempts to fill an entire buffer rather than request
individual slices of a buffer.

This patch simply introduces the API and refactors tests. Subsequent
patches will replace IByteSink callers and remove the API.

Bug: 278637212
Test: cow_api_test
Change-Id: Ib740de5e65fee8d61f603b106752338cc8e95967
2023-04-25 17:07:43 -07:00
David Anderson
405e74f9ce libsnapshot: Disable legacy merge tests.
These are failing on 6.1 kernels. Disabling them since legacy merges +
6.1 is not a supported combination.

Bug: 279009697
Test: vabc_legacy_tests on 6.1 CF
Change-Id: Iddc86f858e1a0101a7823b79fd0e81e221797f33
2023-04-24 15:40:29 -07:00
David Anderson
5c4a265514 snapuserd: Remove dead code from legacy snapuserd.
This code is now dead post xor removal.

Bug: N/A
Test: builds
Change-Id: I16e8357b9ee5c731441c57d7a345a4186d5269c2
2023-04-24 19:26:34 +00:00
David Anderson
657960396c Merge "snapuserd: Remove legacy xor code." 2023-04-14 23:03:39 +00:00
David Anderson
2812fd841e Merge "libsnapshot: Remove various unused components." 2023-04-14 22:42:42 +00:00
David Anderson
3933cbeba0 libsnapshot: Remove various unused components.
PowerTest is no longer relevant with snapuserd, and was not particularly
good at rooting out dm issues anyway.

make_cow_from_ab_ota and estimate_cow_from_nonab_ota are no longer
relevant as they were mainly for estimation during VABC prototyping.

The update_metadata proto was part of the fuzzer which has since been
removed.

Bug: N/A
Test: th builds
Change-Id: Ie341e360f1824ea16794c4625817cfc8f8b8644b
2023-04-14 13:39:14 -07:00
David Anderson
ff92fdb6d3 libsnapshot: Don't run legacy tests in VTS.
Bug: 276035159
Test: builds
Change-Id: I19bca32f636da867445c8b668d3a38ccba5c50f4
2023-04-13 16:39:02 -07:00
David Anderson
07f6f239ee Merge "RESTRICT AUTOMERGE: snapuserd: Run snapuserd threads in-process." 2023-04-13 17:55:40 +00:00
mingzhu.wang
44d88ca703 snapuserd_daemon: Modify incorrect print information
StartServerForUserspaceSnapshots function, if the child resulting from
the separation parameter number is not equal to 4, will be print
Malformed message: expected four sub-arguments, but at present there is
print three

Change-Id: Idc240714a65bc3eeb1e2b9958354de98ca4b329e
2023-04-13 17:55:47 +08:00
mingzhu.wang
d5d7d20d3b dmctl: add repalce command to usage list
If you need to use the replace command, but you can't find it in the
usage list.

Change-Id: I8618ce4b0eff30507fc8846f76aaf858b1a6895e
Signed-off-by: mingzhu.wang <mingzhu.wang@transsion.com>
2023-03-31 15:20:06 +08:00
David Anderson
e74976cccd Merge changes Ib90ae553,I3ca3ad9c
* changes:
  RESTRICT AUTOMERGE: snapuserd: Remove DaemonOps.
  RESTRICT AUTOMERGE: snapuserd: Split the server into two classes.
2023-03-23 18:29:40 +00:00
David Anderson
0172b1cc34 snapuserd: Remove legacy xor code.
Legacy snapuserd had xor optimizations implemented since at the time the
userspace snapshot implementation was a work in progress. We forgot to
ultimately remove the legacy xor code however, so let's do that now.

This should have no impact on devices. An S-frozen device will not be
getting this code because there's no vendor update happening. A T-frozen
device will be using userspace snapshots.

The only scenario this could happen is if a device starts on S vendor,
upgrades to T vendor, enables xor compression, and then updates to U
vendor. This configuration is not supported. Any device with a T+ vendor
should be using userspace snapshots.

Bug: N/A
Test: vabc_legacy_tests
Change-Id: If484a53cb0527393898a7ea703ba6f12d4c9a310
2023-03-23 14:46:28 +00:00
zhangyongpeng
18b3fda9b7 Remove unused variable mount_point
Since aosp/929692, this variable has been discard.

Test: manual

Signed-off-by: zhangyongpeng <zhangyongpeng@xiaomi.com>
Change-Id: I164766c9af06c0c19bc19b258cb866ed7cf36853
2023-03-23 14:14:21 +08:00
David Anderson
1f527c34dd RESTRICT AUTOMERGE: snapuserd: Run snapuserd threads in-process.
This removes the temporary daemon and socket and instead directly embeds
SnapshotHandlerManager. We can also remove the test flags for io_uring
as with this they're no longer necessary.

Bug: 269361087
Test: snapuserd_test
Change-Id: I728186d9bc90c98fabd76d3f86569840488ada96
2023-03-22 19:30:11 +00:00
David Anderson
369c993f5d RESTRICT AUTOMERGE: snapuserd: Remove DaemonOps.
These are only used for a single switch statement. Just compare the
input strings instead.

Bug: 269361087
Test: builds, ota applies

Change-Id: Ib90ae55309ea99c181585c64ed2ac814e5ed6c72
2023-03-22 19:30:04 +00:00
David Anderson
4983e1d33c RESTRICT AUTOMERGE: snapuserd: Split the server into two classes.
This splits server into two classes: one that handles the IPC requests,
and one that managers snapshot handlers. This will allow embedding of
the snapshot handling code, without booting up a server. It'll also make
testing much easier.

The handler code has been further placed behind a virtual interface,
though this is likely unnecessary unless we start testing the server
logic itself (or attempt to unify the S and T+ versions of snapuserd).

Bug: 269361087
Test: ota
Change-Id: I3ca3ad9c8c1fb910a1c7bf9f44165e2f44c930c9
2023-03-22 19:29:15 +00:00
Sam Dubey
55f9fa3911 Remove the libsnapshot fuzzer.
Follow u aosp/2480175

Test Breakage: http://shortn/_YhuHhWTtnw

(cherry picked from https://googleplex-android-review.googlesource.com/q/commit:a3faea8551c03d55924f013b479e869dda8fb166)
Change-Id: I82b0bc9be70fab249b17d6a5b96af09ee14b1586
2023-03-14 11:23:15 +00:00
David Anderson
10c200db28 Merge changes Ife319450,I2ecf838b
* changes:
  snapshotctl: Display merge progress and phases.
  Remove the libsnapshot fuzzer.
2023-03-13 19:27:06 +00:00
David Anderson
8845b1d9c7 Merge changes I262391dd,I18aad4a5,I0786e475
* changes:
  snapuserd: Move socket and io_uring status out of core and into the server.
  snapuserd: Rename UserSnapshotDmUserHandler to HandlerThread.
  snapuserd: Split out the implementation into a libsnapuserd component.
2023-03-13 18:53:28 +00:00
David Anderson
efcc9ae010 snapshotctl: Display merge progress and phases.
Bug: N/A
Test: snapshotctl dump during merge phase
Change-Id: Ife319450b28c8c58268a03127e9f58c47fc224d8
2023-03-13 10:53:54 -07:00
David Anderson
6aaae78047 Remove the libsnapshot fuzzer.
The only bugs in the corpus here were bugs in the fuzzer itself. Part
of the reason is its speed - it's limited by the dm layer so can't fuzz
optimally. That also makes it difficult to maintain, since it's
effectively a functional OTA fuzzer, and is very sensitive to the
implementation of libsnapshot.

After discussing this through with the team, we think the best course is
to sunset this and aim for improved unit and functional OTA testing.

Bug: 227656961
Test: builds
Change-Id: I2ecf838be336ee06459988ef282f0694cebfce51
2023-03-09 08:35:49 -08:00
David Anderson
4c17a07053 libsnapshot: Fix test failures on certain configurations.
Due to how CF is built and tested, VABC is enabled even when not
supported by the kernel. To work around this add some logic in
libsnapshot and the test harness to recognize this situation and
silently flip off the VABC flag.

This also fixes the -force_mode option to vts_libsnapshot_test, so that
it will skip tests that aren't supported by the device.

Bug: 264279496
Test: vts_libsnapshot_test on 11-5.4 with vabc enabled
Change-Id: I4c1e88fac54732c21c88169a7e0dd664e3858b89
2023-03-06 23:21:25 +00:00
Kelvin Zhang
b5a9607232 Initialize COW options using update manifest
Update manifest contains feature specifications such as whether to use
multi-threading or batched writes, initialize COW options using the
manifest allows these features to be enabled/disabled at update install
time.

Test: th
Change-Id: Iff66c43567d4890ae1c88624db49014584b1d96f
2023-02-23 13:21:20 -08:00