Commit graph

3664 commits

Author SHA1 Message Date
David Anderson
f7d07db679 snapuserd: Test both async and sync paths.
Rather than split this into two binaries as we did for libsnapshot, this
uses test parameterization. I went this route because even for async
builds, the sync path must always work as a fallback.

Bug: N/A
Test: snapuserd_test
Change-Id: Ib68c8fd831c345f41a4e14a99fd0a0b8b1d47b6d
2023-10-16 20:23:46 +00:00
Daniel Zheng
452f176ca8 Merge "Add v3 to CreateCowWriter" into main 2023-10-16 15:20:47 +00:00
Yi-Yo Chiang
c0281978b0 Merge "fs_mgr: Don't run clean_scratch_files on non-dynamic devices" into main 2023-10-16 10:24:55 +00:00
Kelvin Zhang
9c949326bd Merge "Make partition metadata write atomic" into main 2023-10-14 00:01:41 +00:00
Kelvin Zhang
78d5352289 Make partition metadata write atomic
We have seen multiple OTA failures with invalid geometry magic
signature. Make partition metadata write atomic by writing to a tmpfile
first and then do a rename.

Test: th
Bug: 303770065
Bug: 298149189
Change-Id: Id1d565de73439b95b665144c2f02fc97273d341c
2023-10-13 15:46:28 -07:00
David Anderson
7e407514dd Remove unused variable in snapuserd_core.
Bug: N/A
Test: builds
Change-Id: I8fc3bd2dd8eb3cfa3c6bfe513f054a4d6a674de1
2023-10-13 21:34:12 +00:00
Daniel Zheng
cab8646a07 Add v3 to CreateCowWriter
Adding v3 writer as a return type to CreateCowWriter

Test: write_cow
Change-Id: Ia68ddf17b9f5f9fae27da4d6ab5fb150078d0c38
2023-10-12 14:10:50 -07:00
Kelvin Zhang
d6fe3032a9 Merge "Revert "Shove type into source_info"" into main 2023-10-11 20:50:02 +00:00
Kelvin Zhang
b6153adde1 Revert "Shove type into source_info"
This reverts commit e90a39aee2.

Reason for revert: 304602386

Change-Id: Ib9d31bdadc7511e67f4c7e5731d18e7bf0162bfc
2023-10-11 19:44:38 +00:00
Nolen Johnson
cb33fb4008 fs_mgr: Don't run clean_scratch_files on non-dynamic devices
* This results in a metric ton of denials on some devices and
  eats up valuable resources on boot, plus there's 0 need for it,
  so kill it.

Change-Id: Ic52d5b3f06724430e9505345024cf0041b37ca49
2023-10-11 13:58:40 -04:00
Edward Liaw
da3caeab43 Merge "liblp_test: Add to kernel-presubmit" into main 2023-10-11 16:53:26 +00:00
Edward Liaw
f5736cb42a Merge "libdm_test: Add to kernel-presubmit" into main 2023-10-11 16:53:10 +00:00
Edward Liaw
eafffc0c9c liblp_test: Add to kernel-presubmit
Bug: 284307085
Test: atest :kernel-presubmit
Change-Id: I2a138cf55a2af07b5c231f163196266f8b3dd524
Signed-off-by: Edward Liaw <edliaw@google.com>
2023-10-10 23:13:30 +00:00
Edward Liaw
942a31e7be libdm_test: Add to kernel-presubmit
Bug: 284307085
Test: atest :kernel-presubmit
Change-Id: I13ae25f17c943825bdf5ee67c00b7d297cf6e319
Signed-off-by: Edward Liaw <edliaw@google.com>
2023-10-10 23:05:09 +00:00
Daniel Zheng
bb45742ba1 Move Cow Header child writer
v2 writer and v3 writer will write different versions of the header.
It's better to have each writer hold it's own header.

Test: th
Change-Id: Ibe310d58b830950ad556aabcd0c1009483fc8d86
2023-10-10 15:06:26 -07:00
Daniel Zheng
ed1a7d5faf Add v3 Cow Header
Adding Version 3 of CowHeader. This will inherit from original
CowHeader and add a compression field. We are no longer supporting
different compressions per operation so having this one field is enough.

Test: cow_api_test
Change-Id: If88dceda139807cc5e647b706ddeb2b3e83c024f
2023-10-10 15:06:26 -07:00
Daniel Zheng
73b3428e1c Rename BLOCK_SZ to block_size
BLOCK_SZ name to block_size as we should reserve all caps for compile
time constants

Test: th
Change-Id: I542632662e81b272ffec50d6b50c290d701b08e3
2023-10-10 12:55:28 -07:00
Daniel Zheng
e90a39aee2 Shove type into source_info
We can shove type into source info to save 8 bits in per cow operation.
We only need 4 bits inside of source_info to enumerate all the types of
Cow Operation:

static constexpr uint8_t kCowCopyOp = 1;
static constexpr uint8_t kCowReplaceOp = 2;
static constexpr uint8_t kCowZeroOp = 3;
static constexpr uint8_t kCowLabelOp = 4;
static constexpr uint8_t kCowClusterOp = 5;
static constexpr uint8_t kCowXorOp = 6;
static constexpr uint8_t kCowSequenceOp = 7;
static constexpr uint8_t kCowFooterOp = -1;

Test: critical ota paths on pixel
Change-Id: I22049db0b39a55bd8f863339f3751d3146d5c1e9
2023-10-10 12:55:27 -07:00
Daniel Zheng
76b1e83fad Move ReadCowHeader to CowReader
Moving this function to cow reader first we have to read the version of
cow from the header before deciding which parser to use. This is a more
logical place for this code to be in

Test: th
Change-Id: Ie5ba53439b4fc3c4c409426b818a2d53c0cbc3e5
2023-10-10 12:55:27 -07:00
Daniel Zheng
c197df70d8 Update API usage for source
replacing any instance of op->source_info with GetCowSourceinfoData as
that's what we're really looking for. This function will return the 48
bits associated with the source and not the extra bits that will store
type + other information. We should never be making a reference to
op->source_info directly as that has no real meaning, we should be
masking for the data that we really need

Test: cow_api_test
Change-Id: I259c790efdd13c61a4599e9edfc75cf0ba000c61
2023-10-09 20:15:34 -07:00
Daniel Zheng
f9f833066d Move Sync to base class
This function will be called in header write for both v3 and v2 writer.
Seems okay to move to base class as theres no unique functionality to
the v2 writer.

Test: cow_api_test

Change-Id: I70c1b08ce67127c9dcbd0f54b574d2cd5ad1d0b5
2023-10-09 20:15:34 -07:00
Daniel Zheng
f7f3b62f24 Merge "Removing compression bit from v3 op" into main 2023-10-10 03:09:58 +00:00
Daniel Zheng
3062a64f06 Merge "Add v3 writer" into main 2023-10-10 02:25:38 +00:00
Daniel Zheng
ce57c58e6b Removing compression bit from v3 op
We don't need the compression bit in v3 op since all operations will
have the same compression per COW Device and it will be stored within the COW header.
We can check to see if an operation contains compressioned data by
checking data_length and see if it's less than BLOCK_SZ

Test: 4 critical OTA paths
Change-Id: I3f86756d83bf54bf6efd15d9cb7ac064eefdd949
2023-10-09 13:31:20 -07:00
Daniel Zheng
04e4c2a6c2 Add v3 writer
Adding v3 writer that works off of Cow Operation v3. Adding test file
that will test this new writer. Adding in stub implementations to v3 writer. None of these functions
have to work yet, we just need the implementations here to compile.

Test: m libsnapshot
Change-Id: If86437d5ceb2c33520d4ca26dea5193984f86546
2023-10-09 13:31:19 -07:00
Daniel Zheng
7248d1b0d8 Merge "Refactor off V2 Cow Ops" into main am: adad3dbefe
Original change: https://android-review.googlesource.com/c/platform/system/core/+/2736101

Change-Id: I0344daaa43ed555c0bcdd377629c70a167f2afa4
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-10-05 19:39:28 +00:00
Daniel Zheng
adad3dbefe Merge "Refactor off V2 Cow Ops" into main 2023-10-05 19:12:56 +00:00
Daniel Zheng
4cda9ec9bb Merge "Adding test for reader compatibility" into main am: 4b3b6e2ff5
Original change: https://android-review.googlesource.com/c/platform/system/core/+/2762279

Change-Id: I3720f7f966c54408c1c4dbe72222dac73ef6dbae
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-10-05 17:45:07 +00:00
Daniel Zheng
4b3b6e2ff5 Merge "Adding test for reader compatibility" into main 2023-10-05 17:04:17 +00:00
Akilesh Kailash
0d38654b34 Merge "snapuserd: I/O request on overlapping blocks during snapshot-merge." into main am: 743c4cdb1c
Original change: https://android-review.googlesource.com/c/platform/system/core/+/2769106

Change-Id: Ie5a42ec523a948382b25200d92cfb1105d972138
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-10-05 14:18:31 +00:00
Akilesh Kailash
cffa413de4 snapuserd: I/O request on overlapping blocks during snapshot-merge.
This fixes the case when all the following conditions are true:

1: Incremental OTA
2: When there are sequence of overlapping COPY operations within one merge-window
   (510 blocks)
3: Device is rebooted when snapshot-merge is in-progress of this
   merge-window. When device reboots, the state of merge-window (of 510 blocks) was
   merge-in-progress (aka - only partial set of blocks were merged in
   this window thereby the state of the base device is in-complete for
   this window)
4: During the next boot, if there any I/O request from the filesystem
   which maps to the merge-window in (3):
   	a: The data has to be retrieved from the scratch space of the
	COW until the snapshot-merge for that window is completed.

	b: Once the snapshot-merge is complete for that window, data
	has to be retrieved from base device.

The bug was in step 4(a) wherein I/O request was getting routed to base
device.

This patch addresses the above flow by fixing step 4(a).

A new vts test has been added to explicitly track this issue.

Additionally, there is no need to re-scan the partition if partition is in merge resume path. This should cut down the overhead of the scan.

Bug: 275296365
Test: 1: 100 iterations of ./vts_snapuserd_test --gtest_filter=SnapuserdTest.Snapshot_COPY_Overlap_Merge_Resume_IO_Validate_TEST
2: Incremental OTA on Pixel 6 Pro with multiple iterations of device
   reboot when merge is in progress

Change-Id: Ib53be7f07ff192a84ec7f7049b2c6be01dad1041
Signed-off-by: Akilesh Kailash <akailash@google.com>
2023-10-05 05:32:31 +00:00
Daniel Zheng
c9770b29b9 Refactor off V2 Cow Ops
Refactor writer, reader + parser to work off v2 version of
CowOperations.

Test: m libsnapshot. ota on cuttlefish
Change-Id: Iec59be91e5f54782272b37702d645942df38c771
2023-10-04 18:09:35 -07:00
Treehugger Robot
72ba5c12be Merge changes Ib7509508,I7e256e8d into main am: 310e7ae496
Original change: https://android-review.googlesource.com/c/platform/system/core/+/2767639

Change-Id: I7a3c878979b3d47963549b78df7d95509ec467a8
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-10-05 00:42:27 +00:00
Daniel Zheng
a16436d5d7 Adding test for reader compatibility
Reader + Parser v3 should be able to read V2 cow format written to disk.
This test reads in a small cow file written by basic_v2_cow_writer and
parses it to ensure this compatibility checks out.

Test: cow_api_test
Change-Id: I46ebf4e3f12cdb3e4716ca5b624aab5836086733
2023-10-04 17:37:26 -07:00
David Anderson
c942daf179 snapuserd: Add an extractor tool.
This is similar to inspect_cow --extract-to, except it uses snapuserd.
It is a diagnostic host tool and uses the tooling added for host
testing.

Usage: snapuserd_extractor -cow COW_FILE -base BASE_FILE -out OUT_FILE
                           -num_sectors NUM_SECTORS

Unlike inspect_cow, this supports xor/copy operations.

The extractor code is separated into a utility file so we can use it for
additional tests later on.

Bug: N/A
Test: manual test
Change-Id: Ib7509508cba45e6c3a0db8c75454e33c2a503e03
2023-10-03 19:54:28 -07:00
David Anderson
8a28163d33 inspect_cow: Add an --extract-to argument.
This adds an --extract-to argument to inspect_cow to verify that full
OTA snapshots contain correct data. It does not yet work for ordered
ops.

Test: inspect_cow
Bug: N/A
Change-Id: I9014da3e83fd4fb5ea54ac1d36e527b3e3e6c9d5

Change-Id: I7e256e8ddec626980cdcf8680bbeac3c2e9d8de1
2023-10-03 16:02:26 -07:00
Treehugger Robot
95879d3e8b Merge "Log the CowOperation when decompression failed" into main am: e877885533
Original change: https://android-review.googlesource.com/c/platform/system/core/+/2768488

Change-Id: Ic4b424caa8183140e53dc28c89990f5883f1e163
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-10-02 21:18:27 +00:00
Treehugger Robot
e877885533 Merge "Log the CowOperation when decompression failed" into main 2023-10-02 20:45:38 +00:00
Kelvin Zhang
ab4c9621d4 Log the CowOperation when decompression failed
Test: th
Bug: 302992208
Change-Id: I617062d75a79ae73dfdff13c2e9d2a62c5dfcfc0
2023-10-02 11:08:18 -07:00
Treehugger Robot
63a7d5bcbf Merge "Adjust unit tests for supporting chain partition with no ab" into main am: b90cd3b5ab
Original change: https://android-review.googlesource.com/c/platform/system/core/+/2765270

Change-Id: I42ca902df13dd623561dcab721c73c45be153cdc
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-10-02 04:49:23 +00:00
Treehugger Robot
b90cd3b5ab Merge "Adjust unit tests for supporting chain partition with no ab" into main 2023-10-02 03:28:37 +00:00
Treehugger Robot
668c95919a Merge "Keep fs_mgr_vendor_overlay_test in internal cf." into main am: ec74e27a61
Original change: https://android-review.googlesource.com/c/platform/system/core/+/2765988

Change-Id: I8e861c8f49cb21acd45ef97c9009ccfb65658870
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-09-28 18:22:23 +00:00
wuhaitao3
250aa5ed72 Adjust unit tests for supporting chain partition with no ab
Test: libfs_avb_internal_test
      libfs_avb_test
Signed-off-by: wuhaitao3 <wuhaitao3@xiaomi.corp-partner.google.com>
Change-Id: I5f91510e72776ed2ed53756ae94a87b1417ae1a5
2023-09-28 11:19:55 +00:00
Dan Shi
69f57f98c5 Keep fs_mgr_vendor_overlay_test in internal cf.
Bug: 302382476
Change-Id: I369927033b407660d2c0e7bdb85494c826c0b406
Test: presubmit
2023-09-28 06:32:31 +00:00
Daniel Zheng
1b53b83925 Merge "Add copyright to libsnapshot/tools" into main am: 26bed5fc09
Original change: https://android-review.googlesource.com/c/platform/system/core/+/2763850

Change-Id: I868a1db6b5920a827aa60c375992a3f21895290a
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-09-27 19:45:07 +00:00
Daniel Zheng
e42df43569 Merge "Add binary to write basic COW" into main am: 0fc7feeafb
Original change: https://android-review.googlesource.com/c/platform/system/core/+/2763849

Change-Id: I57488d1893a80bf44e5c3b12ec6f014691329c77
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-09-27 19:44:16 +00:00
Daniel Zheng
301acae35b Add copyright to libsnapshot/tools
Adding copyright to cow_benchmark + basic_v2_cow_writer

Test: th
Change-Id: Ic6c32b68bbeaa3c0c365f146ef4342356e13706a
2023-09-26 16:51:53 -07:00
Daniel Zheng
d7c5511095 Add binary to write basic COW
Adding small binary to write a simple COW for version 2. We will then
use updated reader to make sure we can read this version 2 cow. Think it
would be a good idea to keep the binary here to see how exactly we
generated the cow (useful for debugging and if we ever need to recreate
this cow)

Test: m basic_v2_cow_writer
Change-Id: I28435025b7a8280fc5c4574876cc9110b391cb0e
2023-09-26 16:51:48 -07:00
Treehugger Robot
8cd8ce3fc0 Merge "Create_cow: Hash of target blocks should not be stored." into main am: a2cd2202f0
Original change: https://android-review.googlesource.com/c/platform/system/core/+/2759698

Change-Id: Ib652154e55c271260cbf0070af9c96f674c31709
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2023-09-24 01:59:18 +00:00
Akilesh Kailash
810274071d Create_cow: Hash of target blocks should not be stored.
Target block hash was in-correctly getting added to map thereby
overriding the source-hash.

Bug: 299011882
Test: Flash Pixel 6 Pro from A->B and from B->A
Change-Id: Ib3887c29cd6b8f2abd50e932273a5cfc2a096bd5
Signed-off-by: Akilesh Kailash <akailash@google.com>
2023-09-23 01:07:47 -07:00