Commit graph

2622 commits

Author SHA1 Message Date
Akilesh Kailash
0ec13ceda9 Merge "libsnapshot: Add Version field which will be used by update-engine manifest" 2021-04-30 18:08:54 +00:00
Akilesh Kailash
acc1c5d189 libsnapshot: Add Version field which will be used by update-engine
manifest

Bug: 183863613
Test: Build
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: If4b6a60669a55eb2ea1167fb2cc49ce04c054748
2021-04-30 06:31:01 +00:00
Daniel Rosenberg
05a73c5788 libsnapshot: Allow inspect_cow to show failed blocks
Adds the -b option to show the bad data block that failed to decompress.
If the block is large enough, display the front as though it were a
CowOperation, as this is the most likely culprit.

Change-Id: I287f13e0794a1ca9d647d4b1099ab238a6202b23
Bug: 183985866
Test: inspect_cow -db <COW_FILE>
2021-04-29 21:58:58 +00:00
Yo Chiang
e222fa4d74 Merge "Enable disabled CtsFsMgrTestCases test cases" 2021-04-29 06:37:59 +00:00
Akilesh Kailash
d79e4f4c19 Merge "libsnapshot: Pass MSG_NOSIGNAL socket flags" 2021-04-28 20:55:34 +00:00
David Anderson
d5349dcf6d Merge "libsnapshot: Add a merge failure code to SnapshotMergeReport." 2021-04-28 19:39:34 +00:00
Yi-Yo Chiang
7e122fedd5 Enable disabled CtsFsMgrTestCases test cases
And use the new fs_mgr C++ API.

Bug: 186504948
Test: atest CtsFsMgrTestCases
Test: atest CtsFsMgrTestCases in DSU
Change-Id: I80d68d72fb3aa9f5a0dc7d9400b4d0f80cc7deb3
2021-04-27 18:46:30 +08:00
Akilesh Kailash
83658e0d36 libsnapshot: Pass MSG_NOSIGNAL socket flags
If one end of the communication socket is closed for some reason, there
is no need to terminate the daemon or the client. Mask
the SIGPIPE using MSG_NOSIGNAL flag - we will
still get EPIPE error but process will not be terminated.

Bug: 186213024
Test: Full OTA
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: Iaa53545c0c4059618f6b49afb9ec24ea5372c7e0
2021-04-27 02:36:18 +00:00
Treehugger Robot
26e5d8f1e2 Merge "Add new dm-verity error handling mode" 2021-04-23 03:35:01 +00:00
Jintao Zhu
84822c8d51 Add function CleanupOldScratchFiles symmetrically
For symmetry, add function CleanupOldScratchFiles in conditional compilation blocks which missing it.

Test: monkey test for one day and one night

Signed-off-by: Jintao Zhu <zhujtcsieee@gmail.com>

Change-Id: Ie754427334c9a9bb7cfed70df45f439c60c9ab16
2021-04-21 14:42:11 +08:00
David Anderson
57c881ebf0 libsnapshot: Add a merge failure code to SnapshotMergeReport.
Bug: 185290850
Test: builds
Change-Id: I56e9e4c8943e420163b2c663c0136d37a45455e4
2021-04-20 13:07:31 -07:00
David Anderson
53a06bffe1 Merge "libsnapshot: Add a merge failure code to the update state." 2021-04-20 16:20:48 +00:00
Yo Chiang
a7e2bbd5f4 Merge "Fix CtsFsMgrTestCases" 2021-04-20 08:43:23 +00:00
Yi-Yo Chiang
f61cb30c27 Fix CtsFsMgrTestCases
As ReadFstabFromFile() may append / remove / modify the fstab read from
the file, we cannot make assumptions about the number of fstab entries.
We can however test that the returned fstab contains at least the
entries we expect.

Fixes: 185826755
Test: atest CtsFsMgrTestCases on GSI & DSU
Change-Id: I539e7eed3f7ae14db7e9983bed7f68754c9fff39
2021-04-20 14:56:41 +08:00
jiahao
bfcee2891c add missing spaces
The parameters "--debug-cache" and path are close to each other, eg:

01-09 17:14:52.058   648   648 I vold    : [libfs_mgr]Running
/system/bin/fsck.f2fs -a -c 10000 --debug-cache/dev/block/dm-10

Add a space between them to separate.

Signed-off-by: jiahao <jiahao@xiaomi.com>
Change-Id: Ie2929b541d5907a521c7b97aaed31fdca381fc1a
2021-04-16 17:07:50 +08:00
David Anderson
558435f6fc libsnapshot: Add a merge failure code to the update state.
Bug: 185290850
Test: vts_libsnapshot_test
Change-Id: I56c62748dd9a24126afbc3e26d1994d80f519333
2021-04-15 16:11:56 -07:00
JeongHyeon Lee
9eb1296ea9 Add new dm-verity error handling mode
The latest kernel supports dm-verity has new error handling mode(panic).
However, there is no code to support in android.

Signed-off-by: JeongHyeon Lee <jhs2.lee@samsung.com>
Change-Id: Ib88ec258adb76ca4c88df1a78636f73f40604b40
2021-04-14 09:10:37 +09:00
Daniel Rosenberg
35ff136a25 libsnapshot: Check cluster size on appends
When appending, if the cluster should end after the given label, ensure
that it does.

Bug: 183985866
Test: cow_api_test#ResumeEndCluster
Change-Id: Ie93d09b3431755d0b9b92761619d55df7f9f6151
2021-04-12 21:52:06 -07:00
Daniel Rosenberg
861e69a4c2 libsnapshot: Zero out leftover ops in CowWriter
When opening in append mode, we could write less than what was present
before. This could result in data blocks referencing beyond the end of
the file, or partially written ops. Zeroing these out will prevent
invalid leftovers from potentially causing confusion.

Bug: 183985866
Test: cow_api_test
Change-Id: I56f0218f3ea5b83c0614d1b86e81a4ca885f5c5e
2021-04-12 21:52:06 -07:00
David Anderson
db25e8e32d libsnapshot: Fix premature truncation in CowWriter.
When opening in append mode, we ftruncate() the COW. This has three side
effects:
 (1) If the COW is never modified, or Finalized(), the state of the COW
     will have changed. Ideally it should only change on an explicit
     write operation.
 (2) Data after the current cluster will be accidentally thrown away.
 (3) The ending "cluster" op will be thrown away if the current cluster
     was incomplete, and thus the last valid label could be invalidated.

Bug: 183985866
Test: cow_api_test
Change-Id: I3c9a38553b7492a3d6e71d177d75ddb1b6490dfe
2021-04-12 21:52:06 -07:00
David Anderson
055f802086 Merge "libsnapshot: Add diagnostics for DM_DEV_REMOVE failures." 2021-04-12 21:04:08 +00:00
Jaegeuk Kim
05ca915431 fs_mgr: add readahead_size_kb in fstab
This patch adds an option, readahead_size_kb in fstab entry option. It supports
to set a proper readahead_size per block/dm devices before using them by mount,
which is useful for low/high-end devices when addressing memory pressure issue.

Bug: 181567573
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
Change-Id: I549962e68f8488417d76bcfb283958bc33fd5d7a
2021-04-09 04:54:11 +00:00
David Anderson
0b903f9910 libsnapshot: Add diagnostics for DM_DEV_REMOVE failures.
Example log line:

    update_engine: Block device was lazily unmounted and is still in-use:
        /dev/block/dm-28; possibly open file descriptor or attached loop device.

This will help diagnose bugs such as b/184715543 in the future.

Bug: N/A
Test: manual test
Change-Id: Ia6b17fe9bd1796d59be7fc0b355218509acfd4af
2021-04-08 11:06:12 -07:00
Treehugger Robot
05e6076ceb Merge "Add MockSnapshotMergeStats" 2021-04-08 18:05:25 +00:00
Yo Chiang
b20e9a3606 Merge "fs_mgr: SkipMountingPartitions() support glob patterns" 2021-04-07 05:04:21 +00:00
Kelvin Zhang
7006048957 Add MockSnapshotMergeStats
Test: th
Change-Id: If4f43e132e9c482418d6a0adbd98ffca9576b39b
2021-04-06 15:47:43 -04:00
Akilesh Kailash
591edca44d Merge "libsnapshot:snapuserd: Terminate daemon after applying OTA." 2021-04-06 18:02:52 +00:00
Akilesh Kailash
e2883c1ea0 libsnapshot:snapuserd: Terminate daemon after applying OTA.
When all threads are terminated, dm-user handler's are removed
from the list. When the last handler is removed, daemon is
shutdown gracefully.

Bug: 183652708
Test: 1: Apply full OTA and verify daemon is terminated; reapply the OTA
      to verify daemon is restarted again.
      2:  vts_libsnapshot_test

Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: Ibd41223fc0eba884993a533fcc95661f72805db2
2021-04-05 21:50:46 +00:00
Yi-Yo Chiang
1a3c050a35 fs_mgr: Refactor ReadDefaultFstab() and ReadFstabFromFile()
* Eliminate redundant std::move() by transforming
    ReadFstabFromDt(&dt_fstab, false);
    *fstab = std::move(dt_fstab);
  to
    fstab->clear();
    ReadFstabFromDt(fstab, false);

* Don't modify output parameter if ReadFstabFromFile() failed.

Bug: 181110285
Test: Presubmit
Change-Id: I4e4d9852cc618a66d79e423780bf97773dca2a58
2021-04-02 05:26:01 +00:00
Yi-Yo Chiang
ea4369d141 fs_mgr: Strengthen ReadFstabFromFile() around gsi_public_metadata_file
ReadFstabFromFile() calls access() to check the existence of DSU
metadata files to determine if device is in DSU running state. This is
error prone because a failed access() can mean non-exsitent file as well
as the caller lacking the permission to path resolute the pathname.

Strengthen ReadFstabFromFile() to check the errno after a failed
access() or open(), if the errno is not ENOENT, then return with error,
as this may be indicating the caller doesn't have sufficient access
rights to call ReadFstabFromFile().

After this change, processes would need these policies to call
ReadFstabFromFile():
  allow scontext { metadata_file gsi_metadata_file_type }:dir search;
And these policies to call ReadFstabFromFile() within a DSU system:
  allow scontext gsi_public_metadata_file:file r_file_perms;

Bug: 181110285
Test: Presubmit
Change-Id: I1a6a796cb9b7b49af3aa5e7a5e8d99cde25e5857
2021-04-02 05:25:46 +00:00
Yi-Yo Chiang
2f2fe4afaf fs_mgr: SkipMountingPartitions() support glob patterns
Enhance skip_mount.cfg to be able to specify glob patterns.

Bug: 184132970
Test: GSI Boot test
Change-Id: I8e6528e4aff6ea176aa028c3c55d6cedaff3e181
2021-04-02 05:23:44 +00:00
Yi-Yo Chiang
20579011cf fs_mgr: Suppress SkipMountingPartitions log in ReadFstabFromFile
Minor refactoring and renaming, goal is to make the follow-up patch
easier to read.

Bug: 184132970
Test: Presubmit
Change-Id: I66416161b30ac310934d901cbaf11bc926e2cbf7
2021-04-02 05:23:02 +00:00
Akilesh Kailash
ef8d3c39b5 Merge "libsnapshot:snapuserd: Fix memory leak" 2021-04-02 05:13:11 +00:00
Akilesh Kailash
e722a1b169 libsnapshot:snapuserd: Fix memory leak
When worker threads were created, snapuserd was converted to a
shared_pointer. Earlier, memory was forcefully released
by setting snapuserd to nullptr which worked as it
was a unique pointer. Now, every worker thread holds
a reference. Clear the vector once all the worker
threads are terminated.

Test: Apply OTA and verify memory is released after OTA is applied
Bug: 183652708
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: I256d26d98b02ad599aff49b92192226546c59b17
2021-04-02 00:49:18 +00:00
David Anderson
7ab8f2eeba Merge "libsnapshot: Ensure we can remove bad snapshots when beginning an update." 2021-03-30 18:48:28 +00:00
David Anderson
bcd19ed73e libsnapshot: Ensure we can remove bad snapshots when beginning an update.
If somehow we wind up with snapshots with a source suffix, we could wind
up trying to unmap an in-use partition. Detect this case and allow the
snapshot to be deleted without the unmap.

Bug: 183567503
Test: vts_libsnapshot_test
Change-Id: I87dd5bb3a7b9be59dede624924374ccc47b563c2
2021-03-24 13:25:27 -07:00
Julien Desprez
b0198cbca7 libfs_avb_internal_test & libfs_avb_test already run in presubmit
remove duplicate postsubmit setup, they already run in host-unit-tests presubmit

Change-Id: I4065d7a50729a14911ec64f10082987d3f9ddcfc
Test:presubmit
Bug: 183622274
2021-03-24 18:29:58 +00:00
Treehugger Robot
2af0504896 Merge "fastboot device: Disallow implicit conversion from unique_fd to int." 2021-03-23 06:48:02 +00:00
Yifan Hong
07e947fc5d fastboot device: Disallow implicit conversion from unique_fd to int.
Do not use the implicit cast from unique_fd to int so
that it is clearer to the reader what the ownership model
is.

Test: pass
Change-Id: I66563eb2bd06f6a712a5afd4c6009f9b25a55de6
2021-03-22 19:30:59 -07:00
Kelvin Zhang
a406c7e2a9 Add CLI entry point for map/unmap snapshot partitions
Test: apply OTA && snapshotctl map

Change-Id: I4d50a70f9b330e313034c6057d574f985d242475
2021-03-22 13:43:14 -04:00
Akilesh Kailash
3e807e9651 libsnapshot:snapuserd: Cut down memory usage
Use sorted std:vector instead of std:map to store
the mapping between chunk-id to COW operation.

Addtionally, use shrink_to_fit to cut down vector
capacity when COW operations are stored.

On a full OTA of 1.8G, Anon RSS usage is
reduced from 120MB to 68MB.  No variance observed
when merge was in progress.

Bug: 182960300
Test: Full and Incremental OTA - verified memory usage
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: I50cacbe0d03837a830dedcf9bd0ac9663fc68fa7
2021-03-19 06:32:52 +00:00
Yifan Hong
4bf547af43 Merge changes from topic "fastboot_fetch"
* changes:
  fuzzy_fastboot: Add tests for fetch:vendor_boot
  fuzzy_fastboot: Add conformance test for getvar:max-fetch-size.
  fastboot driver: repack vendor boot ramdisk
  fastboot driver: Allow colon in partition name when flashing
  fastboot driver: add fetch command in driver
  fastboot driver: RunAndReadBuffer don't allocate too much mem
  fastboot driver: add RunAndReadBuffer helper
  fastboot driver: Fix ownership of fd in fastboot_buffer.
  fastboot driver: fix message
  fastbootd: add O_CLOEXEC/O_BINARY for OpenPartition
  fastbootd: add read arg to OpenPartition
  fastbootd: Add fetch command on device
  fastbootd: Add getvar max-fetch-size.
2021-03-18 23:33:56 +00:00
Yifan Hong
17d469bd57 fastboot driver: RunAndReadBuffer don't allocate too much mem
Instead of allocating the buffer for the whole
upload (which can be arbitrary number of bytes as
the device determines), read 1 MiB at a time.

Test: pass
Bug: 173654501
Change-Id: Ib601b0341b10b7dccbb429cd21aad86a2d3bfda8
2021-03-17 15:38:06 -07:00
David Anderson
102f7e4d7e Merge "libsnapshot: Add boot_complete metrics." 2021-03-16 00:30:46 +00:00
David Anderson
a0992300f2 libsnapshot: Add boot_complete metrics.
Bug: 180535575
Test: apply OTA
Change-Id: Ifea51c4c83ba03012f557927e8694e7c613335a7
2021-03-12 18:35:44 -08:00
David Anderson
bf3020b749 libsnapshot: Fix CowWriter::Finalize() not restoring footer correctly.
Bug: 173432386
Test: cow_api_test
Change-Id: If1b856fcdcd6b6ded5e00342037308c6b35d6fc0
2021-03-11 16:16:36 -08:00
Akilesh Kailash
bd83b72bc8 Merge "libsnapshot:snapuserd: Multithreading support" 2021-03-11 00:51:20 +00:00
Akilesh Kailash
8f0840940e libsnapshot:snapuserd: Multithreading support
Add worker threads per partition to serve the IO request.

Remove memset of buffer in IO path which was impacting
4k IO performance.

update_verifier performance:

1: ~10-12 seconds with this change (both on full OTA and incremental
OTA); ~70 seconds observed without this changeset

2: ~8 seconds without the daemon once merge is completed
and snapshot devices are removed.

Bug: 181293939
Test: update_verifier, full OTA, incremental OTA
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: Id90887f3f4a664ee5d39433715d1c166acbd6c60
2021-03-10 07:53:16 +00:00
Devin Moore
0a799bdfd6 Handle "hardware" bootconfig parameter as "androidboot.hardware"
The parameter "androidboot.hardware" has been removed from bootconfig
and replaced by "hardware" parameter.

Test: launch_cvd with 4.19 and 5.10 kernels
Test: atest CtsFsMgrTestCases
Bug: 173815685
Change-Id: I627426ae1bd0a165b70b8f2584ec184abfb4236f
2021-03-09 14:10:48 -08:00
David Anderson
9865e47e1e Merge "libsnapshot: Collect more COW size data in snapshot merge stats." 2021-03-09 18:48:58 +00:00