Commit graph

2392 commits

Author SHA1 Message Date
David Anderson
4e4cff7e46 Merge "libsnapshot: Add a tool for inspecting COW files." 2020-10-29 00:27:28 +00:00
David Anderson
5fea7df825 Merge "libsnapshot: Add a skeleton API for mapping and unmapping all snapshots." 2020-10-29 00:27:19 +00:00
David Anderson
a1a5fdc6ee Merge "libsnapshot: Stop reading ops once we reach a footer." 2020-10-29 00:27:02 +00:00
David Anderson
49a428b62e Merge changes I2b3f5b33,I464b683b,I8e97c543
* changes:
  libsnapshot: Remove the timeout on client recv().
  libsnapshot: Integrate with snapuserd.
  snapuserd: Add an API call to wait for device deletion.
2020-10-28 01:55:09 +00:00
David Anderson
3fa66c7283 libsnapshot: Add a tool for inspecting COW files.
This simple tool will dump the COW header and included ops to stdout.

Bug: N/A
Test: mm inspect_cow && inspect_cow <file>
Change-Id: I369c4a21a84c95ffc10670bd9eeb2ceccb2a56d6
2020-10-27 18:30:13 -07:00
David Anderson
07ad1b3566 libsnapshot: Add a skeleton API for mapping and unmapping all snapshots.
Bug: 168554689
Test: builds
Change-Id: I0da50275cc4d0a85bd6b176b5c1286659d51a25c
2020-10-27 18:30:13 -07:00
David Anderson
485fe695c1 libsnapshot: Stop reading ops once we reach a footer.
Bug: 168554689
Test: vts_libsnapshot_test
Change-Id: Id8b5ba33220028c856d3761832fe231fd93e94cd
2020-10-27 18:30:13 -07:00
David Anderson
0a03a5a8ec snapuserd: Fix infinite loop when reading ops.
Bug: 168554689
Test: vts_libsnapshot_test
Change-Id: I48e62f258ef2b4c368e8237a132c802a03d7020b
2020-10-26 22:30:21 -07:00
David Anderson
a2b5d9978a libsnapshot: Add Initialize and InitializeAppend methods to ISnapshotWriter.
This is so update engine can resume from the correct label.

Bug: 168554689
Test: vts_libsnapshot_test
Change-Id: Ib04e80e8219f954f105d5a85f86efa7bb9097579
2020-10-26 22:30:21 -07:00
David Anderson
0544f91a9a libsnapshot: Implement OpenReader for CompressedSnapshotWriter.
Bug: 168554689
Test: vts_libsnapshot_test
Test: full OTA with update_device.py
Test: incremental OTA with update_device.py
Change-Id: I3878abfd767d2e47cf8486bc2c06233da2f1ef08
2020-10-26 22:30:21 -07:00
Greg Kaiser
74b8449f58 Fix lseek() check
Since 'pos' is unsigned, we need to tweak the check.

Test: TreeHugger
Change-Id: I97dcb50fb86196b09b11ba725593e6c83c5586d9
2020-10-26 06:37:32 -07:00
Yo Chiang
346e6792b4 Merge "Call GetDsuMetadataKeyDir() for DSU metadata encryption dir" 2020-10-26 05:27:36 +00:00
David Anderson
45662c8941 Merge "snapuserd: Refactor client to allow persistent connections." 2020-10-23 22:01:42 +00:00
David Anderson
8425f6e6e6 Merge "snapuserd: Refactor daemon/server." 2020-10-23 18:33:15 +00:00
Treehugger Robot
d0ce499873 Merge "fs_mgr.cpp: fix the problem that the casefold feature not enabled for the data partition" 2020-10-23 16:47:43 +00:00
Daniel Rosenberg
323060840a Merge changes I3339d552,I126e1583,Iccc6580a
* changes:
  Add GetLastLabel and InitializeAppend
  Switch up Cow Format to be resumable
  reland: Rename Flush to Finalize
2020-10-23 11:08:46 +00:00
Yongqin Liu
c007c43335 fs_mgr.cpp: fix the problem that the casefold feature not enabled for the data partition
The value of entry.mount_point for data partition is "/data"

Fixes: 5ba5b90cd6 ("fs_mgr: try tune2fs for casefolding on /data only")

Test: got "Can't mount with encoding and encryption" problem reported
      by the db845c build with the default 5.4.38 prebuilt kernel

Signed-off-by: Yongqin Liu <yongqin.liu@linaro.org>
Change-Id: I226a2275f5f2ee18503c5a3863ef5a1d2c2ed7be
2020-10-23 13:34:49 +08:00
David Anderson
6494a8ca97 libsnapshot: Remove the timeout on client recv().
Two seconds is a bit aggressive - considering this is analagous to a
synchronous binder call, let's drop the timeout entirely.

Bug: 168554689
Test: vts_libsnapshot_test
Change-Id: I2b3f5b33f79575d72b15ed314dbcc0ad20ebd9a8
2020-10-22 22:07:25 -07:00
David Anderson
8e46846501 libsnapshot: Integrate with snapuserd.
This integrates libsnapshot with dm-user and snapuserd. Tests progress
significantly further now. Tests involving merging still fail as
snapuserd doesn't support this yet.

Bug: 168554689
Test: vts_libsnapshot_test
Change-Id: I464b683b464fe29a646f0f2823b7f4434a878614
2020-10-22 22:07:24 -07:00
David Anderson
6a1e9ca279 snapuserd: Add an API call to wait for device deletion.
This adds a new message to the daemon protocol, which waits for a device
to be deleted. The caller must ensure that the corresponding control
device is actually going away (eg, the device containing the dm-user
table entry has been deleted). Otherwise, this will hang.

This will allow libsnapshot to safely delete the cow since any
outstanding references will be closed.

This also refactors DmUserHandler so that it's freed (and removed from
the handler list) if its corresponding thread exits of its own accord.

Bug: 168554689
Test: vts_libsnapshot_test
Change-Id: I8e97c543eec84874c88795a493470e992dc476fc
2020-10-22 22:07:24 -07:00
David Anderson
fe7585a8b0 snapuserd: Refactor client to allow persistent connections.
This refactors SnapuserdClient so it retains a connection for its
lifetime. This allows SnapshotManager to ensure the daemon is running
and hold a connection open across all of its operations.

The main impetus of this change is to remove the ambiguity between first
and second-stage sockets. SnapshotManager should only ever connect to
the first-stage socket during first-stage init, or, to initiate the
"transition" step during second-stage init.

The transition steps are roughly:
 (1) Start second-stage daemon.
 (2) Load new device-mapper tables.
 (3) Connect second-stage daemon to new dm-user devices.
 (4) Activate the new tables, flushing IO to the first-stage daemon.
 (5) Send a signal to the first-stage daemon to exit.

This patch makes it easier to hold these two separate connections.

Bug: 168554689
Test: manual test
Change-Id: I51cb9adecffb19143ed685e0c33456177ec3d81f
2020-10-22 22:07:23 -07:00
David Anderson
cff35f67ad snapuserd: Refactor daemon/server.
This is in preparation for moving to a traditional client/server model
where clients stay connected and the server multiplexes multiple
connections.

Client has been renamed to DmUserClient to differentiate it from local
socket clients.

poll() responsibilities have been moved into SnapuserdServer. In
addition, the server now tracks all open clients and polls them
together with the listen socket.

SnapuserDaemon is now only responsible for signal masking. These two
classes can probably be merged together - I didn't do that here because
the patch was already large.

Bug: 168554689
Test: manual test
Change-Id: Ibc06f6287d49e832a8e25dd936ec07747a1b0555
2020-10-22 22:00:17 -07:00
Daniel Rosenberg
3d17cb9a57 Add GetLastLabel and InitializeAppend
GetLastLabel returns the last Label that a reader is confident about.
InitializeAppend starts a writer up to append data after the last given
label, assuming all later labels are not relevant data.

Change-Id: I3339d5527bae833d9293cbbc63126136b94bd976
Bug: 168829493
Test: cow_api_test
2020-10-22 21:26:27 -07:00
Daniel Rosenberg
2d2fd72502 Switch up Cow Format to be resumable
This switches up the format to alternate ops with data, followed by a
footer containing additional meta information. This allows the file to
be resumed at arbitrary points if writing gets interrupted by power
loss.
Also adds a label op, which allows labeling future ops as connected.
If the footer is missing, Append will treat the last label as possibly
corrupt, and ignore it.

Change-Id: I126e15837d710776f9396e7afc9b0cd595e26b59
Bug: 168829493
Test: cow_api_test
2020-10-22 21:26:27 -07:00
Daniel Rosenberg
b4a81ccd5a reland: Rename Flush to Finalize
As we change to a more resumable format, flush mostly writes the final
parts of the file that are needed, which would write extra data that is
not needed to continue writing, and would immediately be overwritten.

Additionally, in the next patch we will fsync the file after adding an
op, making the flush built in, and the Finalize name more appropriate.

Bug: 168829493
Test: builds
Change-Id: Iccc6580ac72ff066cfeeb32e3cdaf69c5ba615fc
2020-10-22 02:34:26 -07:00
David Anderson
26878da29b libsnapshot: Add .rc file for snapuserd.
Bug: 168554689
Test: setprop ctl.start snapuserd
Change-Id: I9b8f20a364249135a1edb0ee69f039059c4deaf3
2020-10-21 00:41:45 -07:00
David Anderson
7d66a60012 Merge "libsnapshot: Add a source_partition parameter to OpenSnapshotWriter." 2020-10-21 06:55:03 +00:00
Daniel Rosenberg
bc7a365bbe Merge "Revert "Rename Flush to Finalize"" 2020-10-21 03:50:40 +00:00
Jason Chiu
ebfbf7d920 Revert "Rename Flush to Finalize"
This reverts commit 6177f6f6c4.

Reason for revert: Droidcop-triggered revert due to breakage https://android-build.googleplex.com/builds/quarterdeck?branch=aosp-master&target=sdk_x86-sdk&lkgb=6919873&lkbb=6919923&fkbb=6919923, bug b/171353958

Change-Id: Iea3059b4bf64a7cac930c40b8b312b5b26b0baea
BUG: 168829493
2020-10-21 03:16:42 +00:00
Daniel Rosenberg
6b6a2b9442 Merge "Rename Flush to Finalize" 2020-10-21 02:38:10 +00:00
David Anderson
862b01746e Merge "libsnapshot: Implement OpenSnapshotWriter for compressed snapshots." 2020-10-20 19:16:58 +00:00
David Anderson
f70d801d4b Merge "libsnapshot: Implement OnlineKernelSnapshotWriter::OpenReader." 2020-10-19 20:58:55 +00:00
Daniel Rosenberg
6177f6f6c4 Rename Flush to Finalize
As we change to a more resumable format, flush mostly writes the final
parts of the file that are needed, which would write extra data that is
not needed to continue writing, and would immediately be overwritten.

Additionally, in the next patch we will fsync the file after adding an
op, making the flush built in, and the Finalize name more appropriate.

Change-Id: I1db48315dec097b0620a0dfe7e7fd0098610d482
Bug: 168829493
Test: builds
2020-10-16 22:35:46 -07:00
Akilesh Kailash
b03e4cdccd Integrate snapuserd with dm-user ABI changes.
Create loopback device to simulate system_a/product_a partitions
to test IO path.

Bug: 168259959
Test: cow_snapuserd_test

Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: I9f2a311d3eccfa20c82d0ebdb3e028ea3323a48d
2020-10-17 00:41:44 +00:00
Akilesh Kailash
ba0b4bb900 Merge "libdm: dm-user target creation takes an additional argument" 2020-10-17 00:41:28 +00:00
Palmer Dabbelt
eef49c9e3c libdm: dm-user target creation takes an additional argument
Rather than using some implicit naming scheme, we've decided to just let
userspace pick whatever control device naming scheme it wants.  This requires
an additional argument to be passed to the kernel.

Test: cow_snapuserd_test
Bug: 168259959
Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
Change-Id: Ibf7f378b67abb6ca7457a9fdc9b8739e3d31bee3
Signed-off-by: Akilesh Kailash <akailash@google.com>
2020-10-16 18:16:55 +00:00
David Anderson
3feb3237c0 libsnapshot: Add a source_partition parameter to OpenSnapshotWriter.
Reading from the base device is incorrect, because if the partition
shrinks, we may still have copy operations from the removed area in the
original partition. Ask the caller to explicitly name the source device
for AddCopy() operations.

Bug: 168554689
Test: vts_libsnapshot_test
Test: full OTA with update_device.py
Test: incremental OTA with update_device.py
Change-Id: If388e37c2a2f9288a43d2849312c921bf59d4918
2020-10-14 21:42:03 -07:00
David Anderson
767f3f7216 libsnapshot: Implement OpenSnapshotWriter for compressed snapshots.
This does not implement OpenReader() yet, however, it implements enough
for some vts_libsnapshot_tests to start passing with Virtual A/B
Compression enabled.

Note that OpenSnapshotWriter() has been disabled when linking with init
or recovery. This is to avoid pulling in all the compression libraries
in those places. OpenSnapshotWriter is only designed to be called by
update_engine and will not work outside of normal operation.

Bug: 168554689
Test: vts_libsnapshot_test
Test: full OTA with update_device.py
Test: incremental OTA with update_device.py
Change-Id: I9737d28bdd5c5f4914bc30a2bb72f357d5f44d2b
2020-10-14 21:41:37 -07:00
David Anderson
3e68bd331e libsnapshot: Implement OnlineKernelSnapshotWriter::OpenReader.
Bug: 168554689
Test: vts_libsnapshot_test
Test: full OTA with update_device.py
Test: incremental OTA with update_device.py
Change-Id: Ic5eb99be8ab1f89070a0db231d4660e123ae3967
2020-10-14 21:41:37 -07:00
David Anderson
aca0beaf77 libsnapshot: Partially implement OpenSnapshotWriter.
This is a re-landing of the original CL, with a few changes:
 - The correct device is now returned in MapUpdateSnapshot.
 - The old API is used for tests, and the new API is only tested when
   used on a VABC device.
 - A sync() call has been added to ensure that writes to the base and
   target snapshot devices have been fully flushed. This makes
   IsPartitionUnchanged detect the MapUpdateSnapshot bug.

Implement OpenSnapshotWriter for non-compressed Virtual A/B. This is
done by adding an OnlineKernelSnapshotWriter class, which forwards all
writes to a dm-snapshot block device.

This also introduces a new ISnapshotWriter class which extends
ICowWriter, and adds features specific to libsnapshot (versus ICowWriter
which is intended only for the new COW format). The OpenSnapshotReader
call has been moved here since the writer retains all the information
needed to create the reader.

To test the new call, vts_libsnapshot_test has been modified to use
OpenSnapshotWriter.

As part of this change, all consumers of libsnapshot must now link to
libsnapshot_cow.

Bug: 168554689
Test: vts_libsnapshot_test
Test: full OTA with update_device.py
Test: incremental OTA with update_device.py
Change-Id: I90364a58902a4406a37cb14a816642c57a72bec2
2020-10-14 21:41:36 -07:00
Akilesh Kailash
c81477c5e4 Merge "libsnapshot:snapuserd: Handle signals" 2020-10-08 02:49:04 +00:00
Akilesh Kailash
f594785a26 libsnapshot:snapuserd: Handle signals
1: poll events to accept client request.
2: Mask all signals except SIGINT and SIGTERM.

Bug: 168258493
Test: cow_snapuserd_test

Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: I330170ac15199d2a97cc3b752dbf3b62b7b51a40
2020-10-07 21:45:45 +00:00
Mark Salyzyn
08cd8b70c2 fs_mgr: adb-remount-test.sh: filter out more administrivia mounts.
binder /dev/binderfs binder rw,relatime,max=1048576,stats=global 0 0
/sys/kernel/debug /sys/kernel/debug debugfs rw,seclabel,relatime 0 0
/dev/block/dm-3 240548676 3692368 236856308   2% /data_mirror/cur_profiles

are administrative mounts uninteresting to the adb remount test.

Fix system and vendor devt tests, turn them into warnings instead.

Signed-off-by: Mark Salyzyn <salyzyn@google.com>
Bug: 170256128
Bug: 169988379
Test: adb-remount-test.sh
Change-Id: Ie19722a9ad813dce014bc27610e6db035e6c69e9
2020-10-07 08:36:20 -07:00
Yo Chiang
165d9aee40 Call GetDsuMetadataKeyDir() for DSU metadata encryption dir
So DSU system don't use a hard-coded metadata key dir.

Bug: 168571434
Test: Install and boot a DSU system.
Change-Id: I096ef46a6f7b2d9fc42882288519fc5b93df1f41
2020-10-07 18:59:23 +08:00
Dan Shi
67b7dd570f Remove vts10 related configs
Bug: 169582597
Test: build
Change-Id: I38e1e0f8e32b86ab14b905d040c1c62245a1a9e3
2020-10-06 13:52:44 -07:00
Tianjie Xu
216ce0985f Merge changes from topic "revert-1433573-vab-libsnapshot-linkage-APIHERUSEB"
* changes:
  Revert "libsnapshot: Partially implement OpenSnapshotWriter."
  Revert "libsnapshot: Implement OnlineKernelSnapshotWriter::OpenReader."
2020-10-03 19:27:32 +00:00
Tianjie Xu
5aa52d91dd Revert "libsnapshot: Partially implement OpenSnapshotWriter."
Revert "Link to libsnapshot_cow everywhere libsnapshot is linked."

Revert submission 1433573-vab-libsnapshot-linkage

Reason for revert: b/169981170, update crash for droidfooders.
Reverted Changes:
Ie75bba98c:Link to libsnapshot_cow where libsnapshot is linke...
Ieedfadc55:libsnapshot: Partially implement OpenSnapshotWrite...
I28a5d4a88:Link to libsnapshot_cow everywhere libsnapshot is ...

Exempt-From-Owner-Approval: Revert to unblock dogfood
Change-Id: I0677df77672aca9fd54d94e009ac0be7c88a1a9d
2020-10-03 19:26:18 +00:00
Tianjie Xu
ad583f2a16 Revert "libsnapshot: Implement OnlineKernelSnapshotWriter::OpenReader."
This reverts commit 39f250896a.

Reason for revert: b/169981170, update crash for droidfooders.

Change-Id: I6a3d096df3174596d36195bf59fa8fbc88c982d0
2020-10-03 07:26:29 +00:00
Akilesh Kailash
01085dc1b4 libsnapshot:snapuserd: Transitions for snapuserd
1: Create a static library which exposes APIs
to manage snapuserd daemon.

2: Snapuserd daemon creates communication socket.

Bug: 168258493
Test: cow_snapuserd_test tests all the library API
along with the IO path from dm-snapshot.

Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: I8aaedf89d75e793c145fdad248a4d88e0ce8348c
2020-10-02 18:41:11 +00:00
David Anderson
39f250896a libsnapshot: Implement OnlineKernelSnapshotWriter::OpenReader.
Bug: 168554689
Test: manual test
Change-Id: I91b39c5982c14d80dbe1f38ff6a8bc5745d3e7e8
2020-09-28 11:24:37 -07:00