Commit graph

2379 commits

Author SHA1 Message Date
Greg Kaiser
74b8449f58 Fix lseek() check
Since 'pos' is unsigned, we need to tweak the check.

Test: TreeHugger
Change-Id: I97dcb50fb86196b09b11ba725593e6c83c5586d9
2020-10-26 06:37:32 -07:00
Yo Chiang
346e6792b4 Merge "Call GetDsuMetadataKeyDir() for DSU metadata encryption dir" 2020-10-26 05:27:36 +00:00
David Anderson
45662c8941 Merge "snapuserd: Refactor client to allow persistent connections." 2020-10-23 22:01:42 +00:00
David Anderson
8425f6e6e6 Merge "snapuserd: Refactor daemon/server." 2020-10-23 18:33:15 +00:00
Treehugger Robot
d0ce499873 Merge "fs_mgr.cpp: fix the problem that the casefold feature not enabled for the data partition" 2020-10-23 16:47:43 +00:00
Daniel Rosenberg
323060840a Merge changes I3339d552,I126e1583,Iccc6580a
* changes:
  Add GetLastLabel and InitializeAppend
  Switch up Cow Format to be resumable
  reland: Rename Flush to Finalize
2020-10-23 11:08:46 +00:00
Yongqin Liu
c007c43335 fs_mgr.cpp: fix the problem that the casefold feature not enabled for the data partition
The value of entry.mount_point for data partition is "/data"

Fixes: 5ba5b90cd6 ("fs_mgr: try tune2fs for casefolding on /data only")

Test: got "Can't mount with encoding and encryption" problem reported
      by the db845c build with the default 5.4.38 prebuilt kernel

Signed-off-by: Yongqin Liu <yongqin.liu@linaro.org>
Change-Id: I226a2275f5f2ee18503c5a3863ef5a1d2c2ed7be
2020-10-23 13:34:49 +08:00
David Anderson
fe7585a8b0 snapuserd: Refactor client to allow persistent connections.
This refactors SnapuserdClient so it retains a connection for its
lifetime. This allows SnapshotManager to ensure the daemon is running
and hold a connection open across all of its operations.

The main impetus of this change is to remove the ambiguity between first
and second-stage sockets. SnapshotManager should only ever connect to
the first-stage socket during first-stage init, or, to initiate the
"transition" step during second-stage init.

The transition steps are roughly:
 (1) Start second-stage daemon.
 (2) Load new device-mapper tables.
 (3) Connect second-stage daemon to new dm-user devices.
 (4) Activate the new tables, flushing IO to the first-stage daemon.
 (5) Send a signal to the first-stage daemon to exit.

This patch makes it easier to hold these two separate connections.

Bug: 168554689
Test: manual test
Change-Id: I51cb9adecffb19143ed685e0c33456177ec3d81f
2020-10-22 22:07:23 -07:00
David Anderson
cff35f67ad snapuserd: Refactor daemon/server.
This is in preparation for moving to a traditional client/server model
where clients stay connected and the server multiplexes multiple
connections.

Client has been renamed to DmUserClient to differentiate it from local
socket clients.

poll() responsibilities have been moved into SnapuserdServer. In
addition, the server now tracks all open clients and polls them
together with the listen socket.

SnapuserDaemon is now only responsible for signal masking. These two
classes can probably be merged together - I didn't do that here because
the patch was already large.

Bug: 168554689
Test: manual test
Change-Id: Ibc06f6287d49e832a8e25dd936ec07747a1b0555
2020-10-22 22:00:17 -07:00
Daniel Rosenberg
3d17cb9a57 Add GetLastLabel and InitializeAppend
GetLastLabel returns the last Label that a reader is confident about.
InitializeAppend starts a writer up to append data after the last given
label, assuming all later labels are not relevant data.

Change-Id: I3339d5527bae833d9293cbbc63126136b94bd976
Bug: 168829493
Test: cow_api_test
2020-10-22 21:26:27 -07:00
Daniel Rosenberg
2d2fd72502 Switch up Cow Format to be resumable
This switches up the format to alternate ops with data, followed by a
footer containing additional meta information. This allows the file to
be resumed at arbitrary points if writing gets interrupted by power
loss.
Also adds a label op, which allows labeling future ops as connected.
If the footer is missing, Append will treat the last label as possibly
corrupt, and ignore it.

Change-Id: I126e15837d710776f9396e7afc9b0cd595e26b59
Bug: 168829493
Test: cow_api_test
2020-10-22 21:26:27 -07:00
Daniel Rosenberg
b4a81ccd5a reland: Rename Flush to Finalize
As we change to a more resumable format, flush mostly writes the final
parts of the file that are needed, which would write extra data that is
not needed to continue writing, and would immediately be overwritten.

Additionally, in the next patch we will fsync the file after adding an
op, making the flush built in, and the Finalize name more appropriate.

Bug: 168829493
Test: builds
Change-Id: Iccc6580ac72ff066cfeeb32e3cdaf69c5ba615fc
2020-10-22 02:34:26 -07:00
David Anderson
26878da29b libsnapshot: Add .rc file for snapuserd.
Bug: 168554689
Test: setprop ctl.start snapuserd
Change-Id: I9b8f20a364249135a1edb0ee69f039059c4deaf3
2020-10-21 00:41:45 -07:00
David Anderson
7d66a60012 Merge "libsnapshot: Add a source_partition parameter to OpenSnapshotWriter." 2020-10-21 06:55:03 +00:00
Daniel Rosenberg
bc7a365bbe Merge "Revert "Rename Flush to Finalize"" 2020-10-21 03:50:40 +00:00
Jason Chiu
ebfbf7d920 Revert "Rename Flush to Finalize"
This reverts commit 6177f6f6c4.

Reason for revert: Droidcop-triggered revert due to breakage https://android-build.googleplex.com/builds/quarterdeck?branch=aosp-master&target=sdk_x86-sdk&lkgb=6919873&lkbb=6919923&fkbb=6919923, bug b/171353958

Change-Id: Iea3059b4bf64a7cac930c40b8b312b5b26b0baea
BUG: 168829493
2020-10-21 03:16:42 +00:00
Daniel Rosenberg
6b6a2b9442 Merge "Rename Flush to Finalize" 2020-10-21 02:38:10 +00:00
David Anderson
862b01746e Merge "libsnapshot: Implement OpenSnapshotWriter for compressed snapshots." 2020-10-20 19:16:58 +00:00
David Anderson
f70d801d4b Merge "libsnapshot: Implement OnlineKernelSnapshotWriter::OpenReader." 2020-10-19 20:58:55 +00:00
Daniel Rosenberg
6177f6f6c4 Rename Flush to Finalize
As we change to a more resumable format, flush mostly writes the final
parts of the file that are needed, which would write extra data that is
not needed to continue writing, and would immediately be overwritten.

Additionally, in the next patch we will fsync the file after adding an
op, making the flush built in, and the Finalize name more appropriate.

Change-Id: I1db48315dec097b0620a0dfe7e7fd0098610d482
Bug: 168829493
Test: builds
2020-10-16 22:35:46 -07:00
Akilesh Kailash
b03e4cdccd Integrate snapuserd with dm-user ABI changes.
Create loopback device to simulate system_a/product_a partitions
to test IO path.

Bug: 168259959
Test: cow_snapuserd_test

Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: I9f2a311d3eccfa20c82d0ebdb3e028ea3323a48d
2020-10-17 00:41:44 +00:00
Akilesh Kailash
ba0b4bb900 Merge "libdm: dm-user target creation takes an additional argument" 2020-10-17 00:41:28 +00:00
Palmer Dabbelt
eef49c9e3c libdm: dm-user target creation takes an additional argument
Rather than using some implicit naming scheme, we've decided to just let
userspace pick whatever control device naming scheme it wants.  This requires
an additional argument to be passed to the kernel.

Test: cow_snapuserd_test
Bug: 168259959
Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
Change-Id: Ibf7f378b67abb6ca7457a9fdc9b8739e3d31bee3
Signed-off-by: Akilesh Kailash <akailash@google.com>
2020-10-16 18:16:55 +00:00
David Anderson
3feb3237c0 libsnapshot: Add a source_partition parameter to OpenSnapshotWriter.
Reading from the base device is incorrect, because if the partition
shrinks, we may still have copy operations from the removed area in the
original partition. Ask the caller to explicitly name the source device
for AddCopy() operations.

Bug: 168554689
Test: vts_libsnapshot_test
Test: full OTA with update_device.py
Test: incremental OTA with update_device.py
Change-Id: If388e37c2a2f9288a43d2849312c921bf59d4918
2020-10-14 21:42:03 -07:00
David Anderson
767f3f7216 libsnapshot: Implement OpenSnapshotWriter for compressed snapshots.
This does not implement OpenReader() yet, however, it implements enough
for some vts_libsnapshot_tests to start passing with Virtual A/B
Compression enabled.

Note that OpenSnapshotWriter() has been disabled when linking with init
or recovery. This is to avoid pulling in all the compression libraries
in those places. OpenSnapshotWriter is only designed to be called by
update_engine and will not work outside of normal operation.

Bug: 168554689
Test: vts_libsnapshot_test
Test: full OTA with update_device.py
Test: incremental OTA with update_device.py
Change-Id: I9737d28bdd5c5f4914bc30a2bb72f357d5f44d2b
2020-10-14 21:41:37 -07:00
David Anderson
3e68bd331e libsnapshot: Implement OnlineKernelSnapshotWriter::OpenReader.
Bug: 168554689
Test: vts_libsnapshot_test
Test: full OTA with update_device.py
Test: incremental OTA with update_device.py
Change-Id: Ic5eb99be8ab1f89070a0db231d4660e123ae3967
2020-10-14 21:41:37 -07:00
David Anderson
aca0beaf77 libsnapshot: Partially implement OpenSnapshotWriter.
This is a re-landing of the original CL, with a few changes:
 - The correct device is now returned in MapUpdateSnapshot.
 - The old API is used for tests, and the new API is only tested when
   used on a VABC device.
 - A sync() call has been added to ensure that writes to the base and
   target snapshot devices have been fully flushed. This makes
   IsPartitionUnchanged detect the MapUpdateSnapshot bug.

Implement OpenSnapshotWriter for non-compressed Virtual A/B. This is
done by adding an OnlineKernelSnapshotWriter class, which forwards all
writes to a dm-snapshot block device.

This also introduces a new ISnapshotWriter class which extends
ICowWriter, and adds features specific to libsnapshot (versus ICowWriter
which is intended only for the new COW format). The OpenSnapshotReader
call has been moved here since the writer retains all the information
needed to create the reader.

To test the new call, vts_libsnapshot_test has been modified to use
OpenSnapshotWriter.

As part of this change, all consumers of libsnapshot must now link to
libsnapshot_cow.

Bug: 168554689
Test: vts_libsnapshot_test
Test: full OTA with update_device.py
Test: incremental OTA with update_device.py
Change-Id: I90364a58902a4406a37cb14a816642c57a72bec2
2020-10-14 21:41:36 -07:00
Akilesh Kailash
c81477c5e4 Merge "libsnapshot:snapuserd: Handle signals" 2020-10-08 02:49:04 +00:00
Akilesh Kailash
f594785a26 libsnapshot:snapuserd: Handle signals
1: poll events to accept client request.
2: Mask all signals except SIGINT and SIGTERM.

Bug: 168258493
Test: cow_snapuserd_test

Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: I330170ac15199d2a97cc3b752dbf3b62b7b51a40
2020-10-07 21:45:45 +00:00
Mark Salyzyn
08cd8b70c2 fs_mgr: adb-remount-test.sh: filter out more administrivia mounts.
binder /dev/binderfs binder rw,relatime,max=1048576,stats=global 0 0
/sys/kernel/debug /sys/kernel/debug debugfs rw,seclabel,relatime 0 0
/dev/block/dm-3 240548676 3692368 236856308   2% /data_mirror/cur_profiles

are administrative mounts uninteresting to the adb remount test.

Fix system and vendor devt tests, turn them into warnings instead.

Signed-off-by: Mark Salyzyn <salyzyn@google.com>
Bug: 170256128
Bug: 169988379
Test: adb-remount-test.sh
Change-Id: Ie19722a9ad813dce014bc27610e6db035e6c69e9
2020-10-07 08:36:20 -07:00
Yo Chiang
165d9aee40 Call GetDsuMetadataKeyDir() for DSU metadata encryption dir
So DSU system don't use a hard-coded metadata key dir.

Bug: 168571434
Test: Install and boot a DSU system.
Change-Id: I096ef46a6f7b2d9fc42882288519fc5b93df1f41
2020-10-07 18:59:23 +08:00
Dan Shi
67b7dd570f Remove vts10 related configs
Bug: 169582597
Test: build
Change-Id: I38e1e0f8e32b86ab14b905d040c1c62245a1a9e3
2020-10-06 13:52:44 -07:00
Tianjie Xu
216ce0985f Merge changes from topic "revert-1433573-vab-libsnapshot-linkage-APIHERUSEB"
* changes:
  Revert "libsnapshot: Partially implement OpenSnapshotWriter."
  Revert "libsnapshot: Implement OnlineKernelSnapshotWriter::OpenReader."
2020-10-03 19:27:32 +00:00
Tianjie Xu
5aa52d91dd Revert "libsnapshot: Partially implement OpenSnapshotWriter."
Revert "Link to libsnapshot_cow everywhere libsnapshot is linked."

Revert submission 1433573-vab-libsnapshot-linkage

Reason for revert: b/169981170, update crash for droidfooders.
Reverted Changes:
Ie75bba98c:Link to libsnapshot_cow where libsnapshot is linke...
Ieedfadc55:libsnapshot: Partially implement OpenSnapshotWrite...
I28a5d4a88:Link to libsnapshot_cow everywhere libsnapshot is ...

Exempt-From-Owner-Approval: Revert to unblock dogfood
Change-Id: I0677df77672aca9fd54d94e009ac0be7c88a1a9d
2020-10-03 19:26:18 +00:00
Tianjie Xu
ad583f2a16 Revert "libsnapshot: Implement OnlineKernelSnapshotWriter::OpenReader."
This reverts commit 39f250896a.

Reason for revert: b/169981170, update crash for droidfooders.

Change-Id: I6a3d096df3174596d36195bf59fa8fbc88c982d0
2020-10-03 07:26:29 +00:00
Akilesh Kailash
01085dc1b4 libsnapshot:snapuserd: Transitions for snapuserd
1: Create a static library which exposes APIs
to manage snapuserd daemon.

2: Snapuserd daemon creates communication socket.

Bug: 168258493
Test: cow_snapuserd_test tests all the library API
along with the IO path from dm-snapshot.

Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: I8aaedf89d75e793c145fdad248a4d88e0ce8348c
2020-10-02 18:41:11 +00:00
David Anderson
39f250896a libsnapshot: Implement OnlineKernelSnapshotWriter::OpenReader.
Bug: 168554689
Test: manual test
Change-Id: I91b39c5982c14d80dbe1f38ff6a8bc5745d3e7e8
2020-09-28 11:24:37 -07:00
David Anderson
75b982ad30 libsnapshot: Partially implement OpenSnapshotWriter.
Implement OpenSnapshotWriter for non-compressed Virtual A/B. This is
done by adding an OnlineKernelSnapshotWriter class, which forwards all
writes to a dm-snapshot block device.

This also introduces a new ISnapshotWriter class which extends
ICowWriter, and adds features specific to libsnapshot (versus ICowWriter
which is intended only for the new COW format). The OpenSnapshotReader
call has been moved here since the writer retains all the information
needed to create the reader.

To test the new call, vts_libsnapshot_test has been modified to use
OpenSnapshotWriter.

As part of this change, all consumers of libsnapshot must now link to
libsnapshot_cow.

Bug: 168554689
Test: vts_libsnapshot_test
Change-Id: Ieedfadc557833c1e0540922aabc6e95c80266a64
2020-09-28 11:24:37 -07:00
David Anderson
6560d23765 libsnapshot: Add a maximum block count to CowWriter.
Once COWs are allocated, their size is fixed, and we don't want to
write beyond the end. To make this validation less tedious, the virtual
methods of ICowWriter have been made internal. The user-facing API calls
do validation before calling the internal variants.

Bug: 168554689
Test: cow_writer_test
Change-Id: Ic9ebb5bc4b601180d35d915c47cd9c537bc423fa
2020-09-28 11:24:37 -07:00
David Anderson
f53794d1af libsnapshot: Make the new Open calls look more like MapUpdateSnapshot.
To make the transition to these new calls easier, give them a very
similar signature to MapUpdateSnapshot. Also, allow them to work in
non-compression mode.

Bug: 168554689
Test: builds
Change-Id: I2eb6b41ba8a294dbde1763aabd7701a80c0789f3
2020-09-21 12:06:49 -07:00
David Anderson
2f11ec6a13 libsnapshot: Prototype the new API for mapping writable snapshots.
Since we can't provide a single device or fd anymore, we need to expose
a CowWriter directly. Additionally, we expose an API for reading
snapshots through the FileDescriptor abstraction.

Bug: 168554689
Test: builds
Change-Id: If7e8adbfe69c2a84d34c63d4b0adff2b3365fd82
2020-09-18 14:24:40 -07:00
David Anderson
975ea3217e Merge changes from topic "vab-brotli"
* changes:
  libsnapshot: Add an append mode to CowWriter.
  libsnapshot: Add support for brotli compression.
2020-09-17 19:32:23 +00:00
Akilesh Kailash
516ded7320 Support of multiple device partitions by dm-user driver
Add basic support for daemon creation and handle signals

Test: Add test case to test system and product COW partitions
Bug: 162790322

Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: Icf8dbe49d2237cec6f7dbcdd84256eb6c5afa1cd
2020-09-16 00:35:42 +00:00
David Anderson
a889c87b0f libsnapshot: Add an append mode to CowWriter.
When in append mode, CowWriter will re-open the existing COW and resume
writing at the end of the old data position. All existing operations
will be reimported and buffered in memory.

The size calculation has been simplified to make this work. We now
advance ops_offset and no longer track the number of bytes written.

Additionally, a "header_size" field has been added to the header. This
was missing from the original format and is useful for introducing
forward compatibility later.

Finally, Finalize has been renamed to Flush. It's still mandatory, but
it can be called multiple times to continue appending data without
reopening.

Bug: 168554689
Test: cow_api_test gtest
Change-Id: I637e99ae08a4db5b273c06318e6db523ea8ec7c5
2020-09-15 16:15:41 -07:00
David Anderson
a652877bd6 libsnapshot: Add support for brotli compression.
Bug: 162274240
Test: cow_api_test
Change-Id: I0b0ceec3c3041a6aea4b1e6c4d01ed0a8860d7e8
2020-09-15 16:15:33 -07:00
Palmer Dabbelt
7dd8fe8e70 Add "dmctl uuid" command
This makes looking up device UUIDs a bit easier, as rather than depending on
the device mapper's sysfs layout we can depend on dmctl.  There's some
associated libdm plumbing, but the UUID was already pretty much availiable.

Test: I just ran this by hand.
Signed-off-by: Palmer Dabbelt <palmerdabbelt@google.com>
Change-Id: I7028eec6ab04601308047b67057e51a0ff44c0a7
2020-09-15 18:39:39 +00:00
David Anderson
1f8c18ee55 Merge "libsnapshot: Refactor cow_reader decompression." 2020-09-10 23:05:08 +00:00
David Anderson
511c4bc601 libsnapshot: Refactor cow_reader decompression.
Bug: 162274240
Test: cow_api_test
Change-Id: I12c177f3ebb7bb0550669bd5edbdbbde6f572cfd
2020-09-09 21:54:48 -07:00
Xin Li
e1ebeab9c5 Merge Android R
Bug: 168057903
Merged-In: I88f6774ca9985fb50f015756b8f8508d4691dc62
Change-Id: Iefe2872d29836b77344479ad2552a9c03d81bdcb
2020-09-09 20:21:27 -07:00
Akilesh Kailash
c7c7252289 Merge "libsnapshot:Snapuserd: IO path support with dm-snapshot target" 2020-09-10 01:59:38 +00:00