async merge.
If there are any I/O errors during async merge, we will
retry the I/O in synchronous I/O path. For this to happen,
we have to reset the iterator so that we replay the blocks
which were partially completed during async merge. Furthermore,
we will disable the async merge and continue to do the I/O
in synchronous path.
Additionally, cut down the queue depth to 8 so that
it will decerease the number of the async offload. We don't
want to have a big queue depth with async offload.
Bug: 220991038
Test: Instrument the code to fail the Async I/O's
randomly and make sure merge is completed. Instrumentation
was done both on readahead and merge code path.
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: I0db6d0f46054ca5b8423201a598c726b2c3d21ac
This piece of comment was misleading because it only applies to
BOARD_USES_RECOVERY_AS_BOOT devices. Update the text to give a more
accurate description.
If BOARD_USES_RECOVERY_AS_BOOT is true,
* Recovery ramdisk IS boot ramdisk.
* init_first_stage is actually a symbolic link to
init_second_stage.recovery, which links libfs_mgr.recovery.
If BOARD_USES_RECOVERY_AS_BOOT is not true,
* init_first_stage is a real binary in the generic ramdisk.
* init_first_stage links libfs_mgr.ramdisk.
* During recovery boot, the '/init' binary could be the
init_first_stage from the generic ramdisk (A/B), or
init_second_stage.recovery from the recovery ramdisk (non-A/B;
standalone recovery partition).
Bug: 219811240
Test: None
Change-Id: Ib395a796f61869c13f1a5f1735ef17c224c26c8c
The intention of inserting this synthesised mount entry is for the
FirstStageMount() procedure to eventually fallback to this entry if all
previous mount lines failed.
In order for FirstStageMount() to retry mount, the mount lines have to
be grouped together.
This change ensures that would happen and add test.
Bug: 220074274
Test: atest CtsFsMgrTestCases
Change-Id: Id042a6a6738d27c06a397ef7f4e0977907371c05
Certain code in libfiemap should be unreachable in recovery/ramdisk
builds, but whether intentionally or not, __ANDROID_RECOVERY__ is not
working in vendor_boot.
To make sure this is doing exactly what we want, use __ANDROID_RAMDISK__
instead.
Bug: 205987817
Test: build boot, vendor_boot, recovery
Change-Id: Id419f7d9068e791427a896c22569ae4543548df7
When creating a scratch partition on /data, use either the size of
super, or half the free space on /data - whichever is smaller.
Bug: 213617068
Test: adb remount
Change-Id: If42fe9e1dc53d5a0137ace917a440232f70947fa
This ensures that the dynamic "scratch" partition is removed when doing
a flashall operation.
If "scratch" is on /data, disable the partition. Add IsImageDisabled to
ImageManager so EnsureScratchMapped can skip mapping it. Also, fix
"scratch" not getting unmapped if on /data.
Bug: 205987817
Test: adb remount
adb sync
adb reboot fastboot
fastboot flashall --skip-reboot # no errors
Test: adb-remount-test.sh
Change-Id: I4b9702e1dac15fb663635506fb50a8274e1e10d1
I/O failures seen on one specific device intermittently.
Bug: 219642530
Test: OTA
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: I5b6fbfb13e40b15b25324b70efef11aede916596
* changes:
libsnapshot: Propagate io_uring enablement to daemon
snapuserd: Cut down worker threads
snapuserd: Terminate daemon spin up during first stage init
snapuserd: Avoid checking system properties when daemon launched from fist stage init and during selinux transition
During selinux init transition, system properties
are not yet enabled. Hence, store the io_uring feature
in snapshot.proto and propagate the same snapuserd
daemon
Bug: 214340811
Test: OTA
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: I5458127b00946b5254d5c68d407ab525a0075cb9
We don't need 4 threads per partition all the time.
Cut down to 1 thread for all purposes except during boot time.
During boot, we need multiple worker threads to serve
I/O's to speed up the booting process. For all other
purposes, single thread is sufficient. It will cut
down memory usage ~25MB.
Bug: 214340811
Test: OTA
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: Ib36bf44c63676d3f3169fbda04b7fc3d1760cfbf
from fist stage init and during selinux transition
Bug: 207298357
Test: OTA
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: I449de1939531dbf3cc35e05c80e51237c4dc2b99
This is a race between init process and bionic libc initialization of
snapuserd.
init->fork() ----------------> SecondStageMain() -> PropertyInit()
|
|
v
execveat ---> __libc_init_common() -> __system_properties_init()
(snapuserd)
When init process calls PropertyInit(), /dev/__properties__ directory
is created. When bionic libc of snapuserd daemon invokes __system_properties_init
_after_ init process PropertyInit() function is invoked, libc will
try to initialize the property by reading
/system/etc/selinux/plat_property_contexts. Since any reads on /system
has to be served by snapuserd, this specific read from libc cannot be
serviced leading to deadlock.
Reproduce the race by inducing a sleep of 1500ms just before execveat()
so that init process calls PropertyInit() before bionic libc
initialization. This leads to deadlock
immediately and with additional kernel instrumentation with debug
logs confirms the failure:
======================================================
init: Relaunched snapuserd with pid: 428
ext4_file_open: SNAPUSERD: path /system/etc/selinux/plat_property_contexts - Pid: 428 comm 8
ext4_file_read_iter: SNAPUSERD for path: /system/etc/selinux/plat_property_contexts pid: 428 comm 8
[ 25.418043][ T428] ext4_file_read_iter+0x3dc/0x3e0
[ 25.423000][ T428] vfs_read+0x2e0/0x354
[ 25.426986][ T428] ksys_read+0x7c/0xec
[ 25.430894][ T428] __arm64_sys_read+0x20/0x30
[ 25.435419][ T428] el0_svc_common.llvm.17612735770287389485+0xd0/0x1e0
[ 25.442095][ T428] do_el0_svc+0x28/0xa0
[ 25.446100][ T428] el0_svc+0x14/0x24
[ 25.449825][ T428] el0_sync_handler+0x88/0xec
[ 25.454343][ T428] el0_sync+0x1c0/0x200
=====================================================
Fix:
Before starting init second stage, we will wait
for snapuserd daemon to be up and running. We do a simple probe by
reading system partition. This read will eventually be serviced by
daemon confirming that daemon is up and running. Furthermore,
we are still in the kernel domain and sepolicy has not been enforced yet.
Thus, access to these device mapper block devices are ok even though
we may see audit logs.
Note that daemon will re-initialize the __system_property_init()
as part of WaitForSocket() call. This is subtle but important; since
bionic libc initialized had failed silently, it is important
that this re-initialization is done.
Bug: 207298357
Test: Induce the failure by explicitly delaying the call of execveat().
With fix, no issues observed.
Tested incremental OTA on pixel ~15 times.
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: I86c2de977de052bfe9dcdc002dcbd9026601d0f3
This patch attempts to diagnose snapuserd hangs by performing reads
immediately after entering second-stage init. This is done by spawning
two threads: one to perform the reads, and another to wait for the read
thread to finish. If any aspect of the read fails, or the read thread
does not complete in 10 seconds, then a list of snapuserd's open file
descriptors are logged.
Bug: 207298357
Test: apply working OTA, check logcat for success
apply broken OTA, check logcat for fd map
Change-Id: I549e07b7d576fcdaca9b2d6ff33e0924c3812c07
* changes:
Add a new property to disable io_uring and run vts and snapuserd_test
snapuserd: Async I/O for block verification
snapuserd: Use io_uring api's for snapshot merge
vts and snapuserd_test
This should be run on cuttlefish
Bug: 202784286
Test: vts_libnspahost_test, snapuserd_test
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: I2c28e98f04beca770b8a6efa9474d602fe26f514
using io_uring READ/WRITE opcodes for snapshot merge.
Specifically, it is used only for readahead and ordered ops
code path.
Snapshot merge perf:
===========================================================
Incremental OTA of 300M between two git_master branches on Pixel 6:
===========================================================
On Android S (with dm-snapshot): ~15 minutes:
update_engine: [INFO:cleanup_previous_update_action.cc(330)] Merge finished with state MergeCompleted.
update_engine: [INFO:cleanup_previous_update_action.cc(130)] Stopping/suspending/completing CleanupPreviousUpdateAction
update_engine: [INFO:cleanup_previous_update_action.cc(501)] Reporting merge stats: MergeCompleted in 926508ms (resumed 0 times), using 0 bytes of COW image.
===========================================================
On Android T (with io_uring: ~38 seconds):
update_engine: [INFO:cleanup_previous_update_action.cc(330)] Merge finished with state MergeCompleted.
update_engine: [INFO:cleanup_previous_update_action.cc(130)] Stopping/suspending/completing CleanupPreviousUpdateAction
update_engine: [INFO:cleanup_previous_update_action.cc(501)] Reporting merge stats: MergeCompleted in 38868ms (resumed 0 times), using 0 bytes of COW image.
===========================================================
Bug: 202784286
Test: Full/Incremental OTA
Signed-off-by: Akilesh Kailash <akailash@google.com>
Change-Id: I24ed3ae16730d0a18be0350c162dc67e1a7b74e1
"-o discard" issues UNMAP commands to loopback resulting in punch_hole
on the pinned file in /data. That will break the pinned block map.
Bug: 210589189
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
Change-Id: Ia927c43fc75164ce5929173f5740737eac4de484
