platform_system_core/fs_mgr

Android Overlayfs integration with adb remount

Introduction

Users working with userdebug or eng builds expect to be able to remount the system partition as read-write and then add or modify any number of files without reflashing the system image, which is understandably efficient for a development cycle. Limited memory systems that chose to use readonly filesystems like squashfs, or Logical Resizable Android Partitions which land system partition images right-sized, and with filesystem that have been deduped on the block level to compress the content; means that either a remount is not possible directly, or when done offers little or no utility because of remaining space limitations or support logistics.

Overlayfs comes to the rescue for these debug scenarios, and logic will automatically setup backing storage for a writable filesystem as an upper reference, and mount overtop the lower. These actions will be performed in the adb disable-verity and adb remount requests.

Operations

Cookbook

The typical action to utilize the remount facility is:

$ adb root
$ adb disable-verity
$ adb reboot
$ adb wait-for-device
$ adb root
$ adb remount

Followed by one of the following:

$ adb stop
$ adb sync
$ adb start
$ adb reboot

or

$ adb push <source> <destination>
$ adb reboot

Note that the sequence above:

$ adb disable-verity
$ adb reboot

or

$ adb remount

can be replaced in both places with:

$ adb remount -R

which will not reboot if everything is already prepared and ready to go.

None of this changes if overlayfs needs to be engaged. The decisions whether to use traditional direct filesystem remount, or one wrapped by overlayfs is automatically determined based on a probe of the filesystem types and space remaining.

Backing Storage

When overlayfs logic is feasible, it will use either the /cache/overlay/ directory for non-A/B devices, or the /mnt/scratch/overlay directory for A/B devices that have access to Logical Resizable Android Partitions. The backing store is used as soon as possible in the boot process and can occur at first stage init, or at the mount_all init rc commands.

This early as possible attachment of overlayfs means that sepolicy or init itself can also be pushed and used after the exec phases that accompany each stage.

Caveats

• Space used in the backing storage is on a file by file basis and will require more space than if updated in place. As such it is important to be mindful of any wasted space, for instance BOARD_IMAGE_PARTITION_RESERVED_SIZE being defined will have a negative impact on the overall right-sizing of images and thus free dynamic partition space.
• Kernel must have CONFIG_OVERLAY_FS=y and will need to be patched with "overlayfs: override_creds=off option bypass creator_cred" if kernel is 4.4 or higher. The patch is available on the upstream mailing list and the latest as of Feb 8 2019 is https://lore.kernel.org/patchwork/patch/1009299/. This patch adds an override_creds mount option to overlayfs that permits legacy behavior for systems that do not have overlapping sepolicy rules, principals of least privilege, which is how Android behaves.
• adb enable-verity will free up overlayfs and as a bonus the device will be reverted pristine to before any content was updated. Update engine does not take advantage of this, will perform a full OTA.
• Update engine may not run if fs_mgr_overlayfs_is_setup() reports true as adb remount overrides are incompatible with an OTA resources.
• For implementation simplicity on retrofit dynamic partition devices, take the whole alternate super (eg: if "a" slot, then the whole of "system_b"). Since landing a filesystem on the alternate super physical device without differentiating if it is setup to support logical or physical, the alternate slot metadata and previous content will be lost.
• If dynamic partitions runs out of space, resizing a logical partition larger may fail because of the scratch partition. If this happens, either fastboot flashall or adb enable-verity can be used to clear scratch storage to permit the flash. Then reinstate the overrides and continue.
• File bugs or submit fixes for review.
• There are other subtle caveats requiring complex logic to solve. Have evaluated them as too complex or not worth the trouble, please File a bug if a use case needs to be covered.
  • The backing storage is treated fragile, if anything else has issue with the space taken, the backing storage will be cleared out and we reserve the right to not inform, if the layering does not prevent any messaging.
  • Space remaining threshold is hard coded. If 1% or more space still remains, overlayfs will not be used, yet that amount of space remaining is problematic.
  • Flashing a partition via bootloader fastboot, as opposed to user space fastbootd, is not detected, thus a partition may have override content remaining. adb enable-verity to wipe.
  • Space is limited, there is near unlimited space on userdata, we have made an architectural decision to not utilize /data/overlay/ at this time. Acquiring space to use for backing remains an ongoing battle.
  • First stage init, or ramdisk, can not be overriden.
  • Backing storage will be discarded or ignored on errors, leading to confusion. When debugging using adb remount it is currently advised to confirm update is present after a reboot to develop confidence.