Care maps need to store the original image size, which excludes bytes
used by hash tree or FEC code.
We used to propagate original image size using the global OPTIONS
dictionary. This is bad coding practice, and also fragile because we
have to make sure what's stored in dictionary and what's on disk are
consistent. Instead, let's read the content of images on disk, and parse
the AVB footer. The AVB footer contains the ground truth original image
size.
Test: build OTA, make sure the care maps have valid range
Bug: 246504616
Change-Id: I9250b478ab34dda60578a6b8c23ae6d7a9385788
We are removing VB related functionalities from release tools. This
change removes the hashtreeinfo and generator in verity_utils.py. It
also clears the function args using it.
Bug: 241044073
Test: atest under build/make
Change-Id: Icc298256be6917ffbd40cf4735f8e8092b541c9f
The monotonic property of the range is lost after the recent change to
subtract 0th block from the file range. As a result, we read the block
in the wrong order when pull files from the raw image. This yields a
incorrect hash during validation.
Bug: 150334561
Test: run validate_target_file. unittests
Change-Id: Id4fedaaf273c42c8d21a4549fda4c9a45e22066a
e2fsdroid can export holes as a "0" block, in order to make sure logical
and physical block assignments are maintained properly. Exclude these
hole blocks from range sets in LoadFileBlockMap.
Bug: 139201772
Test: m dist on non-A/B
Change-Id: Ic04e869572229678fdeaaa6fbf6f464d6c672051
These two tools are only used within build system, so not adding them
to otatools.zip.
Bug: 63866463
Test: TreeHugger
Test: Build a non-A/B target that calls make_recovery_patch.
Change-Id: Iaeddc98a2c4a27c6b49d91047500029e460db9e3
Also, move code from build_super_image.py to sparse_img.py.
Test: sparse_img.py on sparse and non-sparse images
Bug: 122377935
Change-Id: Ie91fdfdbb54298ea27eb20d1b5363aeb1470356e
Setting 'uses_shared_blocks' indicates that the block list for a given
file is incomplete, as some of the blocks are "owned" by others. As a
result, such a file will be skipped from imgdiff'ing.
This CL makes a copy of the original block list before removing the
shared blocks. It uses the original RangeSet as the value of
extra['uses_shared_blocks']. validate_target_files.py will try to read
the file as in the original RangeSet, unless the original list is also
incomplete or has non-monotonic ranges.
Test: Run validate_target_files on a target that uses
`BOARD_EXT4_SHARE_DUP_BLOCKS := true`.
Change-Id: I259e871ecc249ba0c14b5796bef413185a1b8242
Converts the following files to Python logging.
add_img_to_target_files.py
blockimgdiff.py
build_image.py
check_ota_package_signature.py
common.py
img_from_target_files.py
make_recovery_patch.py
ota_from_target_files.py
sparse_img.py
verity_utils.py
This separates logging outputs from normal outputs, and allows easier
switching between different verbosity levels. It also supports adjusting
logging config via environment variable (LOGGING_CONFIG).
Test: `m dist`
Test: `python -m unittest discover build/make/tools/releasetools`
Change-Id: Idfc3f7aff83e03f41f3c9b5067d64fd595d5192d
Generate the transfer command "compute_hash_tree" for incremental
updates of the non-A/B devices that enable verified boot 1.0
Other changes include:
i. factor out verity_utils to use both in build_image and blockimgdiff
ii. add a new flag 'hashtree_info_generator' in sparse_image to generate
the hashtree information.
Bug: 25170618
Test: generate a package for aosp_angler; and run simulator
Change-Id: I4d4d7a3e41dc3da810d2cbf8988e85d187c9ab0e
When target defines 'BOARD_EXT4_SHARE_DUP_BLOCKS := true', the generated
system/vendor images may contain shared blocks (i.e. some blocks will
show up in multiple files' block list), which violates the current
assumptions in BBOTA script.
This CL allows generating BBOTAs by considering the first occurrence as
the "owner" of the shared blocks. All the later users of the shared
blocks will have an incomplete block list, whose RangeSet's will be
tagged with 'uses_shared_blocks'.
Files with 'uses_shared_blocks' tag will not be diff'd with imgdiff,
potentially with patch size penalty. Such files will be accounted for in
imgdiff stats report, where we can revisit for a better solution.
Bug: 64109868
Test: Generate BBOTA full and incremental package with targets defining
'BOARD_EXT4_SHARE_DUP_BLOCKS := true'.
Change-Id: I87fbc22eef7fafe2a470a03fdcfa1babf088ea8d
The generator function is not thread safe and is prone to race
conditions. This CL uses a lock to protect this generator and loose the
locks elsewhere, e.g. 'WriteRangeDataToFd()'.
Bug: 71908713
Test: Generate an incremental package several times for angler 4208095 to 4442250.
Change-Id: I9e6f0a182a1ba7904a597f403f2b12fe05016513
The major issue with the existing implementation is unnecessarily
holding too much data in memory, such as HashBlocks() which first reads
in *all* the data to a list before hashing. We can leverage generator
functions to stream such operations.
This CL makes the following changes to reduce the peak memory use.
- Adding RangeSha1() and WriteRangeDataToFd() to Image classes. These
functions perform the operations on-the-fly.
- Caching the computed SHA-1 values for a Transfer instance.
As a result, this CL reduces the peak memory use by ~80% (e.g. reducing
from 5.85GB to 1.16GB for the same incremental, as shown by "Maximum
resident set size" from `/usr/bin/time -v`). It also effectively
improves the (package generation) performance by ~30%.
Bug: 35768998
Bug: 32312123
Test: Generating the same incremental w/ and w/o the CL give identical
output packages.
Change-Id: Ia5c6314b41da73dd6fe1dbe2ca81bbd89b517cec
Pad the sparse image with a zero fill chunk to correctly position
verity and FEC metadata at the end of the partition.
Bug: 27073791
Change-Id: I9f70d579a42e5007d50e9c02a98a608d2815f0ed
(cherry picked from commit 6a8781a251)
For squashfs, we currently don't have a system.map. So the whole system
image will be treated as a single file. But for some unknown bug, the
updater will be killed due to OOM when writing back the patched image to
flash (observed on lenok-userdebug MEA49). Prior to getting a real fix,
we evenly divide the non-zero blocks into smaller groups (currently 1024
blocks or 4MB per group).
Bug: 23227672
Change-Id: Ifeddd8d802f01f8cd2a743a1d1217a284fb6e182
dm-verity may touch some blocks that are not in the care_map due to
block device read-ahead. It will fail if such blocks contain
non-zeroes. As a workaround, we mark them as extended blocks and
zero out explicitly to avoid dm-verity failures.
Bug: 20881595
Change-Id: I54e24e70ad822c0d6d7af43301f74d24505f4461
In certain cases, we may have no mapping of ZERO or NONZERO blocks in
the image (such as squashfs at the moment). Don't write those groups if
they are empty.
Change-Id: I82aa05fbbd79130b08868585edd5298e531c4f5a
Similar to the assertations in file-based OTA, we perform verification
for block-based OTAs (BBOTAs) after updating a partition, for both of
the incremental and full OTAs. It increases the update time (~20s on
Nexus 6), but will capture unnoticed errors right away.
Bug: 21500869
Change-Id: I0f8b27734caaa0f41f9c1b904d55af2112784a68
(cherry picked from commit 68658c0f4f)
In ext4 filesystems, some blocks might be changed even being mounted
R/O, such as the superblock (block 0). We need to exclude such blocks
from integrity verification. Plus such blocks should always be
written to the target by copying instead of patching.
Bug: http://b/20939131
Change-Id: I657025b7b1ad50d4365e7b18dc39308facfe864e
(cherry picked from commit ff7778166b)
This caught a few bugs/syntax errors (a few character classes were not
escaped properly in regex patterns, some indentation was illegal,
etc).
Change-Id: I50637607524e68c4fb9cad7167f58a46b8d26b2c
Replace the xdelta/xz-based block OTA generation with a new system
based on the existing bsdiff/imgdiff tools.
Bug: 16984795
Change-Id: Ia9732516ffdfc12be86260b2cc4b1dd2d210e886
