Caller can optionally specify the verbose flag which overrides
OPTIONS.verbose. The command line won't be outputed with verbose=False.
This is useful for cases that a) those command lines are less useful
(but will spam the output otherwise); b) sensitive info is part of the
invocation.
'verbose=False' will be consumed by common.Run() only, instead of being
passed to subprocess.Popen().
Test: ota_from_target_files.py on a block based OTA.
Change-Id: I7d5b4094d756a60f84f89c6a965e7ccc68e435f8
When bsdiff/imgdiff fails, dump the output along with the src/tgt
filenames and block ranges. Also, exit the script gracefully instead
of raising an exception about iterating over 'NoneType'.
Bug: 31381530
Test: Package generates successfully for angler, and error outputs
correctly with error injection.
Change-Id: I06a2dfe545fbdff7043de05fee34b378453a9291
BBOTA v1 and v2 (introduced in L and L MR1 respectively) don't support
resumable OTA. We shouldn't generate packages using v1/v2 at the risk of
bricking devices.
BBOTA v3 (since M) and v4 (since N) both support resumable OTAs. BBOTA
v4 additionally supports using FEC to possibly recover a corrupted
image.
Bug: 33694730
Test: Generate full and incremental OTAs w/ and w/o the CL. They should
give identical packages (in v4).
Change-Id: Ib89d9cd63ba08e8e9aa4131bed18876b89d244c0
The major issue with the existing implementation is unnecessarily
holding too much data in memory, such as HashBlocks() which first reads
in *all* the data to a list before hashing. We can leverage generator
functions to stream such operations.
This CL makes the following changes to reduce the peak memory use.
- Adding RangeSha1() and WriteRangeDataToFd() to Image classes. These
functions perform the operations on-the-fly.
- Caching the computed SHA-1 values for a Transfer instance.
As a result, this CL reduces the peak memory use by ~80% (e.g. reducing
from 5.85GB to 1.16GB for the same incremental, as shown by "Maximum
resident set size" from `/usr/bin/time -v`). It also effectively
improves the (package generation) performance by ~30%.
Bug: 35768998
Bug: 32312123
Test: Generating the same incremental w/ and w/o the CL give identical
output packages.
Change-Id: Ia5c6314b41da73dd6fe1dbe2ca81bbd89b517cec
Otherwise the comparison is inconsistent between ReviseStashSize() and
WriteTransfers().
Bug: 35775675
Test: Successfully generate a previously failed incremental.
Change-Id: I554a51a210bf322cb5c79e28cf85607a417b094a
We check the needed stash size in ReviseStashSize(), and may not
generate a stash command if it would exceed the max allowed size. This
CL fixes a bug when skipping a stash operation: we shouldn't update the
'stashes' map if a stash command won't be generated.
Bug: 35313668
Test: Successfully generate the package that was failing due to the bug.
Change-Id: If0a3a5fadda9b4a4edad66a2a5826b5f978400ae
Only BBOTA v2 needs to maintain a pool of available 'stash slot id'.
BBOTA v3+ uses the hash of the stashed blocks as the slot id, which
doesn't need the id pool anymore.
Bug: 33694544
Test: Generate v2 and v4 incrementals w/ and w/o the CL. They produce
the same packages respectively.
Change-Id: I8121af5b6b1bee98c3639d54a00b06fd12e378e8
We compute the max stashed_blocks in ReviseStashSize(), prior to calling
WriteTransfers(), to avoid running out of space due to stashing.
There is a bug when computing the to-be-freed stashed blocks, where we
wrongly free the space _before_ executing the transfer command. This leads
to a script failure where the max stash size violates the max allowed
size in WriteTransfers().
Note that this bug doesn't affect already generated packages. It's only
an underestimate in ReviseStashSize(). The check in WriteTransfers() has
been correct to ensure the max stash size.
Bug: 33687949
Test: Successfully generated incremental OTA which failed previously.
Change-Id: I4f4f043c6f521fce81ca5286e6156f22d99bf7f7
set() doesn't keep elements according to the order of insertion. So
Transfers managed with set() in intermediate steps may not appear in the
same order across runs. This leads to slightly different output packages
when generating the same incremental OTA.
This CL fixes the issue by replacing set() with OrderedDict() in
blockimgdiff.GenerateDigraph() and blockimgdiff.FindVertexSequence().
It also adds a testcase that ensures blockimgdiff.GenerateDigraph()
preserves the insertion order for Transfer.goes_after set.
Bug: 32220816
Test: ota_from_target_files.py gives identical package when running
multiple times.
Change-Id: I56d551e5ca926993ab46896e33c80e0ce42e506a
In BBOTA, we generate patches based on _all_ the blocks of a pair of
input files (src and tgt). For security incremental OTAs, one common
pattern is that only a few blocks are changed in odex files (e.g.
headers). We don't really need to stash/patch the unchanged blocks.
This CL analyzes the unchanged blocks in odex files and computes the
diff for the changed blocks only. It reduces the OTA install time by
about 25% to 40% in our experiments, by paying an increase of 5% to 30%
OTA generation time cost.
Bug: 31570716
Test: Generate an incremental and apply on device.
Change-Id: If842c1afeff6894a3d27eb60b7e8f65a179b7977
Limit the number of blocks in each 'new' command to 1024. This should
decrease the chance of fsync error during an OTA update, similiar to
what we have done in b/29535618.
Bug: 29607757
Change-Id: I89f0a845d71eb810766e39860ab667c61b61a417
Limit the number of blocks in command zero to 1024 blocks. This
prevents the target size of one command from being too large and
might help to avoid fsync errors during the OTA update.
Bug: 29535618
Change-Id: Ic630cea2599138516162bd0029e2e4b2af75bf4f
(cherry picked from commit bb848c54a7)
Limit the number of blocks in command zero to 1024 blocks. This
prevents the target size of one command from being too large and
might help to avoid fsync errors during the OTA update.
Bug: 29535618
Change-Id: Ic630cea2599138516162bd0029e2e4b2af75bf4f
We use imgdiff to handle files in zip format (e.g. jar/zip/apk) for
higher compression ratio.
For system/vendor in squashfs, a) all files are compressed in LZ4
format; b) we use 4096-byte block size in their sparse images, but the
files in squashfs may not be laid out as 4K-aligned. So the blocks for
a given file as listed in block map may not form a valid zip file, which
may fail the patch generation with imgdiff.
Disable using imgdiff for squashfs images, and use bsdiff instead.
Bug: 22322817
Change-Id: Ie76aa4cece5c9d38cb1d1a34c505a4a8f37512d3
We used to erase all the unused blocks at the end of an update. In order
to work around the eMMC issue in b/28347095, we move part of the erase
commands, which won't be used during the update, to the beginning. The
reset would be erased at the end. This is in hopes of avoiding eMMC
getting starved for clean blocks which would otherwise fail an OTA.
Bug: 28347095
Change-Id: Ia96b06216b35d6700858687a66734af36d0ec213
For incremental BBOTAs, we used to verify the integrity of all the
blocks in the source partition. In order to reduce the time cost under
recovery, this CL changes to only verify the blocks that will be touched
in the given OTA package (BBOTA >= 3 only). This is a trade-off between
performance and reliability.
Bug: 27813356
Change-Id: I3975ae6f461f0f7e58d24f1df7df46a449d2988b
We use a bitset for blocks in the target image to assert a block hasn't
been touched before reading. Skip checking the blocks that are in the
source image only.
Bug: 27556903
Change-Id: I3a77292da673c813bd20d8dc177ff36419d8ecae
(cherry picked from commit dca2200c8a)
Add "ota-required-cache" into the metadata file in an OTA package,
which shows the minimum free space on /cache to apply the update.
Add "ota-type" into the metadata file, which shows the OTA type for
this package (i.e. one of FILE, BLOCK and AB).
Also add the cache free space check into updater-script when generating
block-based incremental OTAs (we only had such lines for file-based
incrementals before).
Bug: 26731903
Change-Id: Id6ff0fc4cdfb1443636b0b3800b0f8bddb5bb1d0
(cherry picked from commit d8d14bec0d)
Substantially improves the performance of some BlockImageDiff steps on
large images. GenerateDigraph and AssertSequenceGood are now linear in
the number of blocks in the target image rather than quadratic in the
size of the partition. FindVertexSequence tracks the score of each
vertex, the set of sources and sinks, and maintains a heap for finding
the best vertex to use for the greedy step, instead of repeatedly
re-scanning the set of remaining vertexes.
Change-Id: Ifeea5053d80ef4f06a5aa239c9a1797dd5e47841
(cherry picked from commit 2d2dd1526d)
Modify the number of max blocks per transfer to reduce splitting of
large files during patch generation. This will reduce the size of
final incremental OTA updater package. In specific, the size limit
for each file to transfer is set to 1/8 cachesize instead of a fix
number.
Bug: 26472126
Change-Id: Iffb135e378c4c7a91640f2fc4c40e9e93d1fe4bb
To accommodate new changes (such as error correction in [1]) to BBOTA
in N release. We bump up the version to keep the OTA script backward
compatible.
Needs the matching CL in commit
1fdec8685af858c5ff4f45d2e3059186ab5ed2ab.
[1]: commit 0a7b47397db3648afe6f3aeb2abb175934c2cbca
Change-Id: Ib9158b455cd5905fe2d4742ce81feb1b7583054f
CL in [1] added an assertion to make sure an empty RangeSet cannot call
RangeSet.to_raw_string(). However, we may unintentionally hit that when
generating BBOTA v1 script (e.g. for "new" commands that always have
empty src_ranges).
[1] commit e8f756137c
Change-Id: I1723606967f354ace9286d2b888fdbc80ba10e39
With the change in [1], it zeroes out extra blocks that might be read
by dm-verity. We should update the expected number of blocks that get
touched.
[1] commit e9b619108d.
Bug: 23786644
Change-Id: I7a7c3531e5af6bf381d3a14832e77abfa9a8efce
For a DataImage, we may have no mapping of ZERO or NONZERO blocks. Don't
write those groups if they are empty.
Bug: 19024504
Change-Id: I1e921effcdbf0661d4f4b541c104dd456ac0c582
In BBOTAs if the last block of a DataImage is padded, we should always
write the whole block even for incremental OTAs. Because otherwise the
last block may be skipped if unchanged, but would fail the post-install
verification if it has non-zero contents in the padding bytes.
Bug: 23828506
Change-Id: I4b0af7344d18261258cd48d18c029c089d6ff365
For BBOTA v3, we need to stash source blocks to support resumable
feature. However, with the growth of file size and the shrink of the
cache size, source blocks that represent a file are too large to be
stashed as a whole. CL in [1] solves the issue by replacing the diff
command with a "new" command. However, it may increase the generated
package size substantially (e.g. from ~100MB to ~400MB).
With this CL, if a file spans too many blocks, we split it into smaller
pieces by generating multiple commands. For the same case above, it
reduces the package size to ~150MB.
One potential downside is that after splitting, files like .jar,
.apk and .zip can no longer use imgdiff. We may lose the potential
benefit of using imgdiff for patch size reduction.
[1] commit 82c47981bd
Bug: 22430577
Change-Id: I5684fab0ac41c8d94fe18f52a34cef77e06f741c
When generating incremental BBOTAs (v2 and above), we need to ensure
that the needed runtime stash is below the given threshold. If it's
running out of space on /cache, we replace the command that uses a
stash with a "new" command instead.
This may increase the OTA package size, since it is carrying more full
blocks instead of patches. It gets even worse for large files that span
a number of blocks, because currently we will store all the blocks for
the file as "new" blocks if stashing cannot be satisfied. We may further
optimize by splitting them into smaller chunks so that most of them can
still be stashed.
Bug: 22430577
Change-Id: I5a49e361adc7d3d41de2e9c08ee9b08c1e6c091a
In BBOTA v2, the stash size computation is inaccurate. A stash will be
used only once, so we deduct its size from stashed_blocks right after
its use. However, the stash doesn't actually get freed until being
overwritten by another stash or destroyed at the end of an update. This
leads to OTA failures due to insufficient space on /cache. This CL
changes the BBOTA v2 script to generate an explicit free right after
its use.
Bug: 23119955
Change-Id: Icdd8352bb0ff778945937c870990a888c7a4d38b
For some old builds, we may not define cache partition size. Change the
exception into a warning to make the script backward compatible.
Change-Id: Ie94c7fbb1a9f3a7db3f16e8d845e493a534aac5b
With block-based OTA v2 and v3, it requires stash space on the /cache
partition to back up blocks during an update. We need to ensure that
it doesn't exceed the partition size. Since there might be other files
on /cache as well, we use cache_size * threshold as the maximum allowed
size. The threshold defaults to 0.8, which can be overridden by command
line option '--stash_threshold'.
Change-Id: I5c2e353ea4716328d6cdc6a7f758520de8d88b8e
(cherry picked from commit 23ac404212)
dm-verity may touch some blocks that are not in the care_map due to
block device read-ahead. It will fail if such blocks contain
non-zeroes. As a workaround, we mark them as extended blocks and
zero out explicitly to avoid dm-verity failures.
Bug: 20881595
Change-Id: Id097138bfd065c84eac088b3ad49758010b2927b
(cherry picked from commit 2fd2c9bfd6)
Similar to the assertations in file-based OTA, we perform verification
for block-based OTAs (BBOTAs) after updating a partition, for both of
the incremental and full OTAs. It increases the update time (~20s on
Nexus 6), but will capture unnoticed errors right away.
Bug: 21500869
Change-Id: I89ac8fe623b855721b7affd07cf9426a23433ab0
In ext4 filesystems, some blocks might be changed even being mounted
R/O, such as the superblock (block 0). We need to exclude such blocks
from integrity verification. Plus such blocks should always be
written to the target by copying instead of patching.
Bug: http://b/20939131
Change-Id: I991169ec307dfb231b2fe8908a0668595ecb2060
The updater automatically stashes overlapping source blocks to
make it possible to resume if the command performing the update
is interrupted. However, blockimgdiff.py does not take this into
account and therefore, max_stashed_blocks indicates only the
number of explicitly stashed blocks. Change blockimgdiff.py to
correctly calculate the maximum used stash space, including any
automatically stashed blocks.
Bug: 20309033
Change-Id: Ibf5cb1abc38c54e60a6ad3ce4cbce43556b08ac2
This caught a few bugs/syntax errors (a few character classes were not
escaped properly in regex patterns, some indentation was illegal,
etc).
Change-Id: I50637607524e68c4fb9cad7167f58a46b8d26b2c
The generated OTAs are supposed to be repeatable. The use of dict in
blockimgdiff.py breaks the assumption for block-based OTAs. Fixed by
using OrderedDict instead.
Change-Id: I945bdc879912ba174ca330c152b1f8fc7ed211ac
Add source and target block hashes as parameters to transfer list
commands that copy or patch data to a partition. This allows the
updater to verify the status of each command in the transfer list
and makes resuming block based OTAs possible. Due to the changes,
update the transfer list version to 3.
Needs matching changes from
I1e752464134aeb2d396946348e6041acabe13942
Bug: 18262110
Change-Id: Ia5c56379f570047f10f0aa7373a1025439495c98
(cherry picked from commit cac671a9d1)
The erase command we insert at the top for full OTAs was getting
inserted in the wrong place for version 2.
Change-Id: I9caf03a40efbdba79f3428f73e50d4319d9ba371
Generate version 2 of the block_image_update transfer list format.
This improves patch size by a different strategy for dealing with
out-of-order transfers. If transfer A must be done before transfer B
due to B overwriting A's source but we want to do B before A, we
resolve the conflict by:
- before B is executed, we save ("stash") the overlapping region (ie
the blocks B will overwrite that A wants to read)
- when A is executed, it will read those parts of source data from
the stash rather than from the image.
This reverses the ordering constraint; with these additions now B
*must* go before A. The implementation of the stash is left up to the
code that executes the transfer list to apply the patch; it could hold
stashed data in RAM or on a scratch disk such as /cache, if available.
The code retains the ability to build a version 1 block image patch;
it's needed for processing older target-files.
Change-Id: Ia9aa0bd45d5dc3ef7c5835e483b1b2ead10135fe
Move BlockDifference into common and make its script generation code
more complete, so that it can be use by releasetools.py to do diffs on
baseband images.
Bug: 16984795
Change-Id: Iba9afc1c7755458ce47468b5170672612b2cb4b3
Replace the xdelta/xz-based block OTA generation with a new system
based on the existing bsdiff/imgdiff tools.
Bug: 16984795
Change-Id: Ia9732516ffdfc12be86260b2cc4b1dd2d210e886
