Create a wrapper class that handles the payload signing, which unifies
the paths with and without external signer. Also add tests for the newly
added class.
The test keys under testdata/ are created with the script in
development/tools/make_key. testdata/testkey_with_passwd.pk8 uses
password "foo".
Test: python -m unittest test_ota_from_target_files
Test: Get identical A/B OTA packages w/ and w/o the CL.
Change-Id: Ic770aec726498a3babb88ec509985e7f1210fb18
The output (stdout and stderr combined) would be helpful in understanding
the cause of the failure.
Not changing other occurrences in build_image.py yet, since it's less
obvious if a failed call should be considered fatal (thus whether to
dump such outputs). For example, GetVeritySize() simply returns 0 on
failed calls. They deserve further clean-ups in later CLs (with unit
tests).
Bug: 71864688
Test: Inject errors into mke2fs/e2fsck/simg2img. `m snod` prints outputs.
Test: `m snod` remains quiet on successful runs.
Change-Id: I172403f6cd05bce93767265dbcb110271a68e2a8
This CL makes the following changes:
(a) cleans up the similar codes in AddCareMapTxtForAbOta() that handle
'system' and 'vendor' partitions;
(b) fixes an issue with the arcname in AddPackRadioImages() and
AddRadioImagesForAbOta(), where forward slash should always be used
in zip entry names;
(c) refactors the branching statements in AddRadioImagesForAbOta() to
reduce indentation levels.
Test: python -m unittest test_add_img_to_target_files
Test: `m dist` with aosp_marlin-userdebug. Check META/care_map.txt in the
generated target_files.zip.
Change-Id: I3d6f794962d0c68390fbd18eb13c2622acab3ff5
Separate three functions out of AddImagesToTargetFiles(), into
AddCareMapTxtForAbOta(), AddRadioImagesForAbOta() and
AddPackRadioImages() respectively. This CL tries to apply minimal
changes for the refactoring purpose.
Also add tests for AddRadioImagesForAbOta() and AddPackRadioImages().
The tests for AddCareMapTxtForAbOta() require better testing support to
mock sparse_img.SparseImage, which will be added in later CLs.
Test: python -m unittest test_add_img_to_target_files
Test: `m dist` with aosp_marlin-userdebug. Check META/care_map.txt in
the generated target_files.zip.
Change-Id: I1bb723c15237ff721f165cfce0ce996008ce9948
Now they (A/B, non-A/B full and non-A/B incremental) share the same
code. Also add tests for the newly added function.
Test: python -m unittest test_ota_from_target_files
Test: Build A/B and non-A/B packages respectively.
Change-Id: I5499dd9c683ac1a18d8a24f21ff8845fe69f343e
This helps to generate a deterministic package.
Bug: 71770360
Test: Generate a incremental package and transfers are added by file name.
Change-Id: I7562a200b97a1babbb09a77801324cc9408cc01f
In non-A/B OTA path, we've been loading the info dict for the target
build twice (once to have a peek at the OTA type, with a reload after
unzipping the target-files zip). Remove the reloading.
This CL moves the loading of the source info dict up a bit to avoid
having two copies in the A/B and non-A/B paths. It also moves up the
sanity checks of the loaded info dicts before unzipping the target-files
zips.
Test: Generate full and incremental packages for bullhead and marlin.
Change-Id: Iccb953feb0b6ecf62e6f73d6e9ceea00077d098d
GetBuildProp() and GetVendorBuildProp() have been factored into
BuildInfo class. These two functions were meant to be deleted in commit
481bab8d48.
Test: python -m unittest test_ota_from_target_files
Test: No users of these two functions.
Change-Id: Id590a46d597939973ef2339cdfea4f5163548d4d
The fallback-to-full logic doesn't look beneficial, but may hide away
real bugs, especially when it's on by default.
Test: Generate an incremental non-A/B OTA successfully.
Test: Inject bugs into script; observe that incremental OTA
generation errors out instead of silently falling back.
Test: No internal users of the flag from code search.
Change-Id: Ifc627d898552c0bd72463e8c107e36610421608e
We split large apks and generated patches for them in parallel,
resulting in nondeterminate packages between different runs. This CL
sort the split transfers by target name first; and then add them
sequentially to the final transfer list.
Also fix a side effect where we may generate a wrong sha1 for split
ranges due to synchronization error.
Bug: 71770360
Bug: 71759418
Test: Generate the package several times, compare the log and the transfer list.
Change-Id: I2a49e22594d59ffaa98b11edc776be4e3c4c561f
Prior to this CL, we have similar codes in different paths
(WriteFullOTAPackage / WriteBlockIncrementalOTAPackage /
WriteABOTAPackageWithBrilloScript). This CL factors out the common codes
that deal with OEM-specific properties into BuildInfo class, and adds
tests for the new class.
Test: python -m unittest test_ota_from_target_files
Test: Create an incremental package that uses and doesn't use OEM
properties respectively.
Test: Create a full package that uses and doesn't use OEM properties
respectively.
Change-Id: Ida914cace12803d55396fa503bafcac2db2a520e
Test: python -m unittest test_common
Test: Run sign_target_files_apks.py on a target with compressed APKs.
Change-Id: I107a8b8f2f0f82e2d1947f14c8a8b3778f633b11
Mostly cosmetic changes, such as moving print statement to print
function. The only functional change is to wrap the file opening in
Append() with try statement, which would dump the error message as
needed.
Test: `m dist`
Test: python -m unittest test_build_image
Change-Id: I4e6c593517cf737bfcbe51bc533b22a247b10e44
This simplifies the work to track mkdtemp()'d dirs. This CL also cleans
up the temporary file creation in build_image.py.
Test: `m dist`
Test: python -m unittest test_common
Change-Id: Ied7d86126714b9b26e5df7bcca1dfdea21cfcf60
Mostly cosmetic changes, such as replacing print statement with print
function. Also change 'import cStringIO' to optionally look for the one
in io module, to allow Python 2/3 compatibility.
Test: pylint --rcfile=pylintrc sign_target_files_apks.py
Test: Run sign_target_files_apks.py on marlin target_files.zip.
Change-Id: I4dc98b01da6f89e624114bbca5522f659901c1f2
HeapItem defines __bool__(), which contains a logical error that should
return the opposite value.
Note that the bug only manifests while using Python 3, which calls
__bool__(). With Python 2, `if x:` or bool(x) actually calls
x.__nonzero__() or x.__len__(). If a class defines neither __len__() nor
__nonzero__(), as the case in HeapItem, it always returns True.
Test: python -m unittest test_blockimgdiff
Test: python3 -m unittest test_blockimgdiff
Test: Generate an incremental non-A/B OTA package successfully.
Change-Id: Ibe8430e0b495a7d2f430cfffb716d2536ffb53d2
The code is on infeasible path since we already have assertions in
common.BlockDifference().
Also remove the dead code that checks for OPTIONS.info_dict, as we
already set that in ota_from_target_files.main(), for both of A/B and
non-A/B.
Test: Generate incremental OTAs w/ and w/o the CL, and get identical
packages.
Change-Id: Ifb8fc101e78f5ce58c60c8e49028b66ce0d20246
The CL in [1] unintentionally breaks the OEM dict loading logic in the
incremental BBOTA path. We should always require and load the OEM
property dict if _either_ of the source and target builds uses OEM
properties. Otherwise with the current "and" operator, it skips loading
the OEM property dict and thus fails to generate an OTA package that has
OEM property changes (e.g. updating from build with fingerprint to
another one using thumbprint).
The CL in [1] actually makes the right change in the file-based OTA
path, but introduces the bug in the block-based OTA path.
This CL also cleans up the line that reads recovery_mount_options.
[1] commit 7f804ba71f ("releasetools:
allow for multiple OEM property values.").
Test: Genearte an OTA that has OEM property changes successfully.
Change-Id: Idce4ad59825d432618535ce09ab22bd7ddc524f2
We used to have "--gen_verify" that would generate a verify package for
non-A/B devices. Since a) we don't have active users; b) it works with
non-A/B only; c) we have better alternatives (e.g. using fastboot to
compute the partition checksum), this CL removes the support.
Test: `m dist`
Change-Id: Ib4a2c8c2a0394b54e66c106089aa52a0b900f034
We have been shelling out to 'zip -d' to delete existing ZIP entries in
add_img_to_target_files.py. This CL moves the function into common.py,
and calls that for the similar work in ota_from_target_files.py. This CL
also adds unittests for the newly added function.
Test: `m dist`
Test: python -m unittest test_common
Test: ota_from_target_files.py generates identical packages w/ and w/o
the CL (so we know the streaming property computation is intact).
Test: Run 'add_img_to_target_files.py -a' that triggers a call to
ReplaceUpdatedFiles().
Change-Id: Icaa6c3ea3ee2166023f78fa79275295f837ea842
For devices using derived fingerprint (i.e. /system/build.prop doesn't
contain ro.build.fingerprint, but has ro.build.thumbprint instead), the
current code (in android.os.Build) doesn't have a matching logic to do
the same for ro.vendor.build.fingerprint. This means we will see
ro.build.thumbprint in /system/build.prop, while there's no matching
ro.vendor.build.thumbprint in /vendor/build.prop.
From signing script point of view, it should just apply the tag
replacement (e.g. test-keys -> release-keys) for whatever it sees when
signing a target_files.zip.
This CL also adds unit tests for EditTags() and RewriteProps().
Fixes: 27950003
Test: Use 'sign_target_files_apks.py' to sign a target that uses derived
fingerprint and vendor partition. Check VENDOR/build.prop.
Test: python -m unittest test_sign_target_files_apks
Change-Id: I09019da970840cd82f54b68a32b4e94984bc1d8d
The test is mostly trivial, but it ensures the result parsing from
actual calls to mke2fs.
Test: python -m unittest test_build_image
Test: `m dist` on aosp_marlin-userdebug (w/ and w/o defining
PRODUCT_SYSTEM_HEADROOM respectively).
Change-Id: I8b9964213950e76f6d7d5518414a1bab888b4706
This is only used in file-based OTA, where we may reserve space on
/system partition. With the deprecation of file-based OTA since O, the
code has become obsolete.
Test: `m dist`
Change-Id: I2bd686b292f93b566fe28a2bcd74c564ffc75dac
'-w' is a short option for '--wipe_user_data', which happens to be the
same as the whole-file signing flag for signapk.
This CL removes the short version to make it less error-prone. Users
should use '--wipe_user_data' to generate packages with enforced data
wipe.
Bug: 70259309
Test: ota_from_target_files.py generates packages with and without
'--wipe_user_data'.
Test: Not seeing active users that uses 'ota_from_target_files -w'.
Change-Id: Ib8c138663a87aafc6a0b5b6f7c286fb74ef6134b
BuildImage can return false for lots of reasons. Log two of the more
plausible ones. You'll still need to edit this file and pass True to
RunCommand to work out *what* went wrong, but at least this logging will
tell you *where* things went wrong.
Bug: http://b/63142920
Test: debugged my actual problem
Change-Id: If84427058108dcac0a1b5e2216b8bb11eef522ca
This also updates mkf2fsuserimg.sh to call sload.f2fs.
Change-Id: I245f6b3dfcd966fbc6783329d781375879fcd065
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
C:661, 0: Wrong hanging indentation (add 2 spaces).
OPTIONS.input_tmp, "VENDOR_IMAGES")
^ | (bad-continuation)
C:673, 0: Wrong continued indentation (add 14 spaces).
os.path.join("IMAGES", img_name))
^ | (bad-continuation)
C: 54, 0: Import "import datetime" should be placed at the top of the module (wrong-import-position)
C: 55, 0: Import "import hashlib" should be placed at the top of the module (wrong-import-position)
C: 56, 0: Import "import os" should be placed at the top of the module (wrong-import-position)
C: 57, 0: Import "import shlex" should be placed at the top of the module (wrong-import-position)
C: 58, 0: Import "import shutil" should be placed at the top of the module (wrong-import-position)
C: 59, 0: Import "import subprocess" should be placed at the top of the module (wrong-import-position)
C: 60, 0: Import "import tempfile" should be placed at the top of the module (wrong-import-position)
C: 61, 0: Import "import uuid" should be placed at the top of the module (wrong-import-position)
C: 62, 0: Import "import zipfile" should be placed at the top of the module (wrong-import-position)
C: 64, 0: Import "import build_image" should be placed at the top of the module (wrong-import-position)
C: 65, 0: Import "import common" should be placed at the top of the module (wrong-import-position)
C: 66, 0: Import "import rangelib" should be placed at the top of the module (wrong-import-position)
C: 67, 0: Import "import sparse_img" should be placed at the top of the module (wrong-import-position)
W:391,12: Redefining built-in 'dir' (redefined-builtin)
After this CL, it still gives the warning below, which is to be
addressed with follow-ups.
R:635, 2: Too many nested blocks (6/5) (too-many-nested-blocks)
Test: `m dist`
Test: pylint --rcfile=pylintrc add_img_to_target_files.py
Change-Id: I64da184b6b69e93449dbfc989a5d7f46d5223f42
This option u'required-attributes' will be removed in Pylint 2.0
This option u'ignore-iface-methods' will be removed in Pylint 2.0
Test: pylint (1.6.5) no longer gives above warnings.
Change-Id: Ib07cd5aef87407260140c1fca0b11c439cbdf390
In some non-A/B setups, recovery.img is still being used. If AVB is
enabled, we currently don't add a hash footer to recovery.img nor do
we include the hash digest in vbmeta.img. This CL fixes that.
This was tested on a build with the following settings
TARGET_NO_RECOVERY := false
BOARD_USES_RECOVERY_AS_BOOT := false
BOARD_BUILD_SYSTEM_ROOT_IMAGE := false
BOARD_RECOVERYIMAGE_PARTITION_SIZE := 33554432
BOARD_AVB_RECOVERY_ADD_HASH_FOOTER_ARGS := --prop foo:bar
and then it was verified using 'avbtool info_image' that recovery.img
has a hash footer and a 'foo' property with the value 'bar'. This was
also checked successfully for vbmeta.img.
Test: See above.
Bug: None
Change-Id: I98124d5661ea768411416fa8d2a2ae6cc664fdc8
Imgdiff expects the input files to be valid zip archives. During the
split of large apks, imgdiff may fail when there's a hole in the input
file; potentially due to the blocks allocation of mke2fs. This CL solves
the issue by falling back to normal split in case of the imgdiff split
failure. The split transfers will then use bsdiff instead.
Bug: 69624507
Test: generate the incremental package for the failed targets and check the
transfers.
Change-Id: I4882452378123e60ad3434053b57f33e53ac4b82
... to include system/update_engine/scripts if ANDROID_BUILD_TOP is set
(e.g. after lunching a target).
Test: ./build/make/tools/releasetools/check_ota_package_signature.py
works without manually setting PYTHONPATH.
Change-Id: I0c5101fab2dd69cb6a598b909aa04bc4d5f45284
Bug: 68224784
Test: Enable AVB on aosp_bullhead-userdebug. `m dist`. Note that the
prebuilt vendor.img needs to be AVB-signed first.
Test: `m dist` with aosp_walleye-userdebug.
Change-Id: I18235e4d0dde6af71a96e46ec434480cc9a22bef
AddImagesToTargetFiles() takes either a zip file, or a zip root as
input. We used to create IMAGES/ directory only when working with zip
root input. Commit 262bf3f0b5 has changed
to also stage boot / recovery images there when working with a zip file.
This CL makes sure the directory is always available under both modes.
Bug: 63456822
Test: zip -d target_files.zip IMAGES/\*;
add_img_to_target_files.py target_files.zip
Test: sign_target_files_apks.py target_files.zip signed-target_files.zip
Change-Id: Iea91d0403cdec1b16bb93bb71d3ed06856b8f7c3
If only vbmeta.img is missing from the target_files.zip,
'add_img_to_target_files.py -a' will fail to re-create vbmeta.img.
++++ vbmeta ++++
Traceback (most recent call last):
File "./build/make/tools/releasetools/add_img_to_target_files.py", line 693, in <module>
main(sys.argv[1:])
File "./build/make/tools/releasetools/add_img_to_target_files.py", line 687, in main
AddImagesToTargetFiles(args[0])
File "./build/make/tools/releasetools/add_img_to_target_files.py", line 584, in AddImagesToTargetFiles
boot_contents = boot_image.WriteToTemp()
AttributeError: 'NoneType' object has no attribute 'WriteToTemp'
Because it has skipped loading the boot.img, which leads to the above error.
Bug: 63456822
Test: As follows:
$ zip -d target_files.zip vbmeta.img
$ add_img_to_target_files.py -a target_files.zip
Test: `m dist` on aosp_arm64-userdebug
Change-Id: Ieba5f0b6848c3ec5d8bcfd24d48cc5af8b39f06c
When generating full OTAs for AVB-enabled targets, it fails the size
assertion for boot.img.
ERROR: boot size (31457280) is 100.00% of limit (31457280)
The size assumption is no longer true for AVB-signed images, because
the actual image size should be always identical to the limit.
Bug: 67841177
Test: Enable AVB on bullhead. `m dist`.
Change-Id: I9bd432ccd88094628eb71308554839549eccb6d9
With the new implementation of handling large apks, we need to call
imgdiff with block-limit to split the apk and generate the patch at
the same time. The call to imgdiff would significantly increase the
time consumption of the "FindTransfers" function which we used to
execute sequentially. This cl addresses this issue and speeds up the
process by making the imgdiff call parallel.
Bug: 34220646
Test: Create and sideload an incremental package for angler
Change-Id: Id62e348418fc1d22e32ea6c8ac16d9ab3ec92d7b
With the update of brotli to version 1.0.1 the new binary tool is now
called "brotli". This patch updates the scripts to use the new name.
Bug: 34220646
Test: `make dist`; ota_from_target_files on non-A/B device.
Change-Id: Ie8da4333388d029d5960a723a55efd875d8ee820
When signing a target_files.zip, the OTA certificate specified by
default_system_dev_certificate could be replaced with a mapped key. When
that happens, we must explicitly specify --package_key when generating
OTA packages with ota_from_target_files.py. Otherwise the OTA package
will be signed with the wrong key, which leads to verification failures.
This CL updates the default_system_dev_certificate value in
misc_info.txt accordingly.
Test: Sign a target_files.zip and replace the OTA key. Check
META/misc_info.txt in the generated target_files.zip.
$ ./build/make/tools/releasetools/sign_target_files_apks.py -v \
--replace_ota_keys \
-k build/target/product/security/testkey=build/target/product/security/platform \
out/dist/aosp_marlin-target_files-eng.tbao.zip \
signed-marlin-target_files-test.zip
Change-Id: I093234b5add3e27c5b3887cefeffd74e6f0a3e98
test_common constructs a few 2GiB strings in memory, which leads to huge
memory footprint (18GiB). This CL moves away from in-memory strings to
generators, which reduces the memory use down to 41MiB. It also reduces
the time cost from 294s to 139s as an extra benefit for free.
The CL addresses some trivial pylint warnings as well.
* Before
$ /usr/bin/time -v python -m unittest -v test_common
...
----------------------------------------------------------------------
Ran 11 tests in 294.986s
OK
Command being timed: "python -m unittest -v test_common"
User time (seconds): 110.51
System time (seconds): 109.34
Percent of CPU this job got: 74%
Elapsed (wall clock) time (h:mm:ss or m:ss): 4:55.06
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 18894172
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 1
Minor (reclaiming a frame) page faults: 20774908
Voluntary context switches: 48
Involuntary context switches: 3241
Swaps: 0
File system inputs: 184
File system outputs: 8406424
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
* After
$ /usr/bin/time -v python -m unittest -v test_common
...
----------------------------------------------------------------------
Ran 11 tests in 139.100s
OK
Command being timed: "python -m unittest -v test_common"
User time (seconds): 59.00
System time (seconds): 4.73
Percent of CPU this job got: 45%
Elapsed (wall clock) time (h:mm:ss or m:ss): 2:19.17
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 41252
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 106569
Voluntary context switches: 44
Involuntary context switches: 103
Swaps: 0
File system inputs: 8
File system outputs: 8422808
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Fixes: 68988396
Test: See above.
Change-Id: I00f16603a4ee59fb085b189c6f5b5ee9d2378690
imgdiff fails on plain strings after we put more checks the input
data. This leads to failure when running unittest
"test_recovery_from_boot". The CL fixes the issue by serving gzipped
dummy images as the test input.
Bug: 68988329
Test: InstallRecoveryScriptFormatTest passes
Change-Id: I1952b311681ac4f39c0e51aef1de1177aebeac43
$ python rangelib.py
**********************************************************************
File "rangelib.py", line 323, in __main__.RangeSet.next_item
Failed example:
list(rangelib.RangeSet("10-19 3 5 7").next_item())
Exception raised:
Traceback (most recent call last):
File "/usr/lib/python2.7/doctest.py", line 1315, in __run
compileflags, 1) in test.globs
File "<doctest __main__.RangeSet.next_item[2]>", line 1, in <module>
list(rangelib.RangeSet("10-19 3 5 7").next_item())
NameError: name 'rangelib' is not defined
**********************************************************************
1 items had failures:
1 of 3 in __main__.RangeSet.next_item
***Test Failed*** 1 failures.
Test: python rangelib.py
Change-Id: I6c791ff4453f40d90b34f97b881460da9878fe64
It has been long deprecated since commit
fdd8e69c42 (Donut).
Test: `m dist` with aosp_marlin-userdebug and aosp_angler-userdebug.
Test: No user of "--board_config" in code search.
Change-Id: Ic0336c22cac32de5de88161748d09b579f892e99
Commit b937ead5d9 added the fallback to
bsdiff on imgdiff failures. However, it missed setting the transfer
style accordingly, which led to patch header mismatch.
Bug: 68659848
Test: Generate an incremental that has the fallback from imgdiff to
bsdiff. Examine the generated transfer list and verify that it has
"bsdiff" for the fallback command.
Change-Id: I55e46879d590a8af82ea796b9d98ffdb30360408
When generating block based OTAs, we read files from the sparse image
directly with the help of block map file. However, the block map info
might not be accurate if the image is created with mke2fs. Because
mke2fs may skip allocating actual blocks if they contain all zeros.
ota_from_target_files.py consequently passes incomplete APK files to
imgdiff, which fails to generate patches.
This CL works around the issue by falling back from imgdiff to bsdiff on
failures. We should figure out a better way in b/68016761 to remove the
workaround, which would otherwise hide other issues in imgdiff.
Bug: 67824829
Bug: 68016761
Test: ota_from_target_files.py passes on previously failing TF zips.
Change-Id: Ib24c5b5f89812b97a0c87c6bf0dc147ae39bc92f
Test: `m dist` on Linux and macOS.
Test: Use an existing target_files.zip for bullhead. Remove IMAGES/*
and rebuild the images with 'add_img_to_target_files.py -a'. It
generates idential images as in the original TF.zip.
Change-Id: I803656bf2c924dce53f2271dcb967a94fff48440
We have added support in mkuserimg_mke2fs.sh that allows specifying
filesystem UUID and hash seed UUID. This CL generates and passes the
UUIDs based on the image name and build fingerprint. This way we can
rebuild and get identical images.
Note that this only applies to images generated with `m dist` and with
signing tools (sign_target_files_apks.py and
add_img_to_target_files.py). Images under $OUT (built with `make`) are
not affected.
Bug: 64994964
Test: lunch marlin-userdebug; run `m dist` twice (after deleting the
intermediate files under $OUT/obj/PACKAGING), and compare the
generated images.
Change-Id: I41cf4e5869582bb930af2f35a8e9c79bff43b2a2
(cherry picked from commit 3aa21e6bb9)
Adding verified boot metadata with a "disable magic". The resulting
metadata at the end of each image (e.g., system.img, vendor.img) will
be the same as triggering an "adb disable-verity" on an USERDEBUG image.
This can help simplify the code on fs_mgr, which won't have to check if
current image is an ENG build or not.
Bug: 63056044
Test: boot sailfish eng/userdebug builds
Change-Id: I95d23ac7b76c04d6d4483c9c4dc1de16bf0d9c3a
We want the generated images being identical for the same source files.
Currently the generated ext4 image (either from make_ext4fs or mke2fs)
is reproducible, but the AVB footer added by avbtool contain changes
because of the random salt being used.
This CL changes the avbtool invocation to specify "--salt <hexstring>"
(already supported by avbtool) to use reproducible salt that's computed
based on fingerprints (or thumbprints if applicable).
Bug: 67023482
Test: Regenerate images from the same source as follows:
Use a target_files.zip from an AVB-enabled target.
$ zip -d target_files.zip IMAGES/\*
$ ./build/make/tools/releasetools/add_img_to_target_files.py \
-v target_files.zip
Repeat the above commands and compare the generated images.
Change-Id: Id9db17ae0132ca3a820b4be5a5ef06ca3fef71ed
(cherry picked from commit 8f05cca1d9)
We have added support in mkuserimg_mke2fs.sh that allows specifying
filesystem UUID and hash seed UUID. This CL generates and passes the
UUIDs based on the image name and build fingerprint. This way we can
rebuild and get identical images.
Note that this only applies to images generated with `m dist` and with
signing tools (sign_target_files_apks.py and
add_img_to_target_files.py). Images under $OUT (built with `make`) are
not affected.
Bug: 64994964
Test: lunch marlin-userdebug; run `m dist` twice (after deleting the
intermediate files under $OUT/obj/PACKAGING), and compare the
generated images.
Change-Id: I41cf4e5869582bb930af2f35a8e9c79bff43b2a2
(cherry picked from commit 3aa21e6bb9)
vendor.img could be built from the source - in which case we have
unpacked files under VENDOR/ - or dropped in as a prebuilt binary blob.
We should consider either of them as target having a vendor partition.
Because we need to add its AVB hashtree info into vbmeta.img if target
is using AVB. Otherwise libfs_mgr would refuse to mount this
"AVB-enabled" vendor.img.
For targets not using AVB, this change is no-op.
Bug: 65462819
Test: Having vendor.img as prebuilt, `make dist`. Check that the
generated vbmeta.img contains the info from vendor.img.
Test: Build, flash and boot the above image.zip.
Change-Id: Iaeb30e2059cb33fb39f23e5ffd28f338d00ccbfc
Merged-In: Iaeb30e2059cb33fb39f23e5ffd28f338d00ccbfc
(cherry picked from commit b22afea0f0)
vendor.img could be built from the source - in which case we have
unpacked files under VENDOR/ - or dropped in as a prebuilt binary blob.
We should consider either of them as target having a vendor partition.
Because we need to add its AVB hashtree info into vbmeta.img if target
is using AVB. Otherwise libfs_mgr would refuse to mount this
"AVB-enabled" vendor.img.
For targets not using AVB, this change is no-op.
Bug: 65462819
Test: Having vendor.img as prebuilt, `make dist`. Check that the
generated vbmeta.img contains the info from vendor.img.
Test: Build, flash and boot the above image.zip.
Change-Id: Iaeb30e2059cb33fb39f23e5ffd28f338d00ccbfc
Make the build log less verbose; also print the result for
AdjustPartitionSizeForVerity().
Bug: 65292710
Test: Run add_img_to_target_files with and without "-v" and check
the logs.
Change-Id: Icdc86197ce69db036658ea9e77cc4ebe1750463b
We pack _all_ the APK certs info into META/apkcerts.txt at build time,
including the ones that are not installed for the target. When
detecting for compressed APKs, we should only count the ones that are
actually installed.
Bug: 65498015
Test: `check_target_files_signatures.py bullhead-target_files.zip`
Change-Id: I2f32d4667be0efbfd5f9365dde819f009572533d
When creating ext4 images with mke2fs, it may skip allocating some
blocks if they contain all zeros. As a result, there could be less
blocks listed in the block map than the actual file length.
For example, for a file with a length of 112200-byte (27+ blocks),
the listed blocks in block.map could be '43665-43688' (24 blocks).
Because some all-zero blocks are not taking actual space.
The generated ext4 images are perfectly valid - kernel will figure out
that data block is not allocated and writes all zeros into user buffer.
However, we can't fully reconstruct a file from its block list in our
Python script. Ideally this can be avoided by mounting or parsing an
ext4 image directly, which is yet to be supported in our script.
This CL skips checking for such files to avoid failing
validate_target_files.py.
Bug: 65213616
Test: validate_target_files.py passes on targets with mke2fs generated
images (e.g. marlin).
Change-Id: Id9cc59e345b9283844044ef94ceb5702f0ca0526
(cherry picked from commit b418c30e3a)
When creating ext4 images with mke2fs, it may skip allocating some
blocks if they contain all zeros. As a result, there could be less
blocks listed in the block map than the actual file length.
For example, for a file with a length of 112200-byte (27+ blocks),
the listed blocks in block.map could be '43665-43688' (24 blocks).
Because some all-zero blocks are not taking actual space.
The generated ext4 images are perfectly valid - kernel will figure out
that data block is not allocated and writes all zeros into user buffer.
However, we can't fully reconstruct a file from its block list in our
Python script. Ideally this can be avoided by mounting or parsing an
ext4 image directly, which is yet to be supported in our script.
This CL skips checking for such files to avoid failing
validate_target_files.py.
Bug: 65213616
Test: validate_target_files.py passes on targets with mke2fs generated
images (e.g. marlin).
Change-Id: Id9cc59e345b9283844044ef94ceb5702f0ca0526
$ PYTHONPATH=$PYTHONPATH:system/update_engine/scripts \
./build/make/tools/releasetools/check_ota_package_signature.py \
build/target/product/security/testkey.x509.pem \
out/dist/aosp_marlin-ota-eng.zip
Package: out/dist/aosp_marlin-ota-eng.zip
Certificate: build/target/product/security/testkey.x509.pem
...
Whole package signature VERIFIED
Verifying A/B OTA payload signatures...
...
Payload signatures VERIFIED
Bug: 65261072
Test: Signed a package and its payload with the right keys; ran the
command above.
Test: Signed the payload with a different key; ran the command above and
observed the reported verification failure.
Change-Id: If626ecb327a9826cd0956eef94914c939068a7d1
Commit 21803d3574 only added compatibility
metadata for A/B OTA packages, because by that time we didn't use
'ro.treble.enabled' property to determine if a target was Treble-enabled.
This CL uses 'ro.treble.enabled' to guard the packing of compatibility
metadata for both of A/B and non-A/B OTA packages. It also switches to
checking the system/vendor fingerprints to determine if there's an
update to the partition (previously it was computing the SHA-1 of the
images, which may have unintentionally changed due to issues that give
non-repetitive builds).
Bug: 64339310
Test: Generate OTA packages (full and incremental) on Treble-enabled
targets (sailfish, as well as a non-A/B angler target with
Treble-enabled locally); check that the compatibility.zip entry
exists.
Test: Generate OTA packages on Treble-unenabled non-A/B target (angler);
check that the compatibility.zip entry doesn't exist.
Test: Generate OTA packages on Treble-unenabled A/B target; check that
the compatibility.zip entry doesn't exist.
Change-Id: I2a1fcf612439d849ba8ccea217a0faf5d5ba8e14
This entry only exists in non-A/B full OTA path. However, this file has
never been loaded by recovery - recovery was only reading the one from
the recovery image (which also has been removed by a recent change in
[1]).
[1]: commit e35926e1aff2e6b9b54656bd59c8178e295a1b7e in
platform/bootable/recovery.
Bug: 63538434
Test: `m dist` on angler. file_contexts.bin is gone from full OTA
package. Sideloading the generated full OTA on angler.
Change-Id: Iec2e2d36599fa1ed499516a3e74f06cd36ade494
Compressed APKs can be identified by a "compressed=<ext>" entry in
the apkcerts.txt file. When we encounter such an entry, we need to
decompress the file to a temporary location before we process its
certs. When we're signing, we should also recompress the package
after it's signed.
Bug: 64531948
Test: ./build/tools/releasetools/check_target_files_signatures.py
Test: ./build/tools/releasetools/sign_target_files_apks.py
Test: compared signed output before / after this change, verify that
it's bitwise identical when no compressed APKs are present.
Change-Id: Id32e52f9c11023955330c113117daaf6b73bd8c2
In BOARD_AVB_MAKE_VBMETA_IMAGE_ARGS, if we have defined
"--include_descriptors_from_image" with an image file whose path points
to source tree, add_img_to_target_files.py or sign_target_files_apks.py
may fail to find the file. Because these scripts may run without a
source tree, by taking target_files.zip as the only input.
This CL scans additional locations in the input target_files.zip to find
those missing files in avb_vbmeta_args. As long as the files are included
in the target_files.zip, they get a second chance to be found.
Bug: 63910867
Test: As follows:
1. Setup BOARD_AVB_MAKE_VBMETA_IMAGE_ARGS with a local file path;
2. Remove the local file;
3. sign_target_files_apks.py fails without this CL;
4. sign_target_files_apks.py works.
Change-Id: I3c58f80a5535db02b74cfe40d0c0beff72587cf8
(cherry picked from commit 1dc5d47653)
In BOARD_AVB_MAKE_VBMETA_IMAGE_ARGS, if we have defined
"--include_descriptors_from_image" with an image file whose path points
to source tree, add_img_to_target_files.py or sign_target_files_apks.py
may fail to find the file. Because these scripts may run without a
source tree, by taking target_files.zip as the only input.
This CL scans additional locations in the input target_files.zip to find
those missing files in avb_vbmeta_args. As long as the files are included
in the target_files.zip, they get a second chance to be found.
Bug: 63910867
Test: As follows:
1. Setup BOARD_AVB_MAKE_VBMETA_IMAGE_ARGS with a local file path;
2. Remove the local file;
3. sign_target_files_apks.py fails without this CL;
4. sign_target_files_apks.py works.
Change-Id: I3c58f80a5535db02b74cfe40d0c0beff72587cf8
Brotli has a better compression ratio than deflation. So for non-AB
full OTA, we can compress the new.dat with brotli first and store it
in the zip package.
This increase the OTA generation time by ~5 minutes for a full OTA
(measured based on 2.0G system.new.dat)
Bug: 34220646
Test: Generate a full OTA for bullhead
Change-Id: I9c0550af8eafcfa68711f74290c8e2d05a96648f
Accidentally broken by the cherry-pick in commit
f829b40c48 - the original CL in oc-dev
doesn't require the 'import stat' line.
Bug: 63629728
Test: `pylint --rcfile=pylintrc sign_target_files_apks.py`
Test: Successfully sign a build with sign_target_files_apks.py.
Change-Id: I94be613fb2219597148c4339ac33fc93d0cdae47
system_avb_hashtree_enable has been renamed to
avb_system_hashtree_enable in commit
3e599ead66. 'care_map.txt' is missing due
to the change.
Bug: 63142730
Test: `m dist` and check IMAGES/care_map.txt exists in target_files zip.
Change-Id: I60c269b41df844b50353d357bf67c20f15548642
(cherry picked from commit 3f72176da0)
system_avb_hashtree_enable has been renamed to
avb_system_hashtree_enable in commit
3e599ead66. 'care_map.txt' is missing due
to the change.
Bug: 63142730
Test: `m dist` and check IMAGES/care_map.txt exists in target_files zip.
Change-Id: I60c269b41df844b50353d357bf67c20f15548642
Currently we're writing META/misc_info.txt to the new TF.zip during
ReplaceVerityPrivateKey(). We should delay that until we have replaced
everything in need. Otherwise we won't be able to replace/overwrite
that zip entry (unless `zip -d` first).
This CL also cleans up the return value of ReplaceVerityPublicKey() and
ReplaceVerityKeyId(), since the caller no longer needs the values.
Test: sign_target_files_apks.py and check the generated signed TF.zip.
Change-Id: I9fbd7182247728281519e5e3971557f6b018ad65
(cherry picked from commit 46a5999a02)
Radio images are added to INSTALLED_RADIOIMAGE_TARGET, which by default
will be packed under RADIO/ in a target_files zip. This CL introduces
BOARD_PACK_RADIOIMAGES that allows additionally copying them into
IMAGES/, which will then be included into <product>-img.zip.
Bug: 62195105
Test: Define BOARD_PACK_RADIOIMAGES and `m dist`. Check the generated
target_files zip and <product>-img.zip.
Change-Id: I3deafd2dfecd1d4dbfdfc2d002fc40ef22fb60ea
