The usage of datetime.fromtimestamp previously resulted in the build or
signing machine's local timezone affecting the Unix timestamp ultimately
applied to images generated by add_img_to_target_files. The go/ab build
outputs would use 2009-01-01 00:00 UTC, for example, but local builds
and the release signed images (generated through go/ab-sign) would use
2009-01-01 00:00 PST. This change makes the timestamps always use UTC.
Bug: 80600931
Bug: 80093599
Test: 'm -j droid dist' and verified timestamps in resulting
target_files zip.
Change-Id: Ic2a19591519850c249f78254e1464aa6839bfc6c
Split the flag into PRODUCT_USE_DYNAMIC_PARTITION_SIZE
and PRODUCT_BUILD_SUPER_PARTITION. More sub-flags can be
added with the same way.
Also change some checks with USE_LOGICAL_PARTITIONS with
the appropriate sub-flags.
This allows easier device bring-up to fulfill the requirements
separately.
This also enables dynamic partition size without logical patition.
Originally, to enable BOARD_*_PARTITION_RESERVED_SIZE must also
enable PRODUCT_USE_LOGICAL_PARTITIONS. The patch fix the rule to
let dynamic partition size is able to be used independently.
Fixes: 111966003
Test: make superimage with PRODUCT_USE_LOGICAL_PARTITIONS
Change-Id: I45b29c87bf94356d8416bbd58a14c8982039c222
Make names consistent.
Some things that does not change:
- productservicesimage target and intermediates
Test: build product services image
Bug: 111609632
Change-Id: I4c2b975e194577aad3d51b908b103880625deb5e
When building a system image with system_root_image enabled, the size
computation should include files under both of in_dir (i.e. /system
files) and root (pointed by 'root_dir'). Because files from both
locations will end up into the built image. The files under root are
usually only a few MiBs, but should be accounted for especially in the
context of logical partitions (where the partition size will be
allocated based on the actual need). Note that we will still need some
"reserved space" (defined via BOARD_*_PARTITION_RESERVED_SIZE) to cover
the cost for filesystem and/or verity metadata.
This CL moves the combination of the two dirs up, before parsing and
computing other properties. This doesn't affect anything for a
successful image building path. It may however increase the time to
error out in certain error path, since it copies the files earlier now.
Test: python -m unittest test_build_image
Test: `make dist`
Test: Setup a target with PRODUCT_USE_LOGICAL_PARTITIONS == true and
BOARD_SYSTEMIMAGE_PARTITION_RESERVED_SIZE == 20MiB. Build system
image successfully.
Test: Setup a target with PRODUCT_USE_LOGICAL_PARTITIONS == true and
BOARD_SYSTEMIMAGE_PARTITION_RESERVED_SIZE == 20MiB. Write a large
file to root dir (PRODUCT_OUT_ROOT). The image building fails, but
reporting a size that accounts for both of /system and root.
Change-Id: Idfb26b8e259626ba57ec3bd4f85d357c30e56163
When verity is enabled, `partition_size` has the "adjusted" partition
size for holding the filesystem files (i.e. excluding verity hashtree,
and FEC metadata if applicable), whereas `original_partition_size` has
the one for the actual image returned by build_image.py.
`partition_size` must be smaller than `original_partition_size`. As a
result, the later assertion in Makefile (e.g. assert-max-image-size) may
fail to hold when using `partition_size`.
This CL addresses the issue by returning the value in
`original_partition_size` instead, if available. It also changes to
include both values when dumping the size info on image creation
failure.
Fixes: 79106666
Test: `m dist` with aosp_walleye-userdebug
Test: Set up walleye to use logical partitions, with a reserved size of
20MiB. `m systemimage` no longer fails.
Test: Set up walleye that doesn't use verified boot. Use `m systemimage`
to build an oversized image. Check the error message.
Change-Id: I41d77bffa85e8eadfbf4a2a7dcd2e798fadfdab7
For example, verify a target_files.zip that has system AVB-signed as a
chained partition.
$ build/make/tools/releasetools/validate_target_files.py \
signed-target_files-4904652.zip \
--verity_key verifiedboot_pub.pem \
--avb_system_key_path system_pub.pem
Note that verifiedboot_pub.pem should be the key (either public or
private) to verify vbmeta.img, and 'system_pub.pem' should be the key
(either public or private) for the chained partition of system.
testdata/testkey.key is the private key converted from
testdata/testkey.pk8 for testing purpose (`openssl pkcs8 -in
testdata/testkey.pk8 -inform DER -out testdata/testkey.key -nocrypt`).
Bug: 63706333
Test: python -m unittest test_common
Test: python -m unittest test_add_img_to_target_files
Test: `m dist` on aosp_walleye-userdebug; Run validate_target_files.py
on the generated target_files.zip.
Test: Set up walleye with chained system partition; `m dist`; Run
validate_target_files.py on the generated target_files.zip.
Change-Id: I38517ab39baf8a5bc1a6062fab2fe229b68e897d
Drop the parameter of public_key_dir. It's not necessary to stage all
the extracted public keys into the same temporary dir.
Redirect the stderr output to STDOUT while calling avbtool, so that it
can dump the error message accordingly.
Also remove the check for having empty img_path arg. It doesn't look
like a clean logic to allow the function to be called with invalid /
empty arg. The only caller (i.e. AddVBMeta) already asserts img_path
being valid as well.
Test: python -m unittest test_add_img_to_target_files
Test: `m dist` with aosp_walleye-userdebug
Change-Id: Id58c5ae780ac8a22661ffea629144d4836839175
This property indicates the path to the root directory (i.e.
TARGET_ROOT_OUT at build time, or ROOT/ in a target_files.zip).
build_image.py will only use this property when building a target using
system_root_image. Under such a configuration, files under this dir
never land into a ramdisk, but as part of system.img. This CL renames
the property name to avoid confusion.
Note that with this change releasetools keeps working with older
target_files zips that contain 'ramdisk_dir'. Because common.py doesn't
read the value of that property, but sets it to ROOT/ directly.
Test: `m dist` on aosp_marlin-userdebug.
Test: `m dist` on aosp_angler-userdebug.
Test: Run add_img_to_target_files.py with a new target_files.zip and an
old target_files.zip respectively.
Change-Id: Ie9e19dd17d8c2abb4c4a97449e436774d5dd2a36
Clean up the following scripts.
blockimgdiff.py
common.py
edify_generator.py
img_from_target_files.py
ota_from_target_files.py
Seems we don't have a way to force pylint-ing the scripts using
per-directory pylintrc files (and we don't have pylint tool in AOSP
repo), per
https://android.googlesource.com/platform/tools/repohooks/#todo_limitations.
Test: `m dist`
Test: pylint --rcfile=pylintrc <script.py>
Change-Id: Ia6fd1ddc86f4d84c68e500f225d4a89d0fea8ec7
The claim in the original CL doesn't seem to hold. We may still have
devices using both of thumbprint and Treble compatibility check.
Bug: 111640956
Test; python -m unittest test_ota_from_target_files
Test: Build an incremental package for a target using thumbprints.
Change-Id: Ia491d5279901c2385d1c30802abcb99d65d919cc
This CL is largely an adaptation of Change-Id
I774e6a38003734421591e51bed103802ff84f432
It adds the following variables:
- BOARD_AVB_PRODUCT_SERVICES_KEY_PATH
- BOARD_AVB_PRODUCT_SERVICES_ALGORITHM
- BOARD_AVB_PRODUCT_SERVICES_ROLLBACK_INDEX_LOCATION
- BOARD_PRODUCT_SERVICESIMAGE_FILE_SYSTEM_TYPE
- BOARD_PRODUCT_SERVICESIMAGE_EXTFS_INODE_COUNT
- BOARD_PRODUCT_SERVICESIMAGE_EXTFS_RSV_PCT
- BOARD_PRODUCT_SERVICESIMAGE_PARTITION_SIZE
- BOARD_PRODUCT_SERVICESIMAGE_JOURNAL_SIZE
- BOARD_PRODUCT_SERVICESIMAGE_SQUASHFS_BLOCK_SIZE
- BOARD_PRODUCT_SERVICESIMAGE_SQUASHFS_COMPRESSOR
- BOARD_PRODUCT_SERVICESIMAGE_SQUASHFS_COMPRESSOR_OPT
- BOARD_PRODUCT_SERVICESIMAGE_SQUASHFS_DISABLE_4K_ALIGN
- BOARD_PREBUILT_PRODUCT_SERVICESIMAGE
- BOARD_USES_PRODUCT_SERVICESIMAGE
- LOCAL_PRODUCT_SERVICES_MODULE
- PRODUCT_PRODUCT_SERVICES_BASE_FS_PATH
- PRODUCT_PRODUCT_SERVICES_VERITY_PARTITION
- PRODUCT_PRODUCT_SERVICES_PROPERTIES
- TARGET_COPY_OUT_PRODUCT_SERVICES
- TARGET_OUT_PRODUCT_SERVICES
- TARGET_OUT_PRODUCT_SERVICES_*
Bug: 80741439
Test: Successfully built product-services.img with one module in it, and flashed
on device. Also successfully built image with /system/product-services directory
and no /product-services partition.
Change-Id: I5d229f6ac729ea6df9ff1f14cee2e28972cd9b4d
This writes /system/bin/install-recovery.sh with the updated argument
formats. Requires matching changes to bootable/recovery/applypatch.
Bug: 110106408
Test: python -m unittest test_common.InstallRecoveryScriptFormatTest
Test: See additional tests in the other CL.
Change-Id: I2dec6f064819c01b5bb34c3d6fd0e247a5f48f08
The change in [1] removed duplicating radio images from RADIO/ to
IMAGES/. When getting the target_files zip for generating secondary
payload, we need to additionally copy the files under RADIO/ over.
[1] commit 5277d1015f
Test: python -m unittest test_ota_from_target_files
Test: Generate an OTA with secondary payload (--include_secondary).
Change-Id: I096f1642a905fb764e63f5df2edc1396aa6befd8
RemoveBackwardEdges() was used only in BBOTA v1 (which has been
deprecated since O). v2+ calls ReverseBackwardEdges() and
ImproveVertexSequence().
Also remove the imgdiff tag of 'trimmed' that would be set through this
function() only.
Test: python -m unittest test_blockimgdiff
Test: Build an incremental non-A/B OTA.
Test: Code search shows no active user.
Change-Id: I3b58ae048a1fbc283269e70fdfa29eb8d184ede7
With system_root_image, e2fsdroid writes file map with extra leading
slashes in filenames (e.g. "//system/framework/am.jar"). This breaks the
detection of files with incomplete ranges, and thus fails the patch
generation. This CL addresses the issue by stripping out leading
slashes.
Additionally, non-/system files (e.g "//sbin/charger") are not packed
under SYSTEM/ in a target_files.zip, despite being part of system.img.
We need to look for these files under ROOT/ instead.
This CL also asserts the availability of all files listed on a file map,
to avoid silently missing other edge cases.
Bug: 80380658
Test: python -m unittest test_common
Test: Successfully generated an incremental for a target using
system_root_image that was previously failing.
Change-Id: I62a2460e882f3930e99add4d2b44291edf7a51a0
Namely MakeTemporary, Verify and FileCheck. They were once used for
file-based OTA that has been deprecated.
Test: Check that there's no active user, including device-specific
releasetools script.
Change-Id: Ic73054d73271ad91e860b34b2ba156361c58bc1a
BuildInfo already has dict-like getters. This CL adds the matching
setter method to keep it backward compatible with device-specific
scripts that expect a dict.
It also adds items() method that allows iterating the items.
Bug: 111087332
Test: python -m unittest test_ota_from_target_files.BuildInfoTest
Test: Generate an incremental OTA that sets info_dict in device-specific
releasetools script.
Change-Id: Idd033f98a9186740f9da1a300d4c2bdddd9c1345
Test: `make vendorimage`
Test: `make vendorimage` with the following:
- install a large file to vendor image
fails as expected (because _PARTITION_SIZE is exceeded)
Test: `make vendorimage` with the following:
- set PRODUCT_USE_LOGICAL_PARTITIONS to true
- set BOARD_VENDORIMAGE_PARTITION_RESERVED_SIZE
fails as expected (BOARD_VENDORIMAGE_PARTITION_SIZE needs
to be undefined)
Test: `make vendorimage` with the following:
- install a large file to vendor image
- set PRODUCT_USE_LOGICAL_PARTIIONS to true
- add a small BOARD_VENDORIMAGE_PARTITION_RESERVED_SIZE
- remove BOARD_VENDORIMAGE_PARTITION_SIZE
build succeeds.
Bug: 79106666
Change-Id: Ica8fdce64e8f37d91e66e3d49c0c74fadd15a832
- Copy "use_logical_partitions" to *_image_info.txt before sending
it to build_image.py, so that the script can use this variable.
- build_image.py emits an additional properties file to inform
the build system about the system image size.
Test: `make systemimage`
Test: `make systemimage` with the following:
- install a large file to system image
fails as expected (because _PARTITION_SIZE is exceeded)
Test: `make systemimage` with the following:
- set PRODUCT_USE_LOGICAL_PARTITIONS to true
- set BOARD_SYSTEMIMAGE_PARTITION_RESERVED_SIZE
fails as expected (BOARD_SYSTEMIMAGE_PARTITION_SIZE needs
to be undefined)
Test: `make systemimage` with the following:
- install a large file to system image
- set PRODUCT_USE_LOGICAL_PARTIIONS to true
- add a small BOARD_SYSTEMIMAGE_PARTITION_RESERVED_SIZE
- remove BOARD_SYSTEMIMAGE_PARTITION_SIZE
build succeeds.
Test: same for systemotherimage
Bug: 79106666
Change-Id: I574062882acd1ecd633ac38c5a8c5351b90a32d8
We may pack prebuilts that end with ".apk" into target_files zip, via
PRODUCT_COPY_FILES. META/apkcerts.txt won't contain the cert info for
such files, and we want to keep them as is while signing, despite of the
".apk" extension.
This CL adds "--skip_apks_with_path_prefix" option to
sign_target_files_apks.py. APKs with matching prefixes will be copied
verbatim into the signed images. The prefix should match the entry names
in the target_files (e.g. "SYSTEM_OTHER/preloads/"). The option may be
repeated to specify multiple prefixes.
Note that although we may skip signing an APK file with "-e ApkName=".
This would skip *all* the APK files with the matching basename.
"--skip_apks_with_path_prefix" allows matching the exact prefix.
For example:
$ ./build/make/tools/releasetools/sign_target_files_apks.py \
--skip_apks_with_path_prefix SYSTEM_OTHER/preloads/ \
--skip_apks_with_path_prefix PRODUCT/prebuilts/PrebuiltApp1 \
--skip_apks_with_path_prefix VENDOR/app/PrebuiltApp2.apk \
target_files.zip \
signed-target_files.zip
Bug: 110201128
Test: Run the command above and check the logs.
Test: `python -m unittest test_sign_target_files_apks`
Change-Id: I7bd80b360917cef137cf1e7e8cfa796968831f47
Test: Run sign_target_files.py to sign a target_files.zip.
Test: `python -m unittest test_sign_target_files_apks`
Change-Id: Ie795d1bce7bae6af427832283e3d10bfecad80c5
Although the SHA-1 of the source boot image has been encoded in the
filename, the updater code currently doesn't extract that and would fail
to find a matching backup from /cache. This could lead to corrupt
install, if it's interrupted while patching the boot image.
Bug: 80630342
Test: Generate an non-A/B incremental package. Verify the script.
Change-Id: Ie2b1a90836c3ccb4758e1a2d9b69a88de94ae0c7
Bug: 79951743
Test: Run validate_target_files.py on target_files.zip that has
non-monotonic ranges listed in IMAGES/system.map.
Change-Id: I28d3ca6972d361362589cd51e60731af9994a551
Targets can define 'TARGET_USERIMAGES_SPARSE_EXT_DISABLED := true' to
generate non-sparse system images, but validate_target_files.py doesn't
work with such images. This CL adds a workaround to temporarily skip the
file consistency check for such images.
Bug: 79616357
Test: Run validate_target_files.py on a target_files.zip that's not
using sparse image.
Test: Run validate_target_files.py on marlin target_files.zip (which
uses sparse image).
Change-Id: I1f4066c5b3fec595b10cab10283d62c1c5a6c624
We used to write a fake post-timestamp into package metadata for
"--override_timestamp", which allows a package to be pushed based on
the fake timestamp. This CL stops doing that by using the designated
"ota-downgrade=yes" instead, but keeping the "--override_timestamp" flag
for backward compatibility.
Now both "--override_timestamp" and "--downgrade" allow pushing packages
with downgrade timestamps. The only differenece is that "--downgrade"
enforces a data wiping, because it's intended a real downgrade (based on
Android version or security patch level). "--override_timestamp" serves
the path of a legit "upgrade", but unfortunately with build timestamps
in reverse order (e.g. the two builds are cut from different branches).
With this CL, we write "post-timestamp" to all packages.
Bug: 77811049
Test: Generate an incremental pakcage with "--override_timestamp".
Test: Generate an incremental pakcage with "--downgrade".
Test: python -m unittest test_ota_from_target_files
Change-Id: I4ae4ce77cb600a06ffd9b15b6d4c730314aa0b79
ota_from_target_files.py:
Rename PropertyFiles._GetPropertyFilesString to
PropertyFiles.GetPropertyFilesString.
Currently only PropertyFiles.Compute and
PropertyFiles.Finalize gives access to _GetPropertyFilesString.
But Compute force sets reserve_space=True,
and Finalize requires reserved_length.
_GetPropertyFilesString is useful method that can
be used outside releasetools.
One of the usage is in bootable/recovery/updater_sample/.
Test: python -m unittest test_ota_from_target_files
Change-Id: I2cc44ec46a0e68ba071531b003af8cdbfe90e588
Signed-off-by: Zhomart Mukhamejanov <zhomart@google.com>
This reverts commit 2a40cc6996.
Reason for revert: Broke a bunch of builds, platform/system/tools/bpt must be missing from more manifests.
Change-Id: I4c367a1fec5c62c247544227e9d3b596227e31da
...instead of copying from IMAGES/partition-table.bpt during
sign_target_files_apks. This addresses a comment on go/oag/665565, but
is done as a separate change so that it can be cherry-picked later (or
not at all) to downstream branches to avoid breaking signing of existing
target-files zips.
Bug: 72837107
Test: Local sign_target_files_apks run of locally built target-files
Change-Id: Id14c859eefe075fc56b15869f2f21c570eb07e65
For Android Things targets (or any other target which has
BOARD_BPT_INPUT_FILES defined), add_img_to_target_files will generate a
partition-table.img using bpttool. It also adds the final combined .bpt
definition file into target-files in IMAGES/partition-table.bpt.
When we're signing using sign_target_files_apks, add_img_to_target_files
needs to regenerate the partition table, but META/misc_info.txt still
contains the original list of bpt input files from the build that aren't
available. This change extracts the final bpt from the input
target-files, adds it to META/ in the output target-files, and then
updates the board_bpt_input_files property to point to it.
Bug: 72837107
Test: Local sign_target_files_apks run of locally built target-files
Change-Id: Id79125208f31c78b1ac2079172f9c91a9203849b
Output is useful for debugging.
Bug: 72837107
Test: Local sign_target_files_apks run of locally built target-files
Change-Id: I5c27fcc86fa3a51080e0502eb8f5f01a40b033c3
This CL separates the options into three groups (excluding the global
options provided via common.py).
- Non-A/B OTA specific options;
- A/B OTA specific options;
- Common options that apply to both.
It mostly reshuffles the lines, with minor change to "--verify" that
removes the obselete remounting behavior (which doesn't apply to
block-based OTA).
Hopefully this makes the expected behavior of some options less
confusing.
Test: `build/make/tools/releasetools/ota_from_target_files.py`
Change-Id: I194ea52c4f7d6a3c7f34531abbcf3fdc7b7f4fa8
This CL defaults the journal size to 0 for system images (i.e. system,
vendor, system_other, oem, product). We used to do this by explicitly
defining BOARD_{SYSTEM,VENDOR,OEM,PRODUCT}IMAGE_JOURNAL_SIZE to 0 in
device-specific BoardConfig. Targets can still specify a non-zero
journal size as needed.
With this CL, marlin/sailfish gets 32MB / 16MB free space back for
system and vendor respectively.
Bug: 75975085
Test: `m dist`. Check the journal size in the generated images.
Change-Id: Ib3185d07c49be2b4f0fac5fe17ec1a82093ba0c0
(cherry picked from commit 965542fe69)
Defaults the reserved blocks for root to 0% on read only partitions
(system, system_other, vendor, oem). It also adds support for
explicitly specifying the extfs reserved percentage via
BOARD_{SYSTEM,VENDOR,OEM,PRODUCT}IMAGE_EXTFS_RSV_PCT.
This eventually translates down to the -m option for mkfs.
Removing the reserved space can save at least 5% from the default.
dumpe2fs system:
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
Bug: 75975085
Test: Build, verify reserved space is changed accordingly
Change-Id: I212d82741908b636db0d658a1c4847bbaadfd5ba
(cherry picked from commit 5ff758799c)
We've added support in brillo_update_payload that allows additionally
looking for images under RADIO/ in the given target_files zips [1]. This
avoids having duplicate radio images in target_files zips.
Also adjust the unittest in test_ota_from_target_files.py to cover this
path.
As a result of this CL, the radio images will no longer appear in the
image archive (i.e. <target>-img.zip) as well - they are less useful
anyway because we have packed only the _updatable_ pieces that are part
of full bootloader/radio images.
Bug: 77218220
Test: `python -m unittest test_ota_from_target_files`
Test: `python -m unittest test_add_img_to_target_files`
Test: `m dist` produces the same full OTA package
Test: Build marlin-userdebug in internal branch. Check the image zip.
Change-Id: I05579480f0bb9ab90aaeecf75969ee29b6904ad6
It was initially introduced in commit
2e735ca34e, where it packs additional
vendor images into target_files zip in order to generate OTAs. We can
acheive the same goal with INSTALLED_RADIOIMAGE_TARGET, which is the way
being actively used across all targets, including IoT (the former
Brillo) targets.
Bug: 78201540
Test: `m dist` with aosp_marlin-userdebug
Test: Code search shows no active user of BRILLO_VENDOR_PARTITIONS.
Test: `python -m unittest test_add_img_to_target_files`
Change-Id: I8803d5377b5a39304a701cceafb243f9a228347d
This CL defaults the journal size to 0 for system images (i.e. system,
vendor, system_other, oem, product). We used to do this by explicitly
defining BOARD_{SYSTEM,VENDOR,OEM,PRODUCT}IMAGE_JOURNAL_SIZE to 0 in
device-specific BoardConfig. Targets can still specify a non-zero
journal size as needed.
With this CL, marlin/sailfish gets 32MB / 16MB free space back for
system and vendor respectively.
Bug: 75975085
Test: `m dist`. Check the journal size in the generated images.
Change-Id: Ib3185d07c49be2b4f0fac5fe17ec1a82093ba0c0
Defaults the reserved blocks for root to 0% on read only partitions
(system, system_other, vendor, oem). It also adds support for
explicitly specifying the extfs reserved percentage via
BOARD_{SYSTEM,VENDOR,OEM,PRODUCT}IMAGE_EXTFS_RSV_PCT.
This eventually translates down to the -m option for mkfs.
Removing the reserved space can save at least 5% from the default.
dumpe2fs system:
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
Bug: 75975085
Test: Build, verify reserved space is changed accordingly
Change-Id: I212d82741908b636db0d658a1c4847bbaadfd5ba
Non-A/B devices need to include the DTBO image
within the recovery partition to be self-sufficient
and prevent OTA failures.
Test: Ran 'm dist' and verified that the DTBO image
was included in recovery.img using unpack_bootimg.
Also ran 'make' and verified that the DTBO image was
included in recovery.img using unpack_bootimg.
Bug: 74763691
Change-Id: I38c9c395c95d21f4da42cfa646063bd4416f6bd8
Merged-In: I38c9c395c95d21f4da42cfa646063bd4416f6bd8
(cherry picked from commit e74a38bc6d)
Non-A/B devices need to include the DTBO image
within the recovery partition to be self-sufficient
and prevent OTA failures.
Test: Ran 'm dist' and verified that the DTBO image
was included in recovery.img using unpack_bootimg.
Also ran 'make' and verified that the DTBO image was
included in recovery.img using unpack_bootimg.
Also tested that aosp_angler-userdebug could
boot into recovery(including a fake dtbo.img
in recovery image).
Bug: 74763691
Change-Id: I38c9c395c95d21f4da42cfa646063bd4416f6bd8
The following is a buggy pattern that won't capture anything into err.
The issue is benign, since a failed run would be eventually captured by
a subsequent check.
p = Run(["aapt", ...], stdout=subprocess.PIPE)
output, err = p.communicate()
if err:
raise ...
This CL changes the error detection to be based on the return code from
aapt. It also adds some sanity test to ensure the call to aapt works.
The test app is built from AOSP com.android.cts.ctsshim (chosen mostly
because of its small size).
Test: python -m unittest test_common
Change-Id: I337f141bd0fc5f0801dfc628c601b88b7640789c
For the functions in common.py, capture the stdout/stderr outputs when
shelling out to external tools. Dump the outputs on errors.
Bug: 76123422
Test: Inject errors to signapk.jar / brotli / unzip. Check the outputs.
Change-Id: Ib2d4272528b83a50bb727788cf8a5211e2ddade8
build_image.py used to invoke build_verity_metadata.py with a hard-coded
path of 'system/extras/verity/build_verity_metadata.py', which makes it
hard to run unittests from non-$(ANDROID_BUILD_TOP) directory.
This CL adds the dependency on the tool, so that it gets installed to
$(HOST_OUT_EXECUTABLES), then removes the hard-coded path.
Bug: 74544459
Bug: 76015688
Test: `m dist`
Test: python -m unittest test_validate_target_files
Change-Id: I0dcf4eb067a0db6f099cb589eb99a151a05c7f2b
In FinalizeMetadata and PropertyFiles, we need to reserve space between
the calls to Compute() and Finalize(). We used to put a 10-byte
placeholder, in the hope of covering the 'offset:length' space for the
metadata entry, as well as the possible value changes in other entries.
However, this could fail in two possible cases: (a) metadata entry
itself has a large offset (e.g. staying near the end of a 1-GiB package,
where the offset itself has 10-digit); or (b) the offsets for other
entries change substantially due to entry reordering. Note that for case
(b), it's space inefficient to always reserve 15-byte for _each_ token
in the property-files.
This CL handles both of these two cases. For (a), we bump up the 10-byte
to 15-byte, which is large enough to cover a package size up to 10-digit
number (i.e. ~9GiB) with a metadata entry size of 4-digit. All these
15-byte will be used for the metadata token alone.
For (b), we add a fallback flow that would retry one more time, but
based on the already signed package that has entries in desired order.
Bug: 74210298
Test: python -m unittest test_ota_from_target_files
Test: Generate aosp-bullhead full OTA with '--no_signing' flag.
Change-Id: If20487602d2ad09b3797465c01972f2fa792a1f1
(cherry picked from commit 3bf8c65029)
This CL handles a path that uses system-root-image on non-A/B device.
For this path, we can't generate recovery-from-boot patch with imgdiff,
because boot/recovery images contain different number of entries (only
recovery iamge has ramdisk image now).
Using BOARD_USES_FULL_RECOVERY_IMAGE can work around the issue, at the
cost of extra size. Compared to carrying full recovery image, this CL
saves the cost of the kernel size, by putting a patch that's roughly the
size of the recovery ramdisk.
The applypatch executable already detects and handles a bsdiff patch
automatically. No change required to that end.
Note that it won't further reduce the patch size by handling that
ramdisk entry specially, because (a) that's the only difference between
the two images; and (b) there's no corresponding data in boot image to
be diff'd against.
Bug: 72731506
Test: `m dist` with aosp_angler-userdebug. Check the device can install
recovery image successfully (after intentionally corrupting the
recovery image).
Test: Build aosp_angler-userdebug with BOARD_BUILD_SYSTEM_ROOT_IMAGE set.
Verify the generated patch files.
Test: Run validate_target_files.py with the target_files.zips.
Change-Id: I69c06f51ba8c39ae059c5e9a6872a9f10600cf17
Merged-In: I69c06f51ba8c39ae059c5e9a6872a9f10600cf17
(cherry picked from commit 6d5d623987)
This CL exposes ota-property-files flag for non-A/B OTA packages.
Currently the line only contains the info for the METADATA entry, for
example "ota-property-files=metadata:69:286". This allows system updater
to just download the METADATA entry, as opposed to downloading the
entire package, to learn about the info regarding the OTA package (e.g.
post-OTA build fingerprint). Note that this requires the OTA server-side
support to pass down the flag along with the update URL.
Bug: 74210298
Test: python -m unittest test_ota_from_target_files
Test: Generate a non-A/B package and check the property-files string.
Change-Id: I1482c587e18ea7101c8328777ea988c2d8ca06ac
Merged-In: I1482c587e18ea7101c8328777ea988c2d8ca06ac
(cherry picked from commit c0746f4e94)
This CL handles a path that uses system-root-image on non-A/B device.
For this path, we can't generate recovery-from-boot patch with imgdiff,
because boot/recovery images contain different number of entries (only
recovery iamge has ramdisk image now).
Using BOARD_USES_FULL_RECOVERY_IMAGE can work around the issue, at the
cost of extra size. Compared to carrying full recovery image, this CL
saves the cost of the kernel size, by putting a patch that's roughly the
size of the recovery ramdisk.
The applypatch executable already detects and handles a bsdiff patch
automatically. No change required to that end.
Note that it won't further reduce the patch size by handling that
ramdisk entry specially, because (a) that's the only difference between
the two images; and (b) there's no corresponding data in boot image to
be diff'd against.
Bug: 72731506
Test: `m dist` with aosp_angler-userdebug. Check the device can install
recovery image successfully (after intentionally corrupting the
recovery image).
Test: Build aosp_angler-userdebug with BOARD_BUILD_SYSTEM_ROOT_IMAGE set.
Verify the generated patch files.
Test: Run validate_target_files.py with the target_files.zips.
Change-Id: I69c06f51ba8c39ae059c5e9a6872a9f10600cf17
In FinalizeMetadata and PropertyFiles, we need to reserve space between
the calls to Compute() and Finalize(). We used to put a 10-byte
placeholder, in the hope of covering the 'offset:length' space for the
metadata entry, as well as the possible value changes in other entries.
However, this could fail in two possible cases: (a) metadata entry
itself has a large offset (e.g. staying near the end of a 1-GiB package,
where the offset itself has 10-digit); or (b) the offsets for other
entries change substantially due to entry reordering. Note that for case
(b), it's space inefficient to always reserve 15-byte for _each_ token
in the property-files.
This CL handles both of these two cases. For (a), we bump up the 10-byte
to 15-byte, which is large enough to cover a package size up to 10-digit
number (i.e. ~9GiB) with a metadata entry size of 4-digit. All these
15-byte will be used for the metadata token alone.
For (b), we add a fallback flow that would retry one more time, but
based on the already signed package that has entries in desired order.
Bug: 74210298
Test: python -m unittest test_ota_from_target_files
Test: Generate aosp-bullhead full OTA with '--no_signing' flag.
Change-Id: If20487602d2ad09b3797465c01972f2fa792a1f1
This CL exposes ota-property-files flag for non-A/B OTA packages.
Currently the line only contains the info for the METADATA entry, for
example "ota-property-files=metadata:69:286". This allows system updater
to just download the METADATA entry, as opposed to downloading the
entire package, to learn about the info regarding the OTA package (e.g.
post-OTA build fingerprint). Note that this requires the OTA server-side
support to pass down the flag along with the update URL.
Bug: 74210298
Test: python -m unittest test_ota_from_target_files
Test: Generate a non-A/B package and check the property-files string.
Change-Id: I1482c587e18ea7101c8328777ea988c2d8ca06ac
AbOtaPropertyFiles writes 'ota-property-files' into package metadata.
Comparing to StreamingPropertyFiles, AbOtaPropertyFiles additionally
writes a token of 'payload-metadata.bin' for a virtual entry that's head
of the 'payload.bin'. This entry can be used by the system updater to
verify if a payload is applicable to the underlying device.
AbOtaPropertyFiles intends to replace StreamingPropertyFiles, as it
covers all the info of the latter. We keep them both in P, and will
remove the latter in later release.
Bug: 74210298
Test: python -m unittest test_ota_from_target_files
Test: Generate full and incremental A/B OTAs. Check the property-files
in the METADATA entry.
Change-Id: If5872c8ee0a97a8a95b07d4c69b463aa3fe1b9b9
(cherry picked from commit b63046750c)
AbOtaPropertyFiles writes 'ota-property-files' into package metadata.
Comparing to StreamingPropertyFiles, AbOtaPropertyFiles additionally
writes a token of 'payload-metadata.bin' for a virtual entry that's head
of the 'payload.bin'. This entry can be used by the system updater to
verify if a payload is applicable to the underlying device.
AbOtaPropertyFiles intends to replace StreamingPropertyFiles, as it
covers all the info of the latter. We keep them both in P, and will
remove the latter in later release.
Bug: 74210298
Test: python -m unittest test_ota_from_target_files
Test: Generate full and incremental A/B OTAs. Check the property-files
in the METADATA entry.
Change-Id: If5872c8ee0a97a8a95b07d4c69b463aa3fe1b9b9
And move StreamingPropertyFiles as its subclass. We will need similar
PropertyFiles instance for non-A/B OTA as well (to expose the
offset/size for the METADATA entry).
Bug: 74210298
Test: python -m unittest test_ota_from_target_files
Test: Generate an A/B OTA. Check the generated property-files string.
Test: pylint --rcfile=pylintrc \
ota_from_target_files.py \
test_ota_from_target_files.py
Change-Id: If90d97f0b330749fd8a6cde2ed9d0d6cd6ea60a8
And move StreamingPropertyFiles as its subclass. We will need similar
PropertyFiles instance for non-A/B OTA as well (to expose the
offset/size for the METADATA entry).
Bug: 74210298
Test: python -m unittest test_ota_from_target_files
Test: Generate an A/B OTA. Check the generated property-files string.
Test: pylint --rcfile=pylintrc \
ota_from_target_files.py \
test_ota_from_target_files.py
Change-Id: If90d97f0b330749fd8a6cde2ed9d0d6cd6ea60a8
(cherry picked from commit 432f374a1701909ca324f8b047666614684568c0)
For a given (signed) target-files.zip, this CLs allows verifying the
Verified Boot related images. It works with both of VB 1.0 and VB 2.0
images.
As part of the CL, it also moves validate_target_files.py to argparse,
which is more flexible than the traditional getopt module.
Also add unittests for the VB 1.0 path. VB 2.0 tests will be added in
follow-up CL.
Example usage:
- Run the script on aosp_bullhead target-files.zip.
$ ./build/make/tools/releasetools/validate_target_files.py \
--verity_key build/target/product/security/verity.x509.pem \
--verity_key_mincrypt build/target/product/security/verity_key \
aosp_bullhead-target_files-4522605.zip
- Run the script on aosp_walleye target-files.zip.
$ ./build/make/tools/releasetools/validate_target_files.py \
--verity_key external/avb/test/data/testkey_rsa4096.pem \
aosp_walleye-target_files-4627254.zip
Bug: 63706333
Bug: 65486807
Test: Run validate_target_files.py on target_files.zip files.
Test: PYTHONPATH=build/make/tools/releasetools python -m unittest \
test_validate_target_files
Change-Id: I170f14d5828d15f3687d8af0a89a816968069057
This CL breaks down ComputeStreamingMetadata() into mutiple member
functions of StreamingPropertyFiles class, which correspond to the
two-pass logic when generating streaming property files (aka streaming
metadata).
StreamingPropertyFiles.Compute() does the work for the first pass, by
putting placeholders before doing initial signing. Finalize()
corresponds to the second pass, where the placeholders get replaced with
actual data. Verify() can be optionally called to assert the correctness
of the work.
The separation between Compute() and Finalize() is to allow having
multiple StreamingPropertyFiles instances (in coming up CLs). This way
we can call Compute() multiple times for each instance, followed by only
one call to SignOutput(). And similarly for Finalize().
Bug: 74210298
Test: Generate an A/B OTA package. Check the METADATA entry.
Test: python -m unittest test_ota_from_target_files
Change-Id: I45be0372a4863c4405e6d8e20bcb9ccdc29e7e11
(cherry picked from commit ae5e4c30fe)
This CL breaks down ComputeStreamingMetadata() into mutiple member
functions of StreamingPropertyFiles class, which correspond to the
two-pass logic when generating streaming property files (aka streaming
metadata).
StreamingPropertyFiles.Compute() does the work for the first pass, by
putting placeholders before doing initial signing. Finalize()
corresponds to the second pass, where the placeholders get replaced with
actual data. Verify() can be optionally called to assert the correctness
of the work.
The separation between Compute() and Finalize() is to allow having
multiple StreamingPropertyFiles instances (in coming up CLs). This way
we can call Compute() multiple times for each instance, followed by only
one call to SignOutput(). And similarly for Finalize().
Bug: 74210298
Test: Generate an A/B OTA package. Check the METADATA entry.
Test: python -m unittest test_ota_from_target_files
Change-Id: I45be0372a4863c4405e6d8e20bcb9ccdc29e7e11
The file was broken due to earlier touches:
- Missing 'import zipfile';
- Mismatching arguments when calling GetSparseImage().
Bug: 73996151
Test: Run validate_target_files.py with a walleye-target_files.zip.
Test: pylint --rcfile=pylintrc validate_target_files.py
Change-Id: I3692bd51fb27a3da698e06b75155e84502549f66
(cherry picked from commit c63626b4a3)
The file was broken due to earlier touches:
- Missing 'import zipfile';
- Mismatching arguments when calling GetSparseImage().
Bug: 73996151
Test: Run validate_target_files.py with a walleye-target_files.zip.
Test: pylint --rcfile=pylintrc validate_target_files.py
Change-Id: I3692bd51fb27a3da698e06b75155e84502549f66
And add unittests for ComputeStreamingMetadata().
This prepares for the changes that add additional property-files (for
both of A/B and non-A/B).
Bug: 74210298
Bug: 72751683
Test: python -m unittest test_ota_from_target_files
Test: Generate A/B OTA package. Check the ota-streaming-property-files
in the METADATA entry.
Change-Id: Ib4b069f61c2c06c035c0cff73a55112f3936b969
(cherry picked from commit f5110498c0)
And add unittests for ComputeStreamingMetadata().
This prepares for the changes that add additional property-files (for
both of A/B and non-A/B).
Bug: 74210298
Bug: 72751683
Test: python -m unittest test_ota_from_target_files
Test: Generate A/B OTA package. Check the ota-streaming-property-files
in the METADATA entry.
Change-Id: Ib4b069f61c2c06c035c0cff73a55112f3936b969
There's a mismatch in WriteABOTAPackageWithBrilloScript().
temp_zip_file = tempfile.NamedTemporaryFile()
...
common.ZipClose(temp_zip_file)
It's benign since common.ZipClose() happens to be calling
"temp_zip_file.close()". This CL moves the use of tempfile to
common.MakeTempFile(), so that the tempfile will be cleaned up
automatically as part of the call to common.Cleanup(). (Not fixing the
close() directly, since the nearby lines will be refactored into another
function shortly.)
Also remove one assert in the same function, which trivially holds in
the current code.
Test: Generate an A/B OTA.
Change-Id: I53b375d1150820de718dec0ead55abf5f4951071
Also minor cleanups to make it pylint clean.
Test: Run check_target_files_signatures.py with a target-files.zip.
Test: pylint --rcfile=pylintrc check_target_files_signatures.py
Change-Id: Ife3b54c7805c2f2562e87e91ab4b4de355782012
In addition to the unzipping work, common.UnzipTemp() kindly bundles an
open ZipFile object as part of the return value. It doesn't look very
helpful to the callers though. It also looks less obvious that the
caller needs to properly close the handle (missing the close here is
benign though). This CL just removes the ZipFile object out of the
return value, and leaves the work to callers.
Test: `m dist` on both of A/B and non-A/B target.
Test: python -m unittest test_add_img_to_target_files
Test: python -m unittest test_common
Test: python -m unittest test_ota_from_target_files
Test: Check the callers to common.UnzipTemp() in code search.
Change-Id: Id47da3fd42a0e76d6ae8851f05780db319ee48cf
This should be the last case to be moved over.
Test: Generate an incremental BBOTA (which exercises the changed code).
Test: `rgrep mkdtemp` gives no more instance.
Change-Id: I76db069476201cdfaf3a2de9d9635dfe54507f7a
... from the following functions in add_img_to_target_files.py.
AddSystem
AddSystemOther
AddVendor
AddProduct
AddDtbo
AddUserdata
AddVBMeta
AddPartitionTable
AddCache
The last user of the parameter in img_from_target_files.py has been
removed in commit 2bb109709a (in O).
Test: pylint --rcfile=pylintrc add_img_to_target_files.py
Test: Check all the callers to the above functions.
Test: m dist
Change-Id: I551d1683def8f8535062fc90f68dafa0f4252822
The file has been removed from target-files.zip since commit
c19a8d5590 (Gingerbread), whose info has
been consolidated into META/misc_info.txt.
Test: `m dist`
Change-Id: Ic144457954f5742ea082dcd9ffbea71df4afe46e
With test_utils.construct_sparse_image(), creating valid sparse image is
no longer a blocker for testing these functions.
Test: python -m unittest test_add_img_to_target_files
Change-Id: Iff1f15837cc744bad52e2120f9a9ad94d7db50d5
This CL adds a new flag '--skip_postinstall' that allows skipping all
the postinstall hooks when generating an A/B OTA package (default:
False). Note that this discards ALL the hooks, including non-optional
ones. Should only be used if caller knows it's safe to do so (e.g. all
the postinstall work is to dexopt apps and a data wipe will happen
immediately after).
Bug: 73547992
Test: python -m unittest test_ota_from_target_files
Test: Generate a full OTA package for walleye. Examine the generated
payload.
Change-Id: Ifc069e897b4019605051eabfd221230a6a37867c
Since we have been carrying test certificates in testdata/ for other
tests, do the same for test_sign_target_files_apks.py. Copy
verity.x509.pem from build/target/product/security/ to testdata/ for
that purpose.
Also capture the stderr output in ReplaceVerityKeyId().
Test: python -m unittest test_sign_target_files_apks
Change-Id: Ie11e042086952e8a4a5a63950cb0b16cc436b7e6
Also minor clean up to the function, to suppress the following lint
warning.
R:1813, 2: Redefinition of cert type from list to unicode (redefined-variable-type)
Test: python -m unittest -v test_common.CommonApkUtilsTest
Change-Id: Ib4b1dfa8b19e505fc70684b648efc36171c73bbf
Add construct_sparse_image() to test_utils.py, which is a util function
to create sparse images. The new tests also partially cover the recent
changes that add 'incomplete' and 'uses_shard_blocks' tags.
Test: python -m unittest test_common
Change-Id: Ia15f5c4ad12423691216ebbad2c28f95c8427d7e
This is a mandatory field in the generated payload.
Bug: 35724498
Test: Generate OTA with --include_secondary. Check the generation
command for secondary payload.
Change-Id: Ib2e25cc020e294eae5a4be36d6dca297432804b8
Merged-In: Ib2e25cc020e294eae5a4be36d6dca297432804b8
(cherry picked from commit 6048121485)
When target defines 'BOARD_EXT4_SHARE_DUP_BLOCKS := true', the generated
system/vendor images may contain shared blocks (i.e. some blocks will
show up in multiple files' block list), which violates the current
assumptions in BBOTA script.
This CL allows generating BBOTAs by considering the first occurrence as
the "owner" of the shared blocks. All the later users of the shared
blocks will have an incomplete block list, whose RangeSet's will be
tagged with 'uses_shared_blocks'.
Files with 'uses_shared_blocks' tag will not be diff'd with imgdiff,
potentially with patch size penalty. Such files will be accounted for in
imgdiff stats report, where we can revisit for a better solution.
Bug: 64109868
Test: Generate BBOTA full and incremental package with targets defining
'BOARD_EXT4_SHARE_DUP_BLOCKS := true'.
Change-Id: I87fbc22eef7fafe2a470a03fdcfa1babf088ea8d
This CL uses the 'incomplete' tag to skip applying imgdiff to files with
incomplete block list. It's not the ideal fix to address the holes in
ext4 images, but would unhide other imgdiff issues covered by the
unconditional fallback.
Bug: 68016761
Test: Generate an incremental OTA package from images with incomplete
block list. Check the imgdiff stats report.
Test: `python -m unittest test_blockimgdiff`
Change-Id: Ice77686414e70f5e42de35c1757fb31cf02e4fd4
pylint complains about undefined `diff_done`:
W:754, 8: Global variable 'diff_done' undefined at the module level (global-variable-undefined)
W:820,14: Global variable 'diff_done' undefined at the module level (global-variable-undefined)
It would still warn about using global statement after adding the
definition.
W:859, 8: Using the global statement (global-statement)
W:925,14: Using the global statement (global-statement)
This CL computes 'diff_done' via 'len(diff_queue)' instead. It also
moves the progress reporting _before_ the diff work. This way it avoids
showing 100% progress with still changing filenames (because multiple
workers could see an empty queue simultaneously upon finishing their own
works).
There're possible alternatives, such as using the 'nonlocal' keyword in
Python 3 (which we're not there yet), or by using mutable object instead
(e.g. 'diff_done = [0]'). This CL looks cleaner, since it just kills the
var.
Test: Generate a BBOTA incremental. Check the on-screen progress
report.
Test: `pylint --rcfile=pylintrc blockimgdiff.py` no longer complains
about the global diff_done.
Change-Id: I339824735527e1f794b5b1dc99ff3fdb2da85744
The secondary payload should always be applied with
SWITCH_SLOT_ON_REBOOT=0. This CL moves the 'secondary' parameter from
Payload.WriteToZip() to Payload.__init__(). So it can append the flag to
secondary/payload_properties.txt.
Bug: 35724498
Test: Generate an A/B OTA with --include_secondary. Check
secondary/payload_properties.txt entry in the generated ZIP.
Test: `python -m unittest test_ota_from_target_files`
Change-Id: I816c07ab57a1c8a52eff785801634b8b1cb134d4
We have a couple of active imgdiff workarounds (and likely with one more
inbounding that allows having shared blocks in ext4 image). Most of
these workarounds need extending imgdiff's capability. While us not
getting there anytime soon, collect the stats to better understand the
impact of each kind so we can prioritize accordingly.
A sample report is as follows.
Imgdiff Stats Report
========================
APK files diff'd with imgdiff (count: 88)
-------------------------------------------
/system/priv-app/Shell/Shell.apk
...
Large APK files split and diff'd with imgdiff (count: 4)
----------------------------------------------------------
/system/priv-app/Settings/Settings.apk
...
Bug: 68016761
Test: Generate an incremental BBOTA package. Check the stats report.
Test: python -m unittest test_blockimgdiff
Change-Id: I27ad862cde472ab2806db877632ce5a0607420f2
In Transfer class, unbundle 'intact' with the monotonicity of the input
ranges. Negate the logic of 'intact', and thus rename it to 'trimmed'.
Move this property from an attribute of Transfer class as the one in
RangeSet.extra. 'trimmed' indicates whether the source / target ranges
have been modified after creating the Transfer() instance.
The logic that determines whether we can apply imgdiff has been
refactored and consolidated into BlockImageDiff.CanUseImgdiff(). Now
both of the two paths call this single copy, i.e. the one that detects
large APKs (before creating Transfer()'s), and the other that's about to
generate the patch for a given Transfer instance.
Bug: 68016761
Test: python -m unittest test_blockimgdiff
Test: Generate an incremental BBOTA package.
Change-Id: Id07195f63f1fa6c3af6e9091940d251cf09fa104
'monotonic' has been non-optional since [1] (L-MR1). Fix the comment in
RangeSet.parse(), as well as the use in blockimgdiff.py.
[1] commit 8b72aefb5a.
Test: Generate an incremental BBOTA package.
Change-Id: I7f95231683473b4f0f07f9c83fccc0e36a1340cb
Prior to this CL, the call to delta_generator in
check_ota_package_signature.VerifyAbOtaPayload() didn't redirect stderr.
The logs (mostly INFO) on successful verification added noise to the
normal output, which also upset the unittest result parser.
This CL captures stderr outputs from delta_generator, and will only dump
them on error.
Bug: 72884343
Test: `python -m unittest -v test_ota_from_target_files > /dev/null`
gives clean output.
Test: Inject error into delta_generator. The call to
check_ota_package_signature correctly dumps both of stdout and
stderr outputs.
Change-Id: I014a4b21bf758dcf0a4b9963259d6019851935ee
By default, an A/B OTA package doesn't contain the images for the
secondary slot (e.g. system_other.img). Specifying
"--include_secondary" that's introduced in this CL allows generating
a separate payload that will install secondary slot images. Both
payloads will be added to the generated A/B OTA package.
An example A/B OTA package with secondary payload
|
+-- payload.bin
|
+-- payload_properties.txt
|
+-- secondary/payload.bin
|
+-- secondary/payload_properties.txt
|
+-- ...
Such a package needs to be applied in a two-stage manner. During the
first stage, the updater applies the primary payload only. Upon
finishing, it reboots the device into the newly updated slot. It then
continues to install the secondary payload to the inactive slot, but
without switching the active slot at the end (needs the matching support
in update_engine, i.e. SWITCH_SLOT_ON_REBOOT flag).
Due to the special install procedure, the secondary payload will be
always generated as a full payload.
Bug: 35724498
Test: Generate full and incremental OTAs with --include_secondary. Check
the generated OTAs.
Test: python -m unittest test_ota_from_target_files
Change-Id: I975e826bec492e86eb400f99de0c355a32420127
When calling 'openssl x509 -pubkey' to extract the public key from a
certificate, openssl 1.0 and 1.1 handle the '-out' parameter
differently. openssl 1.0 doesn't write the output into the specified
filename, which leads to the payload verification failure in
check_ota_package_signature.VerifyAbOtaPayload(). This CL addresses
the issue by always collecting the output from stdout instead.
It also refactors the two copies into common.ExtractPublicKey(), and
adds unittest. get_testdata_dir() is moved into test_utils.py that holds
common utils for running the unittests.
Bug: 72884343
Test: python -m unittest test_common
Test: python -m unittest test_ota_from_target_files
Test: Run sign_target_files_apks with '--replace_ota_keys' on marlin
target_files zip. Check the payload pubkey replacement.
Test: Trigger the tests with forrest, and tests no longer fail on
machines with openssl 1.0.1.
Change-Id: Ib0389b360f064053e9aa7cc0546d718e7b23003b
This breaks down the current WriteABOTAPackageWithBrilloScript() into
smaller and testable units, which also prepares for the work in
b/35724498.
Bug: 35724498
Test: python -m unittest test_ota_from_target_files
Test: Get identical A/B OTA packages w/ and w/o the CL.
Change-Id: I2ea45ce98e2d2baa58e94fb829b7242f6fe685a7
Merged-In: I2ea45ce98e2d2baa58e94fb829b7242f6fe685a7
(cherry picked from commit 036d721812)
This breaks down the current WriteABOTAPackageWithBrilloScript() into
smaller and testable units, which also prepares for the work in
b/35724498.
Bug: 35724498
Test: python -m unittest test_ota_from_target_files
Test: Get identical A/B OTA packages w/ and w/o the CL.
Change-Id: I2ea45ce98e2d2baa58e94fb829b7242f6fe685a7
(cherry picked from commit 036d721812)
The same thing has been set in GetPackageMetadata(), so they are no-op.
Also replace an occurrence of 'OPTIONS.source_info_dict' with
source_info, which has been missed from previous clean-up CLs.
Test: Generate an A/B OTA package, and an incremental BBOTA with
--wipe_user_data. Check the metadata.
Test: Generate a two-step incremental BBOTA successfully.
Change-Id: I4bb491cac9064d93fb86d12e617c8f38f040e01e
The info comes from the build.prop file of the target build (thus no
backward compatibility concerns). OTA server and client can optionally
use these info to understand the expected behavior of an OTA package.
Bug: 72751683
Test: python -m unittest test_ota_from_target_files
Test: `m dist`, then check the metadata in the generated OTA package.
Change-Id: I5935f67684d2486bb5f00d67ce4bc756589a56ed
This CL detects incomplete block ranges (e.g. due to the holes in
mke2fs created images). Such block ranges will be tagged, so we won't
attempt to imgdiff those files. Note that the change to blockimgdiff.py,
which uses the tag info, will come in a separate CL.
An 'extra' attribute is added to RangeSet class, which defaults to an
empty dict. An 'incomplete' tag will be added into the dict by the
caller of the class. Not adding this tag as an immediate attribute,
because it is not a property regarding the ranges being represented, but
rather some storage space for the caller.
This CL also refactors GetSparseImage and RoundUpTo4K into common.py, so
the same code can be called from both of ota_from_target_files.py and
validate_target_files.py. Not able to add unittests for
GetSparseImage(), as SparseImage requires data in specific format.
Bug: 68016761
Test: Run validate_target_files.py on target-files.zip. It skips
validating files with missing holes as before.
Test: Run ota_from_target_files.py on angler target-files.zip. It gives
identical packages w/ and w/o the CL.
Test: pylint on changed files. There're warnings with common.py, but
unrelated to this change.
Change-Id: I126ccfea13c0d5ebcc8c1b4ff1a4f9200e97423a
C: 73, 0: Wrong hanging indentation (add 4 spaces).
file_name, actual_sha1, expected_sha1)
^ | (bad-continuation)
C:171, 0: Wrong continued indentation (add 20 spaces).
'SYSTEM/etc/recovery.img', expected_recovery_sha1)
^ | (bad-continuation)
C:185, 0: Wrong continued indentation (add 20 spaces).
file_path='IMAGES/boot.img', expected_sha1=boot_info[3])
^ | (bad-continuation)
C:191, 0: Wrong continued indentation (add 20 spaces).
file_path='IMAGES/recovery.img',
^ | (bad-continuation)
C:192, 0: Wrong continued indentation (add 20 spaces).
expected_sha1=expected_recovery_sha1)
^ | (bad-continuation)
W: 67,15: Use % formatting in logging functions and pass the % parameters as arguments (logging-format-interpolation)
W:150,17: Use % formatting in logging functions and pass the % parameters as arguments (logging-format-interpolation)
W:153,15: Use % formatting in logging functions and pass the % parameters as arguments (logging-format-interpolation)
W:194,15: Use % formatting in logging functions and pass the % parameters as arguments (logging-format-interpolation)
C: 27, 0: standard import "import logging" comes before "import common" (wrong-import-order)
C: 28, 0: standard import "import os.path" comes before "import common" (wrong-import-order)
C: 29, 0: standard import "import re" comes before "import common" (wrong-import-order)
C: 31, 0: standard import "import sys" comes before "import common" (wrong-import-order)
Test: pylint --rcfile=pylintrc validate_target_files.py
Test: Run validate_target_files.py with a target-files.zip.
Change-Id: Ie64acdb4cee4326938c4ad5a34b575d7b82478c0
We used to do this in add_img_to_target_files.AddImagesToTargetFiles(),
which didn't cover the path when calling from make_recovery_patch. As a
result, /system/bin/install-recovery.sh contains different SHA values
from the actual images.
Test: Set up aosp_bullhead to use AVB. `m dist`, then run the following
command to verify the generated install-recovery.sh.
$ ./build/make/tools/releasetools/validate_target_files.py \
out/dist/aosp_bullhead-target_files-eng.zip
Change-Id: Id7be8fb17072252fcd4d08db2057b8c4af053376
The generator function is not thread safe and is prone to race
conditions. This CL uses a lock to protect this generator and loose the
locks elsewhere, e.g. 'WriteRangeDataToFd()'.
Bug: 71908713
Test: Generate an incremental package several times for angler 4208095 to 4442250.
Change-Id: I9e6f0a182a1ba7904a597f403f2b12fe05016513
Check that the Sha1 for src&tgt ranges are correct before computing
patches. This adds ~6 seconds overhead for ~2400 commands.
Bug: 71908713
Test: Generate an incremental package from angler 4208095 to 4442250.
Change-Id: I8cf8ce132fb09a22f7d6689274ddb4a27770be76
bsdiff/imgdiff returns non-zero values correctly in case of failures.
So we don't need to check the stderr anymore. This avoids some false
reports of the diff program.
Bug: 72335938
Bug: 71505046
Test: Check the exit value of bsdiff/imgdiff in code search,
generate a package for angler.
Change-Id: I18f0c3882a40a5288d6aee715713a05270e0db2b
The global GetBuildProp() has been factored into BuildInfo class.
ota_from_target_files stderr: Traceback (most recent call last):
File "/tmp/otatest1734786148718419730/releasetools/ota_from_target_files.py", line 1509, in <module>
main(sys.argv[1:])
File "/tmp/otatest1734786148718419730/releasetools/ota_from_target_files.py", line 1421, in main
source_file=OPTIONS.incremental_source)
File "/tmp/otatest1734786148718419730/releasetools/ota_from_target_files.py", line 1117, in WriteABOTAPackageWithBrilloScript
max_timestamp = GetBuildProp("ro.build.date.utc", OPTIONS.source_info_dict)
NameError: global name 'GetBuildProp' is not defined
Test: Build an incremental A/B package with "--downgrade" flag.
Change-Id: I5c304c8b1ea79f3e5465b203bb990cbdffb9b328
Create a wrapper class that handles the payload signing, which unifies
the paths with and without external signer. Also add tests for the newly
added class.
The test keys under testdata/ are created with the script in
development/tools/make_key. testdata/testkey_with_passwd.pk8 uses
password "foo".
Test: python -m unittest test_ota_from_target_files
Test: Get identical A/B OTA packages w/ and w/o the CL.
Change-Id: Ic770aec726498a3babb88ec509985e7f1210fb18
The output (stdout and stderr combined) would be helpful in understanding
the cause of the failure.
Not changing other occurrences in build_image.py yet, since it's less
obvious if a failed call should be considered fatal (thus whether to
dump such outputs). For example, GetVeritySize() simply returns 0 on
failed calls. They deserve further clean-ups in later CLs (with unit
tests).
Bug: 71864688
Test: Inject errors into mke2fs/e2fsck/simg2img. `m snod` prints outputs.
Test: `m snod` remains quiet on successful runs.
Change-Id: I172403f6cd05bce93767265dbcb110271a68e2a8
This CL makes the following changes:
(a) cleans up the similar codes in AddCareMapTxtForAbOta() that handle
'system' and 'vendor' partitions;
(b) fixes an issue with the arcname in AddPackRadioImages() and
AddRadioImagesForAbOta(), where forward slash should always be used
in zip entry names;
(c) refactors the branching statements in AddRadioImagesForAbOta() to
reduce indentation levels.
Test: python -m unittest test_add_img_to_target_files
Test: `m dist` with aosp_marlin-userdebug. Check META/care_map.txt in the
generated target_files.zip.
Change-Id: I3d6f794962d0c68390fbd18eb13c2622acab3ff5
Separate three functions out of AddImagesToTargetFiles(), into
AddCareMapTxtForAbOta(), AddRadioImagesForAbOta() and
AddPackRadioImages() respectively. This CL tries to apply minimal
changes for the refactoring purpose.
Also add tests for AddRadioImagesForAbOta() and AddPackRadioImages().
The tests for AddCareMapTxtForAbOta() require better testing support to
mock sparse_img.SparseImage, which will be added in later CLs.
Test: python -m unittest test_add_img_to_target_files
Test: `m dist` with aosp_marlin-userdebug. Check META/care_map.txt in
the generated target_files.zip.
Change-Id: I1bb723c15237ff721f165cfce0ce996008ce9948
Now they (A/B, non-A/B full and non-A/B incremental) share the same
code. Also add tests for the newly added function.
Test: python -m unittest test_ota_from_target_files
Test: Build A/B and non-A/B packages respectively.
Change-Id: I5499dd9c683ac1a18d8a24f21ff8845fe69f343e
This helps to generate a deterministic package.
Bug: 71770360
Test: Generate a incremental package and transfers are added by file name.
Change-Id: I7562a200b97a1babbb09a77801324cc9408cc01f
In non-A/B OTA path, we've been loading the info dict for the target
build twice (once to have a peek at the OTA type, with a reload after
unzipping the target-files zip). Remove the reloading.
This CL moves the loading of the source info dict up a bit to avoid
having two copies in the A/B and non-A/B paths. It also moves up the
sanity checks of the loaded info dicts before unzipping the target-files
zips.
Test: Generate full and incremental packages for bullhead and marlin.
Change-Id: Iccb953feb0b6ecf62e6f73d6e9ceea00077d098d
GetBuildProp() and GetVendorBuildProp() have been factored into
BuildInfo class. These two functions were meant to be deleted in commit
481bab8d48.
Test: python -m unittest test_ota_from_target_files
Test: No users of these two functions.
Change-Id: Id590a46d597939973ef2339cdfea4f5163548d4d
The fallback-to-full logic doesn't look beneficial, but may hide away
real bugs, especially when it's on by default.
Test: Generate an incremental non-A/B OTA successfully.
Test: Inject bugs into script; observe that incremental OTA
generation errors out instead of silently falling back.
Test: No internal users of the flag from code search.
Change-Id: Ifc627d898552c0bd72463e8c107e36610421608e
We split large apks and generated patches for them in parallel,
resulting in nondeterminate packages between different runs. This CL
sort the split transfers by target name first; and then add them
sequentially to the final transfer list.
Also fix a side effect where we may generate a wrong sha1 for split
ranges due to synchronization error.
Bug: 71770360
Bug: 71759418
Test: Generate the package several times, compare the log and the transfer list.
Change-Id: I2a49e22594d59ffaa98b11edc776be4e3c4c561f
Prior to this CL, we have similar codes in different paths
(WriteFullOTAPackage / WriteBlockIncrementalOTAPackage /
WriteABOTAPackageWithBrilloScript). This CL factors out the common codes
that deal with OEM-specific properties into BuildInfo class, and adds
tests for the new class.
Test: python -m unittest test_ota_from_target_files
Test: Create an incremental package that uses and doesn't use OEM
properties respectively.
Test: Create a full package that uses and doesn't use OEM properties
respectively.
Change-Id: Ida914cace12803d55396fa503bafcac2db2a520e
Test: python -m unittest test_common
Test: Run sign_target_files_apks.py on a target with compressed APKs.
Change-Id: I107a8b8f2f0f82e2d1947f14c8a8b3778f633b11
Mostly cosmetic changes, such as moving print statement to print
function. The only functional change is to wrap the file opening in
Append() with try statement, which would dump the error message as
needed.
Test: `m dist`
Test: python -m unittest test_build_image
Change-Id: I4e6c593517cf737bfcbe51bc533b22a247b10e44
This simplifies the work to track mkdtemp()'d dirs. This CL also cleans
up the temporary file creation in build_image.py.
Test: `m dist`
Test: python -m unittest test_common
Change-Id: Ied7d86126714b9b26e5df7bcca1dfdea21cfcf60
