Commit graph

2500 commits

Author SHA1 Message Date
Kelvin Zhang
6fe2fc031f Add erofs compressor type param
Test: th
Bug: 206729162
Change-Id: I3ddd1a55e1544a8a4d24ed445562f05ad0aaad2c
2022-01-18 10:17:49 -08:00
Kelvin Zhang
554c8bee14 Fix formatting error
Test: th
Bug: 214388458
Change-Id: I2dc72884aad4f440b4b68566e2e29f9f6f5aa73e
2022-01-18 10:17:49 -08:00
Kelvin Zhang
4a376986ea Merge "Fix check_tf failure when duplicate package names are present" 2022-01-18 18:14:02 +00:00
Yi-yo Chiang
8b4e2fd6c0 Merge changes from topic "gki-avb-vts"
* changes:
  releasetools: Android T GKI certification scheme
  Android T GKI certification scheme
2022-01-17 04:52:28 +00:00
Yi-Yo Chiang
36054e2daf releasetools: Android T GKI certification scheme
Companion change of Iaf48a6e3d4b97fa6bfb5e1635a288b045baa248f
To support new GKI certification scheme for boot.img and
init_boot.img on upgrading and launching device combinations.

Bug: 210367929
Bug: 211741246
Bug: 203698939
Test: atest --host releasetools_test:test_common
Test: unpack_bootimg --boot_img boot.img
Test: unpack_bootimg --boot_img init_boot.img
Test: avbtool info_image --image out/boot_signature
Change-Id: I3749297c09c3899046550e4be776acbeea37ef2e
2022-01-14 19:50:32 +08:00
Kelvin Zhang
67553739e3 Fix check_tf failure when duplicate package names are present
Test: check_target_files_signatures sdk_gphone64_x86_64-target_files-8069754.zip
Bug: 214388458

Change-Id: I0a8a0d0596c04a9da4283d4d4caeb3bb62762ccc
2022-01-13 12:30:22 -08:00
Kelvin Zhang
f2728d615e Add option to enable lz4diff
When lz4diff is enabled, we inject the host copy of liblz4.so in
target_file to delta_generator. This is to ensure that host tooling can
produce same compressed output as the device.

Test: th
Bug: 206729162

Change-Id: I2d8206f7ec54fadedea16bf3d811b8353bc9414d
2022-01-12 17:21:00 -08:00
Jiyong Park
efbb6ffa00 Add min_sdk_version to BuildManifest.apk
sign_target_files_apks demands it.

Bug: 213823227
Test: run com.google.android.tradefed.ota.SigningToolTest#Testcase1_Signingtest
Change-Id: I1bba2b25a71449a57d284024e77fb76d1b79a0ee
2022-01-10 20:14:05 +09:00
Treehugger Robot
dcd7e283d2 Merge "Add option to enable zucchini" 2022-01-09 20:38:18 +00:00
Martin Stjernholm
76537be0a9 Merge "Add an option to specify the path to aapt2." 2022-01-09 19:11:49 +00:00
Treehugger Robot
555c3a7dc7 Merge "Write signature size even when it's empty" 2022-01-08 02:20:11 +00:00
Martin Stjernholm
58472e8609 Add an option to specify the path to aapt2.
Thin manifests (e.g. master-art) may not have the aapt2 sources and
instead relies on the prebuilt in prebuilts/sdk/tools/linux/bin. In
that case there's no aapt2 in the search path, so it needs to be
specified.

Test: Heavy presubmit build on mainline_modules_bundles on git_master
Test: vendor/google/modules/ArtGoogle/build-art-module.sh
  with http://ag/16584845 on git_master-art, where only the prebuilt
  aapt2 is present
Bug: 212462945
Change-Id: I41a22e8146f5a88534c2721345b4d9d64f76698c
2022-01-07 23:08:53 +00:00
Kelvin Zhang
410bb389d8 Add option to enable zucchini
Change-Id: Id952826c9c5af912fba679af61f2a2e1645641dd
2022-01-07 14:59:01 -08:00
Kelvin Zhang
98ef7bb157 Fix GetCareMap for small images
Change-Id: Iac518e36dbb2fe9303e84d502cd7dfe74621c0af
2022-01-07 14:59:01 -08:00
Treehugger Robot
7c88c9c1e6 Merge "Move fsverity metadata generation to Makefile" 2022-01-07 03:37:16 +00:00
Treehugger Robot
bd5fd25265 Merge "Avoid packing ramdisk into GKI boot-*.img" 2022-01-07 02:35:06 +00:00
Inseob Kim
3d5ae80f50 Write signature size even when it's empty
Normally this has no effect, but when we generate metadata for small
files (<=4KB), merkle tree isn't generated. In such case, writing zero
will make the metadata format simpler and unconditional.

Test: manual
Change-Id: Ibe18175b580af3409c896a8bb97323792ad9c459
2022-01-07 10:56:17 +09:00
Inseob Kim
135c1f144f Move fsverity metadata generation to Makefile
build_image.py has been handling fsverity metadata generation in the
packing step, but it can cause issues because the metadata files are
missing in the $OUT directory, and they only exist in result system.img.
This change moves the generation logic into Makefile, and makes the
metadata tracked by ninja graph.

Bug: 206326351
Test: PRODUCT_SYSTEM_FSVERITY_GENERATE_METADATA := true and build
Change-Id: I1f910d8ac6e2cc3c54f35916871733c632f18e44
2022-01-07 09:16:42 +09:00
Bowgo Tsai
88fc2bdd19 Avoid packing ramdisk into GKI boot-*.img
If init_boot.img is present, the GKI boot-*.img should
not include the ramdisk because the ramdisk will be packed
into the init_boot.img instead.

The "has_ramdisk" flag incorrectly checks the condition of:
prebuilt_name != "boot.img" to see if it is a boot.img.
Because "has_ramdisk" was added before we packed multiple
GKI boot-*.img, e.g., boot-5.10.img, boot-5.10-lz4.img, etc.,
into the target files.

Fix this by checking the partition_name is "boot" or not.
Also moving the logic into a new function with comments for each
condition.

Bug: 203698939
Bug: 213028932
Test: sign_target_files_apks \
        --gki_signing_key=external/avb/test/data/testkey_rsa4096.pem \
        --gki_signing_algorithm=SHA256_RSA4096 \
        ./out/dist/*-target_files-eng.*.zip signed.zip, then
        unpack_bootimg to checks the signed boot-*.img has no ramdisk
Change-Id: I5354669feb54d547dbe797e51b1b1baa187cb7cf
2022-01-06 17:50:04 +08:00
Jacky Liu
bff997fd1f Merge "Add PRODUCT_EXTRA_OTA_KEYS to add extra ota keys in otacerts.zip" 2022-01-06 06:31:37 +00:00
Jacky Liu
beb0b6927c Add PRODUCT_EXTRA_OTA_KEYS to add extra ota keys in otacerts.zip
Equivalent to PRODUCT_EXTRA_RECOVERY_KEYS but for A/B OTA.

Bug: 211848136
Test: set PRODUCT_EXTRA_OTA_KEYS and check otacerts.zip
Change-Id: I81e27d12a22b405f6227b09c01ed684dfcede19e
2022-01-05 17:54:40 +08:00
Bowgo Tsai
8d4b72405e Fix the signing error on no-system-image targets
Currently when running sign_target_files_apks on a no-system-image
target, it will raise the following error:

  ValueError: max() arg is an empty sequence

This is because there is no APK files in the target_files.zip.
Fixing this by setting maxsize to zero in this case.

Bug: 213028932
Test: lunch gki_arm64-userdebug; make dist
Test: sign_target_files_apks \
        --gki_signing_key=external/avb/test/data/testkey_rsa4096.pem \
        --gki_signing_algorithm=SHA256_RSA4096 \
        --gki_signing_extra_args="--prop gki:prop1 --prop gki:prop2" \
        ./out/dist/*-target_files-eng.*.zip signed.zip
Change-Id: I40daecbc2ff3f89d3e635d1a4a1c1dea31ba9a27
2022-01-04 17:55:19 +08:00
Baligh Uddin
06cc018629 --sign_tool is an optional flag applicable to virtApex.
For not virtApex, this results in a KeyError.
line 151, in <module>\n', '  File
"/usr/local/google/home/baligh/clients/goog/master/out/host/linux-x86/bin/sign_apex/sign_apex.py",
line 144, in main\n', "KeyError: 'sign_tool'\n"]

BUG: 193504286
Test: TH
Change-Id: Id982e5c57086ada78168163d2293813df121847d
2021-12-24 06:17:47 +00:00
Devin Moore
6363a04e79 Merge changes from topic "init_boot_img"
* changes:
  Update releasetools for new init_boot.img
  Build a new init_boot.img that contains the boot ramdisk
2021-12-22 23:21:17 +00:00
Treehugger Robot
ceb81e2bef Merge "Fix some python2/3 migration issues" 2021-12-22 06:33:32 +00:00
Kelvin Zhang
0d0ca5d6a0 Fix some python2/3 migration issues
Exception objects don't necessarily have a .message member, use str()
instead.

Test: th
Change-Id: I57b68dff1ace4937d6ef3b7c564f031e219491b1
2021-12-21 12:31:55 -08:00
Inseob Kim
e7b222aa9e Update comment up to date
nonplat* is renamed to vendor*.

Bug: 210536608
Test: N/A
Change-Id: Ief699bf11395a5b0d9bee22220c09acac6e450f2
2021-12-21 15:57:03 +09:00
Devin Moore
afdd7c74bc Update releasetools for new init_boot.img
This new init_boot.img contains the ramdisk that used to reside in the
boot.img file.

Test: set BOARD_PREBUILT_INIT_BOOT_IMAGE to an external init_boot.img
      - Check that "m" pulls in the init_boot.img to
      out/target/product/vsoc_x86_64/
      - Check that "m dist" adds the init_boot.img to
      aosp_cf_x86_64_phone-img-eng.devinmoore.zip
Test: atest --host releasetools_test
Bug: 203698939
Change-Id: If7ef2cf093e5e525529c7c44333c0f40f6ba0764
2021-12-20 20:54:23 +00:00
Inseob Kim
067492988a fsverity_metadata: Support PEM key
When we have a PEM key, we don't need the process converting a DER key
to PEM format, but we just need to use the PEM key as-is.

Bug: 205987437
Test: build and manual test
Change-Id: I6f61a9088efc0f7193737d3c33b8cfde399b2b6f
2021-12-15 09:41:39 +09:00
Inseob Kim
2d03493341 Add fsverity_metadata_generator helper binary
Making this a host tool will help users generate their own fsverity
metadata easily.

Bug: 205987437
Test: m fsverity_metadata_generator and run it
Change-Id: Iafd228815a74d298d87ca1466c6909c0d24c5874
2021-12-15 09:41:29 +09:00
Tianjie Xu
fab626b7d0 Merge "Revert "Temporarily remove the care_map in the ota package"" 2021-12-01 18:23:41 +00:00
Tianjie Xu
2f76f90616 Revert "Temporarily remove the care_map in the ota package"
This reverts commit 5856cfbf82.

Reason for revert: re-enable care_map b/205541521

Change-Id: I64b4637991a80c9d08a28bdd324c47b4067b1762
2021-12-01 05:54:27 +00:00
David Anderson
009d6f8c73 build_image: Query the actual image size after invoking the final mkfs.
It's not guaranteed that the requested image size to mkfs is precisely
respected, due to metadata alignment and such. For accurate care maps
use the real image size rather than requested.

Bug: 205541521
Test: smartsync to 7892270, check that care_map.pb has the right block
      count
Change-Id: I60fe64f720db13d3c3c4f1d8968341d7293217c9
2021-11-30 15:13:56 -08:00
Treehugger Robot
1fb1088e88 Merge "Fix signapk not found error when running ota_from_target_files locally" 2021-11-19 23:25:05 +00:00
Kelvin Zhang
4fc3aa0bc3 Fix signapk not found error when running ota_from_target_files locally
location of ota_from_target_files changed from
out/host/linux-x86/bin to
out/soong/host/linux-x86/bin . This changes relative position of
signapj.jar. To fix, use ANDROID_HOST_OUT as search path

Change-Id: I5397171566e9d7598b5ef16ae26641f0c183d748
2021-11-19 11:34:54 -08:00
Inseob Kim
f69346e0a8 Add fsverity digest manifest
fsverity digest manifest stores a map from files to fsverity digests.
The manifest is installed as a serialized protobuf file, to a signed apk
system/etc/security/fsverity/BuildManifest.apk.

Bug: 193113311
Test: build with PRODUCT_SYSTEM_FSVERITY_GENERATE_METADATA := true
Change-Id: I55fc10400206b8ce0d5f198faea08fe3930b362c
2021-11-11 11:19:44 +00:00
Inseob Kim
9cda397948 Implement fsverity metadata generator
Using fsverity tool, fsverity metadata for specific artifacts in system
mage can be generated. Users can do that by setting a makefile variable
PRODUCT_SYSTEM_FSVERITY_GENERATE_METADATA to true.

If set to true, the following artifacts will be signed.

- system/framework/*.jar
- system/framework/oat/<arch>/*.{oat,vdex,art}
- system/etc/boot-image.prof
- system/etc/dirty-image-objects

One fsverity metadata container file per one input file will be
generated in system.img, with a suffix ".fsv_meta". e.g. a container
file for "system/framework/foo.jar" will be
"system/framework/foo.jar.fsv_meta".

Bug: 193113311
Test: build with PRODUCT_SYSTEM_FSVERITY_GENERATE_METADATA := true
Change-Id: Ib70d591a72d23286b5debcb05fbad799dfd79b94
2021-11-11 17:30:19 +09:00
Tianjie
b049191e78 Fix python3 incompatiable code
http://aosp/1883069 switch the releasetool to use python3.
But target_files_diff still have py2 code that cause failures.
Fix that to unblock OTA generation.

Bug: 205790608
Test: generate an incremental OTA
Change-Id: Ib4d86dc1842afeae8b35681c7d809da140fac600
2021-11-09 23:37:11 -08:00
Treehugger Robot
984ad44276 Merge "Skip signing key check in some cases" 2021-11-10 00:48:27 +00:00
Tianjie Xu
3fcc52519e Merge "Temporarily remove the care_map in the ota package" 2021-11-09 19:38:14 +00:00
Kelvin Zhang
4b588cf75a Skip signing key check in some cases
Test: th
Bug: 205657204

Change-Id: I3a2fc886cfb3fe3d53619bfb8fe84aa799d80de3
2021-11-09 08:43:01 -08:00
Tianjie
5856cfbf82 Temporarily remove the care_map in the ota package
The care_map has incorrect size for non-sparse images. Temporarily
removes it until the root cause is fixed

Bug: 205541521
Test: build
Change-Id: I76bcd2c0c778566b4a6c69b6d45428952225d406
2021-11-08 22:55:53 -08:00
Cole Faust
c76931fe8b Use python 3 for release tools
The release tools have already been updated in prior cls to
support python 3. To test this, I added code to print the script
name to a temp file at the beginning of every script, and then
ran various builds, ensuring that the scripts were run afterwards.

The builds run:
m brillo_update_payload checkvintf minigzip lz4 toybox unpack_bootimg deapexer (needed for releasetools_test)
m (as both sdk_phone_x86_64 and beagle_x15)
m target-files-package (as both sdk_phone_x86_64 and beagle_x15)
m releasetools_test && out/host/linux-x86/nativetest64/releasetools_test/releasetools_test (25 errors with and without python3)
m check_target_files_signatures apksigner target-files-package && out/host/linux-x86/bin/check_target_files_signatures out/target/product/emulator_x86_64/obj/PACKAGING/target_files_intermediates/sdk_phone_x86_64-target_files-eng.colefaust.zip
m dist
As aosp_cf_x86_64_phone: m dist && sign_target_files_apks  out/dist/aosp_cf_x86_64_phone-target_files-eng.$USER.zip /tmp/signed_target_files.zip && validate_target_files /tmp/signed_target_files.zip

This hit all the scripts except for make_recovery_patch,
merge_builds, and ota_package_parser.

I couldn't find anything that uses merge_builds, so it must
only be run manually.

make_recovery_patch and ota_package_parser are only run if
TARGET_OTA_ALLOW_NON_AB is true (among other things), which
is not the case for any product in aosp.

Test: Described in commit message + presubmits
Change-Id: I1a29eafa7ff1a69973b27055e311de77f7ee628b
2021-11-05 13:47:28 -07:00
Cole Faust
b820bcd829 Prepare release tools for python 3
- Sort dictionaries before looping over them
- Don't call sorted() on lists with Nones
- Open file in binary format when serializing protobufs

Change-Id: If5dbc908f7125f6184014b3c1c7891f833d1d8bf
Bug: 203436762
Test: Presubmits
2021-11-04 17:58:44 -07:00
Kelvin Zhang
2639048c21 Fix a failure in signing test
Pixel moved away from sparse images, so validate_target_files is failing
because it expects sparse images.

Test: th
Change-Id: I322ff10c2afbacfb4d78991be60c11aac92a6d4c
2021-11-03 21:43:22 -07:00
Kelvin Zhang
9b6d4ec01b Add a compressor types option
Test: th
Change-Id: I1ff1bccf179bfd87ce9e08def0e5e38921d7d158
2021-11-03 11:17:04 -07:00
Jooyung Han
8caba5e14e Support custom APEX signing tool
When an APEX specifies its custom signing tool (custom_sign_tool:),
apexkeys.txt contains the info and sign_target_files_apks pass the value
to apex_util.

For now the Virt APEX has its own custom signing tool (sign_virt_apex),
which is added to OTATOOLS.

Bug: 193504286
Test: sign_target_files_apks invokes sign_virt_apex
Change-Id: Iba845723fe3e18f542963324b9c58cd00914c5ba
2021-10-27 16:56:22 +09:00
Jooyung Han
0f5a41da56 sign_apex --sign_tool
A new argument is a custom signing tool for APEX contents. When
specified, apex_util invokes the tool with payload's key and payload
directory.

For now, the Virt APEX has its own custom signing tool (sign_virt_apex)
to re-sign filesystem images in it.

Bug: 193504286
Test: atest releasetools_test
Test: m sign_apex sign_virt_apex
Test: sign_apex --sign_tool sign_virt_apex --payload_key ..
  --container_key .. resigned.apex
      adb install resigned.apex
      reboot & vm run-app
Change-Id: Ic4d369c2ba42a8295044a0a75e054dc8def93208
2021-10-27 16:56:22 +09:00
Jooyung Han
c4b7b34b4b fix releasetools_test
Some test cases are failing because tests are not updated accordinly.

Bug: n/a
Test: atest releasetools_test
Change-Id: I3df071f72f01dedd6df4fa462ca52b8a0b1ffd4e
2021-10-27 16:06:10 +09:00
Treehugger Robot
2992dccd43 Merge "Allow data wipe for full OTAs" 2021-10-27 04:53:16 +00:00