Commit graph

2477 commits

Author SHA1 Message Date
Devin Moore
6363a04e79 Merge changes from topic "init_boot_img"
* changes:
  Update releasetools for new init_boot.img
  Build a new init_boot.img that contains the boot ramdisk
2021-12-22 23:21:17 +00:00
Treehugger Robot
ceb81e2bef Merge "Fix some python2/3 migration issues" 2021-12-22 06:33:32 +00:00
Kelvin Zhang
0d0ca5d6a0 Fix some python2/3 migration issues
Exception objects don't necessarily have a .message member, use str()
instead.

Test: th
Change-Id: I57b68dff1ace4937d6ef3b7c564f031e219491b1
2021-12-21 12:31:55 -08:00
Inseob Kim
e7b222aa9e Update comment up to date
nonplat* is renamed to vendor*.

Bug: 210536608
Test: N/A
Change-Id: Ief699bf11395a5b0d9bee22220c09acac6e450f2
2021-12-21 15:57:03 +09:00
Devin Moore
afdd7c74bc Update releasetools for new init_boot.img
This new init_boot.img contains the ramdisk that used to reside in the
boot.img file.

Test: set BOARD_PREBUILT_INIT_BOOT_IMAGE to an external init_boot.img
      - Check that "m" pulls in the init_boot.img to
      out/target/product/vsoc_x86_64/
      - Check that "m dist" adds the init_boot.img to
      aosp_cf_x86_64_phone-img-eng.devinmoore.zip
Test: atest --host releasetools_test
Bug: 203698939
Change-Id: If7ef2cf093e5e525529c7c44333c0f40f6ba0764
2021-12-20 20:54:23 +00:00
Inseob Kim
067492988a fsverity_metadata: Support PEM key
When we have a PEM key, we don't need the process converting a DER key
to PEM format, but we just need to use the PEM key as-is.

Bug: 205987437
Test: build and manual test
Change-Id: I6f61a9088efc0f7193737d3c33b8cfde399b2b6f
2021-12-15 09:41:39 +09:00
Inseob Kim
2d03493341 Add fsverity_metadata_generator helper binary
Making this a host tool will help users generate their own fsverity
metadata easily.

Bug: 205987437
Test: m fsverity_metadata_generator and run it
Change-Id: Iafd228815a74d298d87ca1466c6909c0d24c5874
2021-12-15 09:41:29 +09:00
Tianjie Xu
fab626b7d0 Merge "Revert "Temporarily remove the care_map in the ota package"" 2021-12-01 18:23:41 +00:00
Tianjie Xu
2f76f90616 Revert "Temporarily remove the care_map in the ota package"
This reverts commit 5856cfbf82.

Reason for revert: re-enable care_map b/205541521

Change-Id: I64b4637991a80c9d08a28bdd324c47b4067b1762
2021-12-01 05:54:27 +00:00
David Anderson
009d6f8c73 build_image: Query the actual image size after invoking the final mkfs.
It's not guaranteed that the requested image size to mkfs is precisely
respected, due to metadata alignment and such. For accurate care maps
use the real image size rather than requested.

Bug: 205541521
Test: smartsync to 7892270, check that care_map.pb has the right block
      count
Change-Id: I60fe64f720db13d3c3c4f1d8968341d7293217c9
2021-11-30 15:13:56 -08:00
Treehugger Robot
1fb1088e88 Merge "Fix signapk not found error when running ota_from_target_files locally" 2021-11-19 23:25:05 +00:00
Kelvin Zhang
4fc3aa0bc3 Fix signapk not found error when running ota_from_target_files locally
location of ota_from_target_files changed from
out/host/linux-x86/bin to
out/soong/host/linux-x86/bin . This changes relative position of
signapj.jar. To fix, use ANDROID_HOST_OUT as search path

Change-Id: I5397171566e9d7598b5ef16ae26641f0c183d748
2021-11-19 11:34:54 -08:00
Inseob Kim
f69346e0a8 Add fsverity digest manifest
fsverity digest manifest stores a map from files to fsverity digests.
The manifest is installed as a serialized protobuf file, to a signed apk
system/etc/security/fsverity/BuildManifest.apk.

Bug: 193113311
Test: build with PRODUCT_SYSTEM_FSVERITY_GENERATE_METADATA := true
Change-Id: I55fc10400206b8ce0d5f198faea08fe3930b362c
2021-11-11 11:19:44 +00:00
Inseob Kim
9cda397948 Implement fsverity metadata generator
Using fsverity tool, fsverity metadata for specific artifacts in system
mage can be generated. Users can do that by setting a makefile variable
PRODUCT_SYSTEM_FSVERITY_GENERATE_METADATA to true.

If set to true, the following artifacts will be signed.

- system/framework/*.jar
- system/framework/oat/<arch>/*.{oat,vdex,art}
- system/etc/boot-image.prof
- system/etc/dirty-image-objects

One fsverity metadata container file per one input file will be
generated in system.img, with a suffix ".fsv_meta". e.g. a container
file for "system/framework/foo.jar" will be
"system/framework/foo.jar.fsv_meta".

Bug: 193113311
Test: build with PRODUCT_SYSTEM_FSVERITY_GENERATE_METADATA := true
Change-Id: Ib70d591a72d23286b5debcb05fbad799dfd79b94
2021-11-11 17:30:19 +09:00
Tianjie
b049191e78 Fix python3 incompatiable code
http://aosp/1883069 switch the releasetool to use python3.
But target_files_diff still have py2 code that cause failures.
Fix that to unblock OTA generation.

Bug: 205790608
Test: generate an incremental OTA
Change-Id: Ib4d86dc1842afeae8b35681c7d809da140fac600
2021-11-09 23:37:11 -08:00
Treehugger Robot
984ad44276 Merge "Skip signing key check in some cases" 2021-11-10 00:48:27 +00:00
Tianjie Xu
3fcc52519e Merge "Temporarily remove the care_map in the ota package" 2021-11-09 19:38:14 +00:00
Kelvin Zhang
4b588cf75a Skip signing key check in some cases
Test: th
Bug: 205657204

Change-Id: I3a2fc886cfb3fe3d53619bfb8fe84aa799d80de3
2021-11-09 08:43:01 -08:00
Tianjie
5856cfbf82 Temporarily remove the care_map in the ota package
The care_map has incorrect size for non-sparse images. Temporarily
removes it until the root cause is fixed

Bug: 205541521
Test: build
Change-Id: I76bcd2c0c778566b4a6c69b6d45428952225d406
2021-11-08 22:55:53 -08:00
Cole Faust
c76931fe8b Use python 3 for release tools
The release tools have already been updated in prior cls to
support python 3. To test this, I added code to print the script
name to a temp file at the beginning of every script, and then
ran various builds, ensuring that the scripts were run afterwards.

The builds run:
m brillo_update_payload checkvintf minigzip lz4 toybox unpack_bootimg deapexer (needed for releasetools_test)
m (as both sdk_phone_x86_64 and beagle_x15)
m target-files-package (as both sdk_phone_x86_64 and beagle_x15)
m releasetools_test && out/host/linux-x86/nativetest64/releasetools_test/releasetools_test (25 errors with and without python3)
m check_target_files_signatures apksigner target-files-package && out/host/linux-x86/bin/check_target_files_signatures out/target/product/emulator_x86_64/obj/PACKAGING/target_files_intermediates/sdk_phone_x86_64-target_files-eng.colefaust.zip
m dist
As aosp_cf_x86_64_phone: m dist && sign_target_files_apks  out/dist/aosp_cf_x86_64_phone-target_files-eng.$USER.zip /tmp/signed_target_files.zip && validate_target_files /tmp/signed_target_files.zip

This hit all the scripts except for make_recovery_patch,
merge_builds, and ota_package_parser.

I couldn't find anything that uses merge_builds, so it must
only be run manually.

make_recovery_patch and ota_package_parser are only run if
TARGET_OTA_ALLOW_NON_AB is true (among other things), which
is not the case for any product in aosp.

Test: Described in commit message + presubmits
Change-Id: I1a29eafa7ff1a69973b27055e311de77f7ee628b
2021-11-05 13:47:28 -07:00
Cole Faust
b820bcd829 Prepare release tools for python 3
- Sort dictionaries before looping over them
- Don't call sorted() on lists with Nones
- Open file in binary format when serializing protobufs

Change-Id: If5dbc908f7125f6184014b3c1c7891f833d1d8bf
Bug: 203436762
Test: Presubmits
2021-11-04 17:58:44 -07:00
Kelvin Zhang
2639048c21 Fix a failure in signing test
Pixel moved away from sparse images, so validate_target_files is failing
because it expects sparse images.

Test: th
Change-Id: I322ff10c2afbacfb4d78991be60c11aac92a6d4c
2021-11-03 21:43:22 -07:00
Kelvin Zhang
9b6d4ec01b Add a compressor types option
Test: th
Change-Id: I1ff1bccf179bfd87ce9e08def0e5e38921d7d158
2021-11-03 11:17:04 -07:00
Jooyung Han
8caba5e14e Support custom APEX signing tool
When an APEX specifies its custom signing tool (custom_sign_tool:),
apexkeys.txt contains the info and sign_target_files_apks pass the value
to apex_util.

For now the Virt APEX has its own custom signing tool (sign_virt_apex),
which is added to OTATOOLS.

Bug: 193504286
Test: sign_target_files_apks invokes sign_virt_apex
Change-Id: Iba845723fe3e18f542963324b9c58cd00914c5ba
2021-10-27 16:56:22 +09:00
Jooyung Han
0f5a41da56 sign_apex --sign_tool
A new argument is a custom signing tool for APEX contents. When
specified, apex_util invokes the tool with payload's key and payload
directory.

For now, the Virt APEX has its own custom signing tool (sign_virt_apex)
to re-sign filesystem images in it.

Bug: 193504286
Test: atest releasetools_test
Test: m sign_apex sign_virt_apex
Test: sign_apex --sign_tool sign_virt_apex --payload_key ..
  --container_key .. resigned.apex
      adb install resigned.apex
      reboot & vm run-app
Change-Id: Ic4d369c2ba42a8295044a0a75e054dc8def93208
2021-10-27 16:56:22 +09:00
Jooyung Han
c4b7b34b4b fix releasetools_test
Some test cases are failing because tests are not updated accordinly.

Bug: n/a
Test: atest releasetools_test
Change-Id: I3df071f72f01dedd6df4fa462ca52b8a0b1ffd4e
2021-10-27 16:06:10 +09:00
Treehugger Robot
2992dccd43 Merge "Allow data wipe for full OTAs" 2021-10-27 04:53:16 +00:00
Kelvin Zhang
81641afae4 Allow data wipe for full OTAs
Test: th
Change-Id: I7a751b461a5442d22ebca07c98624f35c02dca15
2021-10-26 16:03:25 -07:00
Treehugger Robot
d8f17b1fc4 Merge "Add pcluster and deduplication support for erofs images." 2021-10-16 01:56:43 +00:00
Treehugger Robot
777ef8384c Merge "sign_target_files_apks: adding --allow_gsi_debug_sepolicy" 2021-10-15 05:41:55 +00:00
Bowgo Tsai
2a781692ae sign_target_files_apks: adding --allow_gsi_debug_sepolicy
https://android-review.googlesource.com/q/topic:gsi_debug_policy
adds userdebug_plat_sepolicy.cil into the GSI system.img to
reduce the steps of repacking a debug ramdisk.

This CL checks that the file userdebug_plat_sepolicy.cil shouldn't
exist before signing, unless the caller explicitly specifies
--allow_gsi_debug_sepolicy to allow it.

Note: also fixes the indentation around the block.

Bug: 201482141
Test: sign_target_files_apks *-target_files-*.zip signed.zip
Change-Id: I56ed328a9ae70cf49dbd3c6efb5a4a8c54e1b7a7
2021-10-14 16:54:37 +08:00
David Anderson
64b351b882 Add pcluster and deduplication support for erofs images.
This adds BOARD_xIMAGE_EROFS_PCLUSTER_SIZE and
BOARD_EROFS_PCLUSTER_SIZE, which set the "pcluster size" of erofs images
for individual images or all erofs images respectively. The pcluster
size affects the maximum size of a physical compressed block.

This also adds BOARD_EROFS_SHARE_DUP_BLOCKS, which turns on chunk
support in EROFS.

Bug: 201685920
Test: manual test
Change-Id: I27ec0899f89890562796dd9fa567fc74182fbefb
2021-10-13 01:03:10 -07:00
David Anderson
d29e5371e1 Add blocklist support for EROFS.
This also fixes argument ordering for mkerofsimage.sh.

Bug: 201685920
Test: build OTA with erofs
Change-Id: Ic2ba794e37a0feb73a1028eb7522d35bffbf3af6
2021-10-13 01:03:08 -07:00
David Anderson
271dab6ceb Clean up code duplication in build_image.py.
This is another code block where lots of stuff is duplicated, making it
hard to add new partitions or partition features.

Bug: N/A
Test: m otapackage, treehugger
Change-Id: I4c71275303a9246b37c03b24f531925b90d26fc8
2021-10-11 21:20:35 -07:00
Christine Tsai
28b5a7c806 Revert "Clean up code duplication in build_image.py."
This reverts commit aac502fa31.

Reason for revert: Build breakage

Bug: 202674967
Change-Id: I331aef1896db236726b5313580a1172c08173702
2021-10-11 17:25:27 +00:00
David Anderson
aac502fa31 Clean up code duplication in build_image.py.
This is another code block where lots of stuff is duplicated, making it
hard to add new partitions or partition features.

Bug: N/A
Test: m otapackage, misc_info.txt is identical
Change-Id: I435987605fc66b78bfaf454556f1eaa498e3728f
2021-10-08 17:21:43 -07:00
Treehugger Robot
0b08b8dc14 Merge "Fix aosp_x86_64-user build failure" 2021-09-27 15:51:27 +00:00
Bowgo Tsai
aba5c9eacd Fix aosp_x86_64-user build failure
Commit Ia982eb2ee3d1eb64db72c1836e433bcc53e71e3f removes boot-5.4.img
and leaves only a boot-5.10.img file, which makes 'boot_container'
become false. This leads to the failure in AddVbmetaDigest() as it will
search a 'boot.img' based on the 'boot' descriptor from the vbmeta.img.

Add a condition that if boot_images[0] is not 'boot.img' then the
build is also a boot container.

Bug: 199807830
Test: build aosp_x86_64-user
Change-Id: I4a9487b075186f0abf2ba74d3a1cf78072352a05
2021-09-27 14:11:14 +08:00
David Anderson
40a821f600 Allow changing the EROFS compressor.
This adds BOARD_EROFS_COMPRESSOR to change the compression algorithm
globally, and BOARD_{x}IMAGE_EROFS_COMPRESSOR to change it for
individual partitions.

Bug: N/A
Test: manual test
Change-Id: I2ef831558242a4070ee96269140c33b66c689351
2021-09-22 18:04:27 -07:00
David Anderson
dfc9d7912f Merge "Avoid too much reserved_size for erofs image" 2021-09-16 19:36:29 +00:00
Daniel Norman
78554ea568 Build vendor images first when rebuilding with vendor_otatools.
Only supports rebuilding vendor.img and odm.img currently.

This allows "other" images (e.g. vbmeta, super_empty, ...) to
be rebuilt using the latest tooling.

Bug: 188491126
Bug: 192422274
Test: Sign a GRF R+S merged target_files package using:
      sign_target_files_apks \
        --vendor_otatools=<otatools.zip from R> \
	--vendor_partitions=vendor,odm \
	merged-target_files.zip \
	signed-target_files.zip
Change-Id: Ib93aea9f083ee9b722c31a42dcc780b5222053b8
2021-09-15 21:53:25 +00:00
Greg Kaiser
d9c6d5fc91 Merge "Revert "Rebuild vendor images in sign_target_files_apks with vendor otatools."" 2021-09-14 19:38:29 +00:00
Greg Kaiser
e086f7222e Revert "Rebuild vendor images in sign_target_files_apks with vendor otatools."
This reverts commit a1094e9f0c.

Reason for revert: Likely cause for b/199911935

Change-Id: Ib20400cc2f740a26fbbfb0da34fdf20ed7dc620e
2021-09-14 19:32:27 +00:00
Daniel Norman
97e0c2831d Merge "Rebuild vendor images in sign_target_files_apks with vendor otatools." 2021-09-13 20:12:11 +00:00
Huang Jianan
6552727e03 Avoid too much reserved_size for erofs image
After the erofs image is built, it won't be modified again. The reserve
room is only used for the hash footer. There is no need to set an
excessive margin.

Bug: 199248596
Test: erofs image size is smaller than before
Signed-off-by: Huang Jianan <huangjianan@oppo.com>
Change-Id: If2535b9ad2c2d60bea74feeca576a168387ff341
2021-09-13 02:03:13 +00:00
Yifan Hong
1ab1c2ff96 Merge "Avoid repeated building of erofs image" 2021-09-11 00:05:45 +00:00
Tianjie
e7ab38d964 Add option to override minor version
Pass the minor version override to delta_generator. So we can test
features in tradefed before releasing it to droidfoods.

Bug: 197361113
Test: TH
Change-Id: Ie8846935dfa13784f0745513be13811908f44ac3
2021-09-09 12:16:19 -07:00
Daniel Norman
a1094e9f0c Rebuild vendor images in sign_target_files_apks with vendor otatools.
This is needed until image compilation tooling is backwards compatible.

Bug: 188491126
Bug: 192253131
Bug: 192422274
Test: Sign a GRF R+S merged target_files package using:
      sign_target_files_apks \
        --vendor_otatools=<otatools.zip from R> \
	--vendor_partitions=vendor,odm,vbmeta \
	merged-target_files.zip \
	signed-target_files.zip
Change-Id: Iec208f544b56f8ed577344d8d6ca904773d8baab
2021-09-09 17:32:22 +00:00
Huang Jianan
ffa1d57819 Avoid repeated building of erofs image
The image size of erofs is only affected by the content of the input
file, so it doesn't care about the input image_size. In addition, the
build time of erofs image is longer than ext4 due to compression. Only
execute mkfs once to reduce unnecessary time consumption.

Bug: 199248596
Test: erofs image build faster than before
Signed-off-by: Huang Jianan <huangjianan@oppo.com>
Change-Id: I23a4e640d5066485de3fea0f808998045e716d1f
2021-09-09 14:18:59 +08:00
David Anderson
9e95a02a87 Don't sparse right-sized ext4 and erofs images.
When we introduced Dynamic Partitions, we stopped giving readonly
partitions fixed sizes. In addition we introduced deduplication for
ext4. These two factors greatly reduce the impact of sparse images,
since there aren't many fill blocks to optimize.

This patch disables sparsing for images that are rightsized and do not
explicitly specify extra reserved space. This makes the images a little
easier to work with from an engineering perspective. They no longer have
to be unsparsed to interact with any tooling. It also eases a potential
source of bugs, as b/184225422 is not reproducible with sparsing off.

On Pixel, the difference between the sparsed partitions and unsparsed is
12M (out of roughly 4G).

Bug: 198001223
Test: make, treehugger, make target-files-package
      dynamic partitions are no longer sparse images
Change-Id: I74459f8abe74a15a24ba5a40cf701e6af2db8179
2021-09-01 13:12:39 -07:00