Defer edge creation.
Don't create edges until the count is known to avoid repeated allocate+
copy operatios.
Limit resolutions.
Allow only a single resolution condition set per target, and overwrite
intermediate results. Reduces memory and obviates allocations.
Propagate fewer conditions.
Instead of propagating notice conditions to parents in graph during
initial resolve, leave them on leaf node, and attach to ancestors in
the final walk. Reduces copies.
Parallelize resolutions.
Use goroutines, mutexes, and waitgroups to resolve branches of the
graph in parallel. Makes better use of available cores.
Don't accumulate resolutions inside non-containers.
During the final resolution walk, only attach actions to ancestors from
the root down until the 1st non-aggregate. Prevents an explosion of
copies in the lower levels of the graph.
Drop origin for scale.
Tracking the origin of every potential origin for every restricted
condition does not scale. By dropping origin, propagating from top
to bottom can prune many redundant paths avoiding an exponential
explosion.
Conditions as bitmask.
Use bit masks for license conditions and condition sets. Reduces maps
and allocations.
Bug: 68860345
Bug: 151177513
Bug: 151953481
Test: m all
Test: m systemlicense
Test: m listshare; out/soong/host/linux-x86/bin/listshare ...
Test: m checkshare; out/soong/host/linux-x86/bin/checkshare ...
Test: m dumpgraph; out/soong/host/linux-x86/dumpgraph ...
Test: m dumpresolutions; out/soong/host/linux-x86/dumpresolutions ...
where ... is the path to the .meta_lic file for the system image. In my
case if
$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)
... can be expressed as:
${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic
Change-Id: Ia2ec1b818de6122c239fbd0824754f1d65daffd3
Using struct{}{} as the payload for set maps reduces memory use for
large sets.
Bug: 68860345
Bug: 151177513
Bug: 151953481
Test: m all
Test: m systemlicense
Test: m listshare; out/soong/host/linux-x86/bin/listshare ...
Test: m checkshare; out/soong/host/linux-x86/bin/checkshare ...
Test: m dumpgraph; out/soong/host/linux-x86/dumpgraph ...
Test: m dumpresolutions; out/soong/host/linux-x86/dumpresolutions ...
where ... is the path to the .meta_lic file for the system image. In my
case if
$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)
... can be expressed as:
${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic
Change-Id: Ibc831ae80fc50f35e1000348fb28fc0167d0ebed
Use annotation slices looked up from map instead of read from disk to
reduce copies.
Bug: 68860345
Bug: 151177513
Bug: 151953481
Test: m all
Test: m systemlicense
Test: m listshare; out/soong/host/linux-x86/bin/listshare ...
Test: m checkshare; out/soong/host/linux-x86/bin/checkshare ...
Test: m dumpgraph; out/soong/host/linux-x86/dumpgraph ...
Test: m dumpresolutions; out/soong/host/linux-x86/dumpresolutions ...
where ... is the path to the .meta_lic file for the system image. In my
case if
$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)
... can be expressed as:
${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic
Change-Id: Ibf6f7d319092e0d54ab451ffbbd35dcd7d0080e0
sign_target_files_apks demands it.
Bug: 213823227
Test: run com.google.android.tradefed.ota.SigningToolTest#Testcase1_Signingtest
Change-Id: I1bba2b25a71449a57d284024e77fb76d1b79a0ee
Thin manifests (e.g. master-art) may not have the aapt2 sources and
instead relies on the prebuilt in prebuilts/sdk/tools/linux/bin. In
that case there's no aapt2 in the search path, so it needs to be
specified.
Test: Heavy presubmit build on mainline_modules_bundles on git_master
Test: vendor/google/modules/ArtGoogle/build-art-module.sh
with http://ag/16584845 on git_master-art, where only the prebuilt
aapt2 is present
Bug: 212462945
Change-Id: I41a22e8146f5a88534c2721345b4d9d64f76698c
Normally this has no effect, but when we generate metadata for small
files (<=4KB), merkle tree isn't generated. In such case, writing zero
will make the metadata format simpler and unconditional.
Test: manual
Change-Id: Ibe18175b580af3409c896a8bb97323792ad9c459
build_image.py has been handling fsverity metadata generation in the
packing step, but it can cause issues because the metadata files are
missing in the $OUT directory, and they only exist in result system.img.
This change moves the generation logic into Makefile, and makes the
metadata tracked by ninja graph.
Bug: 206326351
Test: PRODUCT_SYSTEM_FSVERITY_GENERATE_METADATA := true and build
Change-Id: I1f910d8ac6e2cc3c54f35916871733c632f18e44
If init_boot.img is present, the GKI boot-*.img should
not include the ramdisk because the ramdisk will be packed
into the init_boot.img instead.
The "has_ramdisk" flag incorrectly checks the condition of:
prebuilt_name != "boot.img" to see if it is a boot.img.
Because "has_ramdisk" was added before we packed multiple
GKI boot-*.img, e.g., boot-5.10.img, boot-5.10-lz4.img, etc.,
into the target files.
Fix this by checking the partition_name is "boot" or not.
Also moving the logic into a new function with comments for each
condition.
Bug: 203698939
Bug: 213028932
Test: sign_target_files_apks \
--gki_signing_key=external/avb/test/data/testkey_rsa4096.pem \
--gki_signing_algorithm=SHA256_RSA4096 \
./out/dist/*-target_files-eng.*.zip signed.zip, then
unpack_bootimg to checks the signed boot-*.img has no ramdisk
Change-Id: I5354669feb54d547dbe797e51b1b1baa187cb7cf
* Now with 4 sections that can be shown/hidden independently.
* After the 'selected_projects' section, add two more sections:
* 'top_directory_section' contains
* a table of directories with at least 1% warnings.
* a 'selected_directory_warnings' subsection to show all warnings
in the clicked directory from the directory table
* 'top_file_section' contains
* a table of files with at least 100 or 1% warnings.
* a 'selected_file_warnings' subsection to show all warnings
in the clicked file from the file table
* Adjust button/section spaces and reduce button font size to 100%.
* Rename drawTable to genTables.
Test: warn.py --url=http://cs/android --separator='?l=' build.log > warnings.html
Test: warn.py --gencsv build.log > warnings.csv
Change-Id: I765b09a46adc111cfe781719ba7aa0f917aa6ffc
Equivalent to PRODUCT_EXTRA_RECOVERY_KEYS but for A/B OTA.
Bug: 211848136
Test: set PRODUCT_EXTRA_OTA_KEYS and check otacerts.zip
Change-Id: I81e27d12a22b405f6227b09c01ed684dfcede19e
Currently when running sign_target_files_apks on a no-system-image
target, it will raise the following error:
ValueError: max() arg is an empty sequence
This is because there is no APK files in the target_files.zip.
Fixing this by setting maxsize to zero in this case.
Bug: 213028932
Test: lunch gki_arm64-userdebug; make dist
Test: sign_target_files_apks \
--gki_signing_key=external/avb/test/data/testkey_rsa4096.pem \
--gki_signing_algorithm=SHA256_RSA4096 \
--gki_signing_extra_args="--prop gki:prop1 --prop gki:prop2" \
./out/dist/*-target_files-eng.*.zip signed.zip
Change-Id: I40daecbc2ff3f89d3e635d1a4a1c1dea31ba9a27
Test: Add a VENDOR_BLOBS_LICENSE entry to an existing product, then `m with-license`
Bug: 203436762
Change-Id: I4bbe77bda7789b6c44bea141518b1cd2e699d326
For not virtApex, this results in a KeyError.
line 151, in <module>\n', ' File
"/usr/local/google/home/baligh/clients/goog/master/out/host/linux-x86/bin/sign_apex/sign_apex.py",
line 144, in main\n', "KeyError: 'sign_tool'\n"]
BUG: 193504286
Test: TH
Change-Id: Id982e5c57086ada78168163d2293813df121847d
Avoid writing test files in the build output directory which fails when
run with Bazel. This happens because Bazel's sandboxing environment
ensures that the test's working directory is unwritable.
See https://docs.bazel.build/versions/main/sandboxing.html for more
information.
Bug: 209687942
Test: atest --bazel-mode zipalign_tests
Test: atest zipalign_tests
Change-Id: Ie22f464830c1ffe4d38a94a16dbd39dafa7fe317
This new init_boot.img contains the ramdisk that used to reside in the
boot.img file.
Test: set BOARD_PREBUILT_INIT_BOOT_IMAGE to an external init_boot.img
- Check that "m" pulls in the init_boot.img to
out/target/product/vsoc_x86_64/
- Check that "m dist" adds the init_boot.img to
aosp_cf_x86_64_phone-img-eng.devinmoore.zip
Test: atest --host releasetools_test
Bug: 203698939
Change-Id: If7ef2cf093e5e525529c7c44333c0f40f6ba0764
When we have a PEM key, we don't need the process converting a DER key
to PEM format, but we just need to use the PEM key as-is.
Bug: 205987437
Test: build and manual test
Change-Id: I6f61a9088efc0f7193737d3c33b8cfde399b2b6f
Making this a host tool will help users generate their own fsverity
metadata easily.
Bug: 205987437
Test: m fsverity_metadata_generator and run it
Change-Id: Iafd228815a74d298d87ca1466c6909c0d24c5874
Performance optimization means not every path will be traversed.
Instead of updating parents via the path, perform a 2nd bottom-up walk
after the top-down walk to propagate the new resolutions to parents.
Note: the 2nd walk method will add resolutions to statically linked
libraries etc. at deeper levels, but those do not affect what gets
reported. In particular, note that test data for dumpresolutions
changes, but none of the test data for listshare, checkshare etc.
changes.
Test: m all systemlicense listshare checkshare dumpgraph dumpresolutions
Bug: 68860345
Bug: 151177513
Bug: 151953481
Change-Id: I76361c4e33bbadbbea38cbec260430e8f9407628
Tune the top-down walk to avoid needlessly walking the same subtree
over and over again with the same condition(s).
Takes walking system image down from 3m to 1.5s.
Test: m all systemlicense listshare checkshare dumpgraph dumpresolutions
Bug: 68860345
Bug: 151177513
Bug: 151953481
Change-Id: I4354800cd8dfc42efd4df274d2ce45eaa3e0a99f
Bug: 68860345
Bug: 151177513
Bug: 151953481
Test: m all
Test: m systemlicense
Test: m listshare; out/soong/host/linux-x86/bin/listshare ...
Test: m checkshare; out/soong/host/linux-x86/bin/checkshare ...
Test: m dumpgraph; out/soong/host/linux-x86/dumpgraph ...
Test: m dumpresolutions; out/soong/host/linux-x86/dumpresolutions ...
where ... is the path to the .meta_lic file for the system image. In my
case if
$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)
... can be expressed as:
${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic
Change-Id: I5d48eababce7bba39795d3668eee86b332cbe43d
package to read, consume, and analyze license metadata and dependency
graph.
Includes the below command-line tools:
listshare outputs csv of projects to share to meet restricted and
reciprocal license requirements with one project per line. The first
field is the path to the project, and subsequent fields identify the
license resolutions as colon-separated target:annotations tuples.
checkshare outputs error messages to stderr for any targets where
policy dictates both sharing and not sharing the source-code, and PASS
or FAIL to stdout. exit status indicates success 0 or conflict found 1
Bug: 68860345
Bug: 151177513
Bug: 151953481
Test: m all
Test: m systemlicense
Test: m listshare; out/soong/host/linux-x86/bin/listshare ...
Test: m checkshare; out/soong/host/linux-x86/bin/checkshare ...
Test: m dumpgraph; out/soong/host/linux-x86/dumpgraph ...
Test: m dumpresolutions; out/soong/host/linux-x86/dumpresolutions ...
where ... is the path to the .meta_lic file for the system image. In my
case if
$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)
... can be expressed as:
${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic
Change-Id: I4ff3f98848f7e6d03a35734300d763ef5f245d53
package to read, consume, and analyze license metadata and dependency
graph.
Includes the below command-line tool:
dumpresolutions outputs the resulting set of resolutions after the
bottom-up and top-down resolves, or after joining 1 or more condition
walks.
Bug: 68860345
Bug: 151177513
Bug: 151953481
Test: m all
Test: m systemlicense
Test: m dumpgraph; out/soong/host/linux-x86/dumpgraph ...
Test: m dumpresolutions; out/soong/host/linux-x86/dumpresolutions ...
where ... is the path to the .meta_lic file for the system image. In my
case if
$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)
... can be expressed as:
${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic
Change-Id: I9869400126cd7ad4b7376b0bab31b46aad732f5d
package to read, consume, and analyze license metadata and dependency
graph.
Includes testdata/ and the the below command-line tool:
dumpgraph outputs edges of the graph as "target dependency annotations"
Bug: 68860345
Bug: 151177513
Bug: 151953481
Test: m all
Test: m systemlicense
Test: m dumpgraph; out/soong/host/linux-x86/dumpgraph ...
where ... is the path to the .meta_lic file for the system image. In my
case if
$ export PRODUCT=$(realpath $ANDROID_PRODUCT_OUT --relative-to=$PWD)
... can be expressed as:
${PRODUCT}/gen/META/lic_intermediates/${PRODUCT}/system.img.meta_lic
Change-Id: I5fe57d361da5155dbcb2c0d369626e9200c9d664
It's not guaranteed that the requested image size to mkfs is precisely
respected, due to metadata alignment and such. For accurate care maps
use the real image size rather than requested.
Bug: 205541521
Test: smartsync to 7892270, check that care_map.pb has the right block
count
Change-Id: I60fe64f720db13d3c3c4f1d8968341d7293217c9
Summary:
In my last diff, I've added mechanism to load private key from keystore.
However, that mechanism will reveal password as part of the java param.
This diff tries to use existing ANDROID_PW_FILE mechanism to support
password for keystore private keys (through stdin)
This diff also fix a null pointer bug in the existing password handling
Test: This diff has been tested locally, and could sign correctly with our
keystore with or without password
Tags:
Change-Id: Ie291ea8702a3b4d270b0f8689b023c3f290980a7
location of ota_from_target_files changed from
out/host/linux-x86/bin to
out/soong/host/linux-x86/bin . This changes relative position of
signapj.jar. To fix, use ANDROID_HOST_OUT as search path
Change-Id: I5397171566e9d7598b5ef16ae26641f0c183d748
Use the proper "**" glob to extract subdirectories as well as files.
The unzip-6.0-fix-recmatch patch shipped on distros like Fedora and ArchLinux
no longer extracts subdirectories when using "*", which breaks vintf checking
on those distros.
Test:
Manually run "unzip -o -q target_files.zip ODM/etc/**" and verify output
run sucessful build with vintf checks on ArchLinux (unzip 6.0-15)
and Ubuntu 20.04 (unzip 6.0-25ubuntu1)
Change-Id: I9e04fa36e071e11427fea89e7369dba8b9c5a4a1
fsverity digest manifest stores a map from files to fsverity digests.
The manifest is installed as a serialized protobuf file, to a signed apk
system/etc/security/fsverity/BuildManifest.apk.
Bug: 193113311
Test: build with PRODUCT_SYSTEM_FSVERITY_GENERATE_METADATA := true
Change-Id: I55fc10400206b8ce0d5f198faea08fe3930b362c
Using fsverity tool, fsverity metadata for specific artifacts in system
mage can be generated. Users can do that by setting a makefile variable
PRODUCT_SYSTEM_FSVERITY_GENERATE_METADATA to true.
If set to true, the following artifacts will be signed.
- system/framework/*.jar
- system/framework/oat/<arch>/*.{oat,vdex,art}
- system/etc/boot-image.prof
- system/etc/dirty-image-objects
One fsverity metadata container file per one input file will be
generated in system.img, with a suffix ".fsv_meta". e.g. a container
file for "system/framework/foo.jar" will be
"system/framework/foo.jar.fsv_meta".
Bug: 193113311
Test: build with PRODUCT_SYSTEM_FSVERITY_GENERATE_METADATA := true
Change-Id: Ib70d591a72d23286b5debcb05fbad799dfd79b94
http://aosp/1883069 switch the releasetool to use python3.
But target_files_diff still have py2 code that cause failures.
Fix that to unblock OTA generation.
Bug: 205790608
Test: generate an incremental OTA
Change-Id: Ib4d86dc1842afeae8b35681c7d809da140fac600
The care_map has incorrect size for non-sparse images. Temporarily
removes it until the root cause is fixed
Bug: 205541521
Test: build
Change-Id: I76bcd2c0c778566b4a6c69b6d45428952225d406
The release tools have already been updated in prior cls to
support python 3. To test this, I added code to print the script
name to a temp file at the beginning of every script, and then
ran various builds, ensuring that the scripts were run afterwards.
The builds run:
m brillo_update_payload checkvintf minigzip lz4 toybox unpack_bootimg deapexer (needed for releasetools_test)
m (as both sdk_phone_x86_64 and beagle_x15)
m target-files-package (as both sdk_phone_x86_64 and beagle_x15)
m releasetools_test && out/host/linux-x86/nativetest64/releasetools_test/releasetools_test (25 errors with and without python3)
m check_target_files_signatures apksigner target-files-package && out/host/linux-x86/bin/check_target_files_signatures out/target/product/emulator_x86_64/obj/PACKAGING/target_files_intermediates/sdk_phone_x86_64-target_files-eng.colefaust.zip
m dist
As aosp_cf_x86_64_phone: m dist && sign_target_files_apks out/dist/aosp_cf_x86_64_phone-target_files-eng.$USER.zip /tmp/signed_target_files.zip && validate_target_files /tmp/signed_target_files.zip
This hit all the scripts except for make_recovery_patch,
merge_builds, and ota_package_parser.
I couldn't find anything that uses merge_builds, so it must
only be run manually.
make_recovery_patch and ota_package_parser are only run if
TARGET_OTA_ALLOW_NON_AB is true (among other things), which
is not the case for any product in aosp.
Test: Described in commit message + presubmits
Change-Id: I1a29eafa7ff1a69973b27055e311de77f7ee628b
- Sort dictionaries before looping over them
- Don't call sorted() on lists with Nones
- Open file in binary format when serializing protobufs
Change-Id: If5dbc908f7125f6184014b3c1c7891f833d1d8bf
Bug: 203436762
Test: Presubmits
Pixel moved away from sparse images, so validate_target_files is failing
because it expects sparse images.
Test: th
Change-Id: I322ff10c2afbacfb4d78991be60c11aac92a6d4c
Make it easier to write tools against .meta_lic files and store complex
data by writing them in textproto.
Test: builds
Change-Id: Ibbb6cfbb1bdddd3d938a86d563673a049d826d66
Gets rid of .meta_module files and instead defers emitting rules until
after all the non-module targets have been processed. Allows direct
dependency on .meta_lic files, which in turn depend on license text
files.
Bug: 68860345
Bug: 151177513
Bug: 151953481
Test: m all
Test: m systemlicense
Test: m reportmissinglicenses
Change-Id: I2c467feac6e13a9366ff66f924889f1dbd48c3f1
When an APEX specifies its custom signing tool (custom_sign_tool:),
apexkeys.txt contains the info and sign_target_files_apks pass the value
to apex_util.
For now the Virt APEX has its own custom signing tool (sign_virt_apex),
which is added to OTATOOLS.
Bug: 193504286
Test: sign_target_files_apks invokes sign_virt_apex
Change-Id: Iba845723fe3e18f542963324b9c58cd00914c5ba
A new argument is a custom signing tool for APEX contents. When
specified, apex_util invokes the tool with payload's key and payload
directory.
For now, the Virt APEX has its own custom signing tool (sign_virt_apex)
to re-sign filesystem images in it.
Bug: 193504286
Test: atest releasetools_test
Test: m sign_apex sign_virt_apex
Test: sign_apex --sign_tool sign_virt_apex --payload_key ..
--container_key .. resigned.apex
adb install resigned.apex
reboot & vm run-app
Change-Id: Ic4d369c2ba42a8295044a0a75e054dc8def93208
Some test cases are failing because tests are not updated accordinly.
Bug: n/a
Test: atest releasetools_test
Change-Id: I3df071f72f01dedd6df4fa462ca52b8a0b1ffd4e
Summary:
Add two flags to load the keys from pkcs#11 keystore. When the option
-loadPrivateKeysFromKeyStore is specified, will load private keys from
the keystore with specified keyStoreName instead of load from file.
Test: make dist for arm_sunfish-user, which includes apk
and ota (wholefile) signing
Test:
- manually call signapk in Java11 (java9 may need additional
change to support), with statically registered pkcs#11 keystore, signed
both apk and ota-package.
- verified using apksigner and extracting otacert from ota-package, both
correct
Change-Id: I3efb8017f73d3d992c07ed4562acfef016a109fe
So that it can be built and installed with
````
(cd build/make/tools/canoninja && go install cmd/canoninja.go)
```
Bug: 201713929
Test: internal
Change-Id: I38133bf26ccfae5ebf8bc3c68bc595b7274576b9
https://android-review.googlesource.com/q/topic:gsi_debug_policy
adds userdebug_plat_sepolicy.cil into the GSI system.img to
reduce the steps of repacking a debug ramdisk.
This CL checks that the file userdebug_plat_sepolicy.cil shouldn't
exist before signing, unless the caller explicitly specifies
--allow_gsi_debug_sepolicy to allow it.
Note: also fixes the indentation around the block.
Bug: 201482141
Test: sign_target_files_apks *-target_files-*.zip signed.zip
Change-Id: I56ed328a9ae70cf49dbd3c6efb5a4a8c54e1b7a7
This adds BOARD_xIMAGE_EROFS_PCLUSTER_SIZE and
BOARD_EROFS_PCLUSTER_SIZE, which set the "pcluster size" of erofs images
for individual images or all erofs images respectively. The pcluster
size affects the maximum size of a physical compressed block.
This also adds BOARD_EROFS_SHARE_DUP_BLOCKS, which turns on chunk
support in EROFS.
Bug: 201685920
Test: manual test
Change-Id: I27ec0899f89890562796dd9fa567fc74182fbefb
This is another code block where lots of stuff is duplicated, making it
hard to add new partitions or partition features.
Bug: N/A
Test: m otapackage, treehugger
Change-Id: I4c71275303a9246b37c03b24f531925b90d26fc8
This is another code block where lots of stuff is duplicated, making it
hard to add new partitions or partition features.
Bug: N/A
Test: m otapackage, misc_info.txt is identical
Change-Id: I435987605fc66b78bfaf454556f1eaa498e3728f
* New "clang-tidy used ... seconds" warnings are reported when
clang-tidy runs for more than 1/2 of TIDY_TIMEOUT seconds.
* Recognize other clang-tidy warnings in separate groups;
they should not be used in normal Android builds.
* Add two more variants of C++ warnings.
Test: warn.py --url=http://cs/android --separator='?l=' build.log > warnings.html
Test: warn.py --gencsv build.log > warnings.csv
Change-Id: I4a2e1c3c817c586cfe3da125b920cca77fcc63b6
1. Fix build break when rebuilding vendor images in
sign_target_files_apks, because of missing SYSTEM/build.prop:
File ".../add_img_to_target_files.py", line 999, in <module>
File ".../add_img_to_target_files.py", line 993, in main
File ".../add_img_to_target_files.py", line 854, in AddImagesToTargetFiles
File ".../add_img_to_target_files.py", line 229, in AddVendor
File ".../add_img_to_target_files.py", line 391, in CreateImage
File "add_img_to_target_files/common.py", line 394, in __init__
self._fingerprint = self.CalculateFingerprint()
File "add_img_to_target_files/common.py", line 537, in CalculateFingerprint
self.GetBuildProp("ro.build.version.release"),
File "add_img_to_target_files/common.py", line 457, in GetBuildProp
raise ExternalError("couldn't find %s in build.prop" % (prop,))
ExternalError: couldn't find ro.build.version.release in build.prop
2. Support more scenarios (non-AB, RADIO...)
Bug: 192422274
Change-Id: I792b0a2c2354ed9312730e322ce28df49b3c2f7f
Commit Ia982eb2ee3d1eb64db72c1836e433bcc53e71e3f removes boot-5.4.img
and leaves only a boot-5.10.img file, which makes 'boot_container'
become false. This leads to the failure in AddVbmetaDigest() as it will
search a 'boot.img' based on the 'boot' descriptor from the vbmeta.img.
Add a condition that if boot_images[0] is not 'boot.img' then the
build is also a boot container.
Bug: 199807830
Test: build aosp_x86_64-user
Change-Id: I4a9487b075186f0abf2ba74d3a1cf78072352a05
Some of the product configuration makefiles use `info` and `warning` Make's
builtins for diagnostics. As running Starlark configuration generates the makefile
as its output, this diagnostics has to go elsewhere. Implement `rblf_log` as
the functional equivalent of `print` that writes to stderr instead of stdout
and use it to implement `info` and `warning` counterparts.
Fixes: 201073196
Test: manual
Change-Id: Ib4d9c10566f9b20310fbee41eda67f0a621b0a84
This adds BOARD_EROFS_COMPRESSOR to change the compression algorithm
globally, and BOARD_{x}IMAGE_EROFS_COMPRESSOR to change it for
individual partitions.
Bug: N/A
Test: manual test
Change-Id: I2ef831558242a4070ee96269140c33b66c689351
Test: Ran the python version of the script and the python3 version
Test: and verified the output is the same. This is for all three
Test: outputs (html, text, xml).
Test: Builds without an error.
Change-Id: I5138b0a1efbbc3ab764c644afc1783c79fd7e250
After the erofs image is built, it won't be modified again. The reserve
room is only used for the hash footer. There is no need to set an
excessive margin.
Bug: 199248596
Test: erofs image size is smaller than before
Signed-off-by: Huang Jianan <huangjianan@oppo.com>
Change-Id: If2535b9ad2c2d60bea74feeca576a168387ff341
Upstream llvm-readobj no longer accepts single dash prefix for long
options. Changed to double dash.
Also let the script to raise exception if rc is non-zero, to avoid
getting confusing error later in the program.
Test: build
Bug: 197230471
Change-Id: I364f51322904431d430549f79af58f455f47e028
Pass the minor version override to delta_generator. So we can test
features in tradefed before releasing it to droidfoods.
Bug: 197361113
Test: TH
Change-Id: Ie8846935dfa13784f0745513be13811908f44ac3
The image size of erofs is only affected by the content of the input
file, so it doesn't care about the input image_size. In addition, the
build time of erofs image is longer than ext4 due to compression. Only
execute mkfs once to reduce unnecessary time consumption.
Bug: 199248596
Test: erofs image build faster than before
Signed-off-by: Huang Jianan <huangjianan@oppo.com>
Change-Id: I23a4e640d5066485de3fea0f808998045e716d1f
* Recognize USE_RBE special environment variable.
* Remove RBE special file path prefix /b/f/w/
and any leading characters.
* Only do this for android now, maybe for chrome later.
* When finding android_root, do not use
/b/f/w/ and /tmp/ file paths.
* Recognize BUILD_ID and add it into .html output.
* Style improvement with continue statements.
Test: warn.py --url=http://cs/android --separator='?l=' build.log > warnings.html
Test: warn.py --gencsv build.log > warnings.csv
Bug: 198657613
Change-Id: I57746f026d5d8b71e792127b8d6ba15f6b28103c
When we introduced Dynamic Partitions, we stopped giving readonly
partitions fixed sizes. In addition we introduced deduplication for
ext4. These two factors greatly reduce the impact of sparse images,
since there aren't many fill blocks to optimize.
This patch disables sparsing for images that are rightsized and do not
explicitly specify extra reserved space. This makes the images a little
easier to work with from an engineering perspective. They no longer have
to be unsparsed to interact with any tooling. It also eases a potential
source of bugs, as b/184225422 is not reproducible with sparsing off.
On Pixel, the difference between the sparsed partitions and unsparsed is
12M (out of roughly 4G).
Bug: 198001223
Test: make, treehugger, make target-files-package
dynamic partitions are no longer sparse images
Change-Id: I74459f8abe74a15a24ba5a40cf701e6af2db8179
--enable_vabc_xor=false to disable, --enable_vabc_xor=true or
--enable_vabc_xor to enable
Also the flag is now enabled by default. But update_engine will
still ignore XOR ops, as update_engine support for XOR is WIP.
Test: th
Change-Id: I038b0dda5cc9efcc1620e72c858626b56ffb1611
Currently CheckVintfIfTrebleEnabled is called after generating payload.
Users have to wait for ~1 hour before vintf is checked. Do the check
early and fail early.
Test: th
Change-Id: I2e4a8c7eaa62c94c92973c127b1d825a26e63b35
The macro find-and-copy finds all the files in the given source tree that
match the given filename patten and create <source>:<dest> pair with the
same relative path in the destination tree.
Bug: 193540681
Test: rbcrun build/make/tests/run.rbc
Change-Id: Ic4315ce2fab7a7791ab55dd9eed039205a1c721a
When computing the data offset of an entry in zip file, we used length
of extra field from central directory. That is correct most of the time
but wrong if the extra field in central directory has different length
than the one in local file directory. Since python's zipfile doesn't
provide an API to access local file header, we need to parse local file
header ourselves and extract length of extra field.
An incorrect offset will cause magic mismatch error from update_engine,
as update_engine expects to find uncompressed payload at the recorded
offset.
Test: th, partner verification
Bug: 191443484
Change-Id: Id670cd79b0bd65adffaaa5224ae4f8065d66b358
When using the VSDK, dexopt is not applied during the vendor build.
To avoid a first-boot time regression, dexopt is applied during the
merge stage, by running dexopt on the vendor apps and rebuilding
the vendor image.
Bug: 188179859
Test: Tested in keystone with VSDK target
Change-Id: Ie8e2d0a82850a2901fa6f250433bcbb43f0a97f2
`zipalign` before `signapk -a 4096` is unnecessary step since `signapk
-a 4096` does the same thing.
And, compressed APEX files don't need entries to be aligned.
Bug: 193362313
Test: atest --host releasetools_test
Change-Id: I0758d1162f7c3eb8cc9c73978b5ae5484f007a3a
A new flag tells that the resulting file size should be aligned as well.
For example, signapk -a 4096 --align-file-size ... generates the output
file of 4K-aligned sized.
Bug: 192991318
Test: check built APEX file size.
APEX files should be sized as 4K-aligned.
Change-Id: I1c287e7219b4551bbb3e5957bdb64d80adfd3c39
During signing, we write the entry directly to the output zip, intead
of a temp directory. Add the logic to write vbmeta_digest.txt to output
zipfile too. So the digest file will show up in the signed target files.
Bug: 189926233
Test: add_img_to_target_files -a <target-files.zip>
Change-Id: Ibf28a8f97512bda8c8c695e06190e1fb6573c53e
fs_config requires the information from the header files in
system/core and bionic/libc. To build the vendor side fs_config,
use the header files in the vendor snashot if the required version
exists.
Bug: 187222756
Test: check ninja depedency with and without snasphots.
Change-Id: Ibf96eab4d9a129745be1a19b2aa2e4c8f57cf6bf
1. system_ext or product may not be a standalone partition,
2. keep symbolic links in vendor image,
3. target-files package may not contain IMAGES folder.
Bug: 192422274
Change-Id: I77867408cc764abb975319a23c387ec63fcf87c3
This allows merged devices to boot using a precompiled_sepolicy built
from merged sources, rather than recompiling this sepolicy at boot
time every boot.
Bug: 178727214
Test: Merge an R+S build using --rebuild-sepolicy and --vendor-otatools.
Observe odm.img is rebuilt by the vendor otatools.zip
when merging.
Observe device boots using ODM's precompiled_sepolicy file.
Test: Same as above, for S+S.
Test: Merge an S+S build using --rebuild-sepolicy and *not*
--vendor-otatools.
Observe odm.img is rebuilt without using a separate otatools.zip.
Observe device boots using ODM's precompiled_sepolicy file.
Change-Id: I9595b8a3296d6deec21db8f0c9bc5b7ec4debd57
This prop is owned by OEM, OEM can set this if they want to disable
VABC.
Test: m dist, make sure generated OTA has VABC disabled
Bug: 185400304
Change-Id: Iceb2fb1f399d38a51722352a86ddf68af05fa24e
Today, the signing script simply ignores capex files, because it
looks for hardcoded '.apex' suffix. Add support to handle capex
as well.
Bug: 190574334
Test: Sign a target file locally
Change-Id: I3085ca7b0396a4fbf1b220f7de44d4eafb60c3d8
So updaters can streaming download the file, and query the apex info
inside the file.
Bug: 190244686
Test: generate an OTA package, check the streaming property
Change-Id: I17078d3f8d60ca53c6afe82f74b232e2fb242467
We have already logged the compressed apexes in the target-files.
Because we want to support the apex metrics during OTA update, also
include the uncompressed apexes in the META/apex_info.pb.
For incremental OTA packages, include the source apex version for
each apex package as well.
Bug: 190244686
Test: unit test
Change-Id: I5cf2647c56c4feb5517f9a81aa1e9abc52515bf1
When generating a partial OTA, filter care_map.pb to include only the
partial partitions, then generate OTA.
Test: Generate a partial OTA, make sure care map is included.
Change-Id: I0eaa12772eb1d06a57451e64f70689d3183f0115
This option is to reduce system partition size.
Bug: 171942852
Signed-off-by: Jaegeuk Kim <jaegeuk@google.com>
Change-Id: Idc849cfce33ac0badb2b9b7953bb821c46a24472
Devices using GKI architecture will use a prebuilt boot.img.
However, we should still sign this prebuilt boot.img with
device-specific AVB keys.
Steps to test the CL.
1. In a device BoardConfig.mk:
# Uses a prebuilt boot.img
TARGET_NO_KERNEL := true
BOARD_PREBUILT_BOOTIMAGE := device/google/redbull/boot.img
# Enable chained vbmeta for the boot image.
# The following can be absent, where the hash descriptor of the
# 'boot' partition will be stored then signed in vbmeta.img instead.
BOARD_AVB_BOOT_KEY_PATH := external/avb/test/data/testkey_rsa4096.pem
BOARD_AVB_BOOT_ALGORITHM := SHA256_RSA4096
BOARD_AVB_BOOT_ROLLBACK_INDEX := $(PLATFORM_SECURITY_PATCH_TIMESTAMP)
BOARD_AVB_BOOT_ROLLBACK_INDEX_LOCATION := 2
2. `make bootimage`, then `avbtool info_image --image $OUT/boot.img`,
checks the image is re-signed with a device-specific key
3. `make dist` to generate out/dist/TF.zip
4. `unzip out/dist/TF.zip IMAGES/boot.img`
5. `avbtool info_image --image out/dist/IMAGES/boot.img`,
checks the image is re-signed with a device-specific key
6. `sign_target_files_apks \
--avb_boot_key=external/avb/test/data/testkey_rsa8192.pem \
--avb_boot_algorithm=SHA256_RSA8192 \
--avb_boot_extra_args="--prop test:sign" \
./out/dist/*-target_files-eng.*.zip signed.zip`, resign the TF.zip
7. `unzip signed.zip IMAGES/boot.img`, then use `avbtool info_image` to
check the boot.img is re-signed with the --avb_boot_key in step 6.
Bug: 188485657
Test: above steps
Change-Id: I7ee8b3ffe6a86aaca34bbb7a8898a97b3f8bd801
Test: th
Test: Manual OTA test on bramble, pause/resume multiple times
Test: verity enabled, VABC enabled OTA
Test: verity enabled, VABC disabled OTA
Test: verity disabled, VABC enabled OTA
Test: verity disabled, VABC disabled OTA
Change-Id: Ia236984b158761f84f54ab7a6d3d49491c249546
For VAB launched device, factory OTA will write system_other
partition to the super image. So we want to check that
sum(dynamic partitions) + system_other + overhead <= super at
build time.
Since we don't know the overhead at build time, we might instead
check sum(all partitions) < super.
Bug: 185809374
Test: m check-all-partition-sizes, unittests
Change-Id: Ia7ba5999d23924a1927e9a9463856a4d0ea90c20
build_image adds additional parameters (uuid, hash_seed) if
prop_dict["ext_mkuserimg"] is set to "mkuserimg_mke2fs".
The comparison does not take paths into consideration, so passing a
full path to mkuserimg_mke2fs would cause the parameters to not
be included.
This is currently not an issue for aosp builds, but could cause problems
for customized build systems.
Bug: 187742822
Test: Manual, using vendor build system, also executed 'm droid'
Change-Id: I7a8973dd0c4d8a39aea5aafcfe1aa69750fb1449
If the build prop ro.build.id isn't set at build time, init will
set it at runtime. The logic is appending the vbmeta digest to
the ro.build.legacy.id.
Make the same change in ota scripts, so the correct build fingerprint
will be saved in the ota metadata.
Bug: 186786987
Test: generate an OTA, check the metadata
Change-Id: I278f59c41c1f98d4cbea749e5d9e4eaf7a6b9565
Commit I8bd8ad3acf324931b47d45fd30bc590206b1927e adds a default
value of "gki_signing_signature_args" in the misc_info.txt for
release signing to work. However, it's better to replace the default
value entirely (e.g., --prop foo:bar) as there is no need to include
them in the final release-signed image.
Bug: 178559811
Bug: 177862434
Test: atest releasetools_test
Test: atest releasetools_py3_test
Change-Id: I060b5a7076ff3e5d883abeb7d72f3db887c9fd69
Background in http://go/compatible-build-fingerprint. If we want
to append unique vbmeta digest to build id, we cannot setup the
prop value at build time. Instead, set the old value as
ro.build.legacy.id; and let init set ro.build.id at runtime.
Bug: 186786987
Test: build a target file with the flag on
Change-Id: Ie139725bb7e5c65bd3f28f43b9975ba48ee10354
Calculate the vbmeta digest if the device builds vbmeta image. The
digest will used later to determine the build fingerprint in new
format.
One sample usage is the ota package generation, where we put the
build fingerprint in the ota metadata. But we don't have the runtime
vbmeta digest provided the bootloader.
Bug: 186786987
Test: unit tests
Change-Id: If572e2b973e295a6c95a9e23a65bb20b3afbf1b0
Downgrade VABC OTA causes users to wait in recovery for merge to
complete, disable by default.
Test: th
Test: generate downgrade OTA, make sure VABC disabled
Test: generate upgrade OTA with --wipe_user_data, make sure VABC
disabled
Test: generate upgrade OTA, make sure VABC is used
Test: generate downgrade OTA with --vabc_downgrade, make sure VABC is
enabled
Bug: 187215486
Change-Id: Ib7e6165252d47f1ecaac4fc2329b580274c8d70e
* add .pylintrc to use 2 space indentation
* rename single-letter local variables
Test: ./warn.py build.log > warnings.html
Change-Id: I2ca56a6cb130a9d6c73328c5592ad7cde8a974ab
If source supports VABC, delta_generator/update_engine will attempt to
use VABC. This dangerous, as the target build won't have snapuserd to
serve I/O request when device boots. Therefore, disable VABC if source
build doesn't supports it.
Test: downgrade from VABC enabled build to a build w/o VABC
Change-Id: Ie8353e00f65354c2242ee5255b6652c6b62483a4
If dynamic partitioning is enabled and the partition size is not set,
we will get a KeyError before image_size or partition_size is calculated
when we try to catch exception in BuildImageMkfs.
Bug: 186704243
Test: build_image.py can correctly throw exception
Change-Id: I3d8c143ad5603d07fe94afb8bb911ead244f0bf7
Signed-off-by: Huang Jianan <huangjianan@oppo.com>
* Add missing function doc strings.
Suppress this warning on trivial functions in *_warn_patterns.py.
* Remove unused g-importing-memeber, g-complex-comprehension.
* Suppress pylint warning on unrecognized g-* options.
* Suppress too-few-public-methods warnings on simple classes.
* Suppress too-many-arguments and missing-function-docstring in
html_writer.py, which will be refactored later.
* Fix bad naming, long lines and line breaks, and bad quotes.
Test: compare output for build.log
Change-Id: Icdb34f014a10ec1e642c2cfe8003fc3ae245b507
