OTA tools should pick up the avbtool, as listed in dict['avb_avbtool'],
from the current PATH (plus bin/ under the dir specified via `--path`),
the same way as handling all other host tools.
Test: `m dist`
Change-Id: I3eb4d2c61979b03d9c23b2403d9a38cf052d87ea
Also makes AddSystem check that an output_zip exists before attempting
to add the recovery patch to the output zip.
Bug: 128838154
Test: Running merge_target_files with --rebuild_recovery and verifying
it passes --rebuild_recovery to add_img_to_target_files.
Change-Id: I19347b2c0dabf29b7196045b18551b5d0687df2c
The keys_info in the touched code is a tuple, which is immutable.
Bug: 123716522
Test: Run sign_target_files_apks.py with '-e foo.apex=bar' that replaces
the APEX container key.
Change-Id: I4e57e46c93a56b7f6646764d021ebb42c19bf7f5
Bug: 123716522
Test: Run sign_target_files_apks.py to sign a target_files with APEXes.
Test: Run check_target_files_signatures.py on signed artifact.
Test: python -m unittest test_sign_target_files_apks
Change-Id: I3fa13e3d9461cf5e0838e0572d436e218164fe41
Only the container certs will be checked and reported. For the payload
within an APEX, we can't easily extract the cert info.
It needs to go along a longer path, if ever needed, by:
- extracting public keys from all the available certs;
- using each of them to verify against an APEX payload to find a match
(`avbtool verify_image --image payload --key public_key`).
Bug: 123716522
Test: Run check_target_files_signatures.py on target_files with APEXes.
Change-Id: I2ef318e05433d2d65ab84e2dff9e01fb6ee3373d
Other modules have switched to logging module. sign_target_files_apks.py
needs to init the logger to get the logs.
Test: Run `sign_target_files_apks.py -v`. Check outputs.
Test: Run `check_target_files_signatures.py -v`.
Change-Id: Ic68c019f6fb14840561885f1194ad6efdfdb7d82
superimage-nodeps and supernod depends
on images from $(ANDROID_PRODUCT_OUT) (not from
target files package). It doesn't rebuild source
images if they are present.
A typical workflow is:
m -j
# change code in system
m snod -j
m supernod -j
Test: For non retrofit, run:
`m snod -j; m supernod -j`
Fixes: 128321505
Change-Id: Ib8c011cadb9c0cd334234aef39f19be6a48fee62
Bug: 124467065
Test: Running `python merge_target_files.py` using the three new flags
and observing that their contents are passed to the merge_target_files() function.
Change-Id: I4de46f041f5ae8bc8be2730313ce873a952bf78e
sign_target_files_apks script looks for the signapk.jar inside the out dir.
If the our dir is set to a different directory via OUT_DIR_COMMON_BASE the script does not work properly.
From now script checks if the OUT_DIR_COMMON_BASE is set, then searches the jar in the proper path.
If OUT_DIR_COMMON_BASE is unset, searches in "out" like it did before.
Test: Build with OUT_DIR_COMMON_BASE set and unset and verify that sign_target_files_apks works in both cases
Change-Id: I9218b98ff79526184f8353705640193405afac9e
For retrofit updates:
ab_partitions -= dynamic_partition_list
ab_partitions += super_block_devices
For example,
if super_block_devices == ["system", "vendor", "odm"] and
dynamic_partition_list == ["system", "vendor", "product"],
product partition needs to be removed from ab_partitions.txt
otherwise brillo_update_payload will generate a payload with
product partition in it.
Test: retrofit update that adds a partition
Fixes: 127425410
Change-Id: Id79a410cee3c611ac50d27f14282916aea34f938
This change adds another utility function to common.py: UnzipToDir, which is
generally useful. Refactor merge_target_files.py to use it, and also refactor
other uses in common.py to use it.
Test: ota_from_target_files.py, validate_target_files.py, test_common.py
Bug: 124464492
Change-Id: Ia571070bceb7d3c8002304836bdf688485bf0dd9
Then refactor the code in merge_target_files.py to adapt to this semantic
change. This makes the code more consistent with existing releasetools code,
and it's easier to follow.
Test: Failure cases (verify exception), success cases (merged target generated)
Bug: 124521133
Change-Id: I56f04e360d8ff8ffcd6245359cdeb79f4565a9c4
The merge_target_files.py script needs fc_sort and sefcontext_compile, so
include these tools into otatools.zip via core/Makefile.
Modify tools/releasetools/merge_target_files.py to use the otatools common argv
processing to take advantage of the '--path' option so that we add point the
'--path' option to an extracted otatools.zip package to gain access to fc_sort,
sefcontext_compile, and soong_zip (previously included in otatools.zip).
Bug: 123600124
Test: extract otatools.zip, use --path option to point to it, verify result
Change-Id: I7d84525981b8741c6bdbcac9984256920fc7f417
The validate_target_files.py checks the 'incomplete' field of the range
in file_map. And range has already considered the shared blocks and
could be smaller in size than the original file range. Therefore, the
'incomplete' flag was set on the original range in common.py; and we
should switch to use the original range also during validation.
I also checked another flag usage in CanUseImgdiff(), and it has
explicitly rejected cases of shared blocks.
Bug: 124868891
Test: unit tests pass
Change-Id: I03959625d7b81fd83420db98f01d23f54064bcd2
Add a flag to write a copy of the metadata to a separate file. Therefore,
users can read the post build fingerprint without extracting the OTA package.
Bug: 124783265
Test: Check the dumped metadata file after generate A/B and non-A/B OTA.
Change-Id: I8918aec87bb81906ef0a7eee774178e9f689d91d
This serves as a workaround to skip the compatibility check for devices
with incompatible kernels.
Bug: 114240221
Test: generate and check the OTA package for wear device
Change-Id: I65b523a66648af7a77fc3ea79176764fe8ae8d02
This script takes as input two partial target files (one contains system bits,
and the other contains non-system, or other, bits). The script merges the
contents of the two partial target files packages to produce a complete target
files package.
Bug: 123430711
Test: Build two partial target files, merge, compare with full target files.
Test: Validate merged target files via validate_target_files.py.
Change-Id: Ic24acf43b86fc703fb4c970688b006291a1861f8
tools/releasetools/add_img_to_target_files.py: This patch excludes the
inclusion of the system path into the target files if it does not exist (which
it will not if we are not building the system image).
Bug: 123430711
Test: Disable building system image, verify that target files builds without system.
Change-Id: Iaf964ede2b1df5ea4e004b572fd91187a366e75e
This commit extracts the AVB key used to sign system_other.img into
system.img, for init to verify system_other's AVB metadata.
The extracted key will locate in:
/system/etc/security/avb/system_other.avbpubkey
Bug: 123611926
Test: build and checks the following is generated
$OUT/system/etc/security/avb/system_other.avbpubkey
Change-Id: Icdc703ff5a0d50f8140bb652507b9b4cbc8a2118
This change allows ro.product.[brand|device|manufacturer|model|name] and
ro.build.fingerprint to be derived at boot time (and in the OTA
generation scripts) from partition-specific properties.
Test: booted system image, verified properties
Test: booted recovery image, verified properties
Test: unpacked OTA package, verified build fingerprint
Bug: 120123525
Change-Id: Iadd230a0577f35c7c37b0f911e91a5c2863ed1fe
For dynamic partitions in retrofit devices, system partition will be
a logical partition but system_other is not. However, current
build system use the same settings (logical) for both system.img and
system_other.img, leading AVB unable to locate the footer from the end
of system_other partition.
This commit support building system_other.img with correct partition size
while building system.img as a dynamic image.
Bug: 123506156
Test: check there is "system_other_size=2952790016" in file
$OUT/obj/PACKAGING/system_other_intermediates/system_other_image_info.txt
Test: build system_other.img, then
`simg2img $OUT/system_other.img system_other.img.raw`, checks the
raw image size.
Change-Id: I748320a7770c694d06f06f4a35bfceb622849aa8
- Add hashtree_info to EmptyImage so that BlockDifference.Compute()
can accept EmptyImage() as target image, which is the case when
a partition is removed.
- BlockDifference also checks source_info_dict to determine
whether a partition is dynamic. When a partition is removed,
its name does not appear in target_info_dict.
- Add tests to ensure DynamicPartitionDifference() still works.
Test: DynamicPartitionDifferenceTest
Test: test_blockimgdiff
Change-Id: Iadb1db075f5dc344db6d5ade358c83b01231e443
This is not used by anyone and the other half of the code to compare
against it is already functionally dead, so remove this.
Test: build
Change-Id: I44ed087cb7735bbc23e30b6c310c80eb3b7b6488
Bug: 111136242
Test: When BOARD_PREBUILT_DTB_DIR is set correctly,
generated $OUT/boot.img contains the DTB image.
Change-Id: I282e31b04cc60383377b9e9b54f8fe64a8140242
Support signing system_other.img but shouldn't include it into the
top-level vbmeta.img. system_other verifiation will not be included
in /vbmeta chains and will be done separately.
Bug: 112103720
Test: avbtool info_image --image $OUT/system_other.img
Test: avbtool info_image --image $OUT/vbmeta.img, checks 'system_other' is NOT included.
Test: Checks $OUT/obj/PACKAGING/system_other_intermediates/system_other_image_info.txt
See the following:
avb_system_other_hashtree_enable=true
avb_system_other_add_hashtree_footer_args=--rollback_index 1551744000
avb_system_other_key_path=external/avb/test/data/testkey_rsa4096.pem
avb_system_other_algorithm=SHA256_RSA4096
Change-Id: Ia152aaab1387dcf556a42222adb39ea76881263a
- When removing a partition, BlockDifference() object
will have tgt=EmptyImage(). Fix the asserts accordingly.
Also, BlockDifference object now allow tgt=None case.
- When adding a partition, BlockDifference() object
will have src=None. Fix the asserts accordingly.
Also, add unit tests to DynamicPartitionsDifference.
Test: create incremental OTA
Test: test_common.DynamicPartitionsDifferenceTest
Bug: 111801737
Change-Id: I3a35378ecf93111b8f44545cff6ae9696b6b4851
We cannot simultaneously stash more blocks than the size limit imposed by
the cache size. As a result, some 'diff' commands will be inevitably
converted to new. We used to do this conversion blindly when iterating
through the transfer list. This leads to an unintended large package.
In order to choose the right transfers to convert, we calculate the size
of the compressed data, and build a heuristic about the package size
increase to remove each stash blocks. After the process, the given
package size for the watch device further reduces from 186M->155M.
In some rare cases, the removed stashed blocks don't directly contribute
to the maximum simultaneously stashed size. For example,
stash A: 10 blocks
stash B: 5 blocks
free B: 5 blocks <-- stash B has been freed before we reach max stashed blocks
stash C: 10 blocks
Converting these blocks lead to an uncertain result. On one hand, patches
are generally smaller than the new data; while on the other hand, the
regenerated graph may have fewer order violation and thus give some size
reduction. But these cases are rare and it seems an overkill to consider all
possible scenarios here.
Bug: 120561199
Test: build non-A/B incrementals and check the size
(p.s. it can be tested on all target files with customed cache threshold)
Change-Id: I599420a91b80f1a1d83d22ee1b336b699050cfb4
To address problems creating real tiny filesystem placeholders,
make sure extra spare inodes margin is greater than 0.
For initial estimate we add 4% of total we request at least 8.
For second pass estimate we add 0.2% of total we request at least 1.
We bumped up the margin for zie on the second pass to 0.3% as the
value was too close on one of the builds.
Test: build
Bug: 122328872
Change-Id: I41707bb6fcc8bbfbdda143a9ce62446cad9c1533
Test: sideload full OTA on cuttlefish
Test: sideload incremental OTA on cuttlefish (that grows
system, shrinks vendor, and move vendor to group foo)
Bug: 111801737
Change-Id: Ie8a267a90b4df9e9e0a2fbcc1b582ab2e353df52
We used to create the intermediate output file as a tempfile, when
adding images to a given zip file. This CL changes OutputFile to write
intermediate files under the given input dir (i.e. OPTIONS.input_tmp
that holds the unzipped files), if the final destination is a zip
archive. This allows image building codes finding the generated images
at consistent locations. The intermediate files will be cleaned up as
part of OPTIONS.input_tmp.
Bug: 122608028
Test: `m dist`
Test: Delete OTA/super*.img from a target_files.zip. Run
`add_img_to_target_files.py -a` that re-generates split super
images.
Change-Id: I97903a59fcc0ca5e43bb9b07c3a0b25e9baa92f9
Bug: 122608028
Test: Run sign_target_files_apks.py on a target-files zip that has split
super images (e.g. OTA/super_system.img).
Change-Id: Iaf7263790961a897ea3f339f5af6b18cf253b946
Allow right-size to support images that are not sparse.
This is in support of cuttlefish.
Test: build
Bug: 122328872
Change-Id: Ic8ef968e750203dffde7044bc6dfc71c1e283158
To address flakiness in image creation, add a .2% margin of safety
for both inodes and space for ext filesystems.
Test: build
Bug: 122328872
Change-Id: I6665bd6fe642291b825dba58dfd09cc55628230f
inode-size will jump around based on filesystem size, however
readonly Android system partitions have a specific pattern of
xattr associated with sepolicy and 256 is the most efficient at
absorbing the content.
Test: manual
Bug: 122328872
Change-Id: I06dd6a503067ab6477293b386d56a89dd86b0e83
We used to build retrofit full OTA package as long as
PRODUCT_RETROFIT_DYNAMIC_PARTITIONS is true. This doesn't work with AOSP
targets that have the flag set but without any available vendor image.
This CL detects such a condition and uses a separate flag to guard the
retrofit OTA building as well as the split super images generation.
Bug: 120852744
Test: `m -j dist` on blueline (w/ vendor images) and aosp_blueline (w/o
vendor images).
Change-Id: I65726f24f8fc546be6802941a6a06590a3804c16
Package size will be unintended large if we stash more blocks than the
stash limit specified by the cache size. To reduce the maximum size of
simultaneous stashed blocks, we will inevitably convert some 'diff'
commands to 'new' commands.
To mitigate the impact, we add a new function to smartly select the
transfers to convert based on their patch size and compressed size.
This cl converts the transfers that have a larger patch size than the
compressed target sizes. And there's a slightly improvement in the
final package size: from 194M -> 185M.
Bug: 120561199
Test: build a non-A/B incremental package, run simulator
Change-Id: Id73ff736ba4e6901d245ad5549d42310d0740284
The current increase of 4 % is insufficient for some
targets.
Test: mmm
Bug: 119115481
Change-Id: Idcba8025b913da9b70794bfc7464d15b4d99ad34
(cherry picked from commit 3e02e34090)
We will call it at an earlier time to compute the patch size; and
choose the transfers to convert to 'new'.
Bug: 120561199
Test: Generate an incremental update on shiner
Change-Id: I29a0c8e75c9e5b66a266c1387186692a86fcbe43
For some internal branches, vendor.img isn't built, so there
is no need to build super image / super split images there.
Test: remove vendor.img and VENDOR/ from target_files_intermediates, then
run add_img_to_target_files
Fixes: 120634805
Change-Id: I2834a27ce232538f203733c204dd257279c789fc
Setting 'uses_shared_blocks' indicates that the block list for a given
file is incomplete, as some of the blocks are "owned" by others. As a
result, such a file will be skipped from imgdiff'ing.
This CL makes a copy of the original block list before removing the
shared blocks. It uses the original RangeSet as the value of
extra['uses_shared_blocks']. validate_target_files.py will try to read
the file as in the original RangeSet, unless the original list is also
incomplete or has non-monotonic ranges.
Test: Run validate_target_files on a target that uses
`BOARD_EXT4_SHARE_DUP_BLOCKS := true`.
Change-Id: I259e871ecc249ba0c14b5796bef413185a1b8242
Use the new script to build super_empty and super split images. No more
transformation to lpmake_args.
Test: build target_files_package for retrofit device
Change-Id: Id5f6bd607654ca869bcdf58d86b7ae300e3927eb
For non-retrofit (launch) devices, super.img is used for factory, so
source images should be from target_files.
In this change, build-superimage-target procedure is converted to a
more flexible script so that it can be built.
Bug: 119322123
Test: build target files for device launch with dynamic partitions
Change-Id: I6ee0cc3e145357dfc74be248f81f5f8f4e51fc5c
On aosp_blueline, system_other gets right-sized to 61,440 bytes. This
messes up the AVB footer calculation, and is too small to store an ext4
image.
Bug: 116608795
Test: build aosp_blueline w/ dynamic partitions
Change-Id: If0138a9d858765ff82cb957532504a066fd113ff
The map files contain the filename-block mapping for the original
images.
Bug: 120109184
Test: Generate an incremental OTA that was previously failing.
Change-Id: I1285d0b24a435477b958e6c2e4f474acf8f7578b
NB: second attempt, issue with scope for mkfs_output.
Separate out BuildImageMkfs from BuildImage, which just makes the
filesystem without any verity, avb, or other decisions. BuildImage
does all the wrapping for such. This will hopefully ease maintenance
and drop the issues surrounding BuildImage reentrancy.
Change right-size estimation path to use BuildImageMkfs, and thus do
so without verity or avb wrappings. Add partition_headroom to space
consideration. This makes the results of the estimation more
accurate and predictable.
Test: build
Bug: 111302946
Change-Id: I2549bd4e403c21290470b2fa1835492ae883f0fd
This reverts commit 2b72b7f01e.
out/target/product/generic_x86/installed-files.txt \${DIST_DIR}/installed-files-rescued.txt; exit 1 ) )"
2018-11-08 19:12:28 - build_image.py - ERROR : Failed to build out/target/product/generic_x86/obj/PACKAGING/systemimage_intermediates/system.img from out/target/product/generic_x86/system
Traceback (most recent call last):
File "build/make/tools/releasetools/build_image.py", line 767, in <module>
main(sys.argv[1:])
File "build/make/tools/releasetools/build_image.py", line 755, in main
BuildImage(in_dir, image_properties, out_file, target_out)
File "build/make/tools/releasetools/build_image.py", line 449, in BuildImage
CheckHeadroom(mkfs_output, prop_dict)
NameError: global name 'mkfs_output' is not defined
Bug: 111302946
Reason for revert: Build problem
Change-Id: I786f232e07af653a7207509055df5a07a6d8bb9e
Separate out BuildImageMkfs from BuildImage, which just makes the
filesystem without any verity, avb, or other decisions. BuildImage
does all the wrapping for such. This will hopefully ease maintenance
and drop the issues surrounding BuildImage reentrancy. Change
right-size estimation path to use BuildImageMkfs, and do so without
verity or avb wrappings.
Test: build
Bug: 111302946
Change-Id: I30e2e2b727f40ecca5164142f34139f5244f6424
Neglected to correct for filesystem size in estimate for verity.
To keep build space optimized and make sure that the existing image
does not confuse the tools, remove the out_file after the test build
image phase.
Test: build
Bug: 111302946
Change-Id: I360775d862d02a721d061bbc43e8b4e2f03bd01e
This is a rollback of Id5ae6645b9aa2d036e6fefe2fb17672e8f8ef6f0
and the followup I6a0f3919e8d45d0b3e7cd558a6ad4d3799012f2c
The magic constant introduced appears to be too aggressive.
Note that the new constant in I0f32c75e80b5da4d47671055ac274ccc2b485992
doesn't appear to fix the broken builds.
Bug: 111302946
Test: make
Change-Id: I29207265172d293c6fee98212e3266bde85895df
Regression introduced by c3d408e3c1
Found a case where 1 + 1/60 reduction in size based on free space
failed when right sizing an ext4 image. Reduced the math to 1 + 1/61.
Test: compile
Bug: 111302946
Change-Id: I0f32c75e80b5da4d47671055ac274ccc2b485992
In Append2Simg and Append in verity_utils.py, we catch and rethrow
exceptions as BuildVerityImageError. However, this suppresses the
traceback from the original exception which usually indicates the actual
cause. We can better handle this with the raise statement in Python 3,
which is however unavailable in Python 2.
This CL logs the exception before rethrowing to retain the useful bits.
Test: Inject an error to append2simg. `m -j systemimage` with
aosp_marlin-userdebug. Check the output.
Change-Id: I0c2f57d6023fa1038256b85fa98d57ad0244a70d
Regression introduced by c3d408e3c1
Found a case where 1 + 1/59 reduction in size based on free space
failed when right sizing an ext4 image. Reduced the math to 1 + 1/60.
Test: compile
Bug: 111302946
Change-Id: I6a0f3919e8d45d0b3e7cd558a6ad4d3799012f2c
Add ext4 dedupe to system_other, product, product_services, odm and
oem images. Experimental savings without any other configuration
changes was 0%, .75%, 3.4% respectively for the raw image file size
for the first three.
Test: manual
Bug: 111302946
Change-Id: Ia8fb5696151acad59bb144ea93f2c2ddac962bbd
To more quickly settle on a no free space result, recognize that
there is roughly 1/58.5 overhead managing the free space pool as
determined experimentally. This algorithm carries with it some
risk of being sensitive to any alterations in ext4.
This also addresses the issue of a much larger apparent free space
available calculated on the sample pass when deploying ext4 dedupe,
and resolves the poorer estimation that happens without accounting
for the overhead.
The alternative of adding a second pass works to a similar, and
likely more comforting algorithm and result. But doing so adds
a minute to the typical incremental build time, or +50% to the
time it takes to determine ext4 right sized images.
Test: manual
Bug: 111302946
Change-Id: Id5ae6645b9aa2d036e6fefe2fb17672e8f8ef6f0
This avoids showing a message as below:
raise BuildImageError("Failed to get tune2fs usage:\n{}".format(output))
UnboundLocalError: local variable 'output' referenced before assignment
The `output` won't be set under such a case, which unfortunately hides
the real cause. This CL addresses the issue by letting it throw
implicitly, which contains the command and outputs.
Test: `m dist`
Test: `python -m unittest test_build_image`
Change-Id: Icf015bab8869d150516246ed73552f6502127cdf
If partition_reserved_size is 0 or undefined, and
use_dynamic_partition_size is true, we should approach no space
and no free inodes automatically.
Estimate the space and number of inodes required, then do a first
pass build to see how much space actually used, and use those values
to refine the estimate.
Depends on tune2fs to report the characteristics of the filesystem,
so only support for ext filesystems. In the future if there has to
be a more generic ability, either a tool per a filesystem has to be
found, or we will need root capabilities to mount the filesystem to
acquire the characteristics live from the host system.
Test: manual + python -m unittest test_build_image
Bug: 111302946
Change-Id: I933a388be43516b6de7b5007b296765bd5556fde
Chained vbmeta images should be included into the top-level vbmeta.img
as chained partitions. It's done in the path of `m`, but not `m dist`.
Bug: 118115607
Test: `m dist` a target that uses chained vbmeta.
a) Check that the `vbmeta.img` in target-img.zip contains the image
descriptor of `vbmeta_system.img` (`avbtool info_image`).
b) `avbtool verify_image --image vbmeta.img \
--expected_chain_partition \
vbmeta_system:1:/path/to/vbmeta_system/pubkey`
c) `avbtool verify_image --image vbmeta_system.img`
Change-Id: I064e583b247c44b9b2f19355838550bb5dbb8f26
Recovery can now parse the pem encoded x509 keys from a zipfile. So
instead of dumping the keys into a text file with some intermediate format,
we can simply create a zipfile with the keys.
Bug: 116655889
Test: make bootimage and check the generated zipfile, run sign_target_files_apks
Change-Id: Ib76feecfb26d6be713a07644e80ec96133759004
This sets up the root logger while running tests. It also logs to
stdout, as our Python test result parser doesn't like outputs from
stderr.
Test: `python -m unittest discover build/make/tools/releasetools \
> /dev/null` doesn't print log lines.
Change-Id: Ic99711bd458bc4b67b38226786fed604c2168476
Converts the following files to Python logging.
add_img_to_target_files.py
blockimgdiff.py
build_image.py
check_ota_package_signature.py
common.py
img_from_target_files.py
make_recovery_patch.py
ota_from_target_files.py
sparse_img.py
verity_utils.py
This separates logging outputs from normal outputs, and allows easier
switching between different verbosity levels. It also supports adjusting
logging config via environment variable (LOGGING_CONFIG).
Test: `m dist`
Test: `python -m unittest discover build/make/tools/releasetools`
Change-Id: Idfc3f7aff83e03f41f3c9b5067d64fd595d5192d
common.RunAndCheckOutput() checks the exit code and will raise on errors.
Test: python -m unittest test_ota_from_target_files
Test: Call ota_from_target_files.py with aosp_marlin target_files zip
Test: Inject an error to the command. Repeat the call without verbose
flag. It dumps the stack trace on error.
Change-Id: I85b765a33b9087bcbcb0571d6e632a07bb86c65c
This prepares for upcoming changes that refactor verity-related
functions into classes.
This CL makes minimal changes to the moved functions, by creating a new
BuildVerityImageError class in verity_utils.py replacing the former
BuildImageError that's specific to build_image.py. As part of the
change, it also moves the tests for AVBCalcMinPartitionSize into
test_verity_utils.py.
Test: python -m unittest test_verity_utils
Test: `m dist` with aosp_marlin-userdebug (Verified Boot 1.0)
Test: `m dist` with aosp_taimen-userdebug (Verified Boot 2.0)
Change-Id: I19b52714d8980705ea1f9484ac03eb0af9483240
All the releasetools unittests extend the common base class of
test_utils.ReleaseToolsTestCase. Define tearDown() in the base class to
do the clean-up works.
Test: `pylint --rcfile=pylintrc test_*.py`
Test: `python -m unittest discover .`
Change-Id: I51775d964ef032dcdf3bb89c55e1a31371cde708
There's a naming convention between the testcase and the class/method
being tested.
Also remove VerifiedBootVersion2HashtreeInfoGenerator as it's merely a
placeholder without telling anything much (e.g. whether it's to be
implemented or working as intended).
Test: python -m unittest test_verity_utils
Change-Id: Ieb156765c865aa551e3882f381e3a6db2cac4cbc
build_image.RunCommand and common.Run are essentially doing the same
work, but with different interfaces and return values. This CL
consolidates them by moving build_image.RunCommand into common.py as
common.RunAndCheckOutput, with the former calling common.Run internally.
common.RunAndCheckOutput checks the exit code: it returns the output
string on success, or raises common.ExternalError otherwise.
Test: `m dist` with aosp_marlin-userdebug
Test: python -m unittest test_build_image
Test: python -m unittest test_common
Change-Id: I3cd0e9301c576572194618eba8dc00db06c3c35f
stdout and stderr will default to subprocess.PIPE and subprocess.STDOUT
respectively (which is the expected behavior from most of the existing
callers), unless caller specifies any of them.
Test: `m dist`
Test: python -m unittest \
test_common \
test_add_img_to_target_files \
test_ota_from_target_files \
test_validate_target_files
Change-Id: I43b3f08edfa8a9bcfe54baf9848dc705c048e327
We don't need verity_root_hash or verity_salt to be in the prop dict.
Test: `m dist` with aosp_marlin-userdebug
Test: python -m unittest test_verity_utils
Change-Id: I5a9c50f4741dfb1083b3f590136335b6bc0e5216
Generate the transfer command "compute_hash_tree" for incremental
updates of the non-A/B devices that enable verified boot 1.0
Other changes include:
i. factor out verity_utils to use both in build_image and blockimgdiff
ii. add a new flag 'hashtree_info_generator' in sparse_image to generate
the hashtree information.
Bug: 25170618
Test: generate a package for aosp_angler; and run simulator
Change-Id: I4d4d7a3e41dc3da810d2cbf8988e85d187c9ab0e
The image building functions in build_image.py have been returning
(success, result) or special values to indicate the validity of the
result. The same logic can be better expressed by raising exceptions
instead, because
a) using a special value relies on caller to check for that magic
value;
b) exceptions can carry additional messages other than a boolean does,
e.g. the output from the failing command;
c) caller can have cleaner code flow without explicitly checking for
the validity of the result.
This CL changes such functions to raise on errors. The majority of these
functions are internal to build_image.py only, except for BuildImage()
that has a few callers in add_img_to_target_files.py (which all die upon
error anyway).
Test: `m dist`
Test: python -m unittest test_build_image
Test: python -m unittest test_add_img_to_target_files
Test: python -m unittest test_validate_target_files
Test: Inject an error to the depended binaries (e.g. avbtool), and check
that build_image.py exits with error messages.
Change-Id: Ibe4d51e267756bb1a00fa9238a213f9d55fd9b58
When AVB is enabled with PRODUCT_USE_DYNAMIC_PARTITION_SIZE, AVB
metadata (e.g., hash tree, fec metadata) will consume _RESERVED_SIZE,
resulting in smaller reserved size in file system (e.g., ext4).
Adding additional space for AVB signing and keep the _RESERVED_SIZE only
for file system. This is done by adding a function to binary search an
optimal partition size for a given image size (disk usage + _RESERVED_SIZE).
Bug: 112322265
Test: Build aosp_arm64-userdebug, calculate the running time of
AVBCalcMinPartitionSize() is about 0.3-0.4 seconds.
Test: python -m unittest test_build_image
Change-Id: I8f0051b57701d6fbba6a9db3756dd194066c74b8
Unless using dynamic partitions, `partition_size` should be a fixed
value that equals to the partition size in BoardConfig. It should stay
the same when building an image for that partition. Only the actual
image size that's used to hold the filesystem could be adjusted. This CL
cleans up the uses of `partition_size` and `image_size` to better
reflect such logic.
With dynamic partitions, the only thing that changes is the need to
compute `partition_size` upfront. Once that's done, `partition_size`
should remain unchanged.
Test: `m dist`
Test: `python -m unittest test_add_img_to_target_files`
Test: `python -m unittest test_validate_target_files`
Change-Id: Idedb3e018c95e8f63dc4d9c423be27f30ebb584f
Reads the fingerprint of a partition from the build props and embeds it
into the care_map. Later, the update_verifier will compare the
fingerprint before it performs the verification.
Bug: 114778109
Test: generate care_map for sailfish and new pixel
Change-Id: I06afa01bf6dd3de4456a08ee7960954facc775fb
The recovery image already contains a copy of first stage init, so we
can boot unconditionally to the recovery image and instruct first
stage init whether or not to boot to Android or to recovery. In this
case, we need neither the kernel to mount /system as / nor a separate
partition for recovery, so this change modifies the build scripts to
allow this combination.
Bug: 114062208
Test: Boot pixel from recovery to Android with BOARD_USES_RECOVERY_AS_BOOT
But without BOARD_BUILD_SYSTEM_ROOT_IMAGE
Change-Id: Icd047afb7f22d2724b3bcaca1aa0c837426dcce7
The new suffix distinguishes the new care_map from the ones in plain
text format; and thus the old update_verifier won't report an error
upon parsing failures.
Bug: 115740187
Test: Generate OTA files for Pixels
Change-Id: Ia782afd8cbb0f4bb8c363edaa00e92ab302d5d1b
Bug: 112007947
Test: `m dist` with aosp_taimen-userdebug.
Test: Set up a target that uses chained VBMeta images of
`vbmeta_mainline` and `vbmeta_vendor`. `m dist` and check the
build log, as well as outputs from
`avbtool info_image --image vbmeta.img`,
`avbtool info_image --image vbmeta_mainline.img`,
`avbtool info_image --image vbmeta_vendor.img`.
Change-Id: Ib1d4e97f583b65245703eae15d211adcd9e83741
Also removing the requirement of BOARD_BOOTIMAGE_PARTITION_SIZE when
BOARD_AVB_ENABLE is set. Some targets (e.g., emulator) doesn't build
boot.img when AVB is enabled.
Bug: 112293933
Test: build a failure case and checks the error output is expected
Change-Id: I290c707719193ddaedcd9fd0b31de566ad17078c
This is in line with the change of mkuserimg_mke2fs.sh to python binary.
Bug: 112555072
Bug: 63866463
Test: unittests pass
Change-Id: I82c0be1e5bbc685edc15120da73aa43fdc9f2f05
Commit d572632f3d added support in build
system to blacklist given vendor properties. When
ro.vendor.build.fingerprint gets blacklisted, the OTA generation script
can't rely on those properties to determine a change in vendor images.
This CL considers such a case as "vendor images must have changed"
between the two builds.
Bug: 113892939
Test: Generate an incremental package with builds not having
ro.vendor.build.fingerprint.
Test: python -m unittest test_ota_from_target_files
Change-Id: I188de9c3cbeecf26132c92b9356e9d5fef75205e
super_empty.img is needed for "fastboot update" to work, as such, it
must be included in the update package. This change adds lpmake
parameters to misc_info.txt for add_img_to_target_files.py, and ensures
that lpmake is packaged with otatools.zip.
The build-superimage-target macro is now split into two functions - one
to generate the device-specific arguments, and another to build the full
command-line for convenience. The former is used to pass arguments
directly to add_img_to_target_files.
Bug: 113524256
Test: make updatepackage builds and includes super_empty.img
make otatools includes lpmake
Change-Id: I25091c964b036beeea2a8b8f738e2c18937c1eb4
It used to accept a non-None input_dir to indicate the need to
re-generate images (called from add_img_to_target_files.py). During that
flow, both of the two parameters redundantly refer to the same input
dir. This CL replaces the second parameter with a bool value instead.
Test: `python -m unittest test_common`
Test: `m dist` with aosp_taimen-userdebug
Test: `zip -d aosp_taimen-target_files-eng.zip IMAGES/\* &&
add_img_to_target_files.py -a aosp_taimen-target_files-eng.zip`
Change-Id: I0a5a164366acb116407f94bb350872a3a0b207d1
In particular, it replaces the generated calls to edify function of
`apply_patch()` and `apply_patch_check()` with `patch_partition()` and
`patch_partition_check()` instead.
It adds two functions to EdifyGenerator: PatchPartition() and
PatchPartitionCheck() for Python releasetools callers. It also tries to
handle the callers of obsolete EdifyGenerator.PatchCheck() and
EdifyGenerator.ApplyPatch(), if their inputs are in known format.
Otherwise it raises an exception to avoid generating an OTA that updater
doesn't understand.
It requires the matching updater changes in the same topic.
Bug: 110106408
Test: Generate an incremental package with the new script and updater.
Apply the package on device.
Test: Revert the change in ota_from_target_files.py, so that it calls
the obsolete EdifyGenerator functions. Check that it generates the
same incremental package.
Change-Id: Ifc55cb40d3a45116fd4d408536d037eca249effa
Support a first stage ramdisk which will include the targets placed
into TARGET_RAMDISK_OUT. This replaces the existing ramdisk on
existing devices.
All system images are now built to be mounted as the root dir.
Devices with a first stage ramdisk will switch root to the system
partition.
BOARD_BUILD_SYSTEM_ROOT_IMAGE remains and is used to specify if the
system partition is going to be directly used as rootfs without the
ramdisk.
Bug: 79173823
Bug: 79758715
Test: hikey boots, sailfish boots
Test: OTA walleye from P to master
Change-Id: Idbb2dccc6340b0235a4bef03e11e420a9ed154b6
Otherwise it gives a floating-point value (i.e. 1230768000.0). e2fsdroid
(i.e. the user of the value) didn't complain though, because it calls
strtoul(3) without checking endptr.
Bug: 80600931
Test: `m dist` with aosp_taimen-userdebug, check the build log.
Change-Id: I7467a60c947134b42f079c7a57fb8a9a9d4277da
The recovery image will be packed under BOOT/RAMDISK only if
system_root_image and recovery_as_boot both are true (e.g. non-A/B
devices launched since P).
Bug: 113191245
Test: Run sign_target_files_apks.py on a target_files file that uses
system-as-root but not recovery-as-boot.
Change-Id: I262a268055c6b5078d21694b5094a1c393d0d37c
The change in [1] moved the recovery etc files from /etc to /system/etc.
However, we may use the latest OTA tools to build incremental OTAs for
old target_files zips. This CL adds a workaround to look at both of the
old and new locations.
[1] commit 696bb33676
Bug: 113073663
Test: Build a previously failing incremental OTA for taimen.
Change-Id: Ie07aa1713e616d523838b1260a992b20f5a11612
When the update package gets larger than 2 GiB, payload.bin offset
mentioned in metadata file for ota-streaming-property-files gets
shifted (CrAU of payload.bin) because ZipInfo FileHeader() returns
incorrect value. To solve the issue, offset is re-calculated from
fixed bytes of central directory file header, filename length and
extra length.
This patch is to sync with update_device.py script.
Test: manually create an A/B update package and run it using
update_device.py
Bug: 111198589
Change-Id: I9bf5a5ca24938cad3206d04af529f70d45e992c0
They used to be installed under recovery/root/etc. This CL moves the
files to the new location and creates a symlink from /etc to /system/etc
(done by the rule in system/core/rootdir). This gives similar layout
between normal boot and recovery, and allows installing prebuilt_etc
files with Soong (`recovery_available: true`).
As part of the change, we no longer need the whitelisting rule for
mke2fs.conf.
Bug: 112780007
Test: Build with other changes in the topic (aosp_taimen-userdebug).
Check the generated files under recovery (/etc being a symlink to
/system/etc).
Test: Boot into recovery. Verify basic functionalities (`adb shell` and
`adb sideload`, factory reset).
Test: `build/soong/build_test.bash --dist`
Change-Id: Ibb6dea6f179a339f0c2d0fd8ba05ec0085b79a12
Attempting to reduce the number of different spellings we have for
"product services" partition in the codebase.
Bug: 112431447
Test: m
Change-Id: I6debd3f5f5abc4bd1e22f2ef11c5fe131915976c
Merged-In: I6debd3f5f5abc4bd1e22f2ef11c5fe131915976c
