Commit graph

1768 commits

Author SHA1 Message Date
Tianjie Xu
ef7737f921 Merge "Check the completeness of original blocks during target files validation" 2019-02-21 03:41:35 +00:00
Bill Peckham
f753e15791 Include merge_target_files.py deps in otatools.zip, use common argv processing.
The merge_target_files.py script needs fc_sort and sefcontext_compile, so
include these tools into otatools.zip via core/Makefile.

Modify tools/releasetools/merge_target_files.py to use the otatools common argv
processing to take advantage of the '--path' option so that we add point the
'--path' option to an extracted otatools.zip package to gain access to fc_sort,
sefcontext_compile, and soong_zip (previously included in otatools.zip).

Bug: 123600124
Test: extract otatools.zip, use --path option to point to it, verify result
Change-Id: I7d84525981b8741c6bdbcac9984256920fc7f417
2019-02-21 00:50:54 +00:00
xunchang
c0f77ee489 Check the completeness of original blocks during target files validation
The validate_target_files.py checks the 'incomplete' field of the range
in file_map. And range has already considered the shared blocks and
could be smaller in size than the original file range. Therefore, the
'incomplete' flag was set on the original range in common.py; and we
should switch to use the original range also during validation.

I also checked another flag usage in CanUseImgdiff(), and it has
explicitly rejected cases of shared blocks.

Bug: 124868891
Test: unit tests pass
Change-Id: I03959625d7b81fd83420db98f01d23f54064bcd2
2019-02-20 15:47:30 -08:00
xunchang
1cfe25155e OTA: Add an option to dump the metadata in ota package
Add a flag to write a copy of the metadata to a separate file. Therefore,
users can read the post build fingerprint without extracting the OTA package.

Bug: 124783265
Test: Check the dumped metadata file after generate A/B and non-A/B OTA.
Change-Id: I8918aec87bb81906ef0a7eee774178e9f689d91d
2019-02-20 15:32:09 -08:00
xunchang
abfa265f81 Add a flag to skip compatibility check
This serves as a workaround to skip the compatibility check for devices
with incompatible kernels.

Bug: 114240221
Test: generate and check the OTA package for wear device
Change-Id: I65b523a66648af7a77fc3ea79176764fe8ae8d02
2019-02-19 16:27:10 -08:00
Treehugger Robot
01a05483ed Merge "Support verifying system_other" 2019-02-18 05:21:27 +00:00
Bill Peckham
e9eb5f96e0 Adding initial merge_target_files.py script to merge two target files packages.
This script takes as input two partial target files (one contains system bits,
and the other contains non-system, or other, bits). The script merges the
contents of the two partial target files packages to produce a complete target
files package.

Bug: 123430711
Test: Build two partial target files, merge, compare with full target files.
Test: Validate merged target files via validate_target_files.py.
Change-Id: Ic24acf43b86fc703fb4c970688b006291a1861f8
2019-02-15 13:25:16 -08:00
Bill Peckham
cc57de3256 Skip some build steps if not building system image.
tools/releasetools/add_img_to_target_files.py: This patch excludes the
inclusion of the system path into the target files if it does not exist (which
it will not if we are not building the system image).

Bug: 123430711
Test: Disable building system image, verify that target files builds without system.
Change-Id: Iaf964ede2b1df5ea4e004b572fd91187a366e75e
2019-02-13 22:57:14 +00:00
Tianjie Xu
b8ac90803c Merge "Code improvement to adjusted the blocks size for care_map file" 2019-02-13 19:49:48 +00:00
Shashikant Baviskar
16a73897c3 Code improvement to adjusted the blocks size for care_map file
Bug: 123931528
Test: python -m unittest test_add_img_to_target_files
Change-Id: I589bc9681bccfa07588cab94f53f69d637d6f0f9
2019-02-13 18:34:50 +00:00
Bowgo Tsai
45db7cefb3 Support verifying system_other
This commit extracts the AVB key used to sign system_other.img into
system.img, for init to verify system_other's AVB metadata.

The extracted key will locate in:
    /system/etc/security/avb/system_other.avbpubkey

Bug: 123611926
Test: build and checks the following is generated
      $OUT/system/etc/security/avb/system_other.avbpubkey

Change-Id: Icdc703ff5a0d50f8140bb652507b9b4cbc8a2118
2019-02-13 17:00:49 +08:00
Steven Laver
9e73e82a12 Allow properties to be derived from partition-specific properties
This change allows ro.product.[brand|device|manufacturer|model|name] and
ro.build.fingerprint to be derived at boot time (and in the OTA
generation scripts) from partition-specific properties.

Test: booted system image, verified properties
Test: booted recovery image, verified properties
Test: unpacked OTA package, verified build fingerprint
Bug: 120123525
Change-Id: Iadd230a0577f35c7c37b0f911e91a5c2863ed1fe
2019-02-11 23:03:00 +00:00
Bowgo Tsai
867ab6606e DAP: building system_other.img for retrofit devices
For dynamic partitions in retrofit devices, system partition will be
a logical partition but system_other is not. However, current
build system use the same settings (logical) for both system.img and
system_other.img, leading AVB unable to locate the footer from the end
of system_other partition.

This commit support building system_other.img with correct partition size
while building system.img as a dynamic image.

Bug: 123506156
Test: check there is "system_other_size=2952790016" in file
      $OUT/obj/PACKAGING/system_other_intermediates/system_other_image_info.txt
Test: build system_other.img, then
      `simg2img $OUT/system_other.img system_other.img.raw`, checks the
      raw image size.

Change-Id: I748320a7770c694d06f06f4a35bfceb622849aa8
2019-01-30 09:58:07 +08:00
Tom Cherry
c683233b64 Merge "Remove ro.expect.recovery_id" 2019-01-29 17:07:51 +00:00
Yifan Hong
bb2658d5e2 releasetools: Really allow removing partitions
- Add hashtree_info to EmptyImage so that BlockDifference.Compute()
can accept EmptyImage() as target image, which is the case when
a partition is removed.

- BlockDifference also checks source_info_dict to determine
whether a partition is dynamic. When a partition is removed,
its name does not appear in target_info_dict.

- Add tests to ensure DynamicPartitionDifference() still works.

Test: DynamicPartitionDifferenceTest
Test: test_blockimgdiff

Change-Id: Iadb1db075f5dc344db6d5ade358c83b01231e443
2019-01-28 11:10:48 -08:00
Tom Cherry
56ff89a90a Remove ro.expect.recovery_id
This is not used by anyone and the other half of the code to compare
against it is already functionally dead, so remove this.

Test: build
Change-Id: I44ed087cb7735bbc23e30b6c310c80eb3b7b6488
2019-01-28 10:56:37 -08:00
Yifan Hong
79997e5a18 dynamic partition update order is deterministic.
Use an OrderedDict.
Test: builds

Change-Id: Ic3782c1961f0c2a735540b054eb3cace720c50bd
2019-01-25 12:36:17 -08:00
Treehugger Robot
d2da2ce398 Merge "Sign system_other.img with AVB" 2019-01-24 05:31:37 +00:00
Yifan Hong
2c0a7bc56d Merge "Really support removing / adding partitions." 2019-01-24 00:51:32 +00:00
Treehugger Robot
2463c3ab2a Merge "Support including DTB image" 2019-01-23 20:56:55 +00:00
Hridya Valsaraju
9683b2f136 Support including DTB image
Bug: 111136242
Test: When BOARD_PREBUILT_DTB_DIR is set correctly,
generated $OUT/boot.img contains the DTB image.

Change-Id: I282e31b04cc60383377b9e9b54f8fe64a8140242
2019-01-23 09:14:55 -08:00
Bowgo Tsai
1e04bf7260 Sign system_other.img with AVB
Support signing system_other.img but shouldn't include it into the
top-level vbmeta.img. system_other verifiation will not be included
in /vbmeta chains and will be done separately.

Bug: 112103720
Test: avbtool info_image --image $OUT/system_other.img
Test: avbtool info_image --image $OUT/vbmeta.img, checks 'system_other' is NOT included.
Test: Checks $OUT/obj/PACKAGING/system_other_intermediates/system_other_image_info.txt
      See the following:
        avb_system_other_hashtree_enable=true
        avb_system_other_add_hashtree_footer_args=--rollback_index 1551744000
        avb_system_other_key_path=external/avb/test/data/testkey_rsa4096.pem
        avb_system_other_algorithm=SHA256_RSA4096

Change-Id: Ia152aaab1387dcf556a42222adb39ea76881263a
2019-01-23 15:17:41 +00:00
Yifan Hong
45433e44dd Really support removing / adding partitions.
- When removing a partition, BlockDifference() object
will have tgt=EmptyImage(). Fix the asserts accordingly.
Also, BlockDifference object now allow tgt=None case.

- When adding a partition, BlockDifference() object
will have src=None. Fix the asserts accordingly.

Also, add unit tests to DynamicPartitionsDifference.

Test: create incremental OTA
Test: test_common.DynamicPartitionsDifferenceTest
Bug: 111801737

Change-Id: I3a35378ecf93111b8f44545cff6ae9696b6b4851
2019-01-18 17:09:33 -08:00
Mark Salyzyn
60fa99d491 build_image: Increase spare inodes to 12.
Test: builds device with DAP
Change-Id: I816a7da54bbe23d4eb1140ac0d101c0fee175f0a
2019-01-17 11:30:12 -08:00
xunchang
b6105dcba3 blockimgdiff: selectively convert 'diff' commands to 'new' to reduce stash size
We cannot simultaneously stash more blocks than the size limit imposed by
the cache size. As a result, some 'diff' commands will be inevitably
converted to new. We used to do this conversion blindly when iterating
through the transfer list. This leads to an unintended large package.

In order to choose the right transfers to convert, we calculate the size
of the compressed data, and build a heuristic about the package size
increase to remove each stash blocks. After the process, the given
package size for the watch device further reduces from 186M->155M.

In some rare cases, the removed stashed blocks don't directly contribute
to the maximum simultaneously stashed size. For example,
stash A: 10 blocks
stash B: 5 blocks
free B: 5 blocks  <-- stash B has been freed before we reach max stashed blocks
stash C: 10 blocks

Converting these blocks lead to an uncertain result. On one hand, patches
are generally smaller than the new data; while on the other hand, the
regenerated graph may have fewer order violation and thus give some size
reduction. But these cases are rare and it seems an overkill to consider all
possible scenarios here.

Bug: 120561199
Test: build non-A/B incrementals and check the size
(p.s. it can be tested on all target files with customed cache threshold)
Change-Id: I599420a91b80f1a1d83d22ee1b336b699050cfb4
2019-01-16 12:59:13 -08:00
Yifan Hong
885787f4d3 Merge "Generate OTA for non-A/B devices with dynamic partitions" 2019-01-16 18:42:58 +00:00
Mark Salyzyn
c25b2bf458 build_image: right-sizing at least some spare inode
To address problems creating real tiny filesystem placeholders,
make sure extra spare inodes margin is greater than 0.
For initial estimate we add 4% of total we request at least 8.
For second pass estimate we add 0.2% of total we request at least 1.

We bumped up the margin for zie on the second pass to 0.3% as the
value was too close on one of the builds.

Test: build
Bug: 122328872
Change-Id: I41707bb6fcc8bbfbdda143a9ce62446cad9c1533
2019-01-16 09:54:31 -08:00
Yifan Hong
10c530d20a Generate OTA for non-A/B devices with dynamic partitions
Test: sideload full OTA on cuttlefish
Test: sideload incremental OTA on cuttlefish (that grows
      system, shrinks vendor, and move vendor to group foo)

Bug: 111801737

Change-Id: Ie8a267a90b4df9e9e0a2fbcc1b582ab2e353df52
2019-01-15 16:04:12 -08:00
Mark Salyzyn
db3c8985be Merge changes I6457933a,I955aa0fa
* changes:
  Revert "Relax inode usage estimation slack for DAP right sizing"
  Revert "Increase slack for inode estimation algorithm"
2019-01-15 20:33:50 +00:00
Mark Salyzyn
9f23b89273 Revert "Relax inode usage estimation slack for DAP right sizing"
This reverts commit bd17fab260.

Test: build
Bug: 122328872
Change-Id: I6457933a0dc51827bd86d9c55b02a65999612bcc
2019-01-14 09:44:40 -08:00
Hridya Valsaraju
9aa42f1721 Revert "Increase slack for inode estimation algorithm"
This reverts commit e8e7958c13.

Test: build
Bug: 122328872
Change-Id: I955aa0fa489ab62ec1643127d106925dcb9675ca
2019-01-14 09:44:26 -08:00
Tao Bao
93e7ebe779 releasetools: OutputFile stages files under input dir.
We used to create the intermediate output file as a tempfile, when
adding images to a given zip file. This CL changes OutputFile to write
intermediate files under the given input dir (i.e. OPTIONS.input_tmp
that holds the unzipped files), if the final destination is a zip
archive. This allows image building codes finding the generated images
at consistent locations. The intermediate files will be cleaned up as
part of OPTIONS.input_tmp.

Bug: 122608028
Test: `m dist`
Test: Delete OTA/super*.img from a target_files.zip. Run
      `add_img_to_target_files.py -a` that re-generates split super
      images.
Change-Id: I97903a59fcc0ca5e43bb9b07c3a0b25e9baa92f9
2019-01-13 23:49:25 -08:00
Tao Bao
774c0f92bf Merge "releasetools: Skip copying split super images while signing." 2019-01-11 23:18:49 +00:00
Tao Bao
33bf268e0b releasetools: Skip copying split super images while signing.
Bug: 122608028
Test: Run sign_target_files_apks.py on a target-files zip that has split
      super images (e.g. OTA/super_system.img).
Change-Id: Iaf7263790961a897ea3f339f5af6b18cf253b946
2019-01-11 12:39:31 -08:00
Tao Bao
4b2306b41c releasetools: Update OWNERS.
Test: N/A
Change-Id: I0c6ac3cd04cc6034787d4b8b90fbd61349c54655
2019-01-11 12:28:09 -08:00
Mark Salyzyn
6541d0a80e build_image: support non-sparse right-sized images
Allow right-size to support images that are not sparse.
This is in support of cuttlefish.

Test: build
Bug: 122328872
Change-Id: Ic8ef968e750203dffde7044bc6dfc71c1e283158
2019-01-10 14:49:34 -08:00
Mark Salyzyn
60a716fe8c build_image: right-sizing add .2% margin for space and inodes
To address flakiness in image creation, add a .2% margin of safety
for both inodes and space for ext filesystems.

Test: build
Bug: 122328872
Change-Id: I6665bd6fe642291b825dba58dfd09cc55628230f
2019-01-10 14:49:24 -08:00
Mark Salyzyn
c777eaa779 build_image: hard code set inode size to 256
inode-size will jump around based on filesystem size, however
readonly Android system partitions have a specific pattern of
xattr associated with sepolicy and 256 is the most efficient at
absorbing the content.

Test: manual
Bug: 122328872
Change-Id: I06dd6a503067ab6477293b386d56a89dd86b0e83
2019-01-10 14:49:12 -08:00
Tao Bao
519d18286e Don't build retrofit OTA package if vendor.img is unavailable.
We used to build retrofit full OTA package as long as
PRODUCT_RETROFIT_DYNAMIC_PARTITIONS is true. This doesn't work with AOSP
targets that have the flag set but without any available vendor image.

This CL detects such a condition and uses a separate flag to guard the
retrofit OTA building as well as the split super images generation.

Bug: 120852744
Test: `m -j dist` on blueline (w/ vendor images) and aosp_blueline (w/o
      vendor images).
Change-Id: I65726f24f8fc546be6802941a6a06590a3804c16
2019-01-07 12:22:48 -08:00
Treehugger Robot
dacde2259f Merge "Increase slack for inode estimation algorithm" 2019-01-05 01:22:43 +00:00
Hridya Valsaraju
e8e7958c13 Increase slack for inode estimation algorithm
This is required to prevent build failure on some targets.

Test: mmm
Bug: 119115481

Change-Id: Idf7c21b163a4e10e2ef435c2b2a7795a42feb200
2019-01-04 09:18:07 -08:00
Yifan Hong
cc46eae994 Put system_other in super.img
... for launch A/B devices.
Test: build super image and lpdump
Bug: 113182233

Change-Id: I79ad9da2f6852b39b23e862ff00f320b6565db2f
2019-01-02 12:05:10 -08:00
xunchang
3df4d5eb6d blockimgdiff: add a function to select and convert 'diff' to new
Package size will be unintended large if we stash more blocks than the
stash limit specified by the cache size. To reduce the maximum size of
simultaneous stashed blocks, we will inevitably convert some 'diff'
commands to 'new' commands.

To mitigate the impact, we add a new function to smartly select the
transfers to convert based on their patch size and compressed size.
This cl converts the transfers that have a larger patch size than the
compressed target sizes. And there's a slightly improvement in the
final package size: from 194M -> 185M.

Bug: 120561199
Test: build a non-A/B incremental package, run simulator
Change-Id: Id73ff736ba4e6901d245ad5549d42310d0740284
2018-12-19 20:21:25 -08:00
Treehugger Robot
7cac3adf1a Merge "Relax inode usage estimation slack for DAP right sizing" 2018-12-18 02:13:10 +00:00
Hridya Valsaraju
bd17fab260 Relax inode usage estimation slack for DAP right sizing
The current increase of 4 % is insufficient for some
targets.

Test: mmm
Bug: 119115481
Change-Id: Idcba8025b913da9b70794bfc7464d15b4d99ad34
(cherry picked from commit 3e02e34090)
2018-12-14 14:13:24 -08:00
Tianjie Xu
8ab919dcce Merge "blockimgdiff: Factor out the diff_worker" 2018-12-13 18:45:30 +00:00
xunchang
2153183acf blockimgdiff: Factor out the diff_worker
We will call it at an earlier time to compute the patch size; and
choose the transfers to convert to 'new'.

Bug: 120561199
Test: Generate an incremental update on shiner
Change-Id: I29a0c8e75c9e5b66a266c1387186692a86fcbe43
2018-12-12 17:02:39 -08:00
Chen, ZhiminX
752439b321 Add recovery_acpio option to make recovery image
Test: can generate recovery image with recovery_acpio
Fixes: 111871613

Change-Id: I9174a4741f571a00bcadc548cd9e1316b4175132
2018-12-11 15:23:28 +08:00
Yifan Hong
e98427ac3f Don't build super image if source images are missing
For some internal branches, vendor.img isn't built, so there
is no need to build super image / super split images there.

Test: remove vendor.img and VENDOR/ from target_files_intermediates, then
      run add_img_to_target_files
Fixes: 120634805

Change-Id: I2834a27ce232538f203733c204dd257279c789fc
2018-12-07 11:01:21 -08:00
Tao Bao
2a20f344fc releasetools: Keep the original RangeSet for 'uses_shared_blocks' case.
Setting 'uses_shared_blocks' indicates that the block list for a given
file is incomplete, as some of the blocks are "owned" by others. As a
result, such a file will be skipped from imgdiff'ing.

This CL makes a copy of the original block list before removing the
shared blocks. It uses the original RangeSet as the value of
extra['uses_shared_blocks']. validate_target_files.py will try to read
the file as in the original RangeSet, unless the original list is also
incomplete or has non-monotonic ranges.

Test: Run validate_target_files on a target that uses
      `BOARD_EXT4_SHARE_DUP_BLOCKS := true`.
Change-Id: I259e871ecc249ba0c14b5796bef413185a1b8242
2018-12-06 10:28:49 -08:00
Yifan Hong
5df29604b9 Merge changes from topic "super_target_files"
* changes:
  add_img_to_target_files uses build_super_image
  Build super_empty with build_super_image.py
  Build super.img from images in target_files
2018-12-04 23:20:21 +00:00
Yifan Hong
055e6cf5c8 add_img_to_target_files uses build_super_image
Use the new script to build super_empty and super split images. No more
transformation to lpmake_args.

Test: build target_files_package for retrofit device
Change-Id: Id5f6bd607654ca869bcdf58d86b7ae300e3927eb
2018-12-03 11:33:44 -08:00
Yifan Hong
2b891ac024 Build super.img from images in target_files
For non-retrofit (launch) devices, super.img is used for factory, so
source images should be from target_files.

In this change, build-superimage-target procedure is converted to a
more flexible script so that it can be built.

Bug: 119322123
Test: build target files for device launch with dynamic partitions
Change-Id: I6ee0cc3e145357dfc74be248f81f5f8f4e51fc5c
2018-12-03 11:33:44 -08:00
David Anderson
6f693f04c2 releasetools: Specify a minimum size for right-sized images.
On aosp_blueline, system_other gets right-sized to 61,440 bytes. This
messes up the AVB footer calculation, and is too small to store an ext4
image.

Bug: 116608795
Test: build aosp_blueline w/ dynamic partitions
Change-Id: If0138a9d858765ff82cb957532504a066fd113ff
2018-11-29 13:27:47 -08:00
Tao Bao
03fecb669c releasetools: Remove the mismatch map files when generating retrofit OTAs.
The map files contain the filename-block mapping for the original
images.

Bug: 120109184
Test: Generate an incremental OTA that was previously failing.
Change-Id: I1285d0b24a435477b958e6c2e4f474acf8f7578b
2018-11-28 10:59:23 -08:00
Treehugger Robot
08d07f5aa9 Merge "Infer prebuilt image size" 2018-11-28 00:36:50 +00:00
Yifan Hong
70fc029ab8 Infer prebuilt image size
Test: call script twice; the second time uses prebuilts
Fixes: 119769527

Change-Id: Icd98a4dbfed91538cbc07aab957c7b042c05b330
2018-11-27 11:21:21 -08:00
Tao Bao
f21a8f5376 Merge "Use common.RunAndCheckOutput() in add_img_to_target_files.py." 2018-11-26 18:41:36 +00:00
Yifan Hong
50611038e9 dynamic_partition_use -> use_dynamic_partitions.
Test: build OTA package
Change-Id: Ia43d23f49fc024ecc767da1c2186c4b446c034de
2018-11-21 13:00:57 -08:00
Tao Bao
2764aee664 Use common.RunAndCheckOutput() in add_img_to_target_files.py.
Test: `m dist` with aosp_marlin-userdebug
Test: python -m unittest test_add_img_to_target_files.py
Change-Id: Icff60a44e5c189c2443fbaba3fc6a2b106b409ea
2018-11-21 11:04:32 -08:00
Yifan Hong
50e7954ac9 Generate retrofit OTA.
Test: m otapackage -j, manual apply on top of P
Bug: 118506262

Change-Id: Iedc389a1058007a604de0482f2348a9ef0d5892a
2018-11-20 11:17:39 -08:00
Yifan Hong
c767f7cbfc Add split super images to target_files.
Test: build target files package
Bug: 118506262
Change-Id: Iee6143987c9bfa6f2d287e20c06ea5ef25e82985
2018-11-20 11:17:39 -08:00
Tao Bao
9788b4ed31 releasetools: Temporarily disable verifying AVB-signed images.
Bug: 119624011
Test: Run validate_target_files.py on aosp_blueline-target_files.zip.
Change-Id: I0c3a93dd2ec9fe26c62be2ac2cc26780fb9fe1a6
2018-11-15 13:46:56 -08:00
Mark Salyzyn
3cd2460edc build: Split out mkfs in BuildImageMkfs (retry)
NB: second attempt, issue with scope for mkfs_output.

Separate out BuildImageMkfs from BuildImage, which just makes the
filesystem without any verity, avb, or other decisions.  BuildImage
does all the wrapping for such.  This will hopefully ease maintenance
and drop the issues surrounding BuildImage reentrancy.

Change right-size estimation path to use BuildImageMkfs, and thus do
so without verity or avb wrappings.  Add partition_headroom to space
consideration.  This makes the results of the estimation more
accurate and predictable.

Test: build
Bug: 111302946
Change-Id: I2549bd4e403c21290470b2fa1835492ae883f0fd
2018-11-08 15:42:22 -08:00
Mark Salyzyn
533558ec96 Revert "build: Split out mkfs in BuildImageMkfs"
This reverts commit 2b72b7f01e.

out/target/product/generic_x86/installed-files.txt \${DIST_DIR}/installed-files-rescued.txt; exit 1 ) )" 
2018-11-08 19:12:28 - build_image.py - ERROR : Failed to build out/target/product/generic_x86/obj/PACKAGING/systemimage_intermediates/system.img from out/target/product/generic_x86/system 
Traceback (most recent call last): 
File "build/make/tools/releasetools/build_image.py", line 767, in <module> 
main(sys.argv[1:]) 
File "build/make/tools/releasetools/build_image.py", line 755, in main 
BuildImage(in_dir, image_properties, out_file, target_out) 
File "build/make/tools/releasetools/build_image.py", line 449, in BuildImage 
CheckHeadroom(mkfs_output, prop_dict) 
NameError: global name 'mkfs_output' is not defined 


Bug: 111302946
Reason for revert: Build problem

Change-Id: I786f232e07af653a7207509055df5a07a6d8bb9e
2018-11-08 19:26:50 +00:00
Mark Salyzyn
2b72b7f01e build: Split out mkfs in BuildImageMkfs
Separate out BuildImageMkfs from BuildImage, which just makes the
filesystem without any verity, avb, or other decisions.  BuildImage
does all the wrapping for such.  This will hopefully ease maintenance
and drop the issues surrounding BuildImage reentrancy.  Change
right-size estimation path to use BuildImageMkfs, and do so without
verity or avb wrappings.

Test: build
Bug: 111302946
Change-Id: I30e2e2b727f40ecca5164142f34139f5244f6424
2018-11-07 15:51:32 -08:00
Mark Salyzyn
41f781de9e build_image: right size image problems.
Neglected to correct for filesystem size in estimate for verity.

To keep build space optimized and make sure that the existing image
does not confuse the tools, remove the out_file after the test build
image phase.

Test: build
Bug: 111302946
Change-Id: I360775d862d02a721d061bbc43e8b4e2f03bd01e
2018-11-07 14:50:33 -08:00
Tao Bao
68b081fd5a Merge "releasetools: Add tests for VerityImageBuilder." 2018-11-07 20:23:12 +00:00
Anton Hansson
0e60f8a42c Make right-sizing less aggressive
This is a rollback of Id5ae6645b9aa2d036e6fefe2fb17672e8f8ef6f0
and the followup I6a0f3919e8d45d0b3e7cd558a6ad4d3799012f2c

The magic constant introduced appears to be too aggressive.
Note that the new constant in I0f32c75e80b5da4d47671055ac274ccc2b485992
doesn't appear to fix the broken builds.

Bug: 111302946
Test: make
Change-Id: I29207265172d293c6fee98212e3266bde85895df
2018-11-07 14:02:42 +00:00
Mark Salyzyn
2b191894c2 build_image still too aggressive with right sizing (part troix)
Regression introduced by c3d408e3c1

Found a case where 1 + 1/60 reduction in size based on free space
failed when right sizing an ext4 image. Reduced the math to 1 + 1/61.

Test: compile
Bug: 111302946
Change-Id: I0f32c75e80b5da4d47671055ac274ccc2b485992
2018-11-06 15:57:03 -08:00
Tao Bao
d0b9758203 Merge "releasetools: Create VerityImageBuilder." 2018-11-06 23:37:15 +00:00
Tao Bao
46901fb8ca releasetools: Log the exceptions before rethrowing.
In Append2Simg and Append in verity_utils.py, we catch and rethrow
exceptions as BuildVerityImageError. However, this suppresses the
traceback from the original exception which usually indicates the actual
cause. We can better handle this with the raise statement in Python 3,
which is however unavailable in Python 2.

This CL logs the exception before rethrowing to retain the useful bits.

Test: Inject an error to append2simg. `m -j systemimage` with
      aosp_marlin-userdebug. Check the output.
Change-Id: I0c2f57d6023fa1038256b85fa98d57ad0244a70d
2018-11-06 10:42:35 -08:00
Tao Bao
4a0d5134e1 releasetools: Add tests for VerityImageBuilder.
`testdata/verity_mincrypt` is generated with the following command.
  $ generate_verity_key \
        -convert testdata/verity.x509.pem \
        testdata/verity_mincrypt

Test: python -m unittest test_verity_utils
Change-Id: I3f2f0cd89da0fd5d2420e0e2b09761cc77f893c5
2018-11-06 10:00:51 -08:00
Tao Bao
7549e5e919 releasetools: Create VerityImageBuilder.
Test: `m dist` with aosp_marlin-userdebug (VB 1.0)
Test: `m dist` with aosp_taimen-userdebug (VB 2.0)
Test: `python -m unittest discover .`
Change-Id: I8c016b7621b73f220fb2a4586cc45a21eeee438d
2018-11-06 10:00:28 -08:00
Mark Salyzyn
4a6ce93f27 build_image too aggressive with right sizing
Regression introduced by c3d408e3c1

Found a case where 1 + 1/59 reduction in size based on free space
failed when right sizing an ext4 image. Reduced the math to 1 + 1/60.

Test: compile
Bug: 111302946
Change-Id: I6a0f3919e8d45d0b3e7cd558a6ad4d3799012f2c
2018-11-01 12:45:29 -07:00
Tao Bao
757dfe8792 Merge "Revert "Work around e2fsck issue for building sparse GSIs"" 2018-10-30 15:18:58 +00:00
Tao Bao
27809667b4 Revert "Work around e2fsck issue for building sparse GSIs"
This reverts commit ec7fa1cd09.

The earlier e2fsck issue has been addressed by [1].

[1] https://android-review.googlesource.com/c/platform/external/e2fsprogs/+/725259/

Bug: 112062612
Test: m PRODUCT-aosp_x86_64-eng \
          SANITIZE_HOST=address ASAN_OPTIONS=detect_leaks=0 \
          -j e2fsck systemimage
Change-Id: I7287cb8639ae45abdf753821421bd070b819a30e
2018-10-29 16:02:47 -07:00
Mark Salyzyn
f0cef8d8bf build_image setup ext4 dedupe on more images.
Add ext4 dedupe to system_other, product, product_services, odm and
oem images.  Experimental savings without any other configuration
changes was 0%, .75%, 3.4% respectively for the raw image file size
for the first three.

Test: manual
Bug: 111302946
Change-Id: Ia8fb5696151acad59bb144ea93f2c2ddac962bbd
2018-10-29 13:37:10 -07:00
Mark Salyzyn
c3d408e3c1 build_image tune right-size image
To more quickly settle on a no free space result, recognize that
there is roughly 1/58.5 overhead managing the free space pool as
determined experimentally.  This algorithm carries with it some
risk of being sensitive to any alterations in ext4.

This also addresses the issue of a much larger apparent free space
available calculated on the sample pass when deploying ext4 dedupe,
and resolves the poorer estimation that happens without accounting
for the overhead.

The alternative of adding a second pass works to a similar, and
likely more comforting algorithm and result.  But doing so adds
a minute to the typical incremental build time, or +50% to the
time it takes to determine ext4 right sized images.

Test: manual
Bug: 111302946
Change-Id: Id5ae6645b9aa2d036e6fefe2fb17672e8f8ef6f0
2018-10-29 13:37:04 -07:00
Tao Bao
f3fc62c8a0 releasetools: Fix the error message.
This avoids showing a message as below:

raise BuildImageError("Failed to get tune2fs usage:\n{}".format(output))
UnboundLocalError: local variable 'output' referenced before assignment

The `output` won't be set under such a case, which unfortunately hides
the real cause. This CL addresses the issue by letting it throw
implicitly, which contains the command and outputs.

Test: `m dist`
Test: `python -m unittest test_build_image`
Change-Id: Icf015bab8869d150516246ed73552f6502127cdf
2018-10-25 12:29:44 -07:00
Mark Salyzyn
780f595fac build_image: improvements to right size for dynamic partitions
If partition_reserved_size is 0 or undefined, and
use_dynamic_partition_size is true, we should approach no space
and no free inodes automatically.

Estimate the space and number of inodes required, then do a first
pass build to see how much space actually used, and use those values
to refine the estimate.

Depends on tune2fs to report the characteristics of the filesystem,
so only support for ext filesystems.  In the future if there has to
be a more generic ability, either a tool per a filesystem has to be
found, or we will need root capabilities to mount the filesystem to
acquire the characteristics live from the host system.

Test: manual + python -m unittest test_build_image
Bug: 111302946
Change-Id: I933a388be43516b6de7b5007b296765bd5556fde
2018-10-23 15:14:00 -07:00
Tao Bao
7ceac730cf Merge "releasetools: Include descriptor of vbmeta_system.img in vbmeta.img." 2018-10-23 20:03:47 +00:00
Tao Bao
7106420add releasetools: Include descriptor of vbmeta_system.img in vbmeta.img.
Chained vbmeta images should be included into the top-level vbmeta.img
as chained partitions. It's done in the path of `m`, but not `m dist`.

Bug: 118115607
Test: `m dist` a target that uses chained vbmeta.
    a) Check that the `vbmeta.img` in target-img.zip contains the image
       descriptor of `vbmeta_system.img` (`avbtool info_image`).
    b) `avbtool verify_image --image vbmeta.img \
            --expected_chain_partition \
                vbmeta_system:1:/path/to/vbmeta_system/pubkey`
    c) `avbtool verify_image --image vbmeta_system.img`
Change-Id: I064e583b247c44b9b2f19355838550bb5dbb8f26
2018-10-22 15:27:54 -07:00
Tianjie Xu
ffbe6b9785 Install the ota keys under recovery as a zipfile
Recovery can now parse the pem encoded x509 keys from a zipfile. So
instead of dumping the keys into a text file with some intermediate format,
we can simply create a zipfile with the keys.

Bug: 116655889
Test: make bootimage and check the generated zipfile, run sign_target_files_apks
Change-Id: Ib76feecfb26d6be713a07644e80ec96133759004
2018-10-22 14:47:15 -07:00
Tao Bao
898a924904 releasetools: Log to stdout when running tests.
This sets up the root logger while running tests. It also logs to
stdout, as our Python test result parser doesn't like outputs from
stderr.

Test: `python -m unittest discover build/make/tools/releasetools \
          > /dev/null` doesn't print log lines.
Change-Id: Ic99711bd458bc4b67b38226786fed604c2168476
2018-10-18 15:05:49 -07:00
Tao Bao
32fcdab512 releasetools: Switch to Python logging.
Converts the following files to Python logging.

add_img_to_target_files.py
blockimgdiff.py
build_image.py
check_ota_package_signature.py
common.py
img_from_target_files.py
make_recovery_patch.py
ota_from_target_files.py
sparse_img.py
verity_utils.py

This separates logging outputs from normal outputs, and allows easier
switching between different verbosity levels. It also supports adjusting
logging config via environment variable (LOGGING_CONFIG).

Test: `m dist`
Test: `python -m unittest discover build/make/tools/releasetools`
Change-Id: Idfc3f7aff83e03f41f3c9b5067d64fd595d5192d
2018-10-17 07:51:04 -07:00
Treehugger Robot
86fb70dcdc Merge "Rename vbmeta_mainline to vbmeta_system." 2018-10-16 22:02:48 +00:00
David Anderson
7709ab2b69 Rename vbmeta_mainline to vbmeta_system.
Bug: 116859651
Test: device builds with chained vbmeta_system enabled
      vbmeta_system.img is generated

Change-Id: I75b52a89feeec8dd0e1e271d2a8c3c94036c932c
2018-10-15 14:42:29 -07:00
Tao Bao
bec89c1e1e releasetools: Clean up the logging while calling external commands.
common.RunAndCheckOutput() checks the exit code and will raise on errors.

Test: python -m unittest test_ota_from_target_files
Test: Call ota_from_target_files.py with aosp_marlin target_files zip
Test: Inject an error to the command. Repeat the call without verbose
      flag. It dumps the stack trace on error.
Change-Id: I85b765a33b9087bcbcb0571d6e632a07bb86c65c
2018-10-15 12:01:42 -07:00
Tao Bao
28f201b417 releasetools: Fix a broken test due to verity utils relocation.
Test: python -m unittest test_validate_target_files
Change-Id: I2267e18f63d4a36ecd5fff911848470a860ab6cf
2018-10-13 19:27:52 -07:00
Tao Bao
71197515bb releasetools: Move verity related functions into verity_utils.py.
This prepares for upcoming changes that refactor verity-related
functions into classes.

This CL makes minimal changes to the moved functions, by creating a new
BuildVerityImageError class in verity_utils.py replacing the former
BuildImageError that's specific to build_image.py. As part of the
change, it also moves the tests for AVBCalcMinPartitionSize into
test_verity_utils.py.

Test: python -m unittest test_verity_utils
Test: `m dist` with aosp_marlin-userdebug (Verified Boot 1.0)
Test: `m dist` with aosp_taimen-userdebug (Verified Boot 2.0)
Change-Id: I19b52714d8980705ea1f9484ac03eb0af9483240
2018-10-12 12:09:34 -07:00
Tao Bao
65b94e92be releasetools: Add a common base class for unittests.
All the releasetools unittests extend the common base class of
test_utils.ReleaseToolsTestCase. Define tearDown() in the base class to
do the clean-up works.

Test: `pylint --rcfile=pylintrc test_*.py`
Test: `python -m unittest discover .`
Change-Id: I51775d964ef032dcdf3bb89c55e1a31371cde708
2018-10-11 23:01:09 -07:00
Tao Bao
5fe287beb4 releasetools: Rename test names in test_verity_utils.py.
There's a naming convention between the testcase and the class/method
being tested.

Also remove VerifiedBootVersion2HashtreeInfoGenerator as it's merely a
placeholder without telling anything much (e.g. whether it's to be
implemented or working as intended).

Test: python -m unittest test_verity_utils
Change-Id: Ieb156765c865aa551e3882f381e3a6db2cac4cbc
2018-10-11 14:39:09 -07:00
Tao Bao
986ee86854 releasetools: Move build_image.RunCommand into common.py.
build_image.RunCommand and common.Run are essentially doing the same
work, but with different interfaces and return values. This CL
consolidates them by moving build_image.RunCommand into common.py as
common.RunAndCheckOutput, with the former calling common.Run internally.

common.RunAndCheckOutput checks the exit code: it returns the output
string on success, or raises common.ExternalError otherwise.

Test: `m dist` with aosp_marlin-userdebug
Test: python -m unittest test_build_image
Test: python -m unittest test_common
Change-Id: I3cd0e9301c576572194618eba8dc00db06c3c35f
2018-10-10 18:13:24 -07:00
Tao Bao
73dd4f45f3 releasetools: Set default stdout and stderr in common.Run().
stdout and stderr will default to subprocess.PIPE and subprocess.STDOUT
respectively (which is the expected behavior from most of the existing
callers), unless caller specifies any of them.

Test: `m dist`
Test: python -m unittest \
          test_common \
          test_add_img_to_target_files \
          test_ota_from_target_files \
          test_validate_target_files
Change-Id: I43b3f08edfa8a9bcfe54baf9848dc705c048e327
2018-10-04 17:05:09 -07:00
Tao Bao
2f057467eb releasetools: build_image.BuildVerityTree() returns a tuple.
We don't need verity_root_hash or verity_salt to be in the prop dict.

Test: `m dist` with aosp_marlin-userdebug
Test: python -m unittest test_verity_utils
Change-Id: I5a9c50f4741dfb1083b3f590136335b6bc0e5216
2018-10-03 16:37:58 -07:00
Tianjie Xu
67c7cbb9c8 Generate the compute_hash_tree command
Generate the transfer command "compute_hash_tree" for incremental
updates of the non-A/B devices that enable verified boot 1.0

Other changes include:
i.  factor out verity_utils to use both in build_image and blockimgdiff
ii. add a new flag 'hashtree_info_generator' in sparse_image to generate
    the hashtree information.

Bug: 25170618
Test: generate a package for aosp_angler; and run simulator
Change-Id: I4d4d7a3e41dc3da810d2cbf8988e85d187c9ab0e
2018-10-03 13:36:35 -07:00
Tao Bao
c6bd70a5e6 releasetools: Raise on image building errors.
The image building functions in build_image.py have been returning
(success, result) or special values to indicate the validity of the
result. The same logic can be better expressed by raising exceptions
instead, because
 a) using a special value relies on caller to check for that magic
    value;
 b) exceptions can carry additional messages other than a boolean does,
    e.g. the output from the failing command;
 c) caller can have cleaner code flow without explicitly checking for
    the validity of the result.

This CL changes such functions to raise on errors. The majority of these
functions are internal to build_image.py only, except for BuildImage()
that has a few callers in add_img_to_target_files.py (which all die upon
error anyway).

Test: `m dist`
Test: python -m unittest test_build_image
Test: python -m unittest test_add_img_to_target_files
Test: python -m unittest test_validate_target_files
Test: Inject an error to the depended binaries (e.g. avbtool), and check
      that build_image.py exits with error messages.
Change-Id: Ibe4d51e267756bb1a00fa9238a213f9d55fd9b58
2018-10-03 11:31:44 -07:00
Bowgo Tsai
040410c104 Prevent AVB signing from using _RESERVED_SIZE
When AVB is enabled with PRODUCT_USE_DYNAMIC_PARTITION_SIZE, AVB
metadata (e.g., hash tree, fec metadata) will consume _RESERVED_SIZE,
resulting in smaller reserved size in file system (e.g., ext4).

Adding additional space for AVB signing and keep the _RESERVED_SIZE only
for file system. This is done by adding a function to binary search an
optimal partition size for a given image size (disk usage + _RESERVED_SIZE).

Bug: 112322265
Test: Build aosp_arm64-userdebug, calculate the running time of
      AVBCalcMinPartitionSize() is about 0.3-0.4 seconds.
Test: python -m unittest test_build_image
Change-Id: I8f0051b57701d6fbba6a9db3756dd194066c74b8
2018-10-03 10:08:45 +08:00
Tao Bao
35f4ebc957 releasetools: Clean up the use of partition_size.
Unless using dynamic partitions, `partition_size` should be a fixed
value that equals to the partition size in BoardConfig. It should stay
the same when building an image for that partition. Only the actual
image size that's used to hold the filesystem could be adjusted. This CL
cleans up the uses of `partition_size` and `image_size` to better
reflect such logic.

With dynamic partitions, the only thing that changes is the need to
compute `partition_size` upfront. Once that's done, `partition_size`
should remain unchanged.

Test: `m dist`
Test: `python -m unittest test_add_img_to_target_files`
Test: `python -m unittest test_validate_target_files`
Change-Id: Idedb3e018c95e8f63dc4d9c423be27f30ebb584f
2018-09-27 18:06:59 -07:00
Tianjie Xu
75844bd171 Merge "Generates the care_map with fingerprint" 2018-09-27 17:32:33 +00:00
Tianjie Xu
861f413fad Generates the care_map with fingerprint
Reads the fingerprint of a partition from the build props and embeds it
into the care_map. Later, the update_verifier will compare the
fingerprint before it performs the verification.

Bug: 114778109
Test: generate care_map for sailfish and new pixel
Change-Id: I06afa01bf6dd3de4456a08ee7960954facc775fb
2018-09-26 23:35:32 -07:00
Tom Cherry
b6eeba5bab Merge "Allow BOARD_USES_RECOVERY_AS_BOOT without BOARD_BUILD_SYSTEM_ROOT_IMAGE" 2018-09-26 22:24:54 +00:00
Tom Cherry
2929cadb3b Allow BOARD_USES_RECOVERY_AS_BOOT without BOARD_BUILD_SYSTEM_ROOT_IMAGE
The recovery image already contains a copy of first stage init, so we
can boot unconditionally to the recovery image and instruct first
stage init whether or not to boot to Android or to recovery.  In this
case, we need neither the kernel to mount /system as / nor a separate
partition for recovery, so this change modifies the build scripts to
allow this combination.

Bug: 114062208
Test: Boot pixel from recovery to Android with BOARD_USES_RECOVERY_AS_BOOT
      But without BOARD_BUILD_SYSTEM_ROOT_IMAGE
Change-Id: Icd047afb7f22d2724b3bcaca1aa0c837426dcce7
2018-09-26 20:30:36 +00:00
Tianjie Xu
5875ca06ce Merge "Rename care map in pb format to care_map.pb" 2018-09-21 00:03:40 +00:00
Tianjie Xu
4c05f4a41e Rename care map in pb format to care_map.pb
The new suffix distinguishes the new care_map from the ones in plain
text format; and thus the old update_verifier won't report an error
upon parsing failures.

Bug: 115740187
Test: Generate OTA files for Pixels
Change-Id: Ia782afd8cbb0f4bb8c363edaa00e92ab302d5d1b
2018-09-18 15:42:40 -07:00
Tao Bao
92cc5d2ebe Merge "Support building chained VBMeta images." 2018-09-11 18:27:37 +00:00
Treehugger Robot
860f9d11a8 Merge "Use the new executable for mkuserimg_mke2fs" 2018-09-08 03:50:32 +00:00
Tao Bao
744c4c7604 Support building chained VBMeta images.
Bug: 112007947
Test: `m dist` with aosp_taimen-userdebug.
Test: Set up a target that uses chained VBMeta images of
      `vbmeta_mainline` and `vbmeta_vendor`. `m dist` and check the
      build log, as well as outputs from
      `avbtool info_image --image vbmeta.img`,
      `avbtool info_image --image vbmeta_mainline.img`,
      `avbtool info_image --image vbmeta_vendor.img`.
Change-Id: Ib1d4e97f583b65245703eae15d211adcd9e83741
2018-09-06 16:55:51 -07:00
Treehugger Robot
71034e3764 Merge "Adding better AVB error message" 2018-09-06 07:45:34 +00:00
Bowgo Tsai
99ed1b4fbd Adding better AVB error message
Also removing the requirement of BOARD_BOOTIMAGE_PARTITION_SIZE when
BOARD_AVB_ENABLE is set. Some targets (e.g., emulator) doesn't build
boot.img when AVB is enabled.

Bug: 112293933
Test: build a failure case and checks the error output is expected
Change-Id: I290c707719193ddaedcd9fd0b31de566ad17078c
2018-09-06 09:38:45 +08:00
Tianjie Xu
573322279a Use the new executable for mkuserimg_mke2fs
This is in line with the change of mkuserimg_mke2fs.sh to python binary.

Bug: 112555072
Bug: 63866463
Test: unittests pass
Change-Id: I82c0be1e5bbc685edc15120da73aa43fdc9f2f05
2018-09-05 20:52:47 +00:00
Tao Bao
ea6cbd0765 releasetools: Handle the case of not having ro.vendor.build.fingerprint.
Commit d572632f3d added support in build
system to blacklist given vendor properties. When
ro.vendor.build.fingerprint gets blacklisted, the OTA generation script
can't rely on those properties to determine a change in vendor images.
This CL considers such a case as "vendor images must have changed"
between the two builds.

Bug: 113892939
Test: Generate an incremental package with builds not having
      ro.vendor.build.fingerprint.
Test: python -m unittest test_ota_from_target_files
Change-Id: I188de9c3cbeecf26132c92b9356e9d5fef75205e
2018-09-05 13:14:55 -07:00
Tao Bao
3e9e6af0ff Merge "releasetools: common.LoadInfoDict() takes a bool parameter for repacking." 2018-08-31 22:08:41 +00:00
Tao Bao
639fe0c7cd Merge "releasetools: Track edify function API changes." 2018-08-31 21:24:31 +00:00
David Anderson
1ef03e2b79 Add super_empty.img to target-files and the update package.
super_empty.img is needed for "fastboot update" to work, as such, it
must be included in the update package. This change adds lpmake
parameters to misc_info.txt for add_img_to_target_files.py, and ensures
that lpmake is packaged with otatools.zip.

The build-superimage-target macro is now split into two functions - one
to generate the device-specific arguments, and another to build the full
command-line for convenience. The former is used to pass arguments
directly to add_img_to_target_files.

Bug: 113524256
Test: make updatepackage builds and includes super_empty.img
      make otatools includes lpmake

Change-Id: I25091c964b036beeea2a8b8f738e2c18937c1eb4
2018-08-30 16:02:23 -07:00
Tao Bao
410ad8bb92 releasetools: common.LoadInfoDict() takes a bool parameter for repacking.
It used to accept a non-None input_dir to indicate the need to
re-generate images (called from add_img_to_target_files.py). During that
flow, both of the two parameters redundantly refer to the same input
dir. This CL replaces the second parameter with a bool value instead.

Test: `python -m unittest test_common`
Test: `m dist` with aosp_taimen-userdebug
Test: `zip -d aosp_taimen-target_files-eng.zip IMAGES/\* &&
       add_img_to_target_files.py -a aosp_taimen-target_files-eng.zip`
Change-Id: I0a5a164366acb116407f94bb350872a3a0b207d1
2018-08-29 22:07:21 -07:00
Tao Bao
5121655643 releasetools: Track edify function API changes.
In particular, it replaces the generated calls to edify function of
`apply_patch()` and `apply_patch_check()` with `patch_partition()` and
`patch_partition_check()` instead.

It adds two functions to EdifyGenerator: PatchPartition() and
PatchPartitionCheck() for Python releasetools callers. It also tries to
handle the callers of obsolete EdifyGenerator.PatchCheck() and
EdifyGenerator.ApplyPatch(), if their inputs are in known format.
Otherwise it raises an exception to avoid generating an OTA that updater
doesn't understand.

It requires the matching updater changes in the same topic.

Bug: 110106408
Test: Generate an incremental package with the new script and updater.
      Apply the package on device.
Test: Revert the change in ota_from_target_files.py, so that it calls
      the obsolete EdifyGenerator functions. Check that it generates the
      same incremental package.
Change-Id: Ifc55cb40d3a45116fd4d408536d037eca249effa
2018-08-29 11:53:19 -07:00
Tom Cherry
04552fed3b Merge "Support a first stage ramdisk via TARGET_RAMDISK_OUT" 2018-08-29 17:05:40 +00:00
Tom Cherry
d14b895665 Support a first stage ramdisk via TARGET_RAMDISK_OUT
Support a first stage ramdisk which will include the targets placed
into TARGET_RAMDISK_OUT.  This replaces the existing ramdisk on
existing devices.

All system images are now built to be mounted as the root dir.
Devices with a first stage ramdisk will switch root to the system
partition.

BOARD_BUILD_SYSTEM_ROOT_IMAGE remains and is used to specify if the
system partition is going to be directly used as rootfs without the
ramdisk.

Bug: 79173823
Bug: 79758715
Test: hikey boots, sailfish boots
Test: OTA walleye from P to master
Change-Id: Idbb2dccc6340b0235a4bef03e11e420a9ed154b6
2018-08-28 12:42:41 -07:00
Tao Bao
a57ab9f210 releasetools: Add test cases for common.LoadInfoDict().
Fixes: 113073663
Test: python -m unittest test_common
Change-Id: I352a2731849d175f94e0f23dd68b6cf28e554f64
2018-08-27 18:18:10 -07:00
Tao Bao
71142ea2c9 Merge "releasetools: Use int value of FIXED_FILE_TIMESTAMP." 2018-08-27 19:39:03 +00:00
Tao Bao
e30a6a6df9 releasetools: Use int value of FIXED_FILE_TIMESTAMP.
Otherwise it gives a floating-point value (i.e. 1230768000.0). e2fsdroid
(i.e. the user of the value) didn't complain though, because it calls
strtoul(3) without checking endptr.

Bug: 80600931
Test: `m dist` with aosp_taimen-userdebug, check the build log.
Change-Id: I7467a60c947134b42f079c7a57fb8a9a9d4277da
2018-08-27 11:03:55 -07:00
Tao Bao
687da3b0a5 releasetools: Fix the path to the OTA keys in recovery image.
The recovery image will be packed under BOOT/RAMDISK only if
system_root_image and recovery_as_boot both are true (e.g. non-A/B
devices launched since P).

Bug: 113191245
Test: Run sign_target_files_apks.py on a target_files file that uses
      system-as-root but not recovery-as-boot.
Change-Id: I262a268055c6b5078d21694b5094a1c393d0d37c
2018-08-26 14:39:58 -07:00
Tao Bao
22f28aa1b5 Merge "A/B update: Replace the zip FileHeader mechanism for update package" 2018-08-24 19:54:10 +00:00
Tao Bao
b4adc06732 releasetools: Look for recovery.fstab at both locations.
The change in [1] moved the recovery etc files from /etc to /system/etc.
However, we may use the latest OTA tools to build incremental OTAs for
old target_files zips. This CL adds a workaround to look at both of the
old and new locations.

[1] commit 696bb33676

Bug: 113073663
Test: Build a previously failing incremental OTA for taimen.
Change-Id: Ie07aa1713e616d523838b1260a992b20f5a11612
2018-08-22 18:41:49 -07:00
Shashikant Baviskar
338856f92b A/B update: Replace the zip FileHeader mechanism for update package
When the update package gets larger than 2 GiB, payload.bin offset
mentioned in metadata file for ota-streaming-property-files gets
shifted (CrAU of payload.bin) because ZipInfo FileHeader() returns
incorrect value. To solve the issue, offset is re-calculated from
fixed bytes of central directory file header, filename length and
extra length.

This patch is to sync with update_device.py script.

Test: manually create an A/B update package and run it using
      update_device.py
Bug: 111198589

Change-Id: I9bf5a5ca24938cad3206d04af529f70d45e992c0
2018-08-23 10:13:20 +09:00
Tao Bao
696bb33676 Install recovery etc files to recovery/root/system/etc.
They used to be installed under recovery/root/etc. This CL moves the
files to the new location and creates a symlink from /etc to /system/etc
(done by the rule in system/core/rootdir). This gives similar layout
between normal boot and recovery, and allows installing prebuilt_etc
files with Soong (`recovery_available: true`).

As part of the change, we no longer need the whitelisting rule for
mke2fs.conf.

Bug: 112780007
Test: Build with other changes in the topic (aosp_taimen-userdebug).
      Check the generated files under recovery (/etc being a symlink to
      /system/etc).
Test: Boot into recovery. Verify basic functionalities (`adb shell` and
      `adb sideload`, factory reset).
Test: `build/soong/build_test.bash --dist`
Change-Id: Ibb6dea6f179a339f0c2d0fd8ba05ec0085b79a12
2018-08-21 10:30:49 -07:00
Dario Freni
924af7d4dd s/product-services/product_services/g
Attempting to reduce the number of different spellings we have for
"product services" partition in the codebase.

Bug: 112431447
Test: m
Change-Id: I6debd3f5f5abc4bd1e22f2ef11c5fe131915976c
Merged-In: I6debd3f5f5abc4bd1e22f2ef11c5fe131915976c
2018-08-20 10:44:59 -07:00
Yifan Hong
35be6ca8f2 Reland "s/PRODUCT-SERVICES/PRODUCT_SERVICES/g"
Test: builds
This reverts commit 87021065b9.

Reason for revert: reland CL

Change-Id: Ic88c2bd612ad7a19d88686b2907c812bd586ee35
Merged-In: Ic88c2bd612ad7a19d88686b2907c812bd586ee35
2018-08-20 10:44:59 -07:00
Treehugger Robot
68883e0717 Merge "Revert "s/PRODUCT-SERVICES/PRODUCT_SERVICES/g"" 2018-08-17 20:32:18 +00:00
Shawn Yang
87021065b9 Revert "s/PRODUCT-SERVICES/PRODUCT_SERVICES/g"
This reverts commit 77c778c040.

Reason for revert: git-master break on 4960355

Change-Id: I2fd272c63ef496067f12852bcf98ffc1194c54bf
2018-08-17 19:02:08 +00:00
Tianjie Xu
ccbae48ac5 Generates the care_map.txt in protobuf format
Call the host binary care_map_generator to generate the care_map in
protobuf format. The parsing part of proto messages has already been
supported in the update_verifier.

Bug: 77867897
Test: unittests pass; run add_image_to_target_files
Change-Id: I40d3184b4b5c48a6dd55203afc84cca73d5765e9
2018-08-17 17:39:00 +00:00
Dario Freni
77c778c040 s/PRODUCT-SERVICES/PRODUCT_SERVICES/g
Attempting to reduce the number of different spellings we have for
"product-services" in the codebase.

Bug: 112431447
Test: m
Change-Id: Iec531122297cf4b86ff8601732d4ab7ccf5c2183
2018-08-17 00:28:17 +01:00
Bowgo Tsai
d624fa63d5 Add support for /odm partition
This CL is largely an adaptation of Change-Id
I5d229f6ac729ea6df9ff1f14cee2e28972cd9b4d

tools/vendor_buildinfo.sh is also renamed to tools/device_buildinfo.sh.
The caller then can invoke device_buildinfo.sh "vendor" | "odm" to
generate properties for vendor.img and odm.img, respectively.

It adds the following variables:

- BOARD_AVB_ODM_KEY_PATH
- BOARD_AVB_ODM_ALGORITHM
- BOARD_AVB_ODM_ROLLBACK_INDEX_LOCATION
- BOARD_ODMIMAGE_FILE_SYSTEM_TYPE
- BOARD_ODMIMAGE_EXTFS_INODE_COUNT
- BOARD_ODMIMAGE_EXTFS_RSV_PCT
- BOARD_ODMIMAGE_PARTITION_SIZE
- BOARD_ODMIMAGE_JOURNAL_SIZE
- BOARD_ODMIMAGE_SQUASHFS_BLOCK_SIZE
- BOARD_ODMIMAGE_SQUASHFS_COMPRESSOR
- BOARD_ODMIMAGE_SQUASHFS_COMPRESSOR_OPT
- BOARD_ODMIMAGE_SQUASHFS_DISABLE_4K_ALIGN
- BOARD_PREBUILT_ODMIMAGE
- BOARD_USES_ODMIMAGE
- LOCAL_ODM_MODULE
- PRODUCT_ODM_BASE_FS_PATH
- PRODUCT_ODM_VERITY_PARTITION
- PRODUCT_ODM_PROPERTIES
- TARGET_COPY_OUT_ODM
- TARGET_OUT_ODM
- TARGET_OUT_ODM_*

Bug: 64195575
Test: boot a Taimen with existing images

Test: `make odmimage` with
      - BOARD_AVB_ENABLE := true
      - BOARD_ODMIMAGE_PARTITION_SIZE := 62914560
      - BOARD_ODMIMAGE_FILE_SYSTEM_TYPE := ext4
      - TARGET_COPY_OUT_ODM := odm
      - PRODUCT_ODM_PROPERTIES += odm.test.build=success

Test: `make odmimage` with
      - BOARD_ODMIMAGE_PARTITION_RESERVED_SIZE := 10485760
      - BOARD_ODMIMAGE_FILE_SYSTEM_TYPE := ext4
      - BOARD_AVB_ENABLE := true
      - TARGET_COPY_OUT_ODM := odm
      - PRODUCT_ODM_PROPERTIES += odm.test.build=success
      - PRODUCT_USE_DYNAMIC_PARTITION_SIZE := true

Change-Id: I4dea7b567ec49a766c7a4683decaf81c7e921d55
2018-08-08 18:59:44 +08:00
Isaac Chen
13d3426f3c Merge "Work around e2fsck issue for building sparse GSIs"
am: a0d3277b8d

Change-Id: I53b45da39145cb10f37bd7bcede55f13cfed801b
2018-08-03 17:07:18 -07:00
Treehugger Robot
a0d3277b8d Merge "Work around e2fsck issue for building sparse GSIs" 2018-08-03 23:29:33 +00:00
Isaac Chen
ec7fa1cd09 Work around e2fsck issue for building sparse GSIs
When trying to support sparse format for GSIs, aosp/720949, a bug in
e2fsck was triggered, and hence failed the build for sparse GSI.

This workaround disables the detection of ASAN ODR so we can build
sparse GSIs successfully like we previously did for the vendors,
some of which can't handle unsparse images.

Bug: 112062612
Test: $ lunch aosp_x86_64-eng
      $ m SANITIZE_HOST=address ASAN_OPTIONS=detect_odr_violation=0 -j
      $ emulator

Change-Id: Ifafd27aa1e57d785b914f78a4d9bfef6a0c33fc9
2018-08-03 11:51:08 +08:00
Yifan Hong
635887d5ac Merge "target_files: copy to PRODUCT-SERVICES instead"
am: 567df6214f

Change-Id: I95bf48eba07daa2b0f6d5b84c45de9b22f7aac0b
2018-08-02 15:31:56 -07:00
Yifan Hong
567df6214f Merge "target_files: copy to PRODUCT-SERVICES instead" 2018-08-02 21:53:57 +00:00
Bryan Henry
a2bf1e5c5c Merge "Fix 2009-01-01 timestamps in releasetools to always be UTC"
am: e118d5bc4a

Change-Id: I12c8cc0664306ba116076b3f6d43e1c01fb9120f
2018-08-01 17:24:41 -07:00
Bryan Henry
e118d5bc4a Merge "Fix 2009-01-01 timestamps in releasetools to always be UTC" 2018-08-02 00:16:06 +00:00
Yifan Hong
4d6b38c47a target_files: copy to PRODUCT-SERVICES instead
The add_img_to_target_files.py script uses upper() to determine
the name of sub-directory under target_files_intermediates. The
name of the image is product-services.img but the sub-directory
name was PRODUCT_SERVICES. Change it to PRODUCT-SERVICES so that
'm dist' works.

Test: m dist
Change-Id: I8cad841c674b5d62c1d803d0383fc3aee14f272a
2018-08-01 16:46:16 -07:00
Yifan Hong
9a7d525849 Merge "Split PRODUCT_USE_LOGICAL_PARTITIONS into sub-flags."
am: d1c308bf40

Change-Id: I834e75dcfc04af0dd8655027bfc251c3d0162815
2018-08-01 11:39:59 -07:00
Bryan Henry
e6d547d53a Fix 2009-01-01 timestamps in releasetools to always be UTC
The usage of datetime.fromtimestamp previously resulted in the build or
signing machine's local timezone affecting the Unix timestamp ultimately
applied to images generated by add_img_to_target_files. The go/ab build
outputs would use 2009-01-01 00:00 UTC, for example, but local builds
and the release signed images (generated through go/ab-sign) would use
2009-01-01 00:00 PST. This change makes the timestamps always use UTC.

Bug: 80600931
Bug: 80093599
Test: 'm -j droid dist' and verified timestamps in resulting
target_files zip.

Change-Id: Ic2a19591519850c249f78254e1464aa6839bfc6c
2018-07-31 18:32:00 -07:00
Yifan Hong
2dae5721d4 Split PRODUCT_USE_LOGICAL_PARTITIONS into sub-flags.
Split the flag into PRODUCT_USE_DYNAMIC_PARTITION_SIZE
and PRODUCT_BUILD_SUPER_PARTITION. More sub-flags can be
added with the same way.

Also change some checks with USE_LOGICAL_PARTITIONS with
the appropriate sub-flags.

This allows easier device bring-up to fulfill the requirements
separately.

This also enables dynamic partition size without logical patition.
Originally, to enable BOARD_*_PARTITION_RESERVED_SIZE must also
enable PRODUCT_USE_LOGICAL_PARTITIONS. The patch fix the rule to
let dynamic partition size is able to be used independently.

Fixes: 111966003

Test: make superimage with PRODUCT_USE_LOGICAL_PARTITIONS
Change-Id: I45b29c87bf94356d8416bbd58a14c8982039c222
2018-08-01 00:39:13 +00:00
Yifan Hong
8883a739b3 Merge changes I1459d62f,I4c2b975e
am: bb937a6b69

Change-Id: Ifdedee1b0251bc9c2a8baf30d28b5d315b5c9e20
2018-07-30 11:35:33 -07:00
Yifan Hong
ebc041af56 productservices => product_services
Make names consistent.
Some things that does not change:
- productservicesimage target and intermediates

Test: build product services image
Bug: 111609632
Change-Id: I4c2b975e194577aad3d51b908b103880625deb5e
2018-07-26 17:19:02 -07:00
Tao Bao
0adc598d82 Merge "releasetools: Fix an issue in image size computation."
am: d578232c95

Change-Id: Idd139fe594e95cd92926da42beb670be239ee113
2018-07-26 12:53:51 -07:00
Tao Bao
4f35d47160 Merge "releasetools: Return the actual image size when building logical partitions."
am: f42fc1a7c3

Change-Id: I714fbd9458c2d6708ade24b36cbfd965a135b840
2018-07-26 10:09:53 -07:00
Tao Bao
c2606eb548 releasetools: Fix an issue in image size computation.
When building a system image with system_root_image enabled, the size
computation should include files under both of in_dir (i.e. /system
files) and root (pointed by 'root_dir'). Because files from both
locations will end up into the built image. The files under root are
usually only a few MiBs, but should be accounted for especially in the
context of logical partitions (where the partition size will be
allocated based on the actual need). Note that we will still need some
"reserved space" (defined via BOARD_*_PARTITION_RESERVED_SIZE) to cover
the cost for filesystem and/or verity metadata.

This CL moves the combination of the two dirs up, before parsing and
computing other properties. This doesn't affect anything for a
successful image building path. It may however increase the time to
error out in certain error path, since it copies the files earlier now.

Test: python -m unittest test_build_image
Test: `make dist`
Test: Setup a target with PRODUCT_USE_LOGICAL_PARTITIONS == true and
      BOARD_SYSTEMIMAGE_PARTITION_RESERVED_SIZE == 20MiB. Build system
      image successfully.
Test: Setup a target with PRODUCT_USE_LOGICAL_PARTITIONS == true and
      BOARD_SYSTEMIMAGE_PARTITION_RESERVED_SIZE == 20MiB. Write a large
      file to root dir (PRODUCT_OUT_ROOT). The image building fails, but
      reporting a size that accounts for both of /system and root.
Change-Id: Idfb26b8e259626ba57ec3bd4f85d357c30e56163
2018-07-26 16:59:57 +00:00
Tao Bao
f42fc1a7c3 Merge "releasetools: Return the actual image size when building logical partitions." 2018-07-26 16:59:36 +00:00
Tao Bao
4251fe97aa releasetools: Return the actual image size when building logical partitions.
When verity is enabled, `partition_size` has the "adjusted" partition
size for holding the filesystem files (i.e. excluding verity hashtree,
and FEC metadata if applicable), whereas `original_partition_size` has
the one for the actual image returned by build_image.py.
`partition_size` must be smaller than `original_partition_size`. As a
result, the later assertion in Makefile (e.g. assert-max-image-size) may
fail to hold when using `partition_size`.

This CL addresses the issue by returning the value in
`original_partition_size` instead, if available. It also changes to
include both values when dumping the size info on image creation
failure.

Fixes: 79106666
Test: `m dist` with aosp_walleye-userdebug
Test: Set up walleye to use logical partitions, with a reserved size of
      20MiB. `m systemimage` no longer fails.
Test: Set up walleye that doesn't use verified boot. Use `m systemimage`
      to build an oversized image. Check the error message.
Change-Id: I41d77bffa85e8eadfbf4a2a7dcd2e798fadfdab7
2018-07-24 11:57:10 -07:00
Tao Bao
fdcb3b5829 Merge "releasetools: Support verifying AVB signed images with chained partitions."
am: 78078646e9

Change-Id: If7656208f92fde69dad358d298cd1c9eb1002ecd
2018-07-24 11:53:57 -07:00
Tao Bao
02a0859b3c releasetools: Support verifying AVB signed images with chained partitions.
For example, verify a target_files.zip that has system AVB-signed as a
chained partition.

  $ build/make/tools/releasetools/validate_target_files.py \
      signed-target_files-4904652.zip \
      --verity_key verifiedboot_pub.pem \
      --avb_system_key_path system_pub.pem

Note that verifiedboot_pub.pem should be the key (either public or
private) to verify vbmeta.img, and 'system_pub.pem' should be the key
(either public or private) for the chained partition of system.

testdata/testkey.key is the private key converted from
testdata/testkey.pk8 for testing purpose (`openssl pkcs8 -in
testdata/testkey.pk8 -inform DER -out testdata/testkey.key -nocrypt`).

Bug: 63706333
Test: python -m unittest test_common
Test: python -m unittest test_add_img_to_target_files
Test: `m dist` on aosp_walleye-userdebug; Run validate_target_files.py
      on the generated target_files.zip.
Test: Set up walleye with chained system partition; `m dist`; Run
      validate_target_files.py on the generated target_files.zip.
Change-Id: I38517ab39baf8a5bc1a6062fab2fe229b68e897d
2018-07-23 15:20:14 -07:00
Tao Bao
f013dc3fed Merge "releasetools: Clean up AppendVBMetaArgsForPartition()."
am: 540a71d2b3

Change-Id: If13ee90c84f75c2452c1e1e8504928414a971ed4
2018-07-23 14:56:22 -07:00
Tao Bao
540a71d2b3 Merge "releasetools: Clean up AppendVBMetaArgsForPartition()." 2018-07-23 21:40:37 +00:00
Yifan Hong
fe421ed263 Merge changes from topic "board_partition_reserved_size"
am: 2d90c246a9

Change-Id: I460ea917b47a61acb0cbb0f17a867d15f61c7d6d
2018-07-23 11:58:52 -07:00
Yifan Hong
2d90c246a9 Merge changes from topic "board_partition_reserved_size"
* changes:
  build_image.py recognize BOARD_PRODUCT_SERVICESIMAGE_PARTITION_RESERVED_SIZE
  build_image.py recognize BOARD_PRODUCTIMAGE_PARTITION_RESERVED_SIZE
2018-07-23 18:38:24 +00:00
Tao Bao
f87ea90d76 Merge "releasetools: Rename the ramdisk_dir in prop dict to root_dir."
am: d094713952

Change-Id: I013c82846a143c5eac050325e66092afc66c958f
2018-07-23 09:41:25 -07:00
Tao Bao
d094713952 Merge "releasetools: Rename the ramdisk_dir in prop dict to root_dir." 2018-07-23 16:33:46 +00:00
Tao Bao
1a3d83d667 Merge "releasetools: Make scripts pylint clean."
am: 23a4070483

Change-Id: I4f4b35565f7d154dd47e4b5b8444e76c6c7227c6
2018-07-23 07:22:59 -07:00
Tao Bao
23a4070483 Merge "releasetools: Make scripts pylint clean." 2018-07-23 14:11:37 +00:00
Tao Bao
3e53ef7b1c releasetools: Clean up AppendVBMetaArgsForPartition().
Drop the parameter of public_key_dir. It's not necessary to stage all
the extracted public keys into the same temporary dir.

Redirect the stderr output to STDOUT while calling avbtool, so that it
can dump the error message accordingly.

Also remove the check for having empty img_path arg. It doesn't look
like a clean logic to allow the function to be called with invalid /
empty arg. The only caller (i.e. AddVBMeta) already asserts img_path
being valid as well.

Test: python -m unittest test_add_img_to_target_files
Test: `m dist` with aosp_walleye-userdebug
Change-Id: Id58c5ae780ac8a22661ffea629144d4836839175
2018-07-22 22:36:27 -07:00
Tao Bao
708ca5f60b Merge "releasetools: Track the change to applypatch arguments."
am: 80810b9d36

Change-Id: I9df643323e5624596b5981e9a4d48fdafb1db2f3
2018-07-20 17:12:39 -07:00
Yifan Hong
9c35a022e1 build_image.py recognize BOARD_PRODUCT_SERVICESIMAGE_PARTITION_RESERVED_SIZE
Test: builds
Bug: 79106666
Change-Id: I00c1a96c09cf7bd9274b65ac0afc47462686171d
2018-07-20 16:51:26 -07:00
Yifan Hong
56a6c3bb34 build_image.py recognize BOARD_PRODUCTIMAGE_PARTITION_RESERVED_SIZE
Test: builds
Bug: 79106666
Change-Id: I26aba694fd3d6a2feed7601524dde52569ba41e2
2018-07-20 16:51:26 -07:00
Tao Bao
8bfd3c7483 releasetools: Rename the ramdisk_dir in prop dict to root_dir.
This property indicates the path to the root directory (i.e.
TARGET_ROOT_OUT at build time, or ROOT/ in a target_files.zip).
build_image.py will only use this property when building a target using
system_root_image. Under such a configuration, files under this dir
never land into a ramdisk, but as part of system.img. This CL renames
the property name to avoid confusion.

Note that with this change releasetools keeps working with older
target_files zips that contain 'ramdisk_dir'. Because common.py doesn't
read the value of that property, but sets it to ROOT/ directly.

Test: `m dist` on aosp_marlin-userdebug.
Test: `m dist` on aosp_angler-userdebug.
Test: Run add_img_to_target_files.py with a new target_files.zip and an
      old target_files.zip respectively.
Change-Id: Ie9e19dd17d8c2abb4c4a97449e436774d5dd2a36
2018-07-20 16:05:36 -07:00
Tao Bao
80810b9d36 Merge "releasetools: Track the change to applypatch arguments." 2018-07-20 23:04:33 +00:00
Dario Freni
5f9ddc95a2 Merge "Add support for /product-services partition"
am: 3cf03b775a

Change-Id: I1b0b3628d1b96967e569093b93b12fbc5acd7ad5
2018-07-20 02:09:37 -07:00
Dario Freni
3cf03b775a Merge "Add support for /product-services partition" 2018-07-20 09:00:25 +00:00
Tao Bao
b1750e2825 Merge "releasetools: Remove the assertion of not allowing thumbprint in Treble world."
am: 805ab25e05

Change-Id: Iee41170414e8b17f05d2defa405b18834bc7e8f3
2018-07-19 15:05:39 -07:00
Tao Bao
76def243ec releasetools: Make scripts pylint clean.
Clean up the following scripts.
  blockimgdiff.py
  common.py
  edify_generator.py
  img_from_target_files.py
  ota_from_target_files.py

Seems we don't have a way to force pylint-ing the scripts using
per-directory pylintrc files (and we don't have pylint tool in AOSP
repo), per
https://android.googlesource.com/platform/tools/repohooks/#todo_limitations.

Test: `m dist`
Test: pylint --rcfile=pylintrc <script.py>
Change-Id: Ia6fd1ddc86f4d84c68e500f225d4a89d0fea8ec7
2018-07-19 10:39:05 -07:00
Tao Bao
4223012099 releasetools: Remove the assertion of not allowing thumbprint in Treble world.
The claim in the original CL doesn't seem to hold. We may still have
devices using both of thumbprint and Treble compatibility check.

Bug: 111640956
Test; python -m unittest test_ota_from_target_files
Test: Build an incremental package for a target using thumbprints.
Change-Id: Ia491d5279901c2385d1c30802abcb99d65d919cc
2018-07-19 10:16:45 -07:00
Dario Freni
5f681e1dd7 Add support for /product-services partition
This CL is largely an adaptation of Change-Id
I774e6a38003734421591e51bed103802ff84f432

It adds the following variables:

- BOARD_AVB_PRODUCT_SERVICES_KEY_PATH
- BOARD_AVB_PRODUCT_SERVICES_ALGORITHM
- BOARD_AVB_PRODUCT_SERVICES_ROLLBACK_INDEX_LOCATION
- BOARD_PRODUCT_SERVICESIMAGE_FILE_SYSTEM_TYPE
- BOARD_PRODUCT_SERVICESIMAGE_EXTFS_INODE_COUNT
- BOARD_PRODUCT_SERVICESIMAGE_EXTFS_RSV_PCT
- BOARD_PRODUCT_SERVICESIMAGE_PARTITION_SIZE
- BOARD_PRODUCT_SERVICESIMAGE_JOURNAL_SIZE
- BOARD_PRODUCT_SERVICESIMAGE_SQUASHFS_BLOCK_SIZE
- BOARD_PRODUCT_SERVICESIMAGE_SQUASHFS_COMPRESSOR
- BOARD_PRODUCT_SERVICESIMAGE_SQUASHFS_COMPRESSOR_OPT
- BOARD_PRODUCT_SERVICESIMAGE_SQUASHFS_DISABLE_4K_ALIGN
- BOARD_PREBUILT_PRODUCT_SERVICESIMAGE
- BOARD_USES_PRODUCT_SERVICESIMAGE
- LOCAL_PRODUCT_SERVICES_MODULE
- PRODUCT_PRODUCT_SERVICES_BASE_FS_PATH
- PRODUCT_PRODUCT_SERVICES_VERITY_PARTITION
- PRODUCT_PRODUCT_SERVICES_PROPERTIES
- TARGET_COPY_OUT_PRODUCT_SERVICES
- TARGET_OUT_PRODUCT_SERVICES
- TARGET_OUT_PRODUCT_SERVICES_*

Bug: 80741439
Test: Successfully built product-services.img with one module in it, and flashed
on device. Also successfully built image with /system/product-services directory
and no /product-services partition.
Change-Id: I5d229f6ac729ea6df9ff1f14cee2e28972cd9b4d
2018-07-19 11:15:11 +01:00
Tao Bao
4948aed1a3 releasetools: Track the change to applypatch arguments.
This writes /system/bin/install-recovery.sh with the updated argument
formats. Requires matching changes to bootable/recovery/applypatch.

Bug: 110106408
Test: python -m unittest test_common.InstallRecoveryScriptFormatTest
Test: See additional tests in the other CL.
Change-Id: I2dec6f064819c01b5bb34c3d6fd0e247a5f48f08
2018-07-17 17:58:14 -07:00
Tao Bao
40e1bb6df9 Merge "releasetools: Fix secondary payload generation."
am: c26dea96a2

Change-Id: I64322df5111736b9276103fb71c36e9ec4a9e1d3
2018-07-12 21:13:29 -07:00
Tao Bao
1248980a8b releasetools: Fix secondary payload generation.
The change in [1] removed duplicating radio images from RADIO/ to
IMAGES/. When getting the target_files zip for generating secondary
payload, we need to additionally copy the files under RADIO/ over.

[1] commit 5277d1015f

Test: python -m unittest test_ota_from_target_files
Test: Generate an OTA with secondary payload (--include_secondary).
Change-Id: I096f1642a905fb764e63f5df2edc1396aa6befd8
2018-07-12 14:52:52 -07:00
Tao Bao
ff4434f433 Merge "releasetools: Remove RemoveBackwardEdges()."
am: d4b9e01f39

Change-Id: If887e4f4e453f1ec2b199ac249d844e753129803
2018-07-11 12:13:04 -07:00
Tao Bao
d4b9e01f39 Merge "releasetools: Remove RemoveBackwardEdges()." 2018-07-11 19:00:32 +00:00
Tao Bao
b047778ebd Merge "releasetools: Address two issues with system_root_image file map."
am: 81421ccf38

Change-Id: I171a9a7b8a4b0e1051e8752f519ca5d81e3b0f4f
2018-07-11 10:14:56 -07:00
Tao Bao
5bab0dd1c4 releasetools: Remove RemoveBackwardEdges().
RemoveBackwardEdges() was used only in BBOTA v1 (which has been
deprecated since O). v2+ calls ReverseBackwardEdges() and
ImproveVertexSequence().

Also remove the imgdiff tag of 'trimmed' that would be set through this
function() only.

Test: python -m unittest test_blockimgdiff
Test: Build an incremental non-A/B OTA.
Test: Code search shows no active user.
Change-Id: I3b58ae048a1fbc283269e70fdfa29eb8d184ede7
2018-07-10 22:13:40 -07:00
Tao Bao
d3554e628f releasetools: Address two issues with system_root_image file map.
With system_root_image, e2fsdroid writes file map with extra leading
slashes in filenames (e.g. "//system/framework/am.jar"). This breaks the
detection of files with incomplete ranges, and thus fails the patch
generation. This CL addresses the issue by stripping out leading
slashes.

Additionally, non-/system files (e.g "//sbin/charger") are not packed
under SYSTEM/ in a target_files.zip, despite being part of system.img.
We need to look for these files under ROOT/ instead.

This CL also asserts the availability of all files listed on a file map,
to avoid silently missing other edge cases.

Bug: 80380658
Test: python -m unittest test_common
Test: Successfully generated an incremental for a target using
      system_root_image that was previously failing.
Change-Id: I62a2460e882f3930e99add4d2b44291edf7a51a0
2018-07-10 16:05:14 -07:00
Tao Bao
27bc4a9b1d Merge "releasetools: Remove three obsolete functions from EdifyGenerator."
am: e9856a835f

Change-Id: Iaf358557ae4ac4e154894868701c84f350ae7a6e
2018-07-06 16:30:56 -07:00
Tao Bao
a24f711bf3 releasetools: Remove three obsolete functions from EdifyGenerator.
Namely MakeTemporary, Verify and FileCheck. They were once used for
file-based OTA that has been deprecated.

Test: Check that there's no active user, including device-specific
      releasetools script.
Change-Id: Ic73054d73271ad91e860b34b2ba156361c58bc1a
2018-07-06 14:21:26 -07:00
Tao Bao
a85cb404cd Merge "releasetools: Add dict-like setter to BuildInfo."
am: ee36b42beb

Change-Id: I97039b14a1664d5cddd2cf44de600686bbf3f26d
2018-07-06 14:05:23 -07:00
Tao Bao
667c7534ed releasetools: Add dict-like setter to BuildInfo.
BuildInfo already has dict-like getters. This CL adds the matching
setter method to keep it backward compatible with device-specific
scripts that expect a dict.

It also adds items() method that allows iterating the items.

Bug: 111087332
Test: python -m unittest test_ota_from_target_files.BuildInfoTest
Test: Generate an incremental OTA that sets info_dict in device-specific
      releasetools script.
Change-Id: Idd033f98a9186740f9da1a300d4c2bdddd9c1345
2018-07-06 10:15:13 -07:00
Yifan Hong
1dce9b6943 Merge "build_image.py: remove extra new lines in error message"
am: 674454f223

Change-Id: Iaa6edf6b28c5d2e5cdbc55068ab2631c17138288
2018-06-22 22:06:16 -07:00
Yifan Hong
92fb231105 build_image.py: remove extra new lines in error message
Test: m dist
Change-Id: I7908e4eb0a9bbcb187e1fa6d15f606537d69864d
2018-06-22 15:05:46 -07:00
Yifan Hong
bb42f7d655 Merge changes from topic "board_partition_reserved_size"
am: 3967e0eb99

Change-Id: I767967f1086ec303c63199d0e38ec02dedf09152
2018-06-22 13:25:18 -07:00
Treehugger Robot
3967e0eb99 Merge changes from topic "board_partition_reserved_size"
* changes:
  build_image.py recognize BOARD_VENDORIMAGE_PARTITION_RESERVED_SIZE
  build_image.py recognize BOARD_SYSTEMIMAGE_PARTITION_RESERVED_SIZE
2018-06-22 20:10:47 +00:00
Yifan Hong
749062d379 build_image.py recognize BOARD_VENDORIMAGE_PARTITION_RESERVED_SIZE
Test: `make vendorimage`

Test: `make vendorimage` with the following:
    - install a large file to vendor image
    fails as expected (because _PARTITION_SIZE is exceeded)

Test: `make vendorimage` with the following:
    - set PRODUCT_USE_LOGICAL_PARTITIONS to true
    - set BOARD_VENDORIMAGE_PARTITION_RESERVED_SIZE
    fails as expected (BOARD_VENDORIMAGE_PARTITION_SIZE needs
    to be undefined)

Test: `make vendorimage` with the following:
    - install a large file to vendor image
    - set PRODUCT_USE_LOGICAL_PARTIIONS to true
    - add a small BOARD_VENDORIMAGE_PARTITION_RESERVED_SIZE
    - remove BOARD_VENDORIMAGE_PARTITION_SIZE
    build succeeds.

Bug: 79106666

Change-Id: Ica8fdce64e8f37d91e66e3d49c0c74fadd15a832
2018-06-22 10:25:54 -07:00
Yifan Hong
bbcba1e353 build_image.py recognize BOARD_SYSTEMIMAGE_PARTITION_RESERVED_SIZE
- Copy "use_logical_partitions" to *_image_info.txt before sending
  it to build_image.py, so that the script can use this variable.

- build_image.py emits an additional properties file to inform
  the build system about the system image size.

Test: `make systemimage`

Test: `make systemimage` with the following:
    - install a large file to system image
    fails as expected (because _PARTITION_SIZE is exceeded)

Test: `make systemimage` with the following:
    - set PRODUCT_USE_LOGICAL_PARTITIONS to true
    - set BOARD_SYSTEMIMAGE_PARTITION_RESERVED_SIZE
    fails as expected (BOARD_SYSTEMIMAGE_PARTITION_SIZE needs
    to be undefined)

Test: `make systemimage` with the following:
    - install a large file to system image
    - set PRODUCT_USE_LOGICAL_PARTIIONS to true
    - add a small BOARD_SYSTEMIMAGE_PARTITION_RESERVED_SIZE
    - remove BOARD_SYSTEMIMAGE_PARTITION_SIZE
    build succeeds.

Test: same for systemotherimage

Bug: 79106666

Change-Id: I574062882acd1ecd633ac38c5a8c5351b90a32d8
2018-06-22 10:25:54 -07:00
Tao Bao
8012498f79 Merge "releasetools: Skip signing APKs with given prefixes."
am: 8acf3c302b

Change-Id: Ie8f30d2c4e95c047653ca7120c62571df9f14842
2018-06-20 16:36:02 -07:00
Tao Bao
93c2a01268 releasetools: Skip signing APKs with given prefixes.
We may pack prebuilts that end with ".apk" into target_files zip, via
PRODUCT_COPY_FILES. META/apkcerts.txt won't contain the cert info for
such files, and we want to keep them as is while signing, despite of the
".apk" extension.

This CL adds "--skip_apks_with_path_prefix" option to
sign_target_files_apks.py. APKs with matching prefixes will be copied
verbatim into the signed images. The prefix should match the entry names
in the target_files (e.g. "SYSTEM_OTHER/preloads/"). The option may be
repeated to specify multiple prefixes.

Note that although we may skip signing an APK file with "-e ApkName=".
This would skip *all* the APK files with the matching basename.
"--skip_apks_with_path_prefix" allows matching the exact prefix.

For example:
$ ./build/make/tools/releasetools/sign_target_files_apks.py     \
    --skip_apks_with_path_prefix SYSTEM_OTHER/preloads/         \
    --skip_apks_with_path_prefix PRODUCT/prebuilts/PrebuiltApp1 \
    --skip_apks_with_path_prefix VENDOR/app/PrebuiltApp2.apk    \
    target_files.zip                                            \
    signed-target_files.zip

Bug: 110201128
Test: Run the command above and check the logs.
Test: `python -m unittest test_sign_target_files_apks`
Change-Id: I7bd80b360917cef137cf1e7e8cfa796968831f47
2018-06-20 13:49:57 -07:00
Tao Bao
023b5104d7 Merge "releasetools: Factor out the check for (compressed) APK file."
am: 2ba547a1c7

Change-Id: I0ae458b3c0d8c0335c8175138185ce714d065471
2018-06-20 09:18:56 -07:00
Tao Bao
11f955c506 releasetools: Factor out the check for (compressed) APK file.
Test: Run sign_target_files.py to sign a target_files.zip.
Test: `python -m unittest test_sign_target_files_apks`
Change-Id: Ie795d1bce7bae6af427832283e3d10bfecad80c5
2018-06-19 14:50:23 -07:00
Tao Bao
e89e50e477 Merge "releasetools: Explicitly pass the SHA-1 of source boot image to PatchCheck."
am: 50869f203a

Change-Id: I21bb21342cde6806e5fd300e1da70813562cad13
2018-06-13 10:31:55 -07:00
Tao Bao
6d8ec5803b releasetools: Explicitly pass the SHA-1 of source boot image to PatchCheck.
Although the SHA-1 of the source boot image has been encoded in the
filename, the updater code currently doesn't extract that and would fail
to find a matching backup from /cache. This could lead to corrupt
install, if it's interrupted while patching the boot image.

Bug: 80630342
Test: Generate an non-A/B incremental package. Verify the script.
Change-Id: Ie2b1a90836c3ccb4758e1a2d9b69a88de94ae0c7
2018-06-12 15:04:08 -07:00
Tao Bao
9ba115e8e2 Merge "releasetools: Skip checking files with non-monotonic ranges." am: 6d2ca1f19e
am: b1ae1cc94a

Change-Id: I41f67881f06223f9a04bc9e1a6213e02f8c9af8a
2018-05-17 21:40:53 -07:00