Commit graph

1087 commits

Author SHA1 Message Date
Tao Bao
2bb109709a Remove the obsolete logic in img_from_target_files.py.
img_from_target_files.py used to handle the case that a given TF.zip not
containing the image entries under IMAGES/. That is only the case for
pre-Lollipop releases.

Also unzip the needed files only since we know that for sure now.

Test: img_from_target_files.py with an existing bullhead-TF.zip gives
      the same bullhead-img.zip.
Change-Id: I892379ba388df80ae63be9d3ce647fbb77fd4753
2017-05-31 11:17:56 -07:00
Tao Bao
3cba374da2 Pack avbtool into otatools.zip.
Also pack the test keys for easier testing.

Bug: 38315721
Test: m otatools-package and avbtool is present in otatools.zip.
Change-Id: Ieb63bf3f4bc211ef1f48ab278cb01b70845d06da
2017-05-31 10:20:25 -07:00
Treehugger Robot
fe7aeb1de7 Merge "Support signing custom images with AVB HASH or AVB HASHTREE" 2017-05-31 03:24:25 +00:00
Tianjie Xu
f7a76f9a30 Merge "Remove the obsolete UnpackPackageDir() in edify generator" 2017-05-31 01:08:15 +00:00
Tianjie Xu
616fbeb144 Move recovery building from sign_target_files_apk to add_img_to_target_files
Currently we're building the boot/recovery image twice, which is
redundant. And b/38455129 shows a problematic case when the image
from two builds doesn't match. We should only build the recovery
image once in the add_img_to_target_files.

Bug: 62021378
Test: call sign_target_files_apk on an angler target file,
recovery-from-boot.p generates successfully; and SHA of recovery.img
matches the one in install-recovery.sh.

Change-Id: I01e033501d80c18a87cbb870300eee5c19a04441
2017-05-26 17:20:22 -07:00
Tianjie Xu
38af07ff20 Update the recovery files under SYSTEM/ if recovery patch is rebuilt
If we pass "rebuild_recovery" to add_img_to_target_files, the recovery
patch is rebuilt. But related files under SYSTEM/ (e.g.
SYSTEM/recovery-from-boot.p && SYSTEM/bin/install-recovery.sh) are not
updated.

This may cause a mismatch between system.img and SYSTEM/, and
may lead to a failure in validate_target_files.py.

Bug: 62096364
Test: Rebuild the system image in the TF and observe the recovery files
under SYSTEM/ get updated.

Change-Id: I7d679a612a86d02cf2eff81d1d120c0067138ed9
2017-05-26 22:40:23 +00:00
Bowgo Tsai
7ea994b21c Support signing custom images with AVB HASH or AVB HASHTREE
`make custom_images` supports to build different kinds of *non-droid* images,
e.g., odm.img. Adding the support of signing them with either AVB HASH footer
or AVB HASHTREE footer. The user can use HASH for small images and
HASHTREE for large images.

Sample signing configurations:
 * AVB HASH footer:
   - CUSTOM_IMAGE_AVB_HASH_ENABLE := true
   - CUSTOM_IMAGE_AVB_ADD_HASH_FOOTER_ARGS := --append_to_release_string my_odm_image

 * AVB HASHTREE footer:
   - CUSTOM_IMAGE_AVB_HASHTREE_ENABLE := true
   - CUSTOM_IMAGE_AVB_ADD_HASHTREE_FOOTER_ARGS := --fec_num_roots 8

 * Using custom signing key:
   - CUSTOM_IMAGE_AVB_ALGORITHM := SHA256_RSA2048
   - CUSTOM_IMAGE_AVB_KEY_PATH := external/avb/test/data/testkey_rsa2048.pem

Bug: 36701014
Test: `make custom_images` with AVB HASH footer
Test: `make custom_images` with AVB HASHTREE footer
Test: `make droid` to check system.img is still properly signed with AVB HASHTREE
Test: `make droid` to check vendor.img is still properly signed with AVB HASHTREE

Change-Id: I8dc420e12e37e9a631345c0cd883339db05d489f
2017-05-24 14:21:03 +08:00
Tianjie Xu
6a324ba588 Remove the obsolete UnpackPackageDir() in edify generator
package_extract_dir is used in file based OTA only and should be killed.

Bug: 37959785
Test: code search shows no usage of this function in aosp.
Change-Id: Id3719b969c24b7ecef0c7f0e4a3af09a72be54d4
2017-05-23 17:15:14 -07:00
Tao Bao
406050bdb6 Fix the symlink detection when signing TF.zip.
We used to check for 'attr >> 16 == 0xa1ff' (i.e. 0o120777) to detect
symlinks in the input target_files zip (TF.zip). This becomes broken
after we switch to soong_zip, which packs symlinks with 0o120700.

This CL fixes the issue by using stat.S_ISLNK() instead.

Note that we don't need to stage the files with the exact permission
bits as in the input TF.zip. Because this part is covered by mkbootfs
by using the canned or the compiled-in fs_config - as long as the
files/directories are accessible and the symlinks are created.

Bug: 38455129
Test: sign_target_files_apks.py on bullhead TF.zip. Check the
      checksums in SYSTEM/bin/install-recovery.sh.
Change-Id: I51c1fc9a257fb3f18c16c2ed71528abaa6f7d9c9
2017-05-22 23:33:11 -07:00
Treehugger Robot
294d5c5f07 Merge "add dtbo support for ota" 2017-05-15 16:49:46 +00:00
Treehugger Robot
bba00b2cb9 Merge "Add the option to reserve headroom for partition images. This is useful for devices with low disk space with different build variants." 2017-05-13 01:37:25 +00:00
Yueyao Zhu
889ee5e7d3 add dtbo support for ota
Test: make dist and locally push ota on a device
Change-Id: I920b98f20e248b437955b2a963eb69ed2ddb8d45
2017-05-12 17:50:46 -07:00
Tianjie Xu
f34d435925 Merge "Add a script to parse an block-based ota package" 2017-05-11 21:52:10 +00:00
Christian Oder
f63e2cd47b OTA Tools: Handle password encrypted keys
* Add logic to handle decrypted keys from common.GetKeyPassword in
WriteABOTAPackageWithBrilloScript.

* Get the keys passwords in main and store them in OPTIONS.key_passwords.
This allows accessing them in WriteABOTAPackageWithBrilloScript and SignOutput
so it's only required to ask for the password once, while allowing to use
decrypted signing keys.

Test: ota_from_target_files.py on marlin and angler respectively.
Change-Id: I7c9b0198855a4b630c52b8552e904f312f09c4ce
2017-05-10 16:54:42 -07:00
Tianjie Xu
ce9d78fcd9 Add a script to parse an block-based ota package
The script prints the meta info, new/patch data size. It also simulates
the transfer commands and calculates amount of I/O as well as cache
needed.

We can add the more options to parse system/vendor.map so that we can
analyze the location change of files.

Bug: 31514709
Test: parse a bullhead incremental/full ota.
Change-Id: I70ed4f4c15e595b1851109e9799d44ac4e815c16
2017-05-09 21:01:19 +00:00
Julius D'souza
001c676b81 Add the option to reserve headroom for partition images.
This is useful for devices with low disk space with different
build variants.

Bug: 37469715
Test: Regular image builds successfully, errors occur when
the headroom size is greater than available partition space.

Change-Id: I526cdd0f84981bbd16e3afcfe1cd7fc43dce98ef
2017-05-08 11:59:25 -07:00
Tao Bao
39451582c4 releasetools: Add a verbose parameter to common.Run().
Caller can optionally specify the verbose flag which overrides
OPTIONS.verbose. The command line won't be outputed with verbose=False.
This is useful for cases that a) those command lines are less useful
(but will spam the output otherwise); b) sensitive info is part of the
invocation.

'verbose=False' will be consumed by common.Run() only, instead of being
passed to subprocess.Popen().

Test: ota_from_target_files.py on a block based OTA.
Change-Id: I7d5b4094d756a60f84f89c6a965e7ccc68e435f8
2017-05-04 11:18:56 -07:00
Tao Bao
21803d3574 Revert "Revert "Add compatibility metadata to OTA package""
This reverts commit b1e59b8a73.

This CL enables generating compatibility zip entry in an A/B OTA
package. We're not enabling the support for non-A/B OTAs for now until
they have the necessary support (e.g. /proc/config.gz exists).

Bug: 36810913
Test: Generate full and incremental packages for marlin and applly under
      normal boot and sideload respectively.

Change-Id: I69f0a3feb7a0efc7f727f92f31c9e63f47ee6530
(cherry picked from commit b63c952cc7)
2017-04-26 14:28:46 -07:00
Bowgo Tsai
9b3776017f AVB: support BOARD_BUILD_SYSTEM_ROOT_IMAGE
Passing --setup_rootfs_from_kernel to avbtool when
BOARD_BUILD_SYSTEM_ROOT_IMAGE is true

Bug: 33590159
Test: 'make' sailfish with BOARD_AVB_ENABLE := true
Test: 'make dist' with BOARD_AVB_ENABLE := true
Change-Id: Ieb58dd9ae6be1eceb90a33c739b85cff5cbc6e0a
2017-04-15 08:53:40 +08:00
Bowgo Tsai
8ee4a3db8c AVB: support signing vendor.img
Uses avbtool to sign vendor.img if BOARD_AVB_ENABLE is set.
It also allows appending additional arguments to avbtool via
BOARD_AVB_VENDOR_ADD_HASHTREE_FOOTER_ARGS.

e.g.,
  BOARD_AVB_ENABLE := true
  BOARD_AVB_VENDOR_ADD_HASHTREE_FOOTER_ARGS := --generate_fec

Bug: 35415839
Test: "make" with the above variables and use avbtool to check vbmeta is
      appended to vendor.img
Test: "make dist" with the above variables

Change-Id: I8ada38dff3def6d34613e77c67944def8a49f464
2017-04-15 08:53:06 +08:00
Tao Bao
afaa0a638b releasetools: Add validate_target_files.py.
Bug: 35408446
Test: validate_target_files.py on existing target_files zips.
Change-Id: I140ef86533eee5adb93c2546510fdd7e9ce4e81a
2017-04-05 09:04:13 -07:00
Tianjie Xu
a011dec9b2 Fix unittest for blockimgdiff
The test broke due to api change in constructor of Transfer.

Bug: 36456381
Test: blockimgdiff test passes
Change-Id: Ic73b66f8fb6141ff6f50c113b0d8cc55d34e0b5d
2017-03-24 16:28:16 -07:00
Tao Bao
5f8ff9319b Add a missing default value for OPTIONS.extracted_input.
Test: ota_from_target_files.py without --extracted_input_target_files.
Change-Id: Ieaba43fa557b544b24f3eeccd68ac289d1fe3d31
2017-03-21 22:36:20 -07:00
Dan Willemsen
cea5cd210e Speed up non-AB ota_from_target_files in the build system
We've just created the target files package and still have the extracted
directory, so pass that to ota_from_target_files so that it doesn't have
to re-extract what we just compressed.

This saves a little bit of time -- for bullhead-userdebug on internal
master, this brings the time to build an OTA from ~340s -> ~310s. Much
of the time is still spent generating and signing the OTA.

Test: lunch bullhead-userdebug; m -j otapackage
Test: bullhead-ota-*.zip is identical before/after
Change-Id: Ib51cf6b02123e9c49280e9957c6590645470a52c
2017-03-21 15:34:27 -07:00
Tao Bao
2f80e83e98 releasetools: Remove a dead line.
system_root_image has been dead since commit
2ce63edab7.

Test: pylint --rcfile=pylintrc add_img_to_target_files.py
Change-Id: Id791747b7313923b82279a21d264a998455a92d4
2017-03-16 11:59:24 -07:00
Tianjie Xu
b59c17fc8f Provide more info for bsdiff/imgdiff exceptions.
When bsdiff/imgdiff fails, dump the output along with the src/tgt
filenames and block ranges. Also, exit the script gracefully instead
of raising an exception about iterating over 'NoneType'.

Bug: 31381530
Test: Package generates successfully for angler, and error outputs
      correctly with error injection.

Change-Id: I06a2dfe545fbdff7043de05fee34b378453a9291
2017-03-14 18:51:21 -07:00
Tao Bao
e5b2ff2a97 Merge "releasetools: Fix the diff_worker progress report." 2017-03-13 19:12:26 +00:00
Tianjie Xu
36e612b753 Merge "Switch command to subprocess in build_image.py" 2017-03-13 18:23:08 +00:00
Tao Bao
33635b1f32 releasetools: Fix the diff_worker progress report.
Test: Observe the progress update during an incremental generation.
Change-Id: Ib0629a0fd9f925076fd20a040345c4f169133c30
2017-03-12 13:02:51 -07:00
Tianjie Xu
e3ad41bb2f Switch command to subprocess in build_image.py
Bug: 36012162
Test: image builds successfully for bullhead and sailfish

Change-Id: Iaa83034b39d392ff8a2154c7b32b21bf33ef552f
2017-03-10 15:16:24 -08:00
Tao Bao
c3e3406125 Merge "releasetools: Drop the support for BBOTA v1 and v2." 2017-03-10 19:25:06 +00:00
Tao Bao
39f3eaf221 releasetools: Fix the missing 'post-build' in full OTAs.
The line was unintentionally removed in commit 7f804ba.

Test: ota_from_target_files.py generates a full OTA. Check the package
      metadata.

Change-Id: Icae88e2a9bb2bfc450a3d0d7ab524d6a6eac9df5
2017-03-09 15:01:11 -08:00
Tao Bao
8fad03e771 releasetools: Drop the support for BBOTA v1 and v2.
BBOTA v1 and v2 (introduced in L and L MR1 respectively) don't support
resumable OTA. We shouldn't generate packages using v1/v2 at the risk of
bricking devices.

BBOTA v3 (since M) and v4 (since N) both support resumable OTAs. BBOTA
v4 additionally supports using FEC to possibly recover a corrupted
image.

Bug: 33694730
Test: Generate full and incremental OTAs w/ and w/o the CL. They should
      give identical packages (in v4).
Change-Id: Ib89d9cd63ba08e8e9aa4131bed18876b89d244c0
2017-03-09 12:25:46 -08:00
Tao Bao
f388104eaa releasetools: Remove dead functions in EdifyGenerator.
Remove the following functions that are needed for file-based OTAs only:
 - SetPermissions()
 - SetPermissionsRecursive()
 - MakeSymlinks()
 - DeleteFiles()
 - DeleteFilesIfNotMatching()
 - RenameFiles()
 - SkipNextActionIfTargetExists()

Bug: 35853185
Test: Verified there's no reference to these functions.
Change-Id: Iff24a9d705476211effaef28eed2a147fa5fcbce
2017-03-08 16:37:07 -08:00
Tao Bao
9bca43858f Merge "releasetools: Drop the support for file-based OTA generation." 2017-03-08 21:00:03 +00:00
Dan Willemsen
c619f99e2f Merge changes Ia5ce6870,I155654cd
* changes:
  Zip after add_img_to_target_files
  Allow add_img_to_target_files to work on a directory
2017-03-08 20:45:56 +00:00
Tao Bao
3b1f260205 Merge "releasetools: Reduce memory footprint for BBOTA generation." 2017-03-08 17:43:34 +00:00
Dan Willemsen
2ee00d5f66 Allow add_img_to_target_files to work on a directory
In addition to the current behavior of add_img_to_target_files working
on an existing zip file, allow passing in a directory where the target
files have already been extracted. When in this mode, it writes the
images out to that directory instead of the zip file.

This allows us to call add_img_to_target_files on the temp directory
used during the build to create the target files package, saving the
time and space of unzipping what we just zipped. This also allows us to
use the parallel soong_zip, which compresses the images much faster.

Test: aosp_marlin target_files zip is the same before/after this change
Test: marlin target_files zip is the same before/after this change
Test: bullhead target_files zip is the same before/after this change
Change-Id: I155654cdc7ad7754ba4ef76ec69c31f504a58198
2017-03-07 14:37:52 -08:00
Tao Bao
457cbf6a8a releasetools: Drop the support for file-based OTA generation.
We have stopped shipping devices with file-based OTAs, and are not
actively maintaining the support. Devices using file-based OTAs
should be moved to block-based, if not A/B OTAs.

We will also need to clean up EdifyGenerator class, which will be
handled in follow-up CLs.

Bug: 35853185
Test: Generate full and incremental OTAs w/ and w/o the CL, and they
      give identical packages.
Test: Not specifying --block also generates block-base OTAs.
Change-Id: I3b0fc8ce5600e109f3251fe41f655534aaa298c7
2017-03-07 00:17:58 -08:00
Tao Bao
7dc54e9761 Merge "releasetools: Remove the obsolete path in GetImage()." 2017-03-07 06:20:04 +00:00
Tao Bao
7e0f160f30 releasetools: Remove the obsolete path in GetImage().
We must have created the images (system.img, system.map etc) prior to
calling ota_from_target_files.py (added by commit
2e0d8fcf08, into Lollipop).

Remove the obsolete suppport for handling "old" target_files zips that
don't have such images. This simplies the logic for BuildSystem() and
BuildVendor(), which now would only be called by
add_img_to_target_files.py itself.

Test: Generating full and incremental OTAs give the same results, w/ and
      w/o this CL.
Change-Id: I0ea243d09d3378249d7982701ae4ec339b04b7b8
2017-03-06 16:22:44 -08:00
Tao Bao
ebce697429 releasetools: Fix two minor issues with _LoadOemDicts().
The first one in WriteVerifyPackage() is mismatching function parameters
that can be trivially fixed.

The other one is in WriteABOTAPackageWithBrilloScript(), where we don't
have edify script instance.

Test: `pylint --rcfile=pylintrc ota_from_target_files.py`.
Change-Id: Ie238ef5b296dfe9e725b61376992447b662d2376
2017-03-06 10:29:13 -08:00
Tao Bao
183e56e83d releasetools: Reduce memory footprint for BBOTA generation.
The major issue with the existing implementation is unnecessarily
holding too much data in memory, such as HashBlocks() which first reads
in *all* the data to a list before hashing. We can leverage generator
functions to stream such operations.

This CL makes the following changes to reduce the peak memory use.
 - Adding RangeSha1() and WriteRangeDataToFd() to Image classes. These
   functions perform the operations on-the-fly.
 - Caching the computed SHA-1 values for a Transfer instance.

As a result, this CL reduces the peak memory use by ~80% (e.g. reducing
from 5.85GB to 1.16GB for the same incremental, as shown by "Maximum
resident set size" from `/usr/bin/time -v`). It also effectively
improves the (package generation) performance by ~30%.

Bug: 35768998
Bug: 32312123
Test: Generating the same incremental w/ and w/o the CL give identical
      output packages.
Change-Id: Ia5c6314b41da73dd6fe1dbe2ca81bbd89b517cec
2017-03-06 08:46:30 -08:00
Tao Bao
6b0b2f9db0 Revert "Revert "releasetools: Use pattern filter when unzipping.""
This reverts commit a7316ce094.

This CL differs from the original CL by not unzipping RADIO/*. This is
because: a) AOSP targets don't have RADIO/ entries in the TF.zip; b)
we're not using the unzipped RADIO files (but reading them from the zip
files directly) - checked all the device-specific releasetools for
angler, bullhead, ryu, shamu, volantis, fugu, marlin and sailfish.

Test: `m dist` with AOSP targets (tested fugu and bullhead).
Change-Id: I4d0c67214ddd6202fc27c71bb79f52b5f4d40c64
2017-03-05 13:11:42 -08:00
Tao Bao
0c3e31c69b Merge "Revert "releasetools: Use pattern filter when unzipping."" 2017-03-05 17:44:58 +00:00
Tao Bao
a7316ce094 Revert "releasetools: Use pattern filter when unzipping."
This reverts commit aa3a04f19d.

Reason for revert: Some AOSP targets don't include RADIO/ in the TF.zip. We may possibly skip unzipping RADIO/, or by always creating a dummy RADIO folder in TF.zip. Revert this CL for now.

Change-Id: I8e90d322706a4fe82129bdfab5ffa1eab989c648
2017-03-05 17:43:33 +00:00
Tao Bao
cc63bc3a91 Merge "Change the default parameter values in common.MakeTempFile()." 2017-03-05 17:00:38 +00:00
Tao Bao
aa3a04f19d releasetools: Use pattern filter when unzipping.
When building BBOTAs, it only needs *some* unzipped entries in the given
target_files zip(s). In particular, it needs 'IMAGES/*', 'META/*',
'RADIO/*'. (It also reads 'SYSTEM/build.prop' and 'OTA/bin/updater', but
directly from the zip file.)

This CL specifies the entries to unzip. It saves the I/O cost, as well as
the temporary storage.

Test: ota_from_target_files.py gives the same package w/ and w/o the CL.
Test: check_target_files_signatures.py still works.
Change-Id: I728428aa0e138879e49f9efbdb46a85892fc7038
2017-03-03 16:38:17 -08:00
Tianjie Xu
d9880a0246 Merge "Do not read the block device when generating the care map" 2017-03-02 21:36:12 +00:00
Treehugger Robot
afc0ea14d8 Merge "releasetools: Drop the support for fstab_version 1." 2017-03-02 19:19:09 +00:00
Tianjie Xu
955629b4aa Do not read the block device when generating the care map
This remove the fstab dependency when building the OTA package for
marlin/sailfish.

Bug: 35811655
Test: OTA package builds successfully for sailfish.
Change-Id: If223d11dddca396c47262042c576f9e7d0cb5b33
(cherry picked from commit 7d051adc3b)
2017-03-02 11:07:12 -08:00
Tao Bao
d1de6f326e releasetools: Drop the support for fstab_version 1.
fstab_version is defined by RECOVERY_FSTAB_VERSION in
bootable/recovery. We have moved to fstab_version 2 since commit
f35d1cef7c19db975a1295e8c23c7fb8bd2489f9 (landed into JB MR2).

Drop the support for fstab_version 1, since we won't run the latest OTA
script over a JB target_files zip.

Test: No impact on building full/incremental OTAs.
Change-Id: Ia87c4e7da6c5e71ce0908fca2e4f1ad1c06ba869
2017-03-01 16:56:45 -08:00
Tao Bao
e889893776 Merge "releasetools: Add support for --override_timestamp." 2017-03-02 00:00:59 +00:00
Tao Bao
9beea2a4a3 releasetools: Merge two identical conditional branches.
The merged two branches have become identical since commit
fc3422ad36 (landed into Nougat).

Test: Get identical incremental packages w/ and w/o the CL.
Change-Id: Id1183f8ed83f684a0dac1a4af87b6e075b08aabc
2017-02-28 20:20:27 -08:00
Tao Bao
3e6161a3b3 releasetools: Add support for --override_timestamp.
We use the timestamps in builds to determine a downgrade, which might
not be always the truth. For examples, two builds cut from different
branches may carry timestamps in a reverse order. An incremental package
won't be able to be pushed nor applied, based on the timestamp
comparison.

We used to handle such a case with manual work, by setting the
post-timestamp to (pre-timestamp + 1) in the package metadata. This CL
automates the process by adding a new flag --override_timestamp.

Note that it doesn't change anything in the installed image, but only
affects the assertions for pushing / installing the package.

With the change in this CL:
 - If it's a downgrade without any extra flag, fail the package
   generation (we only print warnings prior to this CL);
 - If it's a downgrade with --downgrade flag, generate a downgrade
   package with forced data wipe (same as before);
 - If it's a downgrade with --override_timestamp, generate a normal
   incremental with hacked timestamp (pre-timestamp + 1) (new in this CL
   to avoid the manual change);
 - If it's not a downgrade but with any of the above two flags specified,
   fail the package generation.

Bug: 33744169
Test: Generate an incremental from builds with reversed timestamps.
Change-Id: I8b187d32708b4a7c3e20f8c6adb8f9527b73b965
2017-02-28 13:01:13 -08:00
Tao Bao
6cd54739a4 Remove the backwards compatibility support for misc_info.txt.
We introduced META/misc_info.txt to hold the misc info since Gingerbread
(commit 37974731fc). Remove the backwards
compatibility support for building pre-G TF zips.

Test: `m dist` works.
Change-Id: Ibff7aaf69cc7e460634c049d11a004f7196f8f73
2017-02-27 17:00:13 -08:00
Tao Bao
e8c68a03c8 releasetools: Fix an equality check bug when asserting stash size.
Otherwise the comparison is inconsistent between ReviseStashSize() and
WriteTransfers().

Bug: 35775675
Test: Successfully generate a previously failed incremental.
Change-Id: I554a51a210bf322cb5c79e28cf85607a417b094a
2017-02-26 10:58:22 -08:00
Alain Vongsouvanh
7f804ba71f releasetools: allow for multiple OEM property values.
This CL changes the --oem_settings flag to allow a comma seperated list of
property files. All property values will be used when asserting properties such
as ro.product.name.

For example, if two property files are provided with ro.product.name values of
"sprout" and "sprout_a", the resulting otapackage will check that the device's
ro.product.name property matches at least one of them.

Bug: 34191373
Test: manual

Change-Id: I954673511be8f0929982235cc9cbfbd85a9ee1f4
2017-02-21 16:09:34 +00:00
Tao Bao
4f104d1b94 releasetools: Allow using futility-host to sign old TF.zip.
Commit e98fb7a8d3 switched to using
futility-host instead of the prebuilt futility. This CL adds support to
handle signing old TF.zip that still says "futility=prebuilt/..." in
META/misc_info.txt.

Bug: 35467608
Test: Generate otatools.zip and sign an old ryu TF.zip.
Change-Id: I48a9cc918c7afce361e1ec9bc4f85f74fa92566e
2017-02-17 23:27:44 -08:00
Tao Bao
3c5a16d49c releasetools: Fix a bug in ReviseStashSize().
We check the needed stash size in ReviseStashSize(), and may not
generate a stash command if it would exceed the max allowed size. This
CL fixes a bug when skipping a stash operation: we shouldn't update the
'stashes' map if a stash command won't be generated.

Bug: 35313668
Test: Successfully generate the package that was failing due to the bug.
Change-Id: If0a3a5fadda9b4a4edad66a2a5826b5f978400ae
2017-02-13 12:31:42 -08:00
Tao Bao
b31892e5de releasetools: Support generating downgrade A/B OTAs.
We already support generating downgrade OTAs for non-A/B devices (with
mandatory data wipe), but we have missed the --downgrade flag in A/B OTA
path.

This CL factors out the function that writes the downgrade metadata, and
fixes the path for generating A/B OTAs.

Bug: 35094540
Test: Generate incrementals with --downgrade for A/B and non-A/B OTAs.
Change-Id: I30b9bf83e69e8aba3be666507681b555db6ab743
2017-02-07 12:25:59 -08:00
Tao Bao
b09fef9158 Merge "releasetools: Pack the offset/length for metadata." 2017-02-07 04:16:20 +00:00
Tao Bao
82bdf6c1d7 releasetools: Fix the bug when generating care_map.txt.
Commit f1a13180db intended to remove the
verity blocks from care_map.txt, but it added new code without removing
the old one. This leads to a malformed care_map.txt and causes
update_verifier failure.

Bug: 34391662
Test: 'm -j dist' gives a TF.zip with 4-line META/care_map.txt (as
      opposed to a 6-line file).

Change-Id: I7ff1aa525795c4b049af54c1755b0f0ea84f7e0e
2017-02-05 14:31:44 -08:00
Tao Bao
2dd1c48377 releasetools: Pack the offset/length for metadata.
For streaming OTAs, we will also need the info in the metadata entry
(META-INF/com/android/metadata). Compute and pack its offset/length
values into 'ota-streaming-property-files'.

Bug: 34986195
Test: Create an OTA package and check the offset/length values.

Change-Id: Id150700f2bc9bff02467cda9fe8927c8a374412a
2017-02-03 16:50:25 -08:00
Tianjie Xu
3a79c6d784 Merge "Remove verity blocks from care map" 2017-02-03 21:50:07 +00:00
Tao Bao
bfdcb1257c releasetools: Prepend 'ota-' to 'streaming-property-files'.
'streaming-property-files' is a property related to the OTA package
itself. Prepend 'ota-' to make it consistent with others like
'ota-type' and 'ota-required-cache'.

Bug: 34852392
Test: Generate an A/B OTA package and check METADATA entry.
Change-Id: Ia681e6e19ff509e6da0d8718933b42aac997e1cf
2017-01-31 15:53:20 -08:00
Tianjie Xu
f1a13180db Remove verity blocks from care map
When reading /dev/block/dm-X, update_verifier isn't able to access the
verity meta blocks at the end of the system/vendor partition. So we need
to remove these block ranges from the care_map.

Bug: 34391662
Test: care_map generated successfully without verity meta blocks
Change-Id: Id57c602b7e5fd1b0c9d1e1fe5fcdd74e85b6b255
2017-01-25 23:01:29 -08:00
Tao Bao
c96316c89b Revert "Revert "releasetools: Generate streamable A/B OTA packages.""
This reverts commit ea4325baf8 to re-land
commit ef1bb4360f. It fixes the bug when
handling a package without care_map.txt (e.g. dm-verity not enabled).

In order to support streaming A/B OTA packages, we pack
payload_properties.txt and care_map.txt in ZIP_STORED mode. These two
entries along with payload.bin (already in ZIP_STORED prior to this CL)
can be fetched directly based on the offset and length info.

We write the offset and length info into the package metadata entry
(META-INF/com/android/metadata), which can be parsed by the OTA server.

payload_properties.txt and care_map.txt are usually less than 1-KiB. So
the change only incurs marginal size increase.

Bug: 33382114
Test: Generate an A/B OTA package. Verify the 'streaming-property-files'
      entry in the metadata file.
Test: Generate an A/B OTA package on a device with dm-verity not enabled.

Change-Id: I3469c8b62385a1fc58b4fb82e3f9d4690aef52ba
2017-01-24 22:27:02 -08:00
Tao Bao
ea4325baf8 Revert "releasetools: Generate streamable A/B OTA packages."
This reverts commit ef1bb4360f.

It has broken the builds that don't have care_map.txt entry.

Change-Id: I343455e66be5e79457968dfc2813c7c1c234b6dc
2017-01-25 01:41:13 +00:00
Tao Bao
ef1bb4360f releasetools: Generate streamable A/B OTA packages.
In order to support streaming A/B OTA packages, we pack
payload_properties.txt and care_map.txt in ZIP_STORED mode. These two
entries along with payload.bin (already in ZIP_STORED prior to this CL)
can be fetched directly based on the offset and length info.

We write the offset and length info into the package metadata entry
(META-INF/com/android/metadata), which can be parsed by the OTA server.

payload_properties.txt and care_map.txt are usually less than 1-KiB. So
the change only incurs marginal size increase.

Bug: 33382114
Test: Generate an A/B OTA package. Verify the 'streaming-property-files'
      entry in the metadata file.

Change-Id: I04504e834eb36e18876c5f5a5a09289ee05c6f9a
2017-01-19 22:25:12 -08:00
Connor O'Brien
cf4f7eac85 Merge "Pass flash erase & logical block size to mke2fs" 2017-01-17 20:07:11 +00:00
Tao Bao
4a0b494eea Merge "releasetools: Replace print stmt with print()." 2017-01-14 15:36:41 +00:00
Tao Bao
89fbb0f6d5 releasetools: Replace print stmt with print().
So that it's compatible with Python 3.

Test: pylint --pylint=pylintrc

Change-Id: If06c135a492c94bedd713c8cbdf03155a502d5cd
2017-01-13 14:55:14 -08:00
Tao Bao
2a0d1da4d9 releasetools: Delete the obsolete --aslr_mode/-a flag.
It was added in commit 96be7205dc
("Working ASLR implementation.") in 2010, and removed in commit
1807e700a5 ("don't generate retouch
commands in OTA scripts") in 2012.

Remove the obsolete --aslr_mode flag.

Test: ota_from_target_files.py still works (by generating incremental
      and full OTAs respectively).

Change-Id: I6d8e62730ac192f3574d484c4a4b9b43b4ee0a9e
2017-01-13 22:46:27 +00:00
Connor O'Brien
20f08c3e16 Pass flash erase & logical block size to mke2fs
This information can be used to tune ext4 stripe and stride in the
userdata partition for better performance

Test: Build & flash userdata, confirm correct stripe & stride values
Bug: 33243520
Merged-In: Ia97cdd2d0239c3484b895fce49299f692ef911d8
Change-Id: Ia97cdd2d0239c3484b895fce49299f692ef911d8
Signed-off-by: Connor O'Brien <connoro@google.com>
2017-01-13 12:17:05 -08:00
Tao Bao
3a2e350239 blockimgdiff.py: Clean up stash id computation in BBOTA v3+.
Only BBOTA v2 needs to maintain a pool of available 'stash slot id'.
BBOTA v3+ uses the hash of the stashed blocks as the slot id, which
doesn't need the id pool anymore.

Bug: 33694544
Test: Generate v2 and v4 incrementals w/ and w/o the CL. They produce
      the same packages respectively.
Change-Id: I8121af5b6b1bee98c3639d54a00b06fd12e378e8
2016-12-28 09:26:46 -08:00
Tao Bao
304ee27e88 blockimgdiff.py: Add a test case for ReviseStashSize().
Bug: 33687949
Test: python -m unittest test_blockimgdiff
Change-Id: I31e42648d2c0e60b941908c098ce5ccc967861ca
2016-12-19 11:17:12 -08:00
Tao Bao
e27acfdc41 releasetools: Fix the computation in ReviseStashSize().
We compute the max stashed_blocks in ReviseStashSize(), prior to calling
WriteTransfers(), to avoid running out of space due to stashing.

There is a bug when computing the to-be-freed stashed blocks, where we
wrongly free the space _before_ executing the transfer command. This leads
to a script failure where the max stash size violates the max allowed
size in WriteTransfers().

Note that this bug doesn't affect already generated packages. It's only
an underestimate in ReviseStashSize(). The check in WriteTransfers() has
been correct to ensure the max stash size.

Bug: 33687949
Test: Successfully generated incremental OTA which failed previously.
Change-Id: I4f4f043c6f521fce81ca5286e6156f22d99bf7f7
2016-12-16 11:26:05 -08:00
Tao Bao
f9023856ad releasetools: Add prefix when dumping fingerprints.
We used to dump "Source: <fingerprint>" in update logs. The "Source: "
prefix was unintentionally dropped out.

Test: Check the generated incremental BBOTA script.
Change-Id: I4de62333aa38e3fb09a76df0e769b62af48e0313
2016-12-14 11:53:38 -08:00
Patrick Tjin
a1900841bb build_image: add support for specifying number of inodes
Add support for specifying number of inodes when creating
system, vendor, oem partitions.  These are all read-only
and have no use for extra inodes.  Removing extra inodes
saves a lot of space.

Bug: 32246383
Change-Id: I13f1d4614b64a4abc752c42a1c65d3d151481c21
(cherry picked from commit b59eca3586)
2016-12-06 21:07:23 -08:00
The Android Automerger
ada51f3b8f Manually merge commit 'c13da91' into stage-aosp-master
* commit 'c13da91':
  Add support for creating ext4 images with mke2fs

BUG:33304034
2016-12-03 04:02:09 +00:00
Treehugger Robot
c13da91023 Merge "Add support for creating ext4 images with mke2fs" 2016-12-02 22:48:23 +00:00
Tao Bao
d07f9995b8 Merge "Build recovery-two-step.img for two-step OTAs."
am: 06e14e438f

Change-Id: I9c5d6e5f71da88bc0f662373335b024a55ca193e
2016-12-02 15:19:00 +00:00
Tao Bao
d42e97ebb4 Build recovery-two-step.img for two-step OTAs.
In two-step OTAs, we write recovery image to /boot as the first step so
that we can reboot from there and install a new recovery image to
/recovery. However, bootloader will show "Your device is corrupt"
message when booting /boot with the recovery image. Because the recovery
image encodes the path of "/recovery" as part of the signature metadata,
which fails the verified boot.

This CL generates a special "recovery-two-step.img" in addition to the
regular recovery.img. This image encodes "/boot" when being signed,
which will be flashed to /boot at stage 1/3 in a two-step OTA.

Here are the desired changes:

- 'IMAGES/recovery-two-step.img' exists in target_files.zip for non-A/B
targets (e.g. bullhead). The image should not exist for targets that
don't have a recovery partition (e.g. A/B devices like sailfish).

- <device>-img.zip should not contain 'recovery-two-step.img'.

- Nothing should change when building non-two-step OTAs. For two-step
OTAs, 'recovery-two-step.img' should be included in the OTA package;
'updater-script' should flash this image to /boot at stage 1/3.

- When building a two-step OTA with an input TF.zip that doesn't have
  IMAGES/recovery-two-step.img, it should use the existing
  IMAGES/recovery.img instead.

Bug: 32986477
Test: Tested the steps above on bullhead and sailfish.
Change-Id: I34e6c599bcf2011d4cd5c926999418b3975d6d0f
2016-12-01 17:47:59 -08:00
Tao Bao
2e3f1c5258 Merge "releasetools: Remove dead parameter in GetBootableImage()."
am: 8df4fea85f

Change-Id: I192af3927849d0d8f995f0002b8b8f2e191b794f
2016-12-01 17:09:07 +00:00
Tao Bao
2a4336bc04 releasetools: Remove dead parameter in GetBootableImage().
The 'system_img_path' parameter was introduced in commit
d995f4b04d, but became obsolete since
commit 2ce63edab7.

Test: m dist
Change-Id: Iffd496d929db5cc3dfc955a48bfc1b1317bd012f
2016-12-01 00:08:27 -08:00
Adrien Schildknecht
9a072cc0b0 Add support for creating ext4 images with mke2fs
We are investigating replacing make_ext4fs with the upstream tool mke2fs.
To mitigate the trouble that may arise if the new tool behave differently
compared to the old one, there will be a transition period.

Devices that want to use the new way of creating ext4 images can set the
variable "TARGET_USES_MKE2FS" to true in their BoardConfig.mk
By default, the build system will choose the old tool 'make_ext4fs'.

Test: m otapackage with TARGET_USES_MKE2FS={,false,true}
Change-Id: I282bcb9efe335a86c53986283090ca947d65c7f8
2016-11-18 17:06:29 -08:00
Tao Bao
dc04f93e39 Merge "releasetools: Use java_path in sign_target_files_apks.py."
am: 4a71d500c1

Change-Id: I61400574ac3f7d29906ca571f497554631ecd72d
2016-11-08 23:42:26 +00:00
Tao Bao
e95540e060 releasetools: Use java_path in sign_target_files_apks.py.
Prior to this CL, it was calling the hard-coded "java" although it was
accepting a "--java_path" option.

Also switch OPTIONS.java_args from string to list. Otherwise it won't
work when providing multiple args.

Bug: 32737832
Test: Specify "--java_path=" and "--java_args" when invoking
      sign_target_files_apks.py with "-v". Check the commands being
      called.
Change-Id: Id7ef98e778646d532027434de7fba9b7a104dbd0
2016-11-08 12:48:43 -08:00
Tao Bao
9242e1f750 Merge "releasetools: compare size between compressed and patch while build otapackage"
am: 49758959f2

Change-Id: I17b988878cc7ffdfbe490f404c9cd5ab996a8c5f
2016-11-05 00:13:55 +00:00
Tao Bao
49758959f2 Merge "releasetools: compare size between compressed and patch while build otapackage" 2016-11-05 00:09:35 +00:00
Tao Bao
e3554167e2 Merge "releasetools: Make BBOTA incremental generation repeatable."
am: b9bd78d110

Change-Id: I6402481bf300b52d36df1570cceeae6ec3ff4f8f
2016-11-04 21:29:15 +00:00
Tao Bao
3348228423 releasetools: Make BBOTA incremental generation repeatable.
set() doesn't keep elements according to the order of insertion. So
Transfers managed with set() in intermediate steps may not appear in the
same order across runs. This leads to slightly different output packages
when generating the same incremental OTA.

This CL fixes the issue by replacing set() with OrderedDict() in
blockimgdiff.GenerateDigraph() and blockimgdiff.FindVertexSequence().

It also adds a testcase that ensures blockimgdiff.GenerateDigraph()
preserves the insertion order for Transfer.goes_after set.

Bug: 32220816
Test: ota_from_target_files.py gives identical package when running
multiple times.

Change-Id: I56d551e5ca926993ab46896e33c80e0ce42e506a
2016-10-26 09:40:50 -07:00
Tao Bao
f4a88279ed Merge "releasetools: Support verity signer args."
am: 02d2d68aec

Change-Id: Ib1aed8ff11eeef2fd64a6520e92cfb30678dadc7
2016-10-24 18:53:33 +00:00
YOUNG HO CHA
ccc5c40523 releasetools: compare size between compressed and patch while build otapackage
Currently, whether contains patch or verbatim, compute with file size
and patch size.
But ota file must be compressed with zip, so it should be better with
compressed size than uncompressed.

Test: aosp_shamu-user build without proprietary blobs between MOB30P and NRD90S
$ du -k ota_shamu_old.zip ota_shamu_new.zip
217252	ota_shamu_old.zip
216520	ota_shamu_new.zip

Change-Id: If68cb1fbe2f7815067451915a0dcfe93ea5ba8d6
Signed-off-by: YOUNG HO CHA <ganadist@gmail.com>
2016-10-18 17:27:09 +09:00
Tao Bao
458104266f releasetools: Support verity signer args.
system/extras/verity/build_verity_metadata.py now accepts
"--signer_args" to specify verity signer args.

Also remove the duplicate "--verity_signer_args" in
add_img_to_target_files.py, as we already have that in common.py.

Bug: 31500665
Test: Building and signing work w/ and w/o --signer_args.

Change-Id: I02f59c50a1ebf15c5505e9fffd5b9bbbbaa785be
2016-10-17 16:25:22 -07:00
David Zeuthen
4e9c89ab87 resolve merge conflicts of 86180eb to nyc-mr1-dev-plus-aosp
Change-Id: Ia4bfb2571bac509cebe22f70ff43e6696f832423
2016-10-04 18:53:34 -04:00
David Zeuthen
70209ad9f0 Merge "Make room for AVB hashtree and metadata." 2016-10-04 22:32:27 +00:00
David Zeuthen
4014a9daa7 Make room for AVB hashtree and metadata.
While the system.img images currently built with AVB support verify
correctly, mounting the filesystem content fails. This is because
'avbtool add_hashtree_footer' used to claim some of the unused /
DONT_CARE space for stashing the verity tables and this resulting in the
mapped device ending up being smaller causing the mount failure.

Fix this by leaving enough room for AVB hashtree and metadata before
building the image. This is achieved by moving the AVB hashtree support
into build_image.py and using a just added '--calc_max_image_size'
option to 'avbtool add_hashtree_footer' to figure out how much space to
leave out.

This depends on https://android-review.googlesource.com/#/c/281821/

Bug: 31264226
Test: Mounting dm-verity set up from system.img now works.

Merged-In: I4c5de1004c1059f8c582e76b3b8517d427aa1a87

Change-Id: I945a5f1f6782791736cd319f216cfa6b448fb04d
2016-10-04 17:32:16 -04:00
Tao Bao
658af49662 Merge "releasetools: Clean up sign_target_files_apks.py." am: ad49deb5e6 am: 4299e48f0c
am: dfc210c9a4

Change-Id: Iee5cda2c10bc9c474140bf1f303261d8cf31d7b8
2016-10-01 00:24:10 +00:00