Commit graph

5240 commits

Author SHA1 Message Date
Tao Bao
bea20ac722 releasetools: Refactor AddImagesToTargetFiles().
Separate three functions out of AddImagesToTargetFiles(), into
AddCareMapTxtForAbOta(), AddRadioImagesForAbOta() and
AddPackRadioImages() respectively. This CL tries to apply minimal
changes for the refactoring purpose.

Also add tests for AddRadioImagesForAbOta() and AddPackRadioImages().
The tests for AddCareMapTxtForAbOta() require better testing support to
mock sparse_img.SparseImage, which will be added in later CLs.

Test: python -m unittest test_add_img_to_target_files
Test: `m dist` with aosp_marlin-userdebug. Check META/care_map.txt in
      the generated target_files.zip.
Change-Id: I1bb723c15237ff721f165cfce0ce996008ce9948
2018-01-18 23:37:23 -08:00
Treehugger Robot
13f228ebde Merge "releasetools: Remove the reloading of target info dict." 2018-01-18 22:48:15 +00:00
Andreas Gampe
7f494c121f Warn: Change GuardedBy regular expressions
This was modified in commits 20a8664 "Change canonical name for
GuardedByChecker to GuardedBy" and 1f6dc3dc "Merge GuardedByValidator
into GuardedByChecker."

Test: m
Test: manual
Change-Id: I07305e59694a955e52bd69c1cbe53346808b78aa
2018-01-17 21:28:21 -08:00
Tao Bao
d32e78fd3c releasetools: Add a note for the ZipInfo._decodeExtra() workaround.
Test: N/A
Change-Id: I9e84fc27230182f17c2fda0df57a2c149281dcda
2018-01-17 10:10:00 -08:00
Tao Bao
df3a48b43e releasetools: Consolidate the codes that generate metadata dict.
Now they (A/B, non-A/B full and non-A/B incremental) share the same
code. Also add tests for the newly added function.

Test: python -m unittest test_ota_from_target_files
Test: Build A/B and non-A/B packages respectively.
Change-Id: I5499dd9c683ac1a18d8a24f21ff8845fe69f343e
2018-01-16 15:34:27 -08:00
Tianjie Xu
a7229a08b7 Merge "Iterate through the sorted file map when finding transfers" 2018-01-13 00:01:03 +00:00
Tianjie Xu
20a86cdd8d Iterate through the sorted file map when finding transfers
This helps to generate a deterministic package.

Bug: 71770360
Test: Generate a incremental package and transfers are added by file name.
Change-Id: I7562a200b97a1babbb09a77801324cc9408cc01f
2018-01-12 12:21:00 -08:00
Tao Bao
2db1385db5 releasetools: Remove the reloading of target info dict.
In non-A/B OTA path, we've been loading the info dict for the target
build twice (once to have a peek at the OTA type, with a reload after
unzipping the target-files zip). Remove the reloading.

This CL moves the loading of the source info dict up a bit to avoid
having two copies in the A/B and non-A/B paths. It also moves up the
sanity checks of the loaded info dicts before unzipping the target-files
zips.

Test: Generate full and incremental packages for bullhead and marlin.
Change-Id: Iccb953feb0b6ecf62e6f73d6e9ceea00077d098d
2018-01-11 21:59:30 -08:00
Treehugger Robot
896fa043bc Merge "releasetools: Remove the fallback-to-full logic." 2018-01-11 22:24:36 +00:00
Treehugger Robot
155a140c82 Merge "releasetools: Remove two dead functions in ota_from_target_files.py." 2018-01-11 22:23:18 +00:00
Tianjie Xu
c56071b21b Merge "Sort the split transfers to generate a determinate package" 2018-01-11 04:15:28 +00:00
Tao Bao
175df38e6c releasetools: Remove two dead functions in ota_from_target_files.py.
GetBuildProp() and GetVendorBuildProp() have been factored into
BuildInfo class. These two functions were meant to be deleted in commit
481bab8d48.

Test: python -m unittest test_ota_from_target_files
Test: No users of these two functions.
Change-Id: Id590a46d597939973ef2339cdfea4f5163548d4d
2018-01-10 19:57:57 -08:00
Tao Bao
32b80dca9f releasetools: Remove the fallback-to-full logic.
The fallback-to-full logic doesn't look beneficial, but may hide away
real bugs, especially when it's on by default.

Test: Generate an incremental non-A/B OTA successfully.
Test: Inject bugs into script; observe that incremental OTA
      generation errors out instead of silently falling back.
Test: No internal users of the flag from code search.
Change-Id: Ifc627d898552c0bd72463e8c107e36610421608e
2018-01-10 14:03:16 -08:00
Tao Bao
b54f6157a4 Merge "releasetools: Add a BuildInfo class." 2018-01-10 20:37:41 +00:00
Tianjie Xu
d3bf67e1d3 Sort the split transfers to generate a determinate package
We split large apks and generated patches for them in parallel,
resulting in nondeterminate packages between different runs. This CL
sort the split transfers by target name first; and then add them
sequentially to the final transfer list.

Also fix a side effect where we may generate a wrong sha1 for split
ranges due to synchronization error.

Bug: 71770360
Bug: 71759418

Test: Generate the package several times, compare the log and the transfer list.
Change-Id: I2a49e22594d59ffaa98b11edc776be4e3c4c561f
2018-01-10 12:23:57 -08:00
Tao Bao
481bab8d48 releasetools: Add a BuildInfo class.
Prior to this CL, we have similar codes in different paths
(WriteFullOTAPackage / WriteBlockIncrementalOTAPackage /
WriteABOTAPackageWithBrilloScript). This CL factors out the common codes
that deal with OEM-specific properties into BuildInfo class, and adds
tests for the new class.

Test: python -m unittest test_ota_from_target_files
Test: Create an incremental package that uses and doesn't use OEM
      properties respectively.
Test: Create a full package that uses and doesn't use OEM properties
      respectively.
Change-Id: Ida914cace12803d55396fa503bafcac2db2a520e
2018-01-09 15:08:23 -08:00
Elliott Hughes
1a701d93f9 adbs was broken for a year and no-one noticed.
Time to remove it.

Bug: https://b/15270043
Test: N/A
Change-Id: Idb56fc99c213732eca75f81ddd9d7a8a19374b42
2018-01-08 13:55:00 -08:00
Treehugger Robot
e3692095a1 Merge "releasetools: Add tests for common.ReadApkCerts()." 2018-01-05 23:30:53 +00:00
Tao Bao
818ddf5e7e releasetools: Add tests for common.ReadApkCerts().
Test: python -m unittest test_common
Test: Run sign_target_files_apks.py on a target with compressed APKs.
Change-Id: I107a8b8f2f0f82e2d1947f14c8a8b3778f633b11
2018-01-05 11:53:55 -08:00
Colin Cross
96029a1276 Merge changes I9fda7fe3,I0a34b2b4,Ie64a283d
* changes:
  Clear resource_export_package outside conditionals
  Fix adbs after move from build/tools to build/make/tools
  Use proguard.jacoco.flags for EMMA_INSTRUMENT_STATIC
2018-01-05 06:22:35 +00:00
Colin Cross
d31ff8ff32 Fix adbs after move from build/tools to build/make/tools
adbs broke when build/tools was moved to build/make/tools in
aosp/286781.

Bug: 15270043
Test: build/tools/adbs
Change-Id: I0a34b2b446f996bcc53a7dde6381402f8eed9bd0
2018-01-03 14:21:35 -08:00
Tao Bao
c72727ac3f releasetools: Clean up build_image.py.
Mostly cosmetic changes, such as moving print statement to print
function. The only functional change is to wrap the file opening in
Append() with try statement, which would dump the error message as
needed.

Test: `m dist`
Test: python -m unittest test_build_image
Change-Id: I4e6c593517cf737bfcbe51bc533b22a247b10e44
2018-01-03 11:52:28 -08:00
Tao Bao
d8a953d7ab releasetools: Move assertions into CheckHeadroom().
Assert the input arguments inside CheckHeadroom() function, and add
matching tests. Also use common.MakeTemp{File,Dir} in tests.

Test: `m dist`
Test: python -m unittest test_build_image
Change-Id: Ib45f24651af9591a9cfdd72aac3558f6c9a80f44
2018-01-03 11:52:16 -08:00
Tao Bao
e838d1446c releasetools: Clean up ReplaceVerityKeyId and add tests.
Test: python -m unittest test_sign_target_files_apks
Test: Run sign_target_files_apks.py on marlin target_files.zip.
Change-Id: Ic3c3f4f14c73f8f8e48a8341e024e0861e665989
2018-01-02 12:29:31 -08:00
Tao Bao
1c830bfbaa releasetools: Add common.MakeTempDir().
This simplifies the work to track mkdtemp()'d dirs. This CL also cleans
up the temporary file creation in build_image.py.

Test: `m dist`
Test: python -m unittest test_common
Change-Id: Ied7d86126714b9b26e5df7bcca1dfdea21cfcf60
2018-01-02 12:22:01 -08:00
Treehugger Robot
7d100ce03b Merge "releasetools: Clean up sign_target_files_apks.py." 2018-01-02 19:56:03 +00:00
Tao Bao
0c28d2d02d releasetools: Clean up sign_target_files_apks.py.
Mostly cosmetic changes, such as replacing print statement with print
function. Also change 'import cStringIO' to optionally look for the one
in io module, to allow Python 2/3 compatibility.

Test: pylint --rcfile=pylintrc sign_target_files_apks.py
Test: Run sign_target_files_apks.py on marlin target_files.zip.
Change-Id: I4dc98b01da6f89e624114bbca5522f659901c1f2
2018-01-02 09:57:29 -08:00
Tao Bao
186ec99eb9 releasetools: Fix a bug in blockimgdiff.HeapItem.
HeapItem defines __bool__(), which contains a logical error that should
return the opposite value.

Note that the bug only manifests while using Python 3, which calls
__bool__(). With Python 2, `if x:` or bool(x) actually calls
x.__nonzero__() or x.__len__(). If a class defines neither __len__() nor
__nonzero__(), as the case in HeapItem, it always returns True.

Test: python -m unittest test_blockimgdiff
Test: python3 -m unittest test_blockimgdiff
Test: Generate an incremental non-A/B OTA package successfully.
Change-Id: Ibe8430e0b495a7d2f430cfffb716d2536ffb53d2
2018-01-02 09:54:32 -08:00
Dan Shi
9a50168c85 Test config should use manifest package attribute
Bug: 70981774,69929803
Test: m -j SetupWizardActivityTests
Change-Id: I6d50744504eac71d6ecae5db9b14e6d30eb923fe
2017-12-22 13:34:50 -08:00
Tao Bao
be8ea6ec11 Merge "releasetools: Remove the dead code for blockimgdiff_version <= 2." 2017-12-21 22:20:32 +00:00
Dan Shi
3c28ff7bff Merge "Auto generate test config files" 2017-12-21 21:53:12 +00:00
Tao Bao
0582cb639f releasetools: Remove the dead code for blockimgdiff_version <= 2.
The code is on infeasible path since we already have assertions in
common.BlockDifference().

Also remove the dead code that checks for OPTIONS.info_dict, as we
already set that in ota_from_target_files.main(), for both of A/B and
non-A/B.

Test: Generate incremental OTAs w/ and w/o the CL, and get identical
      packages.
Change-Id: Ifb8fc101e78f5ce58c60c8e49028b66ce0d20246
2017-12-21 11:53:27 -08:00
Tao Bao
2f5358d38b releasetools: Fix the bug that skips loading OEM dict.
The CL in [1] unintentionally breaks the OEM dict loading logic in the
incremental BBOTA path. We should always require and load the OEM
property dict if _either_ of the source and target builds uses OEM
properties. Otherwise with the current "and" operator, it skips loading
the OEM property dict and thus fails to generate an OTA package that has
OEM property changes (e.g. updating from build with fingerprint to
another one using thumbprint).

The CL in [1] actually makes the right change in the file-based OTA
path, but introduces the bug in the block-based OTA path.

This CL also cleans up the line that reads recovery_mount_options.

[1] commit 7f804ba71f ("releasetools:
allow for multiple OEM property values.").

Test: Genearte an OTA that has OEM property changes successfully.
Change-Id: Idce4ad59825d432618535ce09ab22bd7ddc524f2
2017-12-20 23:06:02 -08:00
Dan Shi
efb892d858 Auto generate test config files
This change allows one to run atest <module name>. The TradeFed test
config file will be automatically created in testcases directory based
on the settings.

Bug: 69929803, 69932044
Test: m -j NotificationStressTests libminijail_test
add following lines in notificationtests/Android.mk,
before "include $(BUILD_PACKAGE)"
LOCAL_COMPATIBILITY_SUITE := general-test

Change-Id: Ib5e9cd152d957b5d5beb6ed23a899a021a7e06d6
2017-12-20 15:05:17 -08:00
Treehugger Robot
b379762089 Merge "build_image: Factor out CheckHeadroom() and add tests." 2017-12-19 00:00:41 +00:00
Treehugger Robot
b08567cbb6 Merge "releasetools: Fix the tag replacement for ro.build.vendor.fingerprint." 2017-12-18 22:18:14 +00:00
Treehugger Robot
d30b75190d Merge "Remove the support for BOARD_HAS_EXT4_RESERVED_BLOCKS." 2017-12-18 22:17:46 +00:00
Treehugger Robot
e58d385df1 Merge "releasetools: Use delta_generator to verify payload signatures." 2017-12-15 23:38:12 +00:00
Tao Bao
750385e455 releasetools: Use delta_generator to verify payload signatures.
We used to take a hard approach by parsing the payload with Python
script. This can be done by calling deleta_generator directly, which
also avoids the dependency on protobuf.

- Passing case
$ ./build/make/tools/releasetools/check_ota_package_signature.py \
    build/target/product/security/testkey.x509.pem \
    out/dist/aosp_marlin-ota-eng.zip

Package: out/dist/aosp_marlin-ota-eng.zip
Certificate: build/target/product/security/testkey.x509.pem
  ...
Whole package signature VERIFIED

Verifying A/B OTA payload signatures...
[1215/122842:INFO:generate_delta_main.cc(171)] Verifying signed payload.
[1215/122845:INFO:payload_verifier.cc(93)] signature blob size = 264
[1215/122845:INFO:payload_verifier.cc(112)] Verified correct signature 1 out of 1 signatures.
[1215/122845:INFO:payload_verifier.cc(93)] signature blob size = 264
[1215/122845:INFO:payload_verifier.cc(112)] Verified correct signature 1 out of 1 signatures.
[1215/122845:INFO:generate_delta_main.cc(181)] Done verifying signed payload.

Payload signatures VERIFIED

$ echo $?
0

- Failing case
Sign the whole package file with a different key, but leaving payload entries intact.

$ ./build/make/tools/releasetools/check_ota_package_signature.py \
    testkey2.x509.pem \
    marlin-ota-mismatching.zip
Package: marlin-ota-mismatching.zip
Certificate: testkey2.x509.pem
  ...
Whole package signature VERIFIED

Verifying A/B OTA payload signatures...
[1215/123054:INFO:generate_delta_main.cc(171)] Verifying signed payload.
[1215/123056:INFO:payload_verifier.cc(93)] signature blob size = 264
[1215/123056:ERROR:payload_verifier.cc(118)] None of the 1 signatures is correct. Expected:
[1215/123056:INFO:utils.cc(444)] Logging array of length: 256
[1215/123056:INFO:utils.cc(461)] 0x00000000 : 00 01 ff ff ff ff ff ff ff ff ff ff ff ff ff ff
  ...
[1215/123056:ERROR:payload_verifier.cc(121)] But found decrypted hashes:
[1215/123056:INFO:utils.cc(444)] Logging array of length: 256
[1215/123056:INFO:utils.cc(461)] 0x00000000 : 52 68 78 36 f6 9e cd 2d 5e 9f 31 d5 26 03 c9 aa
  ...
[1215/123056:ERROR:payload_signer.cc(333)] PayloadVerifier::VerifySignature( signature_blob, public_key_path, payload_hash) failed.
[1215/123056:INFO:generate_delta_main.cc(177)] VerifySignedPayload failed

    ERROR: Failed to verify payload with delta_generator: marlin-ota-mismatching.zip

$ echo $?
1

Bug: 65261072
Test: See above.
Change-Id: Id2e065655ec49b80dd2b13c6a859f41913be055b
2017-12-15 12:45:09 -08:00
Tao Bao
0c6a414fff releasetools: Remove the support for generating verify package.
We used to have "--gen_verify" that would generate a verify package for
non-A/B devices. Since a) we don't have active users; b) it works with
non-A/B only; c) we have better alternatives (e.g. using fastboot to
compute the partition checksum), this CL removes the support.

Test: `m dist`
Change-Id: Ib4a2c8c2a0394b54e66c106089aa52a0b900f034
2017-12-15 10:20:52 -08:00
Tao Bao
23f7ea9e70 Merge "releasetools: Add common.ZipDelete()." 2017-12-15 16:36:55 +00:00
Tao Bao
89d7ab23e8 releasetools: Add common.ZipDelete().
We have been shelling out to 'zip -d' to delete existing ZIP entries in
add_img_to_target_files.py. This CL moves the function into common.py,
and calls that for the similar work in ota_from_target_files.py. This CL
also adds unittests for the newly added function.

Test: `m dist`
Test: python -m unittest test_common
Test: ota_from_target_files.py generates identical packages w/ and w/o
      the CL (so we know the streaming property computation is intact).
Test: Run 'add_img_to_target_files.py -a' that triggers a call to
      ReplaceUpdatedFiles().
Change-Id: Icaa6c3ea3ee2166023f78fa79275295f837ea842
2017-12-14 17:17:04 -08:00
Treehugger Robot
78b5724622 Merge "releasetools: Remove '-w' from ota_from_target_files options." 2017-12-14 22:18:56 +00:00
Treehugger Robot
973301af79 Merge "Replace build/tools with build/make/tools" 2017-12-13 02:34:42 +00:00
Tao Bao
a7054eede1 releasetools: Fix the tag replacement for ro.build.vendor.fingerprint.
For devices using derived fingerprint (i.e. /system/build.prop doesn't
contain ro.build.fingerprint, but has ro.build.thumbprint instead), the
current code (in android.os.Build) doesn't have a matching logic to do
the same for ro.vendor.build.fingerprint. This means we will see
ro.build.thumbprint in /system/build.prop, while there's no matching
ro.vendor.build.thumbprint in /vendor/build.prop.

From signing script point of view, it should just apply the tag
replacement (e.g. test-keys -> release-keys) for whatever it sees when
signing a target_files.zip.

This CL also adds unit tests for EditTags() and RewriteProps().

Fixes: 27950003
Test: Use 'sign_target_files_apks.py' to sign a target that uses derived
      fingerprint and vendor partition. Check VENDOR/build.prop.
Test: python -m unittest test_sign_target_files_apks
Change-Id: I09019da970840cd82f54b68a32b4e94984bc1d8d
2017-12-08 15:19:47 -08:00
Tao Bao
d4349f2106 build_image: Factor out CheckHeadroom() and add tests.
The test is mostly trivial, but it ensures the result parsing from
actual calls to mke2fs.

Test: python -m unittest test_build_image
Test: `m dist` on aosp_marlin-userdebug (w/ and w/o defining
      PRODUCT_SYSTEM_HEADROOM respectively).
Change-Id: I8b9964213950e76f6d7d5518414a1bab888b4706
2017-12-07 23:39:56 -08:00
Jaegeuk Kim
10c9f5d9d3 Merge "core, release: add sload.f2fs for otatools.zip" 2017-12-08 00:57:42 +00:00
Tao Bao
79d52f8798 Remove the support for BOARD_HAS_EXT4_RESERVED_BLOCKS.
This is only used in file-based OTA, where we may reserve space on
/system partition. With the deprecation of file-based OTA since O, the
code has become obsolete.

Test: `m dist`
Change-Id: I2bd686b292f93b566fe28a2bcd74c564ffc75dac
2017-12-07 14:36:21 -08:00
Treehugger Robot
9a33b96103 Merge "Explain why BuildImage failed." 2017-12-07 21:45:59 +00:00
Colin Cross
fdea893833 Replace build/tools with build/make/tools
Test: m checkbuild
Change-Id: I8db6c27a6d2372f142d10568fe92d6f2e9a59252
2017-12-06 16:12:40 -08:00