Commit graph

270 commits

Author SHA1 Message Date
Tao Bao
73831e0636 Merge "Move error_code.h into otautil."
am: 916e155bab

Change-Id: I53265b03c12bfd022a634f8633f2d2e15a5e641b
2017-10-05 16:09:19 +00:00
Tao Bao
623fe7e701 Move error_code.h into otautil.
This way it stops requiring relative path ".." in LOCAL_C_INCLUDES
(uncrypt and edify). Soong doesn't accept non-local ".." in
"local_include_dirs".

Test: mmma bootable/recovery
Change-Id: Ia4649789cef2aaeb2785483660e9ea5a8b389c62
2017-10-04 08:55:24 -07:00
Tao Bao
8277230a98 Merge "otafault: Remove the use of LOCAL_WHOLE_STATIC_LIBRARIES."
am: 50f5a54a1e

Change-Id: I4f2d63757bb122cd027627321c1ab66c5e9284b6
2017-09-29 17:14:17 +00:00
Tao Bao
646b05a66c otafault: Remove the use of LOCAL_WHOLE_STATIC_LIBRARIES.
Commit d80a99883d has explanation of
potential issues.

Test: mmma bootable/recovery
Change-Id: I25ca9920952b7bbdd8a661d9dc90962431410bc4
2017-09-28 18:25:10 -07:00
Tianjie Xu
eb5e194012 Merge "Output split information for imgdiff when handling large apks"
am: 7f54fe8841

Change-Id: I97ad82d84b16b46e9c0c0f8198d3ab73d43031b0
2017-09-27 00:02:43 +00:00
Tianjie Xu
82582b4562 Output split information for imgdiff when handling large apks
Add a mandatory option in imgdiff to write the split info (i.e.
patch_size, tgt_size, src_ranges) to file when handling large apks.
Therefore, the caller of imgdiff can create split transfers based on
the info.

Bug: 63542719
Test: unit tests pass
Change-Id: I853d55d1f999fd576474faa81077f7307f4d856d
2017-09-21 11:42:16 -07:00
Tianjie Xu
2cb1fb45c0 Merge "Fix the dangling pointer when setting up arguments of imgdiff"
am: ca5a071307

Change-Id: I9411f0bce8d96c598db365d0257d4c9893e24104
2017-09-17 21:42:23 +00:00
Tianjie Xu
8ba7c45e0b Fix the dangling pointer when setting up arguments of imgdiff
Test: unit tests pass
Change-Id: If884e805ccd4df73671ab3436eb90860786ff6c9
2017-09-13 14:05:42 -07:00
Tianjie Xu
a11601da9b Merge "Close cmd_pipe properly after updater test finishes"
am: 64307daf5b

Change-Id: Ic384d9de3c559855cfe0bd122adb5e69fa996ad1
2017-09-12 19:47:11 +00:00
Tianjie Xu
79327ac21d Close cmd_pipe properly after updater test finishes
Otherwise the test may fail after a large number of iterations due to
file open failure.

Bug: 65430057
Test: run recovery_component_test on sailfish for 2000 iterations.
Change-Id: I0d456284d6064467038911d63eade95740cbec2c
2017-09-11 14:38:49 -07:00
Tianjie Xu
e24e90f403 Merge "Improve imgdiff for large zip files"
am: 9f48641784

Change-Id: I999525af80c5423da275025d977de10399140479
2017-09-08 20:46:47 +00:00
Tianjie Xu
2903cddb58 Improve imgdiff for large zip files
Due to the cache size limit for OTA generation, we used to split large
zip files linearly into pieces and do bsdiff on them. As a result, i) we
lose the advantage of imgdiff; ii) if there's an accidental order change
of some huge files inside the zip, we'll create an insanely large patch.

This patch splits the src&tgt more smartly based on the zip entry_name.
If the entry_name is empty or no matching source is found for a target
chunk, we'll skip adding its source and later do a bsdiff against the
whole split source image (this rarely happens in our use cases except
for the metadata inside a ziparchive).

After the split, the target pieces are continuous and block aligned,
while the sources pieces are mutually exclusive. (Some of the source
blocks may not be used if there's no matching entry_name in the target.)
Then we will generate patches accordingly between each split image
pairs.

Afterwards, if we apply imgpatch to each pair of split source/target
images and add up the patched result, we can get back the original
target image.

For example:
Input: [src_image, tgt_image]
Split: [src-0,tgt-0; src-1,tgt-1, src-2,tgt-2]
Diff:  [  patch-0;     patch-1;    patch-2]

Patch: [(src-0,patch-0)=tgt-0; (src-1,patch-1)=tgt-1;
(src-2,patch-2)=tgt-2;]
Append: [tgt-0 + tgt-1 + tgt-2 = tgt_image]

Peformance:
For the small package in b/34220646, we decrease the patch size of
chrome.apk dramatically from 30M to 400K due to the order change of
two big .so files.

On two versions of angler, I also observe decent patch size decrease.
For chrome.apk, we reduced the size from 5.9M to 3.2M; and for
vevlet.apk from 8.0M to 6.5M.

Bug: 34220646
Test: recovery component test && apply imgdiff & imgpatch on two
chrome.apk
Change-Id: I145d802984fa805efbbac9d01a2e64d82ef9728b
2017-09-05 15:09:58 -07:00
Tianjie Xu
44d0b008c6 Merge "Turn on -Wall for recovery modules"
am: 11f68b6b69

Change-Id: I5f0531de7880682c7dfe75a6274f41519e0c64d6
2017-08-29 17:34:28 +00:00
Tianjie Xu
c89d1e7e2a Turn on -Wall for recovery modules
Turn on -Wall for all modules. Also remove the obsolete file_cmp() in
apply_patch test and now() in wear_ui.

The only exception is lib_edify due to the unused functions in the
intermediate cpp files generated from the lex files. It will be handled
in a seperate CL.

Bug: 64939312
Test: mma, unit tests pass
Change-Id: Ic53f76b60b6401ab20db3d98130d674c08e3702f
2017-08-28 21:56:33 -07:00
Tianjie Xu
7379a5290d Merge "Allow comparison against multi serial nums for A/B package" am: 3810046a55 am: e8b02d68e5
am: 968ebdeefd

Change-Id: I21de76c870001cb5faf145d5f39f8fb2cfa58a66
2017-08-22 23:23:07 +00:00
Tianjie Xu
3810046a55 Merge "Allow comparison against multi serial nums for A/B package" 2017-08-22 22:40:47 +00:00
Tianjie Xu
69b9649e9b Allow comparison against multi serial nums for A/B package
The metadata file now can have multiple serial numbers in the format:
serialno=serialno1|serialno2|serialno3 ...
Verifier will pass the check if the device serial number matches any of
these numbers.

Bug: 64802465
Test: Create a metadata file with 1000 numbers and sideload in sailfish.
      The checker detects both match and mismatch cases.

Change-Id: I3f12b75e15f4179df260778e37f4563d65db0fa8
2017-08-22 12:05:52 -07:00
Tianjie Xu
fc569d1516 Merge "Move Image/ImageChunk/PatchChunk declaration into header files" am: b4bc57ed39 am: b127fddf09
am: f5e3cadeca

Change-Id: Ic9056467184e272eec5c0aead3d4a712033b0503
2017-08-19 04:34:03 +00:00
Tianjie Xu
b127fddf09 Merge "Move Image/ImageChunk/PatchChunk declaration into header files"
am: b4bc57ed39

Change-Id: If254ed9e24bc0cafa19db9766ed36643ca0fed49
2017-08-19 04:27:34 +00:00
Tianjie Xu
57dd961995 Move Image/ImageChunk/PatchChunk declaration into header files
1. Move the declaration of the Image classes to the header file to make
testing easier.
2. Also move rangeset.h to bootable/recovery to allow access in imgdiff.

Test: recovery component test
Change-Id: I68a863e60a3f2e7ae46ee48f48eb15391f5f4330
2017-08-18 17:56:22 -07:00
Tao Bao
6f1c38c9ad Merge "tests: Add the missing dependency on libhidlbase." am: 05b2e982ad am: 43df6cfeee
am: 46c13f3f05

Change-Id: Iaa1bd4f403664a87153b9f6b647e04bc7798a33d
2017-08-10 06:10:14 +00:00
Tao Bao
43df6cfeee Merge "tests: Add the missing dependency on libhidlbase."
am: 05b2e982ad

Change-Id: Ie08dca777af1d12a304aebf11f4df25680ff939a
2017-08-10 06:03:45 +00:00
Tao Bao
3e2345e1fe tests: Add the missing dependency on libhidlbase.
It fails to build recovery_component_test with the following errors:

out/soong/.intermediates/hardware/interfaces/boot/1.0/android.hardware.boot@1.0_genc++_headers/gen/android/hardware/boot/1.0/types.h:14:
error: undefined reference to 'android::hardware::hidl_string::hidl_string(android::hardware::hidl_string const&)'
out/soong/.intermediates/hardware/interfaces/boot/1.0/android.hardware.boot@1.0_genc++_headers/gen/android/hardware/boot/1.0/types.h:14:
error: undefined reference to 'android::hardware::hidl_string::operator=(android::hardware::hidl_string
const&)'
out/soong/.intermediates/hardware/interfaces/boot/1.0/android.hardware.boot@1.0_genc++_headers/gen/android/hardware/boot/1.0/types.h:14:
error: undefined reference to 'android::hardware::hidl_string::~hidl_string()'

libupdate_verifier includes <android/hardware/boot/1.0/IBootControl.h>,
which includes the 'types.h' above. In 'types.h', it defines struct
CommandResult that's using android::hardware::hidl_string.

Since libhidlbase doesn't have a static library target, remove
'LOCAL_FORCE_STATIC_EXECUTABLE := true', which isn't required for
running tests.

Test: mmma -j bootable/recovery
Bug: 64538692
Change-Id: Iaa7c08adc241128d787274fcaea9b363e7ff93f4
(cherry picked from commit 102016ce1f)
2017-08-09 22:57:13 -07:00
Tianjie Xu
8b09001e47 Merge "Add implemention of SortedRangeSet" am: 64cba55fbc am: fbd4b10310
am: a09376ed90

Change-Id: I67b2f4e91133239d4eb35524c7f0ed41f18de8dc
2017-08-01 00:38:21 +00:00
Tianjie Xu
fbd4b10310 Merge "Add implemention of SortedRangeSet"
am: 64cba55fbc

Change-Id: I2174e4f55c85fe57014b31625dbc2d06e41350be
2017-08-01 00:31:20 +00:00
Tianjie Xu
64cba55fbc Merge "Add implemention of SortedRangeSet" 2017-08-01 00:22:33 +00:00
Tianjie Xu
b9e7fc7fa9 Add implemention of SortedRangeSet
This is useful in imgdiff to maintain the block ranges of
splitted source image.

Bug: 34220646
Test: mma && unit tests pass
Change-Id: I6427f2ea50f0e3b0aa3dd01880ec0206679b7429
2017-07-31 15:13:55 -07:00
Tao Bao
06db7363b0 Merge "otautil: Clean up dirCreateHierarchy()." am: 610712101b am: 5507c3d63c
am: 0863373376

Change-Id: Ia2326bca2d8a1344fccd2838e2b376cb9a843a82
2017-07-31 00:21:43 +00:00
Tao Bao
5507c3d63c Merge "otautil: Clean up dirCreateHierarchy()."
am: 610712101b

Change-Id: I95350c4b2aab36dd89ea7813f2eb63d407b5f8ac
2017-07-31 00:15:43 +00:00
Tao Bao
ac3d1edca0 otautil: Clean up dirCreateHierarchy().
- Changed to std::string based implementation (mostly moved from the
  former make_parents() in updater/install.cpp);
- Removed the timestamp parameter, which is only neeed by file-based OTA;
- Changed the type of mode from int to mode_t;
- Renamed dirCreateHierarchy() to mkdir_recursively().

Test: recovery_unit_test passes.
Test: No external user of dirCreateHierarchy() in code search.
Change-Id: I71f8c4b29bab625513bbc3af6d0d1ecdc3a2719a
2017-07-27 10:33:07 -07:00
Tao Bao
9c62f67863 Merge "otautil: Delete dirUnlinkHierarchy()." am: 031661d4a9 am: 993cec5cc9
am: 633aeba594

Change-Id: I9833741fe5d785298d7f99f6b2c9539723f8fcd1
2017-07-25 14:35:50 +00:00
Tao Bao
993cec5cc9 Merge "otautil: Delete dirUnlinkHierarchy()."
am: 031661d4a9

Change-Id: I33a793646f6f11757afc20f16fd3a0f700dd09fb
2017-07-25 14:28:52 +00:00
Tao Bao
7934985e0c otautil: Delete dirUnlinkHierarchy().
This function has become obsolete since we've removed file-based OTA
support (it was needed by 'delete_recursive' edify function earlier).

Test: mmma -j bootable/recovery
Test: Code search shows no active user of the function.
Change-Id: If6faaa759d4c849b79acba4e6adb82baadc89f7a
2017-07-24 20:35:48 -07:00
Tao Bao
2357b0c745 Merge "tests: Add a test to cover legacy care_map.txt handling." am: 6eb23c594c am: dbe9da5331
am: 0148a6f9ab

Change-Id: Id568b7cb5043c444f88032368be57a6f128a44c7
2017-07-24 18:32:01 +00:00
Tao Bao
dbe9da5331 Merge "tests: Add a test to cover legacy care_map.txt handling."
am: 6eb23c594c

Change-Id: Iaad43cb44f4c406e4c07c9399e70e96cbaa4db7b
2017-07-24 18:27:00 +00:00
Tao Bao
c319613e06 tests: Add a test to cover legacy care_map.txt handling.
This is to cover the code added by commit
5a1dee01df, where an O update_verifier
should not reject N care_map.txt.

Bug: 63544345
Test: recovery_component_test passes on marlin.
Change-Id: Ia944e16cba3cc635098b3ffd92842d725b570fec
2017-07-24 09:34:06 -07:00
Tianjie Xu
512283a751 Merge "Fix a case when brotli writer fails to write last few blocks of data" am: e45c8f0057 am: 80acaab52f
am: 75ff0107c8

Change-Id: Ieab5e6e412704599e873c7497d1c0de42453e642
2017-07-21 22:46:00 +00:00
Tianjie Xu
80acaab52f Merge "Fix a case when brotli writer fails to write last few blocks of data"
am: e45c8f0057

Change-Id: I337e8ec26f59a5245ab299080d7251331823e2da
2017-07-21 21:04:17 +00:00
Tianjie Xu
e45c8f0057 Merge "Fix a case when brotli writer fails to write last few blocks of data" 2017-07-21 20:59:24 +00:00
Tianjie Xu
6ed175d541 Fix a case when brotli writer fails to write last few blocks of data
receive_new_data may exit too early if the zip processor has sent all
the raw data. As a result, the last few 'new' commands will fail even
though the brotli decoder has more output in its buffer.

Restruct the code so that 'NewThreadInfo' owns the decoder state solely;
and receive_brotli_new_data is responsible for the decompression.

Also reduce the test data size to 100 blocks to avoid the test timeout.

Bug: 63802629
Test: recovery_component_test. on bullhead, apply full updates with and
w/o brotli compressed entries, apply an incremental update.

Change-Id: I9442f2536b74e48dbf7eeb062a8539c82c6dab47
2017-07-21 11:44:00 -07:00
Tianjie Xu
f2a279adb5 Merge "Fix the android-cloexec-* warnings in bootable/recovery" am: 94a8ea1797 am: 6d8827e0d3
am: 96b5bb9601

Change-Id: I55911c112a34797d7c7098e5e325145667b46715
2017-07-19 23:12:20 +00:00
Tianjie Xu
6d8827e0d3 Merge "Fix the android-cloexec-* warnings in bootable/recovery"
am: 94a8ea1797

Change-Id: I57ae57bab58f603540654bb24df9facca9a7d625
2017-07-19 23:06:50 +00:00
Tianjie Xu
de6735e80c Fix the android-cloexec-* warnings in bootable/recovery
Add the O_CLOEXEC or 'e' accordingly.

Bug: 63510015
Test: recovery tests pass
Change-Id: I7094bcc6af22c9687eb535116b2ca6a59178b303
2017-07-19 12:17:41 -07:00
Tianjie Xu
9f8fd32d95 Merge "Add support to decompress brotli compressed new data" am: 918e6ea1b2 am: 43bdf6cad6
am: 8375ebee37

Change-Id: I3eea508486f48d316644b68278f42976ffd4698d
2017-07-10 22:47:10 +00:00
Tianjie Xu
43bdf6cad6 Merge "Add support to decompress brotli compressed new data"
am: 918e6ea1b2

Change-Id: I4fd9cea71716ad1574ecb4bb7f612bc8734711c5
2017-07-10 22:33:37 +00:00
Tianjie Xu
107a34f9fc Add support to decompress brotli compressed new data
Add a new writer that can decode the brotli-compressed system/vendor
new data stored in the OTA zip.

Brotli generally gives better compression rate at the cost of slightly
increased time consumption. The patch.dat is already compressed
by BZ; so there's no point to further compress it.

For the given 1.9G bullhead system image:
Size: 875M -> 787M; ~10% reduction of package size.
Time: 147s -> 153s; ~4% increase of the block_image_update execution time.
(I guess I/O takes much longer time than decompression.)

Also it takes 4 minutes to compress the system image on my local
machine, 3 more minutes than zip.

Test: recovery tests pass && apply a full OTA with brotli compressed
system/vendor.new.dat on bullhead

Change-Id: I232335ebf662a9c55579ca073ad45265700a621e
2017-07-07 16:08:18 -07:00
Tianjie Xu
79c1562dc9 Merge "Fix a rare failure for imgdiff when random data equals gzip header" am: b87a166a16 am: 8542502f1f
am: 233f4b0b4b

Change-Id: If7e81b8e794f34d374d385eb5603353cdfec524f
2017-07-07 18:11:59 +00:00
Tianjie Xu
8542502f1f Merge "Fix a rare failure for imgdiff when random data equals gzip header"
am: b87a166a16

Change-Id: Ic3c7d102cc7a2b992c699add012076a4bda549c6
2017-07-07 17:52:26 +00:00
Tianjie Xu
14ebc1e5ae Fix a rare failure for imgdiff when random data equals gzip header
In a rare case, a random chunk will pass both the gzip header check
and the inflation process; but fail the uncompressed length check in the
footer. This leads to a imgdiff failure. So, we should treat this chunk
as 'normal' instead of 'inflated' while generating the patch.

Bug: 63334984
Test: imgdiff generates patch successfully on previous failing images.
Change-Id: Ice84f22d3653bce9756bda91e70528c0d2f264a0
2017-07-06 16:49:11 -07:00
Tianjie Xu
617b702104 Merge "Remove the obsolete package_extract_dir() test" am: e916cca8a9 am: af913974a5
am: e002d96194

Change-Id: If4887683362c1a00ee20b0d6e447d70bff442c28
2017-06-23 04:58:26 +00:00