Commit graph

173 commits

Author SHA1 Message Date
Narayan Kamath
5c39aa53a9 Merge "zip_archive: Fix a few more badly written tests." am: 93d344d98c
am: 000e0bfff4

Change-Id: I6af4ce2a977e0fbc21974e8e80516fb39ec8156a
2017-12-27 11:47:41 +00:00
Narayan Kamath
c36b80928b zip_archive: Fix a few more badly written tests.
Fix failures in DataDescriptor related tests due to to a bad call to
SetZipString (undefined behaviour). Also fix a typo in the test for
invalid descriptors, we were asserting things on the wrong array.

Test: zip_archive_test
Change-Id: I8c9a632443fdf1d5c115670d6e9317e1f4bf6ef4
2017-12-27 09:42:00 +00:00
Narayan Kamath
780105e6b1 Merge "Get rid of unneeded allocations in Extract...() APIs" am: 18f25d399e
am: 64bf32d999

Change-Id: I0107091854c2d810fc5d19a873c29a270c5cf207
2017-12-22 10:50:24 +00:00
Narayan Kamath
47ede5466a Merge "zip_archive: Fix tests broken by 1f93d71022cca7bb6bb9eec49." am: 492de535c4
am: fede183c53

Change-Id: Ib1e16a1b0799ffe88b822b04d3231c4d30f77917
2017-12-22 10:49:41 +00:00
Narayan Kamath
18f25d399e Merge "Get rid of unneeded allocations in Extract...() APIs" 2017-12-22 10:41:41 +00:00
Narayan Kamath
f37bb8e45b zip_archive: Fix tests broken by 1f93d71022.
Moving to std::hash changed iteration order but these tests should
not have relied on hash_map iteration order anyway.

Test: zip_archive_test

Change-Id: I712bf2307c8770f03ea6f074bfc506a40cdcb066
2017-12-21 12:55:59 +00:00
Yurii Zubrytskyi
834326ce7a Get rid of unneeded allocations in Extract...() APIs
Both Extract...() functions don't need dynamic allocation
for the writers, as those are strictly scoped. This CL
changes heap allocation to stack allocation.

Test: zip_archive_test

Change-Id: Id727e4b9848235cd063cc67ecbe052d21ca21326
2017-12-20 18:36:03 +00:00
Elliott Hughes
1aee775518 Merge "BENCHMARK_MAIN now requires a semicolon." am: 981379b70f
am: 64c59c23f3

Change-Id: I430d1a3abfdc163a0f0ccc215030106ed3a20c6c
2017-12-14 23:34:38 +00:00
Elliott Hughes
62322aae85 BENCHMARK_MAIN now requires a semicolon.
Bug: N/A
Test: ran tests
Change-Id: Ie61d8c775cf23877c1aa7025a50b5f8284ae5b08
2017-12-13 18:19:18 -08:00
Elliott Hughes
aede8375fe Merge "Add OWNERS." am: 3289b9c928
am: 515c8fe572

Change-Id: I378e0205c21a4df5baacde3432b618443f7ef567
2017-12-07 23:51:13 +00:00
Elliott Hughes
3289b9c928 Merge "Add OWNERS." 2017-12-07 23:21:26 +00:00
Elliott Hughes
693d63f9cf Add OWNERS.
Bug: N/A
Test: N/A
Change-Id: Ie785058c0f5eb9b4086c98ccba6e63e3ed411b65
2017-12-07 13:30:03 -08:00
Elliott Hughes
957dc094bf Merge "use std::hash instead of hashing byte by byte" am: aaa0bbce8e
am: bfb96796a4

Change-Id: Ib0cd0e44e34aecbf8af1ccd83c96eb3ebeb71978
2017-11-30 23:34:23 +00:00
Elliott Hughes
bfb96796a4 Merge "use std::hash instead of hashing byte by byte"
am: aaa0bbce8e

Change-Id: Iee321900f8ef386cd118d874c31c5b0d2741551a
2017-11-30 23:25:55 +00:00
Sebastian Pop
1f93d71022 use std::hash instead of hashing byte by byte
This patch uses libcxx's string_view hashing.  The performance is better than the
current implementation, and almost the same as for the hashing implementation amended
by the patch in https://android-review.googlesource.com/404742

The following experiments were conducted on a HiKey aarch64 A53.

No changes:
$ adb push out/target/product/hikey/data/benchmarktest64/ziparchive-benchmarks /data
$ adb shell /data/ziparchive-benchmarks --benchmark_repetitions=5
-----------------------------------------------------------------
Benchmark                          Time           CPU Iterations
-----------------------------------------------------------------
FindEntry_no_match          82279629 ns   81759676 ns          9
FindEntry_no_match          81648056 ns   81627431 ns          9
FindEntry_no_match          81384074 ns   81369057 ns          9
FindEntry_no_match          89618889 ns   82437755 ns          9
FindEntry_no_match          81811389 ns   81785261 ns          9
FindEntry_no_match_mean     83348407 ns   81795836 ns          9
FindEntry_no_match_stddev    3520421 ns     394962 ns          0
Iterate_all_files          137622000 ns  137589032 ns          5
Iterate_all_files          139666333 ns  138409469 ns          5
Iterate_all_files          150070000 ns  140883697 ns          5
Iterate_all_files          138600667 ns  138540646 ns          5
Iterate_all_files          137599833 ns  137567438 ns          5
Iterate_all_files_mean     140711767 ns  138598056 ns          5
Iterate_all_files_stddev     5299929 ns    1354928 ns          0

$ cd system/core/libziparchive
$ git fetch https://android.googlesource.com/platform/system/core refs/changes/42/404742/7 && git cherry-pick FETCH_HEAD
$ mma
$ cd -
$ adb push out/target/product/hikey/data/benchmarktest64/ziparchive-benchmarks /data
$ adb shell /data/ziparchive-benchmarks --benchmark_repetitions=5
-----------------------------------------------------------------
Benchmark                          Time           CPU Iterations
-----------------------------------------------------------------
FindEntry_no_match          53302756 ns   53291178 ns         13
FindEntry_no_match          54314487 ns   54272272 ns         13
FindEntry_no_match          53866923 ns   53851178 ns         13
FindEntry_no_match          53324423 ns   53317296 ns         13
FindEntry_no_match          53289231 ns   53289159 ns         13
FindEntry_no_match_mean     53619564 ns   53604216 ns         13
FindEntry_no_match_stddev     458449 ns     443527 ns          0
Iterate_all_files          128211111 ns  112254783 ns          6
Iterate_all_files          110695000 ns  110677726 ns          6
Iterate_all_files          109351250 ns  109350998 ns          6
Iterate_all_files          109367500 ns  109367796 ns          6
Iterate_all_files          110872222 ns  110591407 ns          6
Iterate_all_files_mean     113699417 ns  110448542 ns          6
Iterate_all_files_stddev     8143723 ns    1194577 ns          0

$ cd system/core/libziparchive
$ git checkout HEAD~
$ git am 0001-use-std-hash.patch
$ mma
$ cd -
$ adb push out/target/product/hikey/data/benchmarktest64/ziparchive-benchmarks /data
$ adb shell /data/ziparchive-benchmarks --benchmark_repetitions=5
-----------------------------------------------------------------
Benchmark                          Time           CPU Iterations
-----------------------------------------------------------------
FindEntry_no_match          55339872 ns   55195112 ns         13
FindEntry_no_match          56232628 ns   56069924 ns         13
FindEntry_no_match          55694103 ns   55255946 ns         13
FindEntry_no_match          55275064 ns   55120136 ns         13
FindEntry_no_match          54971987 ns   54944411 ns         13
FindEntry_no_match_mean     55502731 ns   55317106 ns         13
FindEntry_no_match_stddev     482032 ns     436766 ns          0
Iterate_all_files          114618611 ns  114487804 ns          6
Iterate_all_files          112552917 ns  112229801 ns          6
Iterate_all_files          111288750 ns  111255044 ns          6
Iterate_all_files          111291528 ns  111259045 ns          6
Iterate_all_files          114347222 ns  113677119 ns          6
Iterate_all_files_mean     112819806 ns  112581763 ns          6
Iterate_all_files_stddev     1606214 ns    1454858 ns          0

Change-Id: I1e3413d331bcb460ca38bc2c87e23f89b456cd2f
2017-11-29 09:44:28 -06:00
Xin Li
23e27db576 Merge commit 'a63ccea6abc7ea02e2d98e41c80793ca97237bd3' from
oc-mr1-dev-plus-aosp into stage-aosp-master

Change-Id: Ia33311cd1fd26dfaea59a69317b306fb91203c40
Merged-In: I03d06b10807e8a313c9654c2e1db36bfb59e3f99
2017-11-14 13:19:45 -08:00
Narayan Kamath
70f5571a9a resolve merge conflicts of 5ab03cb41 to oc-mr1-dev-plus-aosp
Test: I solemnly swear I tested this conflict resolution.
Change-Id: Ic749614fb03d60d6cc7e0eb04bb67102e572e9f2
2017-11-01 11:32:17 +00:00
Narayan Kamath
2d1e23f5d6 zip_archive: Allow crc_out to be nullptr in Inflate.
Only compute the crc32 if required. In addition :
- Add unit tests for Inflate that cover this addition.
- Fix an inconsistency in return codes that was revealed
  by this new test.

Bug: 35246701
Test: zip_archive_tests
Test: make; zipalign.
Merged-In: I31d7554378f94fc8995f707471d57cb98311e2c2
Change-Id: I05111bfa665c610f93d1c1dee987a509bf87aa65
2017-10-30 18:26:41 +00:00
Elliott Hughes
cf439ba2d0 Merge "fastboot should fail if it runs out of space while unzipping." am: b88aa023d4
am: ddc4e5cfbf

Change-Id: Icb259596ad6c50c748a47d3224fbd31cab6d8a00
2017-10-28 01:29:36 +00:00
Treehugger Robot
b88aa023d4 Merge "fastboot should fail if it runs out of space while unzipping." 2017-10-28 01:16:29 +00:00
Elliott Hughes
4089d34b93 fastboot should fail if it runs out of space while unzipping.
Previously fastboot would carry on regardless if decompression failed:

  fastboot: archive does not contain 'vbmeta.img'
  fastboot: extracting vendor.img (260 MB)...
  fastboot: W/ziparchive(56777): Zip: unable to allocate  272781472 bytes at offset 0 : No space left on device
  fastboot: failed to extract 'vendor.img': I/O error
  fastboot: archive does not contain 'vendor_other.img'
  fastboot: wiping userdata...

This is because all but "boot" and "system" are considered "optional",
and the implementation of "optional" was "ignore any failures". What it
_should_ have meant was "it's okay if these don't exist, but if they do,
failures matter".

Fix this logic, use die() more aggressively, and remove spurious "\n"s
from die() format strings.

Also fix spurious whitespace in the libziparchive format string. Before:

  Zip: unable to allocate  272781472 bytes at offset 0 : No space left on device

After:

  Zip: unable to allocate 272781472 bytes at offset 0: No space left on device

Bug: http://b/68383022
Test: `fastboot update` on marlin
Change-Id: I3cbf55f1a33ca125f293f873eafbcfb86c880ba8
2017-10-27 14:21:12 -07:00
Narayan Kamath
d93eaf6dd4 Merge "zip_archive: Make Inflate a public API." am: d2ea4471bc
am: c72f5276fb

Change-Id: I482c429dd3afd3093042a1864d98efa19475f952
2017-10-27 08:41:14 +00:00
Narayan Kamath
a7ce0bc493 Merge "zip_archive: generalize deflate method." am: 661b3827a6
am: de0153c20b

Change-Id: I4251f36f21924af358eec85703cf1e87131b8d72
2017-10-27 08:40:36 +00:00
Narayan Kamath
2440323318 Merge "zip_archive: Remove unused ziparchive-host" am: 4adb4ce223
am: 5cc1bd65ed

Change-Id: I78184323adbf40ddd20992b78f108c2e1c692ac7
2017-10-27 08:39:57 +00:00
Narayan Kamath
d2ea4471bc Merge "zip_archive: Make Inflate a public API." 2017-10-27 08:27:20 +00:00
Narayan Kamath
661b3827a6 Merge "zip_archive: generalize deflate method." 2017-10-27 08:25:22 +00:00
Narayan Kamath
485b3640a4 zip_archive: Make Inflate a public API.
For use from zipalign and androidfw::ZipUtils.

Test: make
Bug: 35246701

Change-Id: I1bdeff3770d0833dfc0807d8a69ba9fea525772a
2017-10-26 18:43:49 +01:00
Narayan Kamath
4b97a71cb1 zip_archive: Remove unused ziparchive-host
Test: make
Change-Id: Ice7fdfb59bc939d075d3be216e48bea46a80e82d
2017-10-26 17:50:49 +01:00
Narayan Kamath
8b8faed17d zip_archive: generalize deflate method.
Remove dependencies on zip_archive specific data structures. In follow
up changes, this method will be promoted to a public API and used from
androidfw/ZipUtils as well as tools/zipalign, thereby allowing us to
remove a near-exact copy of this code and to simplify depdencies.

Test: zip_archive_test
Bug: 35246701
Change-Id: If24a9965fbd6fff308c8758859026684fd2af3b9
2017-10-26 17:47:50 +01:00
Dan Willemsen
29108720ce Merge "Rename libz-host -> libz" am: 612b47630f am: b961c07f50
am: 3256982e9b

Change-Id: Id91307a501ceb84204c0fe63db4bade68124015a
2017-09-29 09:35:30 +00:00
Dan Willemsen
5cfa006059 Rename libz-host -> libz
Test: m host
Change-Id: Ib276431d13b3e1bd243f84a80f69aafc12e71aac
2017-09-27 16:24:45 -07:00
Justin Yun
096114d85c Mark libziparchive as VNDK in Android.bp
As a VNDK module, Android.bp must have 'vndk' tag as well as
'vendor_available: true'.

The 'vndk' tag for VNDK module is formated as below:
vndk: {
    enabled: true,
},

VNDK modules will be installed both in system/lib(64) as normal and
in system/lib(64)/vndk as a vendor variant.

Bug: 63866913
Test: build and boot with BOARD_VNDK_VERSION=current
Merged-In: Iec5d3496e91a99f3e6b0c816c67ad279672ff36a
Change-Id: Iec5d3496e91a99f3e6b0c816c67ad279672ff36a
(cherry picked from commit 4e7e5b3ba053d013f2c4ae79d02722b874c629fb)
2017-09-15 01:07:40 +00:00
Narayan Kamath
8310613c37 Merge "zip_archive: reject files that don't start with an LFH signature." into oc-mr1-dev
am: b669c3311b

Change-Id: Id46f20e402c105108309addb0349cb64437d2541
2017-08-16 14:02:30 +00:00
Narayan Kamath
c1a56dcab7 zip_archive: reject files that don't start with an LFH signature.
Bug: 64211847
Test: zip_archive_test

Change-Id: I275e7c4da05ceeb20401b560c72294f29ef63642
2017-08-15 13:44:05 +01:00
Justin Yun
04292879e8 Mark libziparchive as VNDK in Android.bp
am: 077dc8fca4

Change-Id: I73c8d57e95f8c88aefd6beec08b328ca02615de1
2017-08-11 17:22:26 +00:00
Justin Yun
077dc8fca4 Mark libziparchive as VNDK in Android.bp
As a VNDK module, Android.bp must have 'vndk' tag as well as
'vendor_available: true'.

The 'vndk' tag for VNDK module is formated as below:
vndk: {
    enabled: true,
},

VNDK modules will be installed both in system/lib(64) as normal and
in system/lib(64)/vndk as a vendor variant.

Bug: 63866913
Test: build and boot with BOARD_VNDK_VERSION=current
Change-Id: Iec5d3496e91a99f3e6b0c816c67ad279672ff36a
2017-08-12 00:52:14 +09:00
Adam Lesinski
de117e4a49 libziparchive: Use ReadAtOffset exclusively
The use of ReadAtOffset is meant to allow concurrent access
to the zip archive once it has been loaded. There were places
where this was the case, and some places that did a seek + read
combination, which could lead to data races.

NOTE: On Windows, we are not using pread as the implementation of
ReadAtOffset, therefore the guarantees on Windows are weaker.

On Linux, pread allows the file descriptor to be read at a specific
offset without changing the read pointer. This allows inherited fd's
and duped fds to be read concurrently.

On Windows, we use the ReadFile API, which allows for an atomic seek +
read operation, but modifies the read pointer. This means that any mix
use of ReadAtOffset and Read will have races. Just using ReadAtOffset is
safe.

For the Windows case, this is fine as the libziparchive code now only
uses ReadAtOffset.

Bug: 62184114
Bug: 62101783
Test: make ziparchive-tests (existing tests pass)
Change-Id: Ia7f9a30af2216682cdd9d578d26e84bc46773bb9
2017-07-25 18:12:12 +00:00
Andreas Gampe
a10a7e7114 Libziparchive: Export headers in defaults
libziparchive-host needs to include the headers correctly, too.

Bug: 37342627
Test: mmma system/core/libziparchive
Change-Id: I88a6d38ff9e494273040f9b913c71bccdda117ad
2017-07-05 22:03:44 -07:00
Jiyong Park
e7cf680ccc zlib is not exposed to libziparchive headers
By forward-declaring zstream, we don't need to re-export zlib headers.

Bug: 37342627
Test: build
Test: BOARD_VNDK_VERSION=current m -j dumpstate (or any other module
using libziparchive)

Change-Id: Ia69e5d459c79878d491cfe4ca21a7cc8fed4a5d8
2017-07-03 17:44:07 +00:00
Jiyong Park
cdf7ff8e20 libziparchive headers are moved to local directory
libziparchive headers are moved from the global include directory
(/system/core/include) to the local directory inside libziparchive.

Note: /system/core/include/ziparchive still exists as a symlink to
libarchive/include/ziparchive. This will be removed when there is no
header-only dependency to libziparchive.

Bug: 37342627
Test: build
Change-Id: I3631ffc2df7be8a064d64a625d10436090c3bb0f
2017-07-02 04:33:34 +00:00
Jiyong Park
cd997e6094 Ran clang-format on libziparchive sources and headers
Test: build aosp_arm
Change-Id: I469b82b68f2c457f480fb9cd9da2026672985ce3
2017-07-02 11:46:53 +09:00
Tianjie Xu
0fda1cf633 Fix out of bound read in libziparchive
We should check the boundary of central directory before checking its
signature. Swap the order of these two checks.

Bug: 36392138
Test: libziparchive doesn't read the signature after boundary check fails.
Merged-In: Ie89f709bb2d1ccb647116fb7ccb1e23c943e5ab8
Change-Id: Ie89f709bb2d1ccb647116fb7ccb1e23c943e5ab8
(cherry picked from commit 74464a1361)
2017-06-30 17:19:56 +09:00
Tianjie Xu
9e020e2d11 Check filename memory bound when parsing ziparchive
Add a check to ensure the filename boundary doesn't exceed the mapped
memory region. Also add the corresponding unit test.

Bug: 28802225
Test: New unit test passes.
Merged-In: Ibf543a7da3d7898952e9eb332c84cdfc67cf5aa4
Change-Id: Ibf543a7da3d7898952e9eb332c84cdfc67cf5aa4
(cherry picked from commit bcc4431f24)
2017-06-30 17:19:28 +09:00
Tianjie Xu
fba1a36fd9 Fix out of bound access in libziparchive
The boundary check of an invalid EOCD record may succeed due to the
overflow of uint32_t. Fix the check and add a unit test.

Test: Open the crash.apk and libziparchive reports the offset error as expected.

Bug: 31251826
Merged-In: I1d8092a19b73886a671bc9d291cfc27d65e3d236
Change-Id: I1d8092a19b73886a671bc9d291cfc27d65e3d236
(cherry picked from commit ae8180c06d)
2017-06-30 17:18:31 +09:00
Narayan Kamath
d1e9e7bc06 Merge "zip_archive_test: compare error codes directly." 2017-06-20 10:50:27 +00:00
Narayan Kamath
1ef9d2dfbe zip_archive_test: compare error codes directly.
Make them available to tests so that they can compare values
directly instead of comparing error code strings.

Test: zip_archive_test
Change-Id: I1c78c24651ff7549b5125a49a4e67bbd8295c617
2017-06-19 13:03:41 +01:00
Elliott Hughes
3e67944d2e Merge "Add libziparchive-based unzip." 2017-06-15 21:30:56 +00:00
Narayan Kamath
162b705801 zip_archive: validate data descriptor contents.
- Ensure that the compressed and uncompressed lengths in the data
  descriptor match those in the central directory.

- Calculate CRCs for compressed entries and ensure that those CRCs
  match the information in the central directory / data descriptor.
  This is currently guarded by a flag and will be turned on at a
  later stage.

- Finally, always trust the LFH when it comes to indicating the
  existence of a data descriptor. We no longer require this info to
  match the central directory to bring this in line with other zip
  processing tools. (see NOTE in the included change).

Bug: 62047801
Test: zip_archive_test
Change-Id: Icceea66a8b1937c08a4af9aee97bc77ad7218c0f
2017-06-14 14:29:53 +01:00
Adam Lesinski
1d105aa597 libziparchive: verify that gpb flags match
The Central File Header and Local File Header should
have the same general purpose bit flags set for
consistency.

Bug: 36686974
Test: existing tests pass (ziparchive-tests)

(cherry picked from commit 73b3aa541f)

Change-Id: Ia672a28732154a564ea2f2e3520238bb77924a56
2017-06-14 14:25:55 +01:00
Elliott Hughes
55fd293dfe Add libziparchive-based unzip.
Bug: N/A
Test: new toybox unzip.test
Change-Id: I00be388578be9a0a0390a9e2ecfac664c6ab39ca
2017-06-04 11:53:48 -07:00