Commit graph

250 commits

Author SHA1 Message Date
Tianjie Xu
323c09c3d0 Move the implementation of cd entry map to a separate file
Move the entry map classes to a separate file to make the hierarchy
clear.

Test: unittests pass
Change-Id: Ie01d7835359daa4f59af75a0eda204c696d5658e
2020-03-16 17:43:49 -07:00
Tianjie Xu
0ef9783c57 Add std::map implementation for cd entry map
Add the std::map implementation to be used later in zip64 format.
Also move the entry map classes to a separate file to make the hierarchy
clear.

Test: unittests pass
Change-Id: I74d95f53207cc8ca871b955e2a15c184d5497833
2020-03-16 15:42:22 -07:00
Tianjie Xu
28f8eaeffb Create an interface for the cd entry hash table
The current implementation of the hashtable uses less memory than
a std::map. As most of the zip files we encountered don't use the zip64
extension, we should keep the current implementation. And the interface
adds the flexibility for us to switch to std::map for zip64 format.

Bug: 150900468
Test: unit tests pass
Change-Id: Ifd008785c9ff416a27049f9e0c54d9eef985bd85
2020-03-10 11:51:53 -07:00
Elliott Hughes
264a37d12f Merge "Fix SEGV in libziparchive with malformed zip file." 2019-12-18 16:14:54 +00:00
Elliott Hughes
fba2a1a1ec Fix SEGV in libziparchive with malformed zip file.
d77c99ebc3 changed MappedFile to return a
bogus zero-length mapping on failure rather than nullptr. None of the
calling code was changed, though, and it seems like doing so would be a
bad idea. Revert that part of the change.

Add missing tests, and tidy up some of the logging. Also remove
single-use or obfuscatory constants from the tests.

The new "empty.zip" was created by using zip(1) to create a zip file
with one entry, then using `zip -d` to remove it.

The new "zero-size-cd.zip" was created by using zip(1) to create a zip
file containing a single empty file, and then hex editing the two byte
"size of the central directory" field in the "end of central directory
record" structure at the end of the file. (This is equivalent to, but
much smaller than, the example zip file provided by the bug reporter.)

Bug: http://b/145925341
Test: treehugger
Change-Id: Iff64673bce7dae886ccbc9dd6c2bbe18de19f9d2
2019-12-17 08:39:09 -08:00
Elliott Hughes
8748bcc650 Rename unzip.cpp to ziptool.cpp.
It contains unzip and zipinfo, and will likely contain zip too soon.

Test: builds
Change-Id: I017df302108847f29bfdd120f20bf0fd3b9caa5b
2019-12-13 16:47:22 -08:00
Elliott Hughes
f276140d0f cli-test: a tool for testing command-line programs.
Not looking for other users right now, this is just enough to test
unzip/zip/zipinfo.

This includes tests for unzip and ziptool, along with a change to
unzip's behavior to fix AOSP `make dist` when using ziptool unzip.

Also add the boilerplate to run these tests on the device, in presubmit.

Fix command name in --help output.

Test: atest ziptool-tests
Change-Id: I5c0215a3ab8cb2cd5fc517ed9c188f81a7bf4520
2019-12-13 12:23:51 -08:00
Elliott Hughes
2ab5a70b9f ziptool: fix unknown long options.
Previously an unknown long option would cause a crash as we ran off the
end of the array.

Test: `ziptool unzip --unknown`
Change-Id: I7a7b6ac4a0fa157c111f936e837c20143cef9e28
2019-11-16 11:18:50 -08:00
Nick Desaulniers
4e7507ffc2 libziparchive: fix -Wimplicit-int-float-conversion
The value of uncompressed may not be precisely representable when
implicitly casted to an IEEE 754 single precision float.

Assuming the code doesn't need a precise compression ratio, accept the
potential imprecision via explicit cast.

system/core/libziparchive/unzip.cpp:114:68: error: implicit conversion
from 'int64_t' (aka 'long') to 'float' may lose precision
[-Werror,-Wimplicit-int-float-conversion]
  return static_cast<float>(100LL * (uncompressed - compressed)) / uncompressed;
                                                                 ~ ^~~~~~~~~~~~

Change-Id: If46cfa4eb2bb16a7491e52bb5d1c212ed5d59079
Bug: 139945549
Test: lunch hikey960-userdebug && mm
Signed-off-by: Nick Desaulniers <ndesaulniers@google.com>
2019-11-13 13:04:22 -08:00
Elliott Hughes
008759711f unzip/zipinfo: use float percentages like the RI.
Test: my new test runner, specifically developed for ziptool
Change-Id: I1237d02daaf2939eebc4fd5ec19ccdd0de291ad5
2019-11-12 13:53:43 -08:00
Treehugger Robot
16366e36d5 Merge "ziparchive: add a corpus for the fuzzer." 2019-11-08 16:22:41 +00:00
Elliott Hughes
9e63ba787d ziparchive: add a corpus for the fuzzer.
Reuse the existing test data.

Test: ran fuzzer on host, saw a lot more log spam
Change-Id: If57e4b8708832d7296b118e6926d41951d4a3ca9
2019-11-07 14:24:04 -08:00
Elliott Hughes
f1b255a6ad Darwin: include <libgen.h> for basename(3).
Test: treehugger
Change-Id: I11858430f13cbb22895c9dd0befb8054308efb19
2019-11-04 19:27:33 -08:00
Elliott Hughes
bcd810622b unzip: fix Mac build.
Turns out the Mac doesn't have <error.h>. Add our own "die" function
instead, and use it everywhere so the Mac isn't using an untested
codepath.

One upside to this is that we'll now call ourselves "unzip" even when
run as `ziptool unzip ...`, which was awkward to fix with <error.h> but
trivial if we're rolling our own anyway.

Test: still works on Linux
Change-Id: I9cb1922595a21cd9f6d55a70d67e30090f8b7f21
2019-11-03 08:30:33 -08:00
Elliott Hughes
d3aee6653f unzip: add -Z for "zipinfo mode".
But don't document it because it's a silly idea. Just call zipinfo
directly if you want zipinfo!

There are multiple uses of `unzip -Z` in the AOSP build, though, so we
may as well support it if people are already using it.

Test: manual
Change-Id: I04b05795badf63febe1210fbeaa96e3bd27237f1
2019-10-29 20:47:16 -07:00
Elliott Hughes
d50952587d zipinfo: support DOS attributes.
golang doesn't include Unix mode by default.

Also show all the deflate variants ("defN" versus "defX").

Cope better with being called directly rather than via symlink.

Test: manual
Change-Id: I23b441c847ce9a557ea866b3c43bdf0542b26f10
2019-10-29 07:55:16 -07:00
Elliott Hughes
2672413d91 libziparchive: add zipinfo(1).
Useful for debugging and hermetic builds. (Various places in the build
check to see that a file was stored uncompressed.)

Test: manual
Change-Id: I127e5689cd493ab06739b765beed50912dc9cc1d
2019-10-25 10:07:08 -07:00
Elliott Hughes
f66460b92a libziparchive: add trivial fuzzer.
Didn't find anything when I ran it, but it did get me to fix the
const/non-const void* in the API.

Test: treehugger
Change-Id: If3849d974965e3e5ffcbdaf5e47921316d717410
2019-10-22 11:45:49 -07:00
Donald Chai
e170d7fe85 Avoid using data descriptors in ZIP files when possible.
These add 16 bytes per ZIP entry, and are usually avoidable.  APKs contain thousands of
deflated entries, so this overhead adds up to tens of kilobytes.

Bug: 135470635
Change-Id: Ib928aa41dd55cacc41f7394c218c4340d3bbd570
2019-07-23 06:58:53 +00:00
Elliott Hughes
2226fe6ddf Merge "Finally remove ZipString." 2019-06-20 19:20:39 +00:00
Elliott Hughes
50ef29a170 Finally remove ZipString.
Bug: http://b/129068177
Test: treehugger
Change-Id: If8c009f96931c9c2672255d8d0fe01d7992282af
2019-06-19 15:26:38 -07:00
Yurii Zubrytskyi
a6633d7739 [zip] Save 1 malloc and memset for each added file in ZipWriter
+ add a benchmark for the function.

This change speeds up the function by about 3%: 910ns->880ns

Change-Id: I33c8c31de18d10eb38f109917ecbcbdda45b4034
2019-06-18 21:49:16 -07:00
Yurii Zubrytskyi
2b283118a0 [zip] Change const char* to string_view in ZipWriter
This would allow adding entries from one zip archive into
a new one without copying, directly from a ZipString object

Change-Id: I52f91008f497e798e044c43f57a6481cf4bec36d
2019-06-18 21:00:43 -07:00
Elliott Hughes
1e40c30b0c ziparchive: add a std::string_view overload to Next.
Recovery wanted this, and frameworks/base/ wants it too.

Bug: http://b/129068177
Test: treehugger
Change-Id: I8ee3f7c058fc9c1cde829da613ed15be5ce7b41e
2019-06-12 12:12:47 -07:00
Elliott Hughes
e06a808037 Add a std::string overload to Next.
All but one existing caller actually wants a std::string.

Bug: http://b/129068177
Test: treehugger
Change-Id: I428c4453edaae74451db56e9542e4e462f08d43a
2019-05-22 19:05:44 -07:00
Elliott Hughes
13a45c01f2 libziparchive: remove now-unused StartIteration overload.
Bug: http://b/129068177
Test: treehugger
Change-Id: If494c3031aee2bd3e72eda57de4c334f11f5a5df
2019-05-10 15:00:37 -07:00
Elliott Hughes
a22ac0f07e libziparchive: start moving to a non-ZipString StartIteration API.
Same issue as with FindEntry: using ZipString in the API forces all
callers to make sure they don't hit the ZipString length limits. Switch
to std::string_view and uniformly use the empty string as a way to
signal no prefix/suffix rather than nullptr.

Also use default arguments to make the common case of no prefix and no
suffix more convenient.

Also just use std::string to increase the lifetime of the provided
prefix/suffix rather than manual memory management.

Bug: http://b/129068177
Test: treehugger
Change-Id: I6675e39ce62fadd766386d77d27423013c17d6f7
2019-05-08 11:00:32 -07:00
Elliott Hughes
1d5745fb40 Merge "libziparchive: remove now-unused FindEntry overload." 2019-05-08 17:05:46 +00:00
Dimitry Ivanov
9a45f8ff4c Merge "Enable native_bridge_support" 2019-05-07 21:47:41 +00:00
Elliott Hughes
a5ff19e7e9 libziparchive: remove now-unused FindEntry overload.
Bug: http://b/129068177
Test: treehugger
Change-Id: I53da90bb61b0299aca545f9a1420f64e3f909657
2019-05-07 09:27:59 -07:00
dimitry
a808b1150a Enable native_bridge_support
Enable native bridge support for libbase, liblog,
libziparchive and libpropertyinfoparser.

This makes it possible to use them in binaries for translated
architectures.

Bug: http://b/77159578
Test: make
Change-Id: If67ce92288b17a052ea1e79a268e284f7d941439
2019-05-06 14:05:05 +02:00
Elliott Hughes
b17bf521d5 libziparchive: report errors on over-long names.
Switch FindEntry and the ZipString constructor to std::string_view. This
lets us accept an over-long name so that we can reject it as too long.

Also fastboot changes to track the API change.

Bug: http://b/129068177
Test: treehugger
Change-Id: I7df7acd1fe2c46380b789c25f8909e0553e2d55e
2019-05-04 08:41:12 -07:00
Treehugger Robot
5664489a31 Merge "Ziparchive: Enable -Wconversion" 2019-04-19 03:33:19 +00:00
Elliott Hughes
5f8b309883 unzip: support shell globs in include/exclude lists.
Bug: http://b/113928508
Test: manual
Change-Id: Ic9b11486ce07bf3b385e0c1180c4d1bde61d1628
2019-04-08 12:46:56 -07:00
Nick Kralevich
ba80ab1373 Merge "zip_archive.cc: Use static cast instead of masking" 2019-04-08 16:23:51 +00:00
Andreas Gampe
964b95cf61 Ziparchive: Enable -Wconversion
Enable -Wconversion (but not -Wsign-conversion). Fix up code. Handle
some actual error cases:

* too many files
* files too large

Bug: 130039052
Test: atest ziparchive-tests
Change-Id: I632af317b9e65dbc728851efefd0d11a2b5c29b9
2019-04-05 13:50:48 -07:00
Nick Kralevich
2871411902 zip_writer_test.cc: fix implicit integer truncation
converting from a size_t to a uint8_t results in a loss of precision,
which triggers ubsan's implicit-unsigned-integer-truncation checker.
Make the conversion explicit.

This change allows the ziparchive-tests to pass with ubsan enabled.

Test: atest ziparchive-tests
Test: compiles and boots
Bug: 122975762
Change-Id: I63f28b58f1ca1f4c57323494cb1f4a41e0f34fba
2019-04-05 10:08:44 -07:00
Nick Kralevich
c0bf36647b zip_archive.cc: Use static cast instead of masking
Although ubsan's implicit-unsigned-integer-truncation sanitizer may be
happy, this code still performs an implicit conversion from a wider
width data structure to a narrower width data structure. Rather than
masking the bits, make the conversion explicit. This keeps ubsan happy
as well as addressing a -Wconversion warning.

This change addresses comments from the post-submit review
of a4e5433660.

Test: compiles and boots.
Bug: 122975762
Change-Id: I1fa6d6f8a6fcfb93ba9916b7d2b3564ca1d8caf3
2019-04-05 09:32:14 -07:00
Nick Kralevich
a4e5433660 zip_archive.cc: fix ubsan false positive
std::hash returns a 64 bit value, which is truncated to a 32 bit value
in ComputeHash. ubsan's implicit-unsigned-integer-truncation doesn't
like this implicit truncation and crashes the program. Explicitly strip
off the top order bits after computing the hash.

Remove the windows specific version of the hash computation. The windows
compile now uses clang, so this code is obsolete. This also avoids us
having to add __attribute__((no_sanitize("integer"))) to the windows
code.

This is needed to support Android booting with ubsan's
implicit-unsigned-integer-truncation option enabled.

Test: compiles and boots
Bug: 122975762
Change-Id: I2f05fbf5ffee8e90a66a6fda32e80de9cca246c0
2019-04-04 14:53:11 -07:00
Nick Kralevich
6d3c80b85a zip_writer.cc: update powerof2 and use it unconditionally
Update the powerof2 macro to one that is integer overflow safe.
This definition is the same one used by bionic, but cleaned up using
clang_format.

Unconditionally use this powerof2 macro, to ensure consistency across
all the build environments.

Test: compiles and boots
Change-Id: I21bfaf2360d1f77caf96365e20490b1deadfdfe0
2019-03-27 10:24:44 -07:00
Treehugger Robot
80e72443d0 Merge "libziparchive: add integer checks" 2019-03-27 13:03:16 +00:00
Nick Kralevich
832bd17a0b Merge "zip_archive.cc: delete android_errorWriteLog" 2019-03-23 03:08:35 +00:00
Nick Kralevich
104771e55e zip_archive.cc: delete android_errorWriteLog
The process of determining whether or not to emit a safetynet error
entry while processing a malformed file relies on addition overflow.
Since this is only logging, and logging which isn't used, delete the
code instead of trying to fix the logic which is causing the integer
overflow.

This change is necessary to enable integer sanitization on this code.

Somewhat related to Bug: 122975762
Test: atest ziparchive-tests

Change-Id: I6b41ccf7881348cb4e5236324eaa44a05662a725
2019-03-21 09:58:56 -07:00
Nick Kralevich
c5da03f448 entry_name_utils-inl.h: clear top bit before shifting
Attempting to shift a uint8_t which has the top bit set results in a
value outside of the range allowable for a uint8_t. The process of
converting this value back to a uint8_t triggers ubsan's
implicit-conversion sanitizer.

Ensure the high order bit is stripped before shifting, so that the
shifted value fits into the allowable uint8_t range.

This change is necessary to enable integer sanitization on this code.

Somewhat related to Bug: 122975762
Test: atest ziparchive-tests

Change-Id: I940c8b2828ac88348d80f731274b990e35de6631
2019-03-21 09:29:31 -07:00
Nick Kralevich
30dec070dd libziparchive: add integer checks
The code in libziparchive has lots of questionable looking but not
obviously wrong integer operations. In order to shake out integer bugs
in libziparchive (for example, commit
1ee4892e66 from bug 31251826) and provide
protection against security bugs, enable some integer sanitization
options in libziparchive.

Bug: 122975762
Bug: https://bugs.chromium.org/p/chromium/issues/detail?id=941802
Test: device boots and no obvious problems.
Change-Id: I215d81892a6eff12d692648c69a03e8200b334d7
2019-03-20 13:04:02 -07:00
Julien Desprez
f4f54b1d69 Revert the explicit config, see topic
Other CL in topic address the issue of the file pusher.
The explicit config for this module will not be required
anymore.

Test: atest -v ziparchive-tests
Bug: 124515549
Change-Id: I4dad8adbce0817009158bc191c2cce86c38d9e3e
2019-02-20 13:29:11 -08:00
Andreas Gampe
881e2e198b Ziparchive: Give tests a test config
To work around problems with the autogenerated one.

Bug: 117891984
Bug: 124515549
Test: atest ziparchive-tests
Change-Id: Ia4b352b7404255a4fe9e644a56ae9c5d41c79886
2019-02-15 11:53:49 -08:00
Nick Kralevich
3bdf744a55 zip_archive.cc: add O_CLOEXEC
Prevent file descriptors from leaking across an exec() boundary.

Bug: 120983106
Test: compiles and boots
Change-Id: I392b0767674b557b1e4404a2ba63bc48e3e37b24
2018-12-18 12:48:06 -08:00
Elliott Hughes
f2228e0cc5 Merge "TEST_MAPPING: add the rest of our actively maintained tests." 2018-11-28 00:56:36 +00:00
Josh Gao
5db84229e1 Fix double closes in ziparchive tests.
Bug: http://b/119632446
Test: setprop debug.fdsan fatal && /data/native64/ziparchive-tests/ziparchive-tests
Change-Id: I3a9e6f3823ba49c56602041779cb085a6971ca6d
2018-11-16 19:54:41 +00:00