Commit graph

203 commits

Author SHA1 Message Date
Nick Kralevich
3bdf744a55 zip_archive.cc: add O_CLOEXEC
Prevent file descriptors from leaking across an exec() boundary.

Bug: 120983106
Test: compiles and boots
Change-Id: I392b0767674b557b1e4404a2ba63bc48e3e37b24
2018-12-18 12:48:06 -08:00
Elliott Hughes
f2228e0cc5 Merge "TEST_MAPPING: add the rest of our actively maintained tests." 2018-11-28 00:56:36 +00:00
Josh Gao
5db84229e1 Fix double closes in ziparchive tests.
Bug: http://b/119632446
Test: setprop debug.fdsan fatal && /data/native64/ziparchive-tests/ziparchive-tests
Change-Id: I3a9e6f3823ba49c56602041779cb085a6971ca6d
2018-11-16 19:54:41 +00:00
Mark Salyzyn
9f1cf25332 switch to using android-base/file.h instead of android-base/test_utils.h
Test: compile
Bug: 119313545
Change-Id: I4f7ad84743e974b4b4d1d7256088f6c8b749a237
2018-11-14 09:35:34 -08:00
Elliott Hughes
e038f49bca TEST_MAPPING: add the rest of our actively maintained tests.
We can't add the ones that need a shared library because the
infrastructure doesn't work yet. (We also can't comment this in the file,
because there's no support for comments :-( .)

Bug: N/A
Test: N/A
Change-Id: I4d84f962bbf48fc708df336726c18e48fe206492
2018-10-30 11:24:57 -07:00
Elliott Hughes
b70d19e75e libziparchive: use lseek directly.
The Mac doesn't have lseek64, and this code is built with
_FILE_OFFSET_BITS=64 anyway.

Bug: N/A
Test: ran tests
Change-Id: Ibda49c44ecfbe21b304e163e48f7ea42d97e38a7
2018-10-26 21:27:38 -07:00
Yuchao Zhou
1e6f4d8dab Merge "Yet another Mac fix." 2018-10-26 23:20:24 +00:00
Elliott Hughes
9c8bd669e8 Yet another Mac fix.
system/core/libziparchive/zip_archive.cc:847:36: error: use of undeclared identifier 'lseek64'; did you mean 'lseek'?
  const off64_t current_offset = lseek64(fd, 0, SEEK_CUR);

Bug: N/A
Test: builds
Change-Id: If762011722d53376bb6dab35c6ee8031762e5a95
2018-10-26 16:14:21 -07:00
Elliott Hughes
a56a729c14 More Mac build fixes.
The libziparchive public headers that refer to `off64_t` also need the
Mac workaround.

In fastboot, there's a stray `lseek64` but since it's only for offset 0,
any kind of seek is fine.

Bug: N/A
Test: builds
Change-Id: I68b4f95202623ebf07ffe6c3e0e21437e7922c5b
2018-10-26 10:53:24 -07:00
Elliott Hughes
e8f4b14301 Add a simple MappedFile to libbase and switch fastboot and libziparchive over.
This allows us to remove libziparchive's dependency on libutils.

Bug: http://b/79112958
Test: ran libbase and libziparchive tests, ran fastboot manually
Change-Id: I95c651976dad222863e5b8c37d4514b778f5dce7
2018-10-23 13:20:49 -07:00
Ryan Prichard
c394f7f7ac Merge "Change ZipArchiveHandle from void* to ZipArchive*" 2018-10-15 23:23:29 +00:00
Ryan Prichard
3673f99dd2 Change ZipArchiveHandle from void* to ZipArchive*
A typedef to void* allows an implicit conversion from ZipArchiveHandle*
(or any other pointer type) to ZipArchiveHandle.

See I95d79809b6e118fb3c39c7b98b8055c8e324db1a in platform/bionic.

Bug: none
Test: m checkbuild
Change-Id: I3dd426cb64c46ef81e1dd81b4a2e4f40ac2701df
2018-10-11 14:18:08 -07:00
Pirama Arumuga Nainar
a2ccce870c Merge changes from topic "windows-libcxx"
* changes:
  Do not enable -Wold-style-cast for Windows
  Add sys/types.h in utf8.h
  Adapt to switch to libc++ for Windows
2018-10-11 17:00:53 +00:00
Chih-Hung Hsieh
747eb149d0 Add noexcept to move constructors and assignment operators.
Bug: 116614593
Test: build with WITH_TIDY=1
Change-Id: I5a7461386946ca623ab509609092aa0ac8418b80
2018-10-05 16:43:47 +00:00
Pirama Arumuga Nainar
53f5945609 Do not enable -Wold-style-cast for Windows
Bug: http://b/91353691

Enable -Wold-style-cast only for non-Windows targets.  _islower_l,
_isupper_l etc. in MinGW locale_win32.h (included from
libcxx/include/__locale) have an old-style-cast.

Test: Build and test Windows modules under Wine.
Change-Id: Ib7594559a43096885b0cc1c656cf59db8b52d38b
2018-09-26 02:14:02 +00:00
Josh Gao
3da42a6c05 Merge "libziparchive: encode type in fdsan owner tag." 2018-09-20 19:22:03 +00:00
Zimuzo
5a503efaf0 Reduce libziparchive internal hashtable memory size
Debugging memory allocations on the nexus launcher unveiled significant memory allocations for the hashatable used in libziparchive, ~1MB.
This is partly because of the ZipString struct storing an entry in the table. The struct stored a pointer to a string (on 64 bit, 8 bytes) and the length to read from that pointer, 2 bytes. Because of alignment, the structure consumed 16 bytes, wasting 6 bytes.
Now, we store entries in the hashtable as a ZipStringOffset. This new structure stores a 4 byte offset from a fixed location in the memory mapped file instead of the entire address, consuming 8 bytes with alignment.

Bug: 79416399
Test: Builds successfully and manual testing by opening launcher on Pixel 2 shows precisely 50% decrease in memory allocated for the hashtable. From 909312 bytes to 454656.
Change-Id: I28b43699233fbee7f63fccae2d4fe96fcc07e5c4
2018-09-19 19:20:21 +01:00
Josh Gao
abdfc248f3 libziparchive: encode type in fdsan owner tag.
Test: debuggerd `pidof system_server` | grep " fd "
Change-Id: I05577adb94cf58b549aa0af03b42653e345bcd11
2018-09-07 14:09:10 -07:00
Elliott Hughes
ab08933f97 Make unzip available.
Bug: N/A
Test: N/A
Change-Id: I7f7a2e5d494536e5eb5f5e8744af224f31351c15
2018-09-04 13:33:30 -07:00
Ryan Mitchell
c77f9d380f Open zips using _wopen on windows.
Allows for opening zip files usng paths longer than 260 characters and
with unicode characters on Windows.

Bug: 113110184
Test: manual and libziparchive_tests
Change-Id: I9ce96ac2f1b1e448ae2a2f69c1d4cb3395ea79ee
2018-08-28 09:31:26 -07:00
Josh Gao
9404f2da14 Merge changes from topic "looper_unique_fd"
am: 7e7cefa2c7

Change-Id: Id920135cff272f593b71ef8cb7ff41ee16de4484
2018-07-23 14:36:43 -07:00
Josh Gao
1b49634f83 libziparchive: use fdsan in ZipArchive.
Test: treehugger
Change-Id: I8586b8ad27c4f1eda1a5266867da8dbbf4870c5e
2018-07-18 18:11:46 -07:00
Jiyong Park
4ebda908de Shared libs are supported in recovery mode
am: 011ee12b1d

Change-Id: I867dbafecf11215b82d952c532774cdf12c6e782
2018-06-13 16:43:09 -07:00
Jiyong Park
011ee12b1d Shared libs are supported in recovery mode
adbd has been built as a static executable since the same binary was
copied to the recovery partition where shared library is not supported.
However, since we now support shared library in the recovery partition,
adbd is built as a dynamic executable.

In addition, the dependency from adbd to libdebuggerd_handler is removed
as debuggerd is handled by the dynamic linker.

A few more modules in /system/core are marked as recovery_available:
true as they are transitive dependencies of the dynamic linker.

This change also includes ld.config.recovery.txt which is the linker
config file for the recovery mode. It is installed to /etc/ld.config.txt
and contains linker namespace config for the dynamic binaries under
/sbin.

Bug: 63673171
Test: `adb reboot recovery; adb devices` shows the device ID
Test: Select 'mount /system' in the recovery mode, then `adb shell`.
$ lsof -p `pidof adbd` shows that libm.so, libc.so, etc. are loaded from
the /lib directory.

Change-Id: I363d5a787863f1677ee40afb5d5841321ddaae77
2018-06-10 08:32:24 +09:00
Elliott Hughes
499b03579e Merge "Make ziparchive-tests run standalone." am: c8a8771b6b
am: e9f842142c

Change-Id: Ideb61d3e2bf01b4cdd341a9fdd7bbe3ca4d62852
2018-04-25 14:57:29 -07:00
Elliott Hughes
bde0ec9e00 Make ziparchive-tests run standalone.
We need to (a) tell soong to copy our data and (b) automatically find
our data relative to our executable.

The real point of this is to be able to run these tests in APCT and
presubmit.

Bug: N/A
Test: ran tests on host and device, from a variety of directories
Change-Id: I4c0be1ac60f03953fdd5ba6e3d15b1aaa37ed019
2018-04-25 12:50:17 -07:00
Jiyong Park
337bc0551d Merge "Mark libziparchive as double_loadable" am: a7112f4c9b
am: b36004c96d

Change-Id: I59160f9ff7ad6a4539617b51b10b9a42b7522479
2018-04-20 21:02:40 -07:00
Jiyong Park
91687878c5 Mark libziparchive as double_loadable
libziparchive is explicitly marked as double_loadable since it is one of the
(indirect) dependencies of the LLNDK library libvulkan
and at the same time the lib itself is marked as VNDK. Such lib can be
double loaded inside a vendor process.

Note: even without this change, the library is already capable of being
double loaded due to the dependency graph around it. This change is to
make it explicit so that double loading of a library is carefully
tracked and signed-off by the owner of the lib.

Bug: 77155589
Test: m -j

Merged-In: Id0a731d553bbb68b84bca421500c94b7b35eca14
Change-Id: Id0a731d553bbb68b84bca421500c94b7b35eca14
(cherry picked from commit 730728cbb4)
2018-04-18 11:46:52 +09:00
Adam Lesinski
fbd68ec85e Merge "Remove empty zip warning on host builds" am: fd42c09813
am: c9c0460e99

Change-Id: Ie22226ad9a9c0dc5f3fbc4a841936cdecabf3cc4
2018-03-02 02:32:13 +00:00
Adam Lesinski
b354dcef98 Remove empty zip warning on host builds
Change-Id: Iaafe208930c316cc32f39ce2687be555b9eeb35f
2018-03-01 21:33:49 +00:00
Narayan Kamath
5c39aa53a9 Merge "zip_archive: Fix a few more badly written tests." am: 93d344d98c
am: 000e0bfff4

Change-Id: I6af4ce2a977e0fbc21974e8e80516fb39ec8156a
2017-12-27 11:47:41 +00:00
Narayan Kamath
c36b80928b zip_archive: Fix a few more badly written tests.
Fix failures in DataDescriptor related tests due to to a bad call to
SetZipString (undefined behaviour). Also fix a typo in the test for
invalid descriptors, we were asserting things on the wrong array.

Test: zip_archive_test
Change-Id: I8c9a632443fdf1d5c115670d6e9317e1f4bf6ef4
2017-12-27 09:42:00 +00:00
Narayan Kamath
780105e6b1 Merge "Get rid of unneeded allocations in Extract...() APIs" am: 18f25d399e
am: 64bf32d999

Change-Id: I0107091854c2d810fc5d19a873c29a270c5cf207
2017-12-22 10:50:24 +00:00
Narayan Kamath
47ede5466a Merge "zip_archive: Fix tests broken by 1f93d71022cca7bb6bb9eec49." am: 492de535c4
am: fede183c53

Change-Id: Ib1e16a1b0799ffe88b822b04d3231c4d30f77917
2017-12-22 10:49:41 +00:00
Narayan Kamath
18f25d399e Merge "Get rid of unneeded allocations in Extract...() APIs" 2017-12-22 10:41:41 +00:00
Narayan Kamath
f37bb8e45b zip_archive: Fix tests broken by 1f93d71022.
Moving to std::hash changed iteration order but these tests should
not have relied on hash_map iteration order anyway.

Test: zip_archive_test

Change-Id: I712bf2307c8770f03ea6f074bfc506a40cdcb066
2017-12-21 12:55:59 +00:00
Yurii Zubrytskyi
834326ce7a Get rid of unneeded allocations in Extract...() APIs
Both Extract...() functions don't need dynamic allocation
for the writers, as those are strictly scoped. This CL
changes heap allocation to stack allocation.

Test: zip_archive_test

Change-Id: Id727e4b9848235cd063cc67ecbe052d21ca21326
2017-12-20 18:36:03 +00:00
Elliott Hughes
1aee775518 Merge "BENCHMARK_MAIN now requires a semicolon." am: 981379b70f
am: 64c59c23f3

Change-Id: I430d1a3abfdc163a0f0ccc215030106ed3a20c6c
2017-12-14 23:34:38 +00:00
Elliott Hughes
62322aae85 BENCHMARK_MAIN now requires a semicolon.
Bug: N/A
Test: ran tests
Change-Id: Ie61d8c775cf23877c1aa7025a50b5f8284ae5b08
2017-12-13 18:19:18 -08:00
Elliott Hughes
aede8375fe Merge "Add OWNERS." am: 3289b9c928
am: 515c8fe572

Change-Id: I378e0205c21a4df5baacde3432b618443f7ef567
2017-12-07 23:51:13 +00:00
Elliott Hughes
3289b9c928 Merge "Add OWNERS." 2017-12-07 23:21:26 +00:00
Elliott Hughes
693d63f9cf Add OWNERS.
Bug: N/A
Test: N/A
Change-Id: Ie785058c0f5eb9b4086c98ccba6e63e3ed411b65
2017-12-07 13:30:03 -08:00
Elliott Hughes
957dc094bf Merge "use std::hash instead of hashing byte by byte" am: aaa0bbce8e
am: bfb96796a4

Change-Id: Ib0cd0e44e34aecbf8af1ccd83c96eb3ebeb71978
2017-11-30 23:34:23 +00:00
Elliott Hughes
bfb96796a4 Merge "use std::hash instead of hashing byte by byte"
am: aaa0bbce8e

Change-Id: Iee321900f8ef386cd118d874c31c5b0d2741551a
2017-11-30 23:25:55 +00:00
Sebastian Pop
1f93d71022 use std::hash instead of hashing byte by byte
This patch uses libcxx's string_view hashing.  The performance is better than the
current implementation, and almost the same as for the hashing implementation amended
by the patch in https://android-review.googlesource.com/404742

The following experiments were conducted on a HiKey aarch64 A53.

No changes:
$ adb push out/target/product/hikey/data/benchmarktest64/ziparchive-benchmarks /data
$ adb shell /data/ziparchive-benchmarks --benchmark_repetitions=5
-----------------------------------------------------------------
Benchmark                          Time           CPU Iterations
-----------------------------------------------------------------
FindEntry_no_match          82279629 ns   81759676 ns          9
FindEntry_no_match          81648056 ns   81627431 ns          9
FindEntry_no_match          81384074 ns   81369057 ns          9
FindEntry_no_match          89618889 ns   82437755 ns          9
FindEntry_no_match          81811389 ns   81785261 ns          9
FindEntry_no_match_mean     83348407 ns   81795836 ns          9
FindEntry_no_match_stddev    3520421 ns     394962 ns          0
Iterate_all_files          137622000 ns  137589032 ns          5
Iterate_all_files          139666333 ns  138409469 ns          5
Iterate_all_files          150070000 ns  140883697 ns          5
Iterate_all_files          138600667 ns  138540646 ns          5
Iterate_all_files          137599833 ns  137567438 ns          5
Iterate_all_files_mean     140711767 ns  138598056 ns          5
Iterate_all_files_stddev     5299929 ns    1354928 ns          0

$ cd system/core/libziparchive
$ git fetch https://android.googlesource.com/platform/system/core refs/changes/42/404742/7 && git cherry-pick FETCH_HEAD
$ mma
$ cd -
$ adb push out/target/product/hikey/data/benchmarktest64/ziparchive-benchmarks /data
$ adb shell /data/ziparchive-benchmarks --benchmark_repetitions=5
-----------------------------------------------------------------
Benchmark                          Time           CPU Iterations
-----------------------------------------------------------------
FindEntry_no_match          53302756 ns   53291178 ns         13
FindEntry_no_match          54314487 ns   54272272 ns         13
FindEntry_no_match          53866923 ns   53851178 ns         13
FindEntry_no_match          53324423 ns   53317296 ns         13
FindEntry_no_match          53289231 ns   53289159 ns         13
FindEntry_no_match_mean     53619564 ns   53604216 ns         13
FindEntry_no_match_stddev     458449 ns     443527 ns          0
Iterate_all_files          128211111 ns  112254783 ns          6
Iterate_all_files          110695000 ns  110677726 ns          6
Iterate_all_files          109351250 ns  109350998 ns          6
Iterate_all_files          109367500 ns  109367796 ns          6
Iterate_all_files          110872222 ns  110591407 ns          6
Iterate_all_files_mean     113699417 ns  110448542 ns          6
Iterate_all_files_stddev     8143723 ns    1194577 ns          0

$ cd system/core/libziparchive
$ git checkout HEAD~
$ git am 0001-use-std-hash.patch
$ mma
$ cd -
$ adb push out/target/product/hikey/data/benchmarktest64/ziparchive-benchmarks /data
$ adb shell /data/ziparchive-benchmarks --benchmark_repetitions=5
-----------------------------------------------------------------
Benchmark                          Time           CPU Iterations
-----------------------------------------------------------------
FindEntry_no_match          55339872 ns   55195112 ns         13
FindEntry_no_match          56232628 ns   56069924 ns         13
FindEntry_no_match          55694103 ns   55255946 ns         13
FindEntry_no_match          55275064 ns   55120136 ns         13
FindEntry_no_match          54971987 ns   54944411 ns         13
FindEntry_no_match_mean     55502731 ns   55317106 ns         13
FindEntry_no_match_stddev     482032 ns     436766 ns          0
Iterate_all_files          114618611 ns  114487804 ns          6
Iterate_all_files          112552917 ns  112229801 ns          6
Iterate_all_files          111288750 ns  111255044 ns          6
Iterate_all_files          111291528 ns  111259045 ns          6
Iterate_all_files          114347222 ns  113677119 ns          6
Iterate_all_files_mean     112819806 ns  112581763 ns          6
Iterate_all_files_stddev     1606214 ns    1454858 ns          0

Change-Id: I1e3413d331bcb460ca38bc2c87e23f89b456cd2f
2017-11-29 09:44:28 -06:00
Xin Li
23e27db576 Merge commit 'a63ccea6abc7ea02e2d98e41c80793ca97237bd3' from
oc-mr1-dev-plus-aosp into stage-aosp-master

Change-Id: Ia33311cd1fd26dfaea59a69317b306fb91203c40
Merged-In: I03d06b10807e8a313c9654c2e1db36bfb59e3f99
2017-11-14 13:19:45 -08:00
Narayan Kamath
70f5571a9a resolve merge conflicts of 5ab03cb41 to oc-mr1-dev-plus-aosp
Test: I solemnly swear I tested this conflict resolution.
Change-Id: Ic749614fb03d60d6cc7e0eb04bb67102e572e9f2
2017-11-01 11:32:17 +00:00
Narayan Kamath
2d1e23f5d6 zip_archive: Allow crc_out to be nullptr in Inflate.
Only compute the crc32 if required. In addition :
- Add unit tests for Inflate that cover this addition.
- Fix an inconsistency in return codes that was revealed
  by this new test.

Bug: 35246701
Test: zip_archive_tests
Test: make; zipalign.
Merged-In: I31d7554378f94fc8995f707471d57cb98311e2c2
Change-Id: I05111bfa665c610f93d1c1dee987a509bf87aa65
2017-10-30 18:26:41 +00:00
Elliott Hughes
cf439ba2d0 Merge "fastboot should fail if it runs out of space while unzipping." am: b88aa023d4
am: ddc4e5cfbf

Change-Id: Icb259596ad6c50c748a47d3224fbd31cab6d8a00
2017-10-28 01:29:36 +00:00
Treehugger Robot
b88aa023d4 Merge "fastboot should fail if it runs out of space while unzipping." 2017-10-28 01:16:29 +00:00