Commit graph

17 commits

Author SHA1 Message Date
Zhenhuang Wang
b8451b8448 Support storing SHA256 checksum for files in soong_zip
Add a -sha256 argument to soong_zip that computes SHA256 checksum for
each file and stores the value in file header. The SHA information can
then be used by downstream systems that use content addressing.

Bug: 259513199
Test: zip_test.go
Test: soong_zip -o test.zip -D test_dir -sha256
Change-Id: I20e9f424bd0a4604f0dc7cc77bd65f10eb49a163
2023-01-25 10:00:58 +08:00
Colin Cross
7ddd08ad2b Allow duplicate files inputs in soong_zip
Accept duplicate file inputs in soong_zip when they are the same
source file.  This came up when trying to zip lint srcs, as some
java modules have duplicate source files that seem to be ignored
by javac.

Test: TestZip
Bug: 216456886
Change-Id: I8c43df9aded8cf094afaed79cca2b9eb091cc861
2022-08-15 16:05:03 -07:00
Chris Gross
fa5b4e909b Use broader permissions when archiving files.
Using broader premissions for archived files allows them to be more
easily used when extracted. For example, defaulting regular files to
0644 will allow other tooling to use a file without the need to change
permissions manually.

Bug: 189919409
Test: m dist and inspected file permissions of archived files
Change-Id: I4a0f8075206391254c639ecf865639bb9e8df0bf
2021-06-02 16:14:45 -07:00
Colin Cross
e070c3a26b Always call SetMode in soong_zip
In addition to setting the mode, SetMode also marks the file
as created by Unix, which seems to cause Debian's unzip tool
to handle UTF-8 filenames correctly.

Fixes: 184575223
Test: TestZip
Change-Id: I0e839dc3d27aaa9abced0eb1d3c4c0f8eed3e3c4
2021-05-11 16:14:39 -07:00
Colin Cross
fd708b5651 Move response file handling to a separate package
sbox is going to need to read and write response files, move ReadRspFile
to its own package.

Test: response_test.go
Change-Id: Iecb5486b4aaeb2531828743ad8ef784df675e18e
2021-03-25 11:06:45 -07:00
Colin Cross
caf4d4c270 soong_zip: don't glob RSP file entries
File arguments are considered globs, but RSP file entries should be
considered filenames without globbing.  Escape RSP file entries so
they don't get treated as globs later.

Bug: 179297188
Test: zip_test.go
Change-Id: I1de7829c668538204fc4a2ac793a4f88dbc886c9
2021-02-03 16:06:20 -08:00
Sasha Smundak
8eedba6184 Document usage, perform minor cleanups.
Test: treehugger
Bug: 173466220
Change-Id: I5aa09c2941c0eceb8ae635b9e613ad932d38aebb
2020-11-16 19:00:27 -08:00
Colin Cross
405af07859 Revert "Make lots of tests run in parallel"
This reverts commit 323dc60712.

Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests

Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
2020-10-09 18:34:24 -07:00
Colin Cross
323dc60712 Make lots of tests run in parallel
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.

Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
2020-10-06 15:12:22 -07:00
Colin Cross
053fca10c9 Support ninja rsp files in soong_zip
Add a -r argument to soong_zip that reads a list of files from a file
like the -l argument but treats it as a Ninja rsp file with escaping.
Replace the -l arguments in Soong that are using rsp files with -r.

Fixes: 162435077
Test: TestReadRespFile, TestZip
Change-Id: I4605312e99406ab1bd0c37af9c5ad212393f0403
2020-08-19 21:18:56 +00:00
Jiyong Park
04bbf98e06 -l option soong_zip can accept a file having space separated list
-l option of soong_zip is used to specify the list of input files that
should be zipped. However, it only accepted newline-separated list in the
file. This prevented us from using a rspfile as paths in rspfile are
space-separated in a single line. Fixing the problem by splitting the
file content by any unicode whitespace character including newline and
space.

Test: zip_test
Change-Id: Iba572109e30c01540eacf20bd2794ba60c870fa3
2019-11-04 14:23:07 +09:00
Colin Cross
9cb51dbc0b Support moving sources in srcjars in soong_zip
Add a -srcjar argument to soong_zip that causes it to read the
package statement of each .java file and use that to place the
source file at a path that matches the package.

Test: jar_test.go, zip_test.go
Change-Id: I36017e42445ba3b0a82a10a8d81e8ac0cca096f2
2019-06-18 13:33:20 -07:00
Colin Cross
4be8f9e2a3 soong_zip: add --ignore_missing_files flag
soong_zip builds a list of files to zip early and then starts
zipping them all.  If a directory being zipped is concurrently
modified, a file that existed when soong_zip started may not
still exist.  Add a flag that continues when an expected file
does not exist.  Print a warning, since this should be rare
in normal usages but is a sign of a problem if it happens
regularly.

Test: zip_test.go
Test: m checkbuild
Test: m platform
Change-Id: I78426fe66fded8528ddd436c0f71a7442183cfeb
2018-09-28 16:14:52 -07:00
Colin Cross
09f11056f8 Add a --symlinks argument to soong_zip
Add a --symlinks argument that defaults to true to soong_zip.
Passing --symlinks=false will cause it to follow symlinks instead
of storing them in the zip file.

Relands I4deb98daa9d4ba9f94e3d7670c117fe00381d2ba with tests.

Bug: 112843624
Test: glob_test.go
Test: zip_test.go
Test: m checkbuild
Change-Id: I0eff9c1f2dba79e873fda381ff585df55d5aaaad
2018-09-28 14:01:33 -07:00
Colin Cross
1d98ee23a3 soong_zip: support globs in -f and -D arguments
-f and -D arguments can now take globs in the Soong format.

Also update the use of soong_zip that jars resources to escape the
globs in the arguments, and then shell-escape them when writing to
the rsp file so the glob escape are not intepreted by ReadRespFile.

Also remove an unused argument to the buildAAR rule that could
have contained values that needed escaping.

Relands I7f20bb169dc01f952d2a7681ec6ee9c05737ed37 with a fix for
trailing "\n" in list files, which causes a call to pathtools.Glob("")
that returns "./", which could then get incorrectly translated to
"../../../" in the zip file. Also adds tests.

Test: m checkbuild
Test: zip_test.go
Change-Id: I54b8eef9231875e6042a32c9f8bcc5c2f779922a
2018-09-28 14:01:29 -07:00
Colin Cross
05518bc13b soong_zip: Add tests
Add test that cover basic command line usage of soong_zip.  -D
is not covered yet as the implementation will be replaced with
one that is also more easily testable in the next patch.

Bug: 116751500
Test: zip_test.go
Change-Id: I5a1bcee74ebc9cb3cf332c36f89bc12c0e807ad2
2018-09-28 13:56:06 -07:00
Nan Zhang
674dd932d8 Add Respfile support for soong_zip.
Sometime the size of our command line passed to soong_zip go program
exceeds the cmdline size limit. So add an RespFile support with "@" special
character prefix.

The args in the cmdline will be considered together with the
args in RespFile during soong_zip running.

Test: real tests in my local machine, and compare the
res/libphonenumber.jar before and after changes.

./cmd -o test.zip '""'-C -> [./cmd,-o,test.zip,""-C]
./cmd -o test.zip '-C -f -> [./cmd,-o,test.zip,-C -f]
./cmd -o test.zip '\"'-C -f -> [./cmd,-o,test.zip,\"-C -f]
./cmd -o test.zip '\\'-C -f -> [./cmd,-o,test.zip,\\-C -f]
./cmd -o test.zip '\a'-C -f -> [./cmd,-o,test.zip,\a-C -f]

./cmd -o test.zip \'-C -> [./cmd,-o,test.zip,'-C]
./cmd -o test.zip \\-C -> [./cmd,-o,test.zip,\-C]
./cmd -o test.zip \"-C -> [./cmd,-o,test.zip,"-C]

./cmd -o test.zip "'"-C -> [./cmd,-o,test.zip,'-C]
./cmd -o test.zip "\\"-C -f -> [./cmd,-o,test.zip,\a-C -f]
./cmd -o test.zip "\""-C -f -> [./cmd,-o,test.zip,"a-C -f]

Bug: b/72484223

Change-Id: I83c3630b70c8396c8e8a3f266244d868d754c4e8
2018-02-13 15:13:01 -08:00