We want to measure how much of build time is spent on writing large
zips to disk. We can do this by doing the rest of the work but
skipping the write itself.
Some local and trivial testing shows this is probably a lot of the
cost of these rules:
$ python -m timeit -s "import os" -r 10 "os.system('soong_zip -write_if_changed -o foo.zip -D prebuilts/sdk')"
1 loop, best of 10: 8.37 sec per loop
$ python -m timeit -s "import os" -r 10 "os.system('soong_zip -o foo.zip -D prebuilts/sdk')"
1 loop, best of 10: 5.39 sec per loop
$ python -m timeit -s "import os" -r 10 "os.system('soong_zip -n -o foo.zip -D prebuilts/sdk')"
1 loop, best of 10: 3.66 sec per loop
Bug: None
Test: treehugger
Change-Id: I43acdb08d0e00efaba9e5d7add972a7ec41646d2
soong_zip normally takes the name of the input source file as the
name of the file in the zip, which is ideal for zipping up
directories but not for constructing arbitrary zip files. Add a
-e argument that explicitly sets the path in the zip file for
the next -f argument.
Bug: 254867347
Test: zip_test.go
Change-Id: If9d62c1a0064a485aebddf6d2a661f63f3e60b0f
Add a -sha256 argument to soong_zip that computes SHA256 checksum for
each file and stores the value in file header. The SHA information can
then be used by downstream systems that use content addressing.
Bug: 259513199
Test: zip_test.go
Test: soong_zip -o test.zip -D test_dir -sha256
Change-Id: I20e9f424bd0a4604f0dc7cc77bd65f10eb49a163
This is an improvement over directly referencing the prebuilt because
bp2build can now emit BUILD files that contain a dep to
//build/soong/zip/cmd:soong_zip even though we're not converting the
soong_zip module.
It helps to make progress despite not having go rules support to
convert/build soong_zip. This unblocks many modules, like genrules and
gensrcs that use soong_zip in tools. soong_zip doesn't change often, so
it's not risky to depend on the prebuilt instead of source version.
Bug: 194644518
Bug: 248222296
Test: bp2build CI
Change-Id: Idef0c27ec0fb43a9097d750ec72fba00644c9f2a
sbox is going to need to read and write response files, move ReadRspFile
to its own package.
Test: response_test.go
Change-Id: Iecb5486b4aaeb2531828743ad8ef784df675e18e
Add a -r argument to soong_zip that reads a list of files from a file
like the -l argument but treats it as a Ninja rsp file with escaping.
Replace the -l arguments in Soong that are using rsp files with -r.
Fixes: 162435077
Test: TestReadRespFile, TestZip
Change-Id: I4605312e99406ab1bd0c37af9c5ad212393f0403
Add a -srcjar argument to soong_zip that causes it to read the
package statement of each .java file and use that to place the
source file at a path that matches the package.
Test: jar_test.go, zip_test.go
Change-Id: I36017e42445ba3b0a82a10a8d81e8ac0cca096f2
It was not printing out the flag descriptions because it was using the
default FlagSet instead of the custom FlagSet when calling
PrintDefaults.
Test: soong_zip --help
Change-Id: I8a5705f8701ce0c05dde144b9a14962faa384e65
soong_zip builds a list of files to zip early and then starts
zipping them all. If a directory being zipped is concurrently
modified, a file that existed when soong_zip started may not
still exist. Add a flag that continues when an expected file
does not exist. Print a warning, since this should be rare
in normal usages but is a sign of a problem if it happens
regularly.
Test: zip_test.go
Test: m checkbuild
Test: m platform
Change-Id: I78426fe66fded8528ddd436c0f71a7442183cfeb
Add a --symlinks argument that defaults to true to soong_zip.
Passing --symlinks=false will cause it to follow symlinks instead
of storing them in the zip file.
Relands I4deb98daa9d4ba9f94e3d7670c117fe00381d2ba with tests.
Bug: 112843624
Test: glob_test.go
Test: zip_test.go
Test: m checkbuild
Change-Id: I0eff9c1f2dba79e873fda381ff585df55d5aaaad
Add test that cover basic command line usage of soong_zip. -D
is not covered yet as the implementation will be replaced with
one that is also more easily testable in the next patch.
Bug: 116751500
Test: zip_test.go
Change-Id: I5a1bcee74ebc9cb3cf332c36f89bc12c0e807ad2
Profiling is only used by the standalone soong_zip command,
move it out of the shared zip library.
Bug: 116751500
Test: m checkbuild
Change-Id: I443c34fb39cf8955e163a7720d6f7ed585e4172a
Add a builder that can construct []FileArg for zip.Run to use.
This will help writing tests that cover the command line parsing.
Test: later cl
Bug: 116751500
Change-Id: I15b5144f5094ab154f13de9c8a84b82db2da4e67
This reverts commit 08e28abc4e.
Reason for revert: inserts extra ../../ entries in the zip file.
Bug: 116737386
Change-Id: I048f94889a66fe618058e37a5827d0cf6b1a6b68
Add a --symlinks argument that defaults to true to soong_zip.
Passing --symlinks=false will cause it to follow symlinks instead
of storing them in the zip file.
Bug: 112843624
Test: glob_test.go
Test: m checkbuild
Change-Id: I4deb98daa9d4ba9f94e3d7670c117fe00381d2ba
-f and -D arguments can now take globs in the Soong format.
Also update the use of soong_zip that jars resources to escape the
globs in the arguments, and then shell-escape them when writing to
the rsp file so the glob escape are not intepreted by ReadRespFile.
Also remove an unused argument to the buildAAR rule that could
have contained values that needed escaping.
Test: m checkbuild
Change-Id: I7f20bb169dc01f952d2a7681ec6ee9c05737ed37
When -j is specified ignore the path to the source file and just
zip it with its filename. -j overrides -C, and -C overrides -j,
so -j -f path1 -C dir -f path2 will junk the path for path1, but
treat path2 as relative to dir.
Remove the filepath.Clean for the FileArgs, it would convert ""
to "." sometimes, and everything gets cleaned in zip already for
the non-command-line use cases.
Test: m checkbuild
Change-Id: I7d90572c622ee0c6f35967ff31d067b5312c72eb
Sometime the size of our command line passed to soong_zip go program
exceeds the cmdline size limit. So add an RespFile support with "@" special
character prefix.
The args in the cmdline will be considered together with the
args in RespFile during soong_zip running.
Test: real tests in my local machine, and compare the
res/libphonenumber.jar before and after changes.
./cmd -o test.zip '""'-C -> [./cmd,-o,test.zip,""-C]
./cmd -o test.zip '-C -f -> [./cmd,-o,test.zip,-C -f]
./cmd -o test.zip '\"'-C -f -> [./cmd,-o,test.zip,\"-C -f]
./cmd -o test.zip '\\'-C -f -> [./cmd,-o,test.zip,\\-C -f]
./cmd -o test.zip '\a'-C -f -> [./cmd,-o,test.zip,\a-C -f]
./cmd -o test.zip \'-C -> [./cmd,-o,test.zip,'-C]
./cmd -o test.zip \\-C -> [./cmd,-o,test.zip,\-C]
./cmd -o test.zip \"-C -> [./cmd,-o,test.zip,"-C]
./cmd -o test.zip "'"-C -> [./cmd,-o,test.zip,'-C]
./cmd -o test.zip "\\"-C -f -> [./cmd,-o,test.zip,\a-C -f]
./cmd -o test.zip "\""-C -f -> [./cmd,-o,test.zip,"a-C -f]
Bug: b/72484223
Change-Id: I83c3630b70c8396c8e8a3f266244d868d754c4e8
Simplify making restat rules by adding a --write_if_changed argument
to soong_zip that generates the output file into memory and then
compares it to the version on disk before writing it out.
Test: builds
Change-Id: I5059a1e3a11e79b0ca538b3b829bc7479c126ce6
to make it faster/easier to invoke from other Go programs
(such as multiproduct_kati)
Bug: 67478260
Test: m -j
Change-Id: Idd2671a44290550197c88f53dd11a6dd39c85cc5
