The make javac rule uses [ -s srcjar-list ] to decide whether or
not to run javac. zipsync was putting a trailing "\n" in the
list file, so the file was never empty. There was one case of a
package that contained no source files (so java-source-list was
empty) and had a res directory with an xml file that declared no
resources (so aapt.jar was created but empty). The 1-byte
srcjar-list file caused javac to run and error with:
javac: no source files
Test: m checkbuild
Change-Id: I26b394b66bf81e4f5abbd27e4dc06bee1d9420a8
extract_srcjars.sh uses zipinfo and unzip, which fail with an
error on an empty zip file. Instead of trying to hack around
this (which is hard to make guarantees for since they are
non-hermetic host tools), replace them with a go tool to unzip
a set of zip files into a directory.
Bug: 73885582
Test: m checkbuild
Change-Id: I151fed347ed5196726e36866ffc27bc831799afb
Each Python module will generate a zip file containing source & data
files. The Python binary will collect all its dependencies and use
merge_zips to merge each zip file to create a final .par file.
Test: m -j checkbuild && real examples:
Bug: b/70568913
Change-Id: I9ff232d461d33e1c06026e7dcb5b124bf02c3ce5
Allow native modules to specify use_version_lib, which will make
an android::build::GetBuildNumber() function available. For host
builds, the function will return the build number at the time that
the module was linked. For device modules it will return the
value of the ro.build.version.incremental property.
Bug: 71719137
Test: build_version_test
Test: m build_version_test && touch build/make/core/Makefile build/soong/cc/libbuildversion/tests/build_version_test.cpp && m build_version_test shows different build numbers for binary and library tests.
Change-Id: I6f7d40b7574bb8206866c4e39bad9c710c796e32
Parse the machine-specific symbol tables into a custom symbol table
object and use a single function to find the offset and size of the
symbol in the custom table. Make the functions to convert a
machine-specific symbol table into the custom table also take mocks,
and provide functions to dump mocks from real files. Add tests
that take the mocks and verify they produce the right offset and
size.
Test: cmd/symbol_inject/*_test.go
Change-Id: I25654032b5017dd13a9a1fe29f8b8826ce5bc82b
Some exe files have a .data symbol at the same address as the
soong_build_number symbol. If the .data symbol is after
soong_build_number in the symbol list, symbol_inject would think
the end address was the same as the start address, and use
uint32(-1) as the size.
Use sort.Search to find the first symbol whose section number is
the same as the target symbol, but whose address is higher than
the target symbol.
Test: manual
Change-Id: I51d6e53c6b906222ba68c5cf93be944843e23550
macho symbol table entries are not always in order, which breaks
finding the next symbol to find the size of the target symbol.
Test: build_version_test
Change-Id: I41d1c3c3ff9929694e9ec2b034553d6b7ddef937
The path to the output directory may be arbitrarily long, use
__SBOX_OUT_DIR__ in the list of output files passed to sbox
to avoid expanding it multiple times in the command line.
Fixes:
ninja: fatal: posix_spawn: Argument list too long
09:40:14 ninja failed with: exit status 1
when building libchrome with a long OUT or OUT_DIR_COMMON_BASE.
Bug: 73726635
Test: m checkbuild
Change-Id: I59024b2164287c8e531711afd9273b692ce9c28a
Only include dependencies for scope "compile."
Bug: 73263586
Test: ./update_current.py -s -t <build-id>
Change-Id: I2ac6055cb4c1ad1f8d7924869f54f50e5e662742
Move __init__.py files population to merge_zips, and add options: -p, and
-e. Since par needs entry_point.txt to search the program entry point, so
add -e option to support it.
Test: ./merge_zips -p -e entry_point.txt out in1.zip in2.zip
Bug: 70568913
Change-Id: Ida10faf125e1b94dffc98b30240db3b90fd75b25
Add a rule in soong that re-executes soong_build in order to
generate build documentation. This allows Soong to customize
the documentation.
Bug: 70516282
Test: m soong_docs
Change-Id: If143cfacd6ac20274cd7bb8d8fab0c07025a5553
On large branches (250+ configurations), the open file count can go
over the default 1024 soft limit on Ubuntu. Many systems have increased
that default, but for the ones that haven't, at least opt into the
hard limit until this can be refactored to use fewer open files.
Bug: 70370883
Test: prlimit -n256:4096 build/soong/build_test.bash -only-config
Test: ulimit -Sn 256; build/soong/build_test.bash -only-config (darwin)
Change-Id: I7a952ffc89a0149ab65b04db1523d348daa6ab3e
This mostly reverts commit 178d5fefc0
and mostly reapplies change I6d3e52ef62c4cabe85b9a135a54de0e1a6aab29c .
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
--products=aosp_arm \
'build/blueprint:work^ build/soong:work^' \
'build/blueprint:work build/soong:work'
# and see that the only changes were:
# 1. adding some new files
# 2. changing some line numbers
Test: m -j nothing # which runs unit tests
Change-Id: I32baae00277a547fdcdd1c2219fe6625ee0e45d7
which should make debugging faster.
Bug: 66921582
Test: ./out/soong/host/linux-x86/bin/sbox --output-root outs --sandbox-path out/.temp/sbox-work -c "cd __SBOX_OUT_DIR__ && mkdir asubdir && mkdir b && touch asubdir/child a b c d e f g h i j k l m n o p" outs/a outs/b outs/z # and observe the below output:
mismatch between declared and actual outputs
in sbox command(cd out/.temp/sbox-work/sbox343858828 && mkdir asubdir && mkdir b && touch asubdir/child a b c d e f g h i j k l m n o p)
in sandbox out/.temp/sbox-work/sbox343858828,
failed to create 2 files:
b: not a file
z: does not exist
did create 16 files:
a
asubdir/child
c
d
e
f
g
h
i
j
...6 more
Change-Id: I75e37834c44d4279dec874701d67ce8bb01b872c
This reverts commit 63a250a336.
Reason for revert: Some failures:
namespace_test.go:648: dir1/Blueprints:2:4: a namespace must be the first module in the file
such as New Build Breakage: aosp-master/build_test @ 4475274
Change-Id: I1b5db8eb934e51ff22241bfca44199d886b1393b
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
--products=aosp_arm \
'build/blueprint:work^ build/soong:work^' \
'build/blueprint:work build/soong:work'
# and see that the only changes were:
# 1. adding some new files
# 2. changing some line numbers
Test: m -j nothing # which runs unit tests
Change-Id: I6d3e52ef62c4cabe85b9a135a54de0e1a6aab29c
This reverts commit fb941913a3.
Reapplies I4933187e8b72f2ef0c32d18ffea756e2c6fa417c with fixes
to disable the check for mac builds, where many modules are
disabled and the check is just going to cause more problems.
Test: m checkbuild
Change-Id: If6712c90ececd5d015fcdcdeefe0c3d4f5590711
Add a --ignore_duplicates option to cause merge_zips to
silently take the first entry in the case of duplicate entries
in the zips to be merged. This will be used for soong Jacoco
support to combine the instrumented jar with the original jar
to pick up any classes that were excluded.
Bug: 69629238
Test: m EMMA_INSTRUMENT=true EMMA_INSTRUMENT_FRAMEWORK=true SKIP_BOOT_JARS_CHECK=true WITH_DEXPREOPT=false
Change-Id: I49ede3f07ce0ed7701b4db7058da2e4f11b8043e
Jacoco support will use zip2zip to create a jar that is a subset
of another jar, and will need exclusion filters and recursive
globs. Switch zip2zip from filepath.Match to pathtools.Match,
and check each included file against the exclusion list.
Bug: 69629238
Test: zip2zip_test.go
Change-Id: Ibe961b0775987f52f1efa357e1201c3ebb81ca9c
Get a list of leaf nodes in the dependency graph
from ninja, and make sure none of them are in the
output directory. This ensures that there are no
rules that depend on a file in the output directory
that doesn't have rule to generate it. The check
will catch a common set of build failures where
a rule to generate a file is deleted (either by
deleting a module in an Android.mk file, or by
modifying the build system incorrectly). These
failures are often not caught by a local incremental
build because the previously built files are still
present in the output directory.
Bug: 36843214
Bug: 68062417
Test: manual
Change-Id: I4933187e8b72f2ef0c32d18ffea756e2c6fa417c
Instead of throwing it away (though it would also be in soong.log along
with the verbose messages).
Test: build/soong/build_test.bash with ctx.Println added
Change-Id: I64f8b11ab6752a6be8a5934fb41e30439238e331
For unbundled builds, LOCAL_SDK_VERSION needs to be set, otherwise we'll
try to use frameworks-res, which isn't available.
Bug: 64723465
Test: Use with prebuilts/sdk/update_current.py
Change-Id: If5e1233b4bcb09b3d0e2251e8dd1a8ecfc87eb19
Instead of deleting artifacts/logs from successful build (unless -keep
is set), and keeping unsuccessful artifacts, keep all logs and remove
all artifacts (unless -keep is set, then we'll compress the artifacts).
If -dist is passed in, we'll put an archive of the logs into the
DIST_DIR. Even compressed, the rest of the artifacts are still a bit too
large to dist (~5.6GB on AOSP).
Test: build/soong/build_test.bash
Test: build/soong/build_test.bash -keep
Test: build/soong/build_test.bash -dist
Change-Id: I87f55978c18c8ff2e517b8271554ba383003742f
Android Support Library 26+ does not specify dependency types, so parse
all pom files first, then propagate missing dependency types if we've
got the information.
Bug: 64723465
Test: cd prebuilts/maven_repo/android; pom2mk -use-version 26.0.0-beta2 com/android/support
Change-Id: Ieaff757ff198c9a7b4b006623340b382728c1fd4
Also removeTempDir -> !keepOutDir
Also shorten the help comment
Bug: 68336760
Test: rm out -rf && m -j
Test: sbox -c # and observe that the output does not have a stacktrace
Change-Id: I6fbb385132128ab1273c8398ff2425191049751e
For repositories that contain multiple versions of the same artifact,
we'd only like to create makefiles for one of the versions of the
artifact.
Bug: 64723465
Test: cd prebuilts/maven_repo/android; pom2mk -use-version 25.3.1 com/android/support
Change-Id: I5bdbba8d2143a47610d56d679381eb01b3cf176d
Most notably, the sandbox depfile path should be passed into the tool.
Bug: 68336760
Test: m -j checkbuild
Change-Id: I22f944a3f57d613fda26de0ea777a915cafcd020
ALLOW_MISSING_DEPENDENCIES=true can cause missing dependencies to be
hidden from GenerateAndroidBuildActions, which is triggering the
len(jars) == 0 check in compileJavaHeader. For the
ALLOW_MISSING_DEPENDENCIES=true case the check is unnecessary, as the
build rules will be replaced with runtime errors. On the off chance
the check would have been hit in a real build, allow merge_zips to
create an empty zip file if it is asked to.
Test: m ALLOW_MISSING_DEPENDENCIES=true
Change-Id: I3171f921d51229ddf38a1a647d32566658c673fa
to make it faster/easier to invoke from other Go programs
(such as multiproduct_kati)
Bug: 67478260
Test: m -j
Change-Id: Idd2671a44290550197c88f53dd11a6dd39c85cc5
javac warns whenever -source is used with a version that does not
match javac, because it wants an rt.jar provided for the given
source level. We are not using -source to prevent usage of APIs
that don't exist in the newer rt.jar, so hide the messages.
Test: javac_wrapper_test.go
Change-Id: Ic94b61767436848fc164e377dc3cc5a63726f1cf
build/soong/soong_ui.bash --dumpvars-mode \
--vars="..." \
--abs-vars="..." \
--var-prefix="..." \
--abs-var-prefix="..."
is similar to the previous:
CALLED_FROM_SETUP=true BUILD_SYSTEM=build/core \
make -f build/core/config.mk dump-many-vars \
DUMP_MANY_VARS="..." \
DUMP_MANY_ABS_VARS="..." \
DUMP_VAR_PREFIX="..." \
DUMP_ABS_VAR_PREFIX="..."
and
build/soong/soong_ui.bash --dumpvar-mode [--abs] VAR
is similar to the previous:
CALLED_FROM_SETUP=true BUILD_SYSTEM=build/core \
make -f build/core/config.mk dumpvar-[abs-]-VAR
But uses soong_ui and ckati, so that we use a consistent make parser and
sandboxing configurations.
One major output difference between the pure make implementation and
this one is that report_config in Go is implemented using embedded
newlines in single quotes, while the make implementation uses `` with
embedded echo commands. This seems to work fine for both bash and zsh,
and report_config isn't meant to be machine-parsed anyways.
Test: build/soong/soong_ui.bash --dumpvar-mode report_config
Test: build/soong/soong_ui.bash --dumpvar-mode TARGET_DEVICE
Test: build/soong/soong_ui.bash --dumpvar-mode --abs PRODUCT_OUT
Test: build/soong/soong_ui.bash --dumpvar-mode --abs ALL_PRODUCTS
Test: build/soong/soong_ui.bash --dumpvars-mode --vars="report_config TARGET_DEVICE" --abs-vars="ALL_PRODUCTS"
Test: build/soong/soong_ui.bash --dumpvars-mode --vars=TARGET_DEVICE --abs-vars=PRODUCT_OUT --var-prefix=v_ --abs-var-prefix=a_
Change-Id: I0fbd0732bbf6fcfcd24084cf3c830a91a4b6bfc2
This speeds up dumping make variables from ~380ms using make to ~220ms
using ckati. It also means that we're consistently using the same parser
for builds (with the same .KATI_READONLY/etc extensions).
envsetup.sh (lunch) / other scripts still use make, changing those to go
through soong_ui will be a future change.
Test: m clean; m nothing
Test: USE_GOMA=true m nothing
Test: m PRODUCT-aosp_x86-sdk
Test: m APP-Calculator
Test: build/soong/build_test.bash -only-config (on AOSP and internal master)
Change-Id: I6ca554de8de4955fb869001d06d29969b75751cc
This can make it easier to add build/soong/build_test.sh to continuous
build branches to catch new breaks, but where some products are known to
be broken.
Test: build/soong/build_test.bash --skip-products aosp_arm,aosp_arm64
Change-Id: Ib240570b68957077047ca75b78658429cb8c6a49
Putting resources in dex jars is going to need to pull files from
classpath jars that do not match *.class.
Test: m -j checkbuild
Change-Id: Ia37b8b9387e5f5636769afc6937b9aa3c9ec1ced
* changes:
Extract the linker and embed it into host bionic binaries
Add prefix_symbols support to cc_object
Fix ToolchainClangCflags for host bionic
Don't add a rpath to the linker
The linux kernel requires that the ELF interpreter (runtime linker)
that's referenced by PT_INTERP be either an absolute path, or a relative
path from the current working directory. We'd prefer a relative path
from the binary, similarly to how we handle looking up shared libraries,
but that's not supported.
Instead, extract the load sections from the runtime linker ELF binary
and embed them into each host bionic binary, omitting the PT_INTERP
declaration. The kernel will treat it as a static binary, and we'll use
a special entry point (linker_wrapper) to fix up the arguments passed by
the kernel before jumping to the embedded linker. From the linker's
point of view, it looks like the kernel loaded the linker like normal.
Bug: 31559095
Test: Enable host bionic,
out/soong/host/linux_bionic-x86/nativetest64/libdemangle_test/libdemangle_test
Change-Id: I8d0aea9790b5e86fcc3ea6e2d00cfa33907e2853
Jar always puts default MANIFEST.MF files in if none was specified.
Copying that behavior in soong_zip causes problems with merge_zips,
because it ends up taking the default manifest from the classes.jar
instead of the user's manifest from res.jar. We don't want the
user's manifest in the classes.jar, otherwise a change to the
manifest will cause all the class files to rebuild. Instead,
move the manifest insertion to the final merge_zips stage.
Test: m -j checkbuild
Change-Id: Id6376961dbaf743c2fb92843f9bdf2e44b963be0
In some cases, we need delete META-INF dir to void mis-using some
classes files in the Java runtime.
For now, when we delete META-INF, we need keep Manifest file in the jar
since merge_zips doesn't have ability to add a default one.
Bug: b/65455145
Test: m clean && m -j
Change-Id: Iea44b1a98d348cd8df1fa7374907733b1add6935
Also filter out META-INF/TRANSITIVE dir, and report warnings when
merge_zips see duplicates entries with different CRC hash.
Bug: b/65455145
Test: m clean && m -j java (locally)
Change-Id: I47172ffa27df71f3280f35f6b540a7b5a0c14550
Allow filename collisions for directories, which may happen if
multiple globs of resources include the same directory names.
Continue to report errors if collisions occur between files, and
also add checks for collisions between files and directories.
Test: manual
Change-Id: Iac19fbd325c53fbb41552ea230d813c8dddf9439
Combining the results of a javac command into a jar needs to take
a whole directory, add -D to specify a directory to search for all
files.
Test: m -j checkbuild
Change-Id: Id4707189d57b664f0262a0782630f893af397900
This will enable writing a modified manifest without having to create a temporary file first
Bug: 64536066
Test: soong_zip --jar -o /tmp/out.zip -C . -l files.list && \
# make sure that the output is binary equal
# with and without this patch
Change-Id: I559d653e0e72e641e1ee6745924cb835bb0a355b
Bug: 64536066
Test: soong_zip --jar -o /tmp/out.zip -C . -l files.list && \
zipdetails /tmp/out.zip | less \
# and check that the first entry contains the \
# "CAFE" extra attribute as shown below:
0000001E Filename 'META-INF/'
00000027 Extra ID #0001 CAFE 'Java Executable'
00000029 Length 0000
Change-Id: I12c4078591f2ce2afc1af5b9db20393b26601bca
Bug: 64536066
Test: soong_zip --jar -o /tmp/out.zip -C . -l files.list && \
zipdetails /tmp/out.zip | less \
# and check that the META-INF entries are earlier in \
# the list than other entries
Change-Id: Id5c6ea9ce8c3a6fbfb8366db753e6603a076dbf8
This reverts commit 526416b1e4.
Figured out a fix for the deadlocks; resubmitting the patch.
The first version was deadlocking because the switch statement in
zipWriter.write would choose a specific zip entry to start writing,
but the individual chunks may not have all necessarily been compressed
yet. When each individual chunk was made to require to request its own
allocations, the compression of the chunks of the file being currently
written could be blocked waiting for memory to be freed by chunks from
other files that hadn't yet started being written.
This patch is much like the original except it preallocates the memory
for the entire file upfront (and happens to use the total file size
rather than the compressed size, but I didn't observe that to cause any
performance differences).
Bug: 64536066
Test: m -j dist showcommands # which runs soong_zip to package everything
Change-Id: Id1d7ff415e54d3a6be71188abbdbbbab5a719fcf
This wasn't intended to change the behavior, but it does slightly.
Previously any requests to acquire memory wouldn't block; only
(subsequent) requests for execution would block (if another
caller had already consumed the memory quota). Now the requests
for memory can also also block.
It turns out in a brief test on my computer that soong_zip
runs about 10 to 20% faster with this change than without it.
The final step involving soong_zip decreased from about
3.6 sec to about 3.3 sec in tests on my computer.
When testing the process of re-zipping the contents of
angler-target_files*.zip , the time decreased from about
6.3 sec to about 5.3 sec in tests on my computer, and the
peak memory usage reported by `top` decreased from about
1.5g to 1g
Bug: 64536066
Test: m -j dist showcommands # which runs soong_zip to package everything
Change-Id: I0422e4c363c675bb7a93309fac4616c768bfbe8f
The Finder runs roughly 200ms faster than findleaves.py in aosp,
and runs roughly 400ms faster in internal master.
Bug: 64363847
Test: m -j
Change-Id: I62db8dacc90871e913576fe2443021fb1749a483
Fix mutator registration for tests to allow different tests
in the same package to register different mutators.
Allow tests to track the resulting ModuleBuildParams objects
to use in assertions, and provide helpers for getting them.
For example:
config := android.TestConfig(buildDir)
ctx := android.NewTestContext()
ctx.RegisterModuleType(...)
ctx.MockFileSystem(...)
ctx.ParseBlueprintsFile("Android.bp")
ctx.PrepareBuildActions(config)
ctx.Register()
// Get the Inputs value passed to the javac rule for the foo module
inputs := ctx.ModuleForTests("foo".Rule("javac").Inputs
Test: java_test.go
Change-Id: I10c82967f5f3586d2c176f169906b571ed82fc73
This was causing android/soong/ui/build to rebuild sometimes when it
didn't need to.
Test: `m -j nothing; grep "B compile" out/.soong_ui.trace` repeatedly
Change-Id: I5d33271a7b037f53674a0d312d8da1746eed8eaa
It will cd to $TOP before running soong_ui itself, so that soong_ui
still runs from the top of the tree. ORIGINAL_PWD is saved so that we
can reference that later (for example, to move 'mma' implementation into
Go).
Test: cd system; ../build/soong/soong_ui.bash
Test: Set absolute / relative OUT_DIR or OUT_DIR_COMMON_BASE and repeat
Change-Id: Icb67a3ee6a3358cca50300755c8953419fc19437
Jars have a strange sorting order; the META-INF/ directory should
come first, then META-INF/MANIFEST.MF, then any other files in
META-INF/, and then any files outside META-INF. Add a -j argument
to zip2zip that sorts using jar ordering.
Test: zip2zip_test
Change-Id: I80e2bc7e284ef74f6561c26cb6541298834db1bc
When '-s' is passed, any globbed arguments will have their results
sorted. When there are multiple arguments, the files will still be
inserted in argument order.
A bare '**' is now special cased to mean every file in the input zip.
This allows zip2zip to sort entire zip files efficiently by using
`zip2zip -s -i <> -o <> '**'`. This can be useful if your original zip
program used filesystem ordering which was not reproducible.
Test: m -j blueprint_tools (new tests pass)
Change-Id: Ic3512c5fe14c94c6f3e134296905121d2ff8b58a
Cleanup of some failure messages
Also, this leaves the temp directory untouched if a declared output was not created
Bug: 35562758
Test: make
Change-Id: I8ef1315af80eb327752501f12a331dbdf52ba3e9
This causes Soong to put the outputs of each genrule into a temporary
location and copy the declared outputs back to the output directory.
This gets the process closer to having an actual sandbox.
Bug: 35562758
Test: make
Change-Id: I8048fbf1a3899a86fb99d71b60669b6633b07b3e
This is still fairly simplistic, but good enough to test some build
system changes.
Bug: 33381544
Test: run, inspect output
Change-Id: Ia5c19570493116dca01cb65605cdf20becf8c1d0
Adds a -dist argument that will use DIST_DIR to save logs.
Also writes a summary of each std.log to stderr on errors, so that the
error is more likely to show up in the error reporting. This output is
prefixed with "> " to differentiate it from the progress reports from
multiproduct_kati itself.
Test: multiproduct_kati -only-config
Test: DIST_DIR=dist build/soong/build_test.bash -dist (introducing errors)
Change-Id: I5005b5f3f200c876bc004dd9b0e01e7b6edf5be2
Move common microfactory functions to cmd/microfactory/microfactory.bash
so that they can be used to build both soong_ui for normal build and
multiproduct_kati for build system tests.
Test: m -j
Test: build/soong/build_test.bash
Change-Id: I9512642d846ce54d05a027b6d33a2b3029b3f90b
It now uses the same output style as ninja, overwriting status lines in
smart terminals.
Test: multiproduct_kati
Test: multiproduct_kati | cat
Change-Id: I8db5198ffdc5ebc5503241ac492379753d92978e
1. Added ability to keep the mixed "-f"/"-l" order as same as
command-line flags order.
2. Added "-s" flag to specify which target file within zip is stored uncompressed.
Test: manual
Change-Id: I338b25a7bd6bf1b7e9cc29ad3324575167630fb7
Piping the output of javac through a filter makes it hard to capture
the exit status. Convert it to a wrapper that executes javac and
propagates the exit status.
Bug: 36666657
Test: javac_wrapper_test
Change-Id: I9b56cc3794023aabc9328138a68830e26e980f97
