This will allow us to track (and eventually limit) the commands that the
build references via $PATH. These are mostly implicit dependencies on
the host system -- for Linux, we assume something similar to Ubuntu
14.04 with a few extra packages, but this will let us better define
that.
This will not catch uses of tools with absolute paths (/bin/bash, etc),
but most uses shouldn't be relying on absolute path names anyways.
Adds ~400ms on the first startup, ~140ms on subsequent runs, and
overhead of a few ms for every forwarded execution.
Test: m
Test: build/soong/build_test.bash
Test: Add `gcc --version`, TEMPORARY_DISABLE_PATH_RESTRICTIONS=true m
Change-Id: Id68cbb1c8ceef65bbbb10751e83722c7662d2351
The find commands used to locate the BoardConfig.mk can't be optimized
by Kati, so we're currently spending ~125ms three times during every
build (dumpvars, cleanspec, and the main kati run). Preserve the value
of TARGET_DEVICE_DIR from the dumpvars run so that we only need to run
the find commands once.
Bug: 78020936
Test: out/build-taimen.ninja is identical
Test: out/soong.log shows that we're not running these finds again
Change-Id: Iee56b454c3661de2b58c161169218ecaf2135398
I got a report of a user exporting CDPATH in their shell and causing
strange issues in the build. We should never need this value (and it
probably shouldn't be exported from the shell either -- that was their
workaround).
NDK_ROOT causes issues with Android.mk files thinking that they're
building with the ndk (like external/googletest/googletest/Android.mk).
Bug: 78933670
Test: CDPATH=. NDK_ROOT=test m; check out/soong.log
Change-Id: Icce43d7e31ed5e5e1fb7a4e37fd4dfbf421af4b1
Many board still have this problem, but if we can switch some over,
we'll prevent global problems and have the ability to clean boards up
one-by-one.
Bug: 77611511
Test: lunch aosp_arm-eng; m nothing
Test: lunch aosp_marlin-eng; m nothing
Test: build_test on all downstream branches
Change-Id: I78bee44adde2059d01188658b9050927748d2028
Some people have run into issues where /tmp is on a smaller partition
than their main source tree, and if it fills up (or only has a few GB
free), their builds can start failing.
There may also be a subset of people whose /tmp may be slower than their
out directory -- if they put their src and out trees on a SSD, but keep
/tmp on a spinning HDD.
Bug: 71755844
Test: check out/soong.log for TMPDIR
Test: do a build, watch out/soong/.temp
Change-Id: Ib749ec736ad0f6c8ce7453861a91b3e99ddbd92d
Soong_UI will update timestamp to build_date.txt, and export variables
to kati/ninja.
Test: m -j32
Bug: b/70351683
Change-Id: I153897afdf2d3f39a32d757d4c3ae7515caea52d
This ensures that the current locale supports UTF-8, and that we're
getting a consistent (but still supported by the system) locale for
every configuration except user-facing messages. This should eliminate
any reproducibility problems around sorting, formatting, etc for all
built products, while still showing localized error messages where
available.
Bug: 71573630
Test: LANG=es_ES LANGUAGE=es: m (check env in soong.log)
Change-Id: If33311899eaed8c44573113ee35c5a71cee503a0
This architecture only existed for unbundled use, but even the NDK is
removing support in their r17 release, so just remove support for it.
Test: build/soong/build_test.bash -only-soong
Change-Id: I4bd23babf567128d2d242cbdee3311abb198dd7c
This is cherry-picked from attempt 3, which was reverted
due to http://b/70862583.
Before this CL topic, the build toolchain for .java source files
used OpenJDK 8, targeting 1.8 (v52 class files) by default.
This CL topic switches the default to OpenJDK 9, but still
targeting 1.8 (v52 class files) by default. If USE_ERROR_PRONE
is set to true, then the default remains OpenJDK 8.
Code in the Android platform should generally be unaffected,
but if host tools that are now compiled and run using
OpenJDK 9 are causing problems for your team, then let me
know.
To manually switch back to the old behavior for now (continue
using OpenJDK 8), run this command in your shell:
export EXPERIMENTAL_USE_OPENJDK9=false
Bug: 69449021
Test: Treehugger
Test: art/test/testrunner/run_build_test_target.py -j40 art-interpreter
Test: dalvik/dx/tests/run-all-tests
(cherry picked from commit 13f23a2753)
Change-Id: I57abae73f9bdb21ef004a5118ff0e4ef70418ed9
Merged-in: Iac78122f58df0ebbb55134d55021ce6c57351b5f
Previously, these variables were exported only by makevars.go,
but those values are not available to config.mk. This CL adds
the variable to ui/build/config.go, which also makes it
available to config.mk.
Test: Treehugger
Bug: 70862583
Bug: 70521453
Change-Id: Ib54660e4b08ab751265b30004630cf1bb8c0041b
After this revert CL topic, the default toolchain for Android
goes back to being OpenJDK 8.
This revert is being prepared ahead of time in case of
any problems with CL topic:
https://r.android.com/#/q/topic:bug69449021_attempt2
Bug: 69449021
Test: Treehugger
Change-Id: I9ead8d569226bd487baee3c6d5be9ec7033eb56a
(This is cherry-picked from the first attempt to submit this CL
topic, which was reverted after 3 hours because of bug 70286093;
robolectric 3.{1.1,4.2} now stick with OpenJDK 8 to avoid that bug).
Before this CL topic, the build toolchain for .java source files
used OpenJDK 8, targeting 1.8 (v52 class files) by default.
This CL topic switches the default to OpenJDK 9, but still
targeting 1.8 (v52 class files) by default. If USE_ERROR_PRONE
is set to true, then the default remains OpenJDK 8.
Code in the Android platform should generally be unaffected,
but if host tools that are now compiled and run using
OpenJDK 9 are causing problems for your team, then let me
know.
To manually switch back to the old behavior for now (continue
using OpenJDK 8), run this command in your shell:
export EXPERIMENTAL_USE_OPENJDK9=false
Bug: 69449021
Test: Treehugger
Test: Running robolectric tests succeeds on internal-master
after cherry-picking this CL topic, using the command
line from http://b/70286093#comment1
(cherry picked from commit 0ae8b548af)
Change-Id: Ide6a7e55126d919a44f89ef8e0bd14fb12ff470e
After this revert CL topic, the default toolchain for Android
goes back to being OpenJDK 8.
This revert is being prepared ahead of time in case of
any problems with original change topic,
https://r.android.com/#/q/topic:bug_69449021
Bug: 69449021
Test: Treehugger
This reverts commit 0ae8b548af.
Change-Id: Ief7646a94f1a264085cd299b4327d244b78a1537
Before this CL topic, the build toolchain for .java source files
used OpenJDK 8, targeting 1.8 (v52 class files) by default.
This CL topic switches the default to OpenJDK 9, but still
targeting 1.8 (v52 class files) by default. If USE_ERROR_PRONE
is set to true, then the default remains OpenJDK 8.
Code in the Android platform should generally be unaffected,
but if host tools that are now compiled and run using
OpenJDK 9 are causing problems for your team, then let me
know.
To manually switch back to the old behavior for now (continue
using OpenJDK 8), run this command in your shell:
export EXPERIMENTAL_USE_OPENJDK9=false
Bug: 69449021
Test: Treehugger
Test: "make core-oj", checked that compilation now uses
OpenJDK 9 javac -target 1.8
Test: Checked that this is still compiled using OpenJDK 8.
export EXPERIMENTAL_USE_OPENJDK9=false
make core-oj
Change-Id: Ic87e9bb2a2e5da0ff13a2e51845b5365901c1507
This reverts commit fb941913a3.
Reapplies I4933187e8b72f2ef0c32d18ffea756e2c6fa417c with fixes
to disable the check for mac builds, where many modules are
disabled and the check is just going to cause more problems.
Test: m checkbuild
Change-Id: If6712c90ececd5d015fcdcdeefe0c3d4f5590711
This fourth possible value currently has the same semantics
as a default/unset EXPERIMENTAL_USE_OPENJDK9, but allows
people to explicitly switch back to the old semantics when
the default changes.
Test: make showcommands core-oj (in environments with
EXPERIMENTAL_USE_OPENJDK9 set to "", "1.8", and "true").
Bug: 38177295
Change-Id: I25accf14344a05349a6e97572d7c2c1f6a7f2063
Get a list of leaf nodes in the dependency graph
from ninja, and make sure none of them are in the
output directory. This ensures that there are no
rules that depend on a file in the output directory
that doesn't have rule to generate it. The check
will catch a common set of build failures where
a rule to generate a file is deleted (either by
deleting a module in an Android.mk file, or by
modifying the build system incorrectly). These
failures are often not caught by a local incremental
build because the previously built files are still
present in the output directory.
Bug: 36843214
Bug: 68062417
Test: manual
Change-Id: I4933187e8b72f2ef0c32d18ffea756e2c6fa417c
This way config.mk no longer needs to check which java is in PATH and
fix it. It'll be consistent for all build steps under soong_ui.
Also unify handling of ANDROID_JAVA_HOME / JAVA_HOME with
OVERRIDE_ANDROID_JAVA_HOME / EXPERIMENTAL_USE_OPENJDK9.
Test: m nothing
Test: build/soong/soong_ui.bash --make-mode nothing (w/o envsetup.sh)
Test: aosp_arm ninja files are the same before/after
Test: before/after ninja files match with OVERRIDE_ANDROID_JAVA_HOME
Test: before/after ninja files match with EXPERIMENTAL_USE_OPENJDK9
Change-Id: Icdb65093d9c346524074de239a4f895e4230a24d
This reverts commit a3e6c520ac. I fixed
a few heavy stack users, and we've got better debugging.
Bug: 36182021
Test: m dist
Change-Id: Iefd8589185720e44c40a2e0ceff5fd1438211b86
They don't really affect anything general in android except for the
output path.
Bug: 65453318
Test: TARGET_BUILD_TYPE=debug m
Change-Id: I98c17fefae61f9f7936e184bd1f8441086bb7d23
The Finder runs roughly 200ms faster than findleaves.py in aosp,
and runs roughly 400ms faster in internal master.
Bug: 64363847
Test: m -j
Change-Id: I62db8dacc90871e913576fe2443021fb1749a483
This way we only have one way to start a build, which always has logging
/ tracing / etc, even if we don't need Kati.
There's two ways to use this:
As a direct replacement for mkdir out; cd out; ../bootstrap.bash;
./soong -- as long as --skip-make is always passed, we'll never run
Kati, and Soong will run outside of it's "make" mode. This preserves
most of the speed, and allows full user control over the Soong
configuration.
A (experimental, dangerous) way to temporarily bypass the product
variable and kati steps of a build. As long as a user is sure that
nothing has changed from the last build, and they know exactly which
Ninja targets they want to build (which may not be the same as the
arguments normally passed to 'm'), this can lead to shorter build
startup times.
Test: rm -rf out; m --skip-make libc
Test: rm -rf out; m libc; m --skip-make libc
Test: rm -rf out; mkdir out; cd out; ../bootstrap.bash; ./soong libc
Test: build/soong/scripts/build-ndk-prebuilts.sh
Change-Id: Ic0f91167b5779dba3f248a379fbaac67a75a946e
MAKELEVEL and MAKEFLAGS were are both unset by makeparallel when invoked
from Make. In preparation for removing the Make wrapper (and
makeparallel), strip them in soong_ui as well.
Add MFLAGS for completeness.
Test: m -j
Change-Id: I029f9464d11b432e979b32ede360450f4facb038
Make supports specifying all types of variables on the command line
(using =, :=, +=, and other variable references. When running soong_ui
through make/makeparallel these all effectively become environment
variables.
So in preparation to remove the Make wrapper, support a simplified form
of this syntax, roughly equivalent to what the shell supports if
specified before the command (<NAME>=<VALUE>).
Test: m -j blueprint_tools
Change-Id: I08fa2b86710f282e619b0cc324a3e5bbaf62d26e
In preparation to remove Make/makeparallel from soong_ui startup, we
need to preserve compatibility with the different ways that make
supports the -j option.
Nothing changes unless Make/makeparallel is removed from the startup.
Once that is removed, not specifying a -j value will be equivalent to
'-j' instead of '-j1', like Ninja. A value will also be supported when
specifying -k, like Ninja (though specifying it alone will be equivalent
to '-k 0').
Test: m -j blueprint_tools
Change-Id: I9d5d59bedd4f6e5cca76bdb4cd47e0b5b7d523f0
This causes Soong to put the outputs of each genrule into a temporary
location and copy the declared outputs back to the output directory.
This gets the process closer to having an actual sandbox.
Bug: 35562758
Test: make
Change-Id: I8048fbf1a3899a86fb99d71b60669b6633b07b3e
This speeds things up a little bit, but the major win is that we don't
need to recursively call into the build during the auto installclean.
Test: m -j installclean
Test: m -j dataclean
Change-Id: I09ded8dbd8a2015c7848bc0042dfd3adad1b31c9
This way kati won't need to be run as often (either initially, or when
switching products with the same device).
Bug: 35970961
Test: m clean; m -j blueprint_tools; m -j blueprint_tools; m -j blueprint_tools
Test: lunch aosp_arm-eng; m -j blueprint_tools; lunch full-eng; m -j blueprint_tools; <repeat>
Change-Id: Ie9fca3c8f1dd412459ea47c7090c7c5fdb0bcf6e
When kati keeps state around, it has to regenerate the ninja file every
time the state is changed. So move the java version checking into
soong_ui, where we can parallelize it with other operations instead of
only checking it occasionally.
Bug: 35970961
Test: Put java7 in PATH, m -j
Test: Put java8-google in PATH, m -j
Test: Put a space in TOP, m -j
Test: OUT_DIR=<case-preserving fs> m -j
Test: OUT_DIR=<path with space> m -j
Test: DIST_DIR=<path with sapce> m -j
Change-Id: I3245c8dd6d856240d17d54cb05d593dc9df71a27
Otherwise Make and soong_ui disagree about where the soong.variables
file should be. For example, when "OUT_DIR=out/":
Error dumping make vars: Failed to parse make line: "make: Nothing to be done for `out/soong/soong.variables'."
We would need to pass out//soong/soong.variables into make, since it
doesn't attempt to match clean/unclean paths.
Bug: 37553659
Test: OUT_DIR=out/ m -j
Change-Id: Iebeea4868b96b11e8b08735f272e22be5db1d701
We're seeing some rare crashes that appear to be stack overflows. To get
better debugging, use the asan version of ckati when 'dist' is
specified.
Bug: 36182021
Test: m -j
Test: m -j dist
Test: make -j dist (w/o lunch)
Change-Id: Ic3a5590974bfd718bf1929355d344b5933ac1d4f
I missed this when converting to soong_ui.
Test: m -j blueprint_tools (check soong.log)
Test: SANITIZE_HOST=address m -j blueprint_tools
Change-Id: I01eb567db6848dc36dd679557291a4e600a63bba
We keep having makefiles reading $DISPLAY, and GREP_OPTIONS with
--color=always has caused problems with shell commands in the past.
Bug: 37333696
Test: GREP_OPTIONS="--color=always" m -j
Test: DISPLAY=test m -j (modify an Android.mk to check DISPLAY)
Change-Id: Iafab37d61ecde5c6fdb35a05f18709abc17e8554
Currently, Kati is the only one that is respecting OUT_DIR_COMMON_BASE,
causing the build to fail when ninja tries to match up the Soong and
Kati ninja files.
The soong_ui.bash script change is necessary to move the microfactory
built outputs to the correct location.
The config.go change takes care of setting OUT_DIR in the environment of
every subprocess, so that we could remove the OUT_DIR_COMMON_BASE
handling from Make once the USE_SOONG_UI=false path goes away.
Bug: 35929763
Test: m -j blueprint_tools
Test: OUT_DIR=... m -j blueprint_tools
Test: OUT_DIR_COMMON_BASE=... m -j blueprint_tools
Change-Id: I4184f2b4392d32c5bea51e51162a48df0e5c90d5
This is a replacement for build/tools/kati_all_products.sh using the new
Soong ui/build package. It doesn't even attempt to run ninja, and it can
be configured to run only the product config, or only the product config
and Soong.
For AOSP on my machine:
-only-config 1.4s
-only-soong 1m20s
<none> 13m
Test: multiproduct_kati
Change-Id: Ie3e6e7bdf692e46a8b8eb828f437190f8003500b
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
