Debugging issues on the build servers can be difficult because the
intermediate files are not visible. Gzip ninja file and Makefiles
generated by Soong and the ninja files generated by Kati to the dist
directory, and also copy all of the finder output files.
Bug: 157656545
Test: m dist nothing
Change-Id: I48d75305e551ccae81c7a55721981cf58acd838b
The metrics uploader was currently running on foreground where it
would copy the metrics files in a separate directory and then forked
into the background for the upload process. As a result, running the
lunch command would take a second longer to run since each metrics
uploader run had an average of half a second.
Bug: 140638454
Test: * Wrote and updated unit test cases.
* Set ANDROID_ENABLE_METRICS_UPLOAD to point to the latest
metrics_uploader bash script. Executed the "lunch 1" command
and measured the running time. Executed "m nothing" command
and checked that the metrics were uploaded.
* Ran "lunch 1" and "m nothing" with
ANDROID_ENABLE_METRICS_UPLOAD=""
* Removed oauth from metrics_uploader and ran "m nothing" and
"lunch 1". The oauth Message appeared only to "m nothing"
Change-Id: I13c61e666c8f44613dee291a704cef6a27335188
Add the following build actions {BUILD_MODULES_IN_A_DIRECTORY,
BUILD_MODULES_IN_DIRECTORIES} in soong_ui config so the bash code version of
build commands (m, mm, mma, mmm, mmma) in build/make/envsetup.sh can be deprecated.
This is to allow up to date bug fixes on the build commands.
Bug: b/130049705
Test: Unit test cases
Change-Id: I772db1d4e9c1da5273374d1994eb5e8f17cd52f2
This adds a new status package that merges the running of "actions"
(ninja calls them edges) of multiple tools into one view of the current
state, and gives that to a number of different outputs.
For inputs:
Kati's output parser has been rewritten (and moved) to map onto the
StartAction/FinishAction API. A byproduct of this is that the build
servers should be able to extract errors from Kati better, since they
look like the errors that Ninja used to write.
Ninja is no longer directly connected to the terminal, but its output is
read via the protobuf frontend API, so it's just another tool whose
output becomes merged together.
multiproduct_kati loses its custom status routines, and uses the common
one instead.
For outputs:
The primary output is the ui/terminal.Status type, which along with
ui/terminal.Writer now controls everything about the terminal output.
Today, this doesn't really change any behaviors, but having all terminal
output going through here allows a more complicated (multi-line / full
window) status display in the future.
The tracer acts as an output of the status package, tracing all the
action start / finish events. This replaces reading the .ninja_log file,
so it now properly handles multiple output files from a single action.
A new rotated log file (out/error.log, or out/dist/logs/error.log) just
contains a description of all of the errors that happened during the
current build.
Another new compressed and rotated log file (out/verbose.log.gz, or
out/dist/logs/verbose.log.gz) contains the full verbose (showcommands)
log of every execution run by the build. Since this is now written on
every build, the showcommands argument is now ignored -- if you want to
get the commands run, look at the log file after the build.
Test: m
Test: <built-in tests>
Test: NINJA_ARGS="-t list" m
Test: check the build.trace.gz
Test: check the new log files
Change-Id: If1d8994890d43ef68f65aa10ddd8e6e06dc7013a
Some tests make their temporary directories readonly. If they fail or
crash before cleaning up, they could leave these readonly directories
behind with files in them. os.RemoveAll fails with an error in this
case, and we can't start the build until they're removed.
Test: `m blueprint_tools` to run the new go tests
Change-Id: I761f96579e96167ebfd98c6cca59765bd50536ec
This way config.mk no longer needs to check which java is in PATH and
fix it. It'll be consistent for all build steps under soong_ui.
Also unify handling of ANDROID_JAVA_HOME / JAVA_HOME with
OVERRIDE_ANDROID_JAVA_HOME / EXPERIMENTAL_USE_OPENJDK9.
Test: m nothing
Test: build/soong/soong_ui.bash --make-mode nothing (w/o envsetup.sh)
Test: aosp_arm ninja files are the same before/after
Test: before/after ninja files match with OVERRIDE_ANDROID_JAVA_HOME
Test: before/after ninja files match with EXPERIMENTAL_USE_OPENJDK9
Change-Id: Icdb65093d9c346524074de239a4f895e4230a24d
This causes Soong to put the outputs of each genrule into a temporary
location and copy the declared outputs back to the output directory.
This gets the process closer to having an actual sandbox.
Bug: 35562758
Test: make
Change-Id: I8048fbf1a3899a86fb99d71b60669b6633b07b3e
1. Collapses the "including .../Android.mk ..." lines like ninja does,
so that we overwrite the last line if there wasn't anything else to
print.
2. Strips ansi control codes so that log files don't include unreadable
characters.
Test: m -j
Test: m -j | tee output.log (with colored output)
Change-Id: Ib18437f6f9d37084360097a9d586800c833072c5
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
