If Kati ever wrote a line over 64k characters, the scanner in soong_ui
would error, and Kati would hang trying to write into the pipe. Now if
the Scanner errors out, fall back to copying directly from the pipe to
the output.
Test: Add $(warning $(shell find frameworks)), does not hang
Change-Id: I86412ad2f53f2fe3cbda901ee673abb904d56d3c
This way we strip out ANSI codes when using dumb terminals. It's likely
overkill to use katiRewriteOutput, but we should rarely see any output
on stderr while dumping variables. This also lets us turn on kati_stats.
Bug: 71729611
Test: lunch missing-eng (colored error)
Test: lunch missing-eng 2>&1 | cat (non-colored)
Test: TERM=dumb lunch missing-eng (non-colored)
Change-Id: Ic63fd42d82a4a64e5c68aecd9ae0f242a0d703f1
Pass --dump-stats to ckati, but filter out the '*kati*' lines so that
they only end up in our verbose output. That way we've always got access
to the statistics.
Bug: 36182021
Test: m nothing; cat out/soong.log
Change-Id: Iaf7a814fc67f3e475c913faf69924a7f4e2ae3b3
This speeds up dumping make variables from ~380ms using make to ~220ms
using ckati. It also means that we're consistently using the same parser
for builds (with the same .KATI_READONLY/etc extensions).
envsetup.sh (lunch) / other scripts still use make, changing those to go
through soong_ui will be a future change.
Test: m clean; m nothing
Test: USE_GOMA=true m nothing
Test: m PRODUCT-aosp_x86-sdk
Test: m APP-Calculator
Test: build/soong/build_test.bash -only-config (on AOSP and internal master)
Change-Id: I6ca554de8de4955fb869001d06d29969b75751cc
Instead of pairing the reading of CleanSpec.mk files with the reading of
Android.mk files, split them into separate, individually cachable steps.
This way we only read Android.mk files once after a clean/sync. We'll
still read the CleanSpec.mk files multiple times, but that's
significantly faster than reading all the Android.mk files.
This adds about 50ms if kati doesn't need to reread the CleanSpec.mk
files. Reading all the CleanSpec.mk files takes about a second.
Bug: 35970961
Test: m clean; m nothing; m nothing
Test: Add CleanSpec.mk line, see it executed.
Change-Id: I83bad15c50709510959d5b8b673a907b8aa7de82
Kati sets detect_leaks=0 by default now.
Test: unset ASAN_OPTIONS; SANITIZE_HOST=address m nothing
Change-Id: I0cbf04ae9ed40de520abca3fb35c65e7f543682d
We're seeing some rare crashes that appear to be stack overflows. To get
better debugging, use the asan version of ckati when 'dist' is
specified.
Bug: 36182021
Test: m -j
Test: m -j dist
Test: make -j dist (w/o lunch)
Change-Id: Ic3a5590974bfd718bf1929355d344b5933ac1d4f
Wrap os/exec.Cmd to use our Context and Config interfaces for automatic
logging and error handling. It also simplifies environment modification
based on the Config's environment.
This also adds sandboxing on Macs using sandbox-exec. A simple profile
is provided that only logs on violations, though multiproduct_kati on
AOSP has no violations. This isn't applied to ninja, only make / soong /
kati to start with. I measured <5% time increase in reading all
makefiles, and no noticable difference when kati doesn't regenerate.
I'd like to spin up a process to dump violation logs into our log file,
but the log reporting changed over the range of Mac versions that we
support, so that's going to be more complicated. Opening Console.app
works in all cases if you're local -- just search/filter for sandbox.
Linux sandboxing will be implemented later -- the sandbox definition is
opaque enough to support a different implementation.
Test: multiproduct_kati on AOSP master on Mac
Change-Id: I7046229333d0dcc8f426a493e0f7380828879f17
I missed this when converting to soong_ui.
Test: m -j blueprint_tools (check soong.log)
Test: SANITIZE_HOST=address m -j blueprint_tools
Change-Id: I01eb567db6848dc36dd679557291a4e600a63bba
We can start removing out directories again in multiproduct_kati, since
the opendir bug has been fixed.
Add --color_warnings to the Kati command line. Since this is different
from Make, take this opportunity to reorder the command line to make
more sense. This wasn't done before because kati forces a regen whenever
the command line changes.
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I5ad03359fbc16db482722946202297c1ae0f2b90
1. Collapses the "including .../Android.mk ..." lines like ninja does,
so that we overwrite the last line if there wasn't anything else to
print.
2. Strips ansi control codes so that log files don't include unreadable
characters.
Test: m -j
Test: m -j | tee output.log (with colored output)
Change-Id: Ib18437f6f9d37084360097a9d586800c833072c5
This creates a rotating build.trace.gz in the out directory that can be
loaded with chrome://tracing. It'll include start and end timings for
make/soong/kati/ninja, and it will import and time-correct the ninja log
files.
Test: m -j; load out/build.trace.gz in chrome://tracing
Test: multiproduct_kati -keep; load out/multiproduct*/build.trace.gz
Change-Id: Ic060fa9515eb88d95dbe16712479dae9dffcf626
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
