Commit graph

215 commits

Author SHA1 Message Date
Dan Willemsen
269a8c78e7 Add exec.Cmd wrapper for logging / sandboxing
Wrap os/exec.Cmd to use our Context and Config interfaces for automatic
logging and error handling. It also simplifies environment modification
based on the Config's environment.

This also adds sandboxing on Macs using sandbox-exec. A simple profile
is provided that only logs on violations, though multiproduct_kati on
AOSP has no violations. This isn't applied to ninja, only make / soong /
kati to start with. I measured <5% time increase in reading all
makefiles, and no noticable difference when kati doesn't regenerate.

I'd like to spin up a process to dump violation logs into our log file,
but the log reporting changed over the range of Mac versions that we
support, so that's going to be more complicated. Opening Console.app
works in all cases if you're local -- just search/filter for sandbox.

Linux sandboxing will be implemented later -- the sandbox definition is
opaque enough to support a different implementation.

Test: multiproduct_kati on AOSP master on Mac
Change-Id: I7046229333d0dcc8f426a493e0f7380828879f17
2017-05-05 16:25:40 -07:00
Dan Willemsen
f173d595c2 Support ASAN for ckati and ninja
I missed this when converting to soong_ui.

Test: m -j blueprint_tools (check soong.log)
Test: SANITIZE_HOST=address m -j blueprint_tools
Change-Id: I01eb567db6848dc36dd679557291a4e600a63bba
2017-04-27 14:33:24 -07:00
Dan Willemsen
68a09854a4 Blacklist DISPLAY / GREP_OPTIONS
We keep having makefiles reading $DISPLAY, and GREP_OPTIONS with
--color=always has caused problems with shell commands in the past.

Bug: 37333696
Test: GREP_OPTIONS="--color=always" m -j
Test: DISPLAY=test m -j (modify an Android.mk to check DISPLAY)
Change-Id: Iafab37d61ecde5c6fdb35a05f18709abc17e8554
2017-04-18 14:02:38 -07:00
Yoshisato Yanagisawa
13fd3ff296 Export JAVAC_WRAPPER environment.
To allow to inject a wrapper program for compiling java code, let
me allow soong to export JAVAC_WRAPPER.

Test: Set JAVAC_WRAPPER and ANDROID_COMPILE_WITH_JACK=false envs,
Test: and make -j 32

Change-Id: Ib77fe00bd781f8f4ddf965c5e7905eab466a057a
2017-04-05 11:35:24 +09:00
Dan Willemsen
0c3919e996 Fix OUT_DIR_COMMON_BASE with soong_ui
Currently, Kati is the only one that is respecting OUT_DIR_COMMON_BASE,
causing the build to fail when ninja tries to match up the Soong and
Kati ninja files.

The soong_ui.bash script change is necessary to move the microfactory
built outputs to the correct location.

The config.go change takes care of setting OUT_DIR in the environment of
every subprocess, so that we could remove the OUT_DIR_COMMON_BASE
handling from Make once the USE_SOONG_UI=false path goes away.

Bug: 35929763
Test: m -j blueprint_tools
Test: OUT_DIR=... m -j blueprint_tools
Test: OUT_DIR_COMMON_BASE=... m -j blueprint_tools
Change-Id: I4184f2b4392d32c5bea51e51162a48df0e5c90d5
2017-03-02 19:42:21 -08:00
Dan Willemsen
c38d366d78 Updates for the new ckati drop
We can start removing out directories again in multiproduct_kati, since
the opendir bug has been fixed.

Add --color_warnings to the Kati command line. Since this is different
from Make, take this opportunity to reorder the command line to make
more sense. This wasn't done before because kati forces a regen whenever
the command line changes.

Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I5ad03359fbc16db482722946202297c1ae0f2b90
2017-02-24 10:53:23 -08:00
Treehugger Robot
fd96d2421a Merge "Improve signal handling in soong_ui" 2017-02-22 20:29:53 +00:00
Dan Willemsen
9af5fb9ac8 Improve signal handling in soong_ui
Bug: 35214134
Test: ctrl-C during build
Test: add for{} to hang soong_ui in multiple places, ensure it exits
Change-Id: Ic71eedd4b1814ab2f3c441ae61a97570eda4fe16
2017-02-22 10:44:19 -08:00
Dan Willemsen
29f88279ba Filter kati output to collapse verbose messages
1. Collapses the "including .../Android.mk ..." lines like ninja does,
so that we overwrite the last line if there wasn't anything else to
print.
2. Strips ansi control codes so that log files don't include unreadable
characters.

Test: m -j
Test: m -j | tee output.log   (with colored output)
Change-Id: Ib18437f6f9d37084360097a9d586800c833072c5
2017-02-21 19:40:49 -08:00
Dan Willemsen
62944779e2 Merge "Ship the build log and trace on the build servers" 2017-02-09 02:10:07 +00:00
Dan Willemsen
8a073a8b7b Ship the build log and trace on the build servers
Test: m -j; m -j dist
Change-Id: I4718e7a0cda6719bbab4243038ea2666c212cf19
2017-02-07 16:51:11 -08:00
Dan Willemsen
223e3ae102 ui: Fix soong-executed ninja wrapping
For proper ninja smart terminal support, we need to pass stdin to
./soong. Otherwise it starts a new line if the terminal isn't wide
enough.

Test: `rm -rf out/soong/.bootstrap; m -j` in narrow terminal
Change-Id: I643a526001adc2323a420a03fa1df282554c7886
2017-02-06 20:59:11 -08:00
Dan Willemsen
d9f6fa28d6 Add build tracing
This creates a rotating build.trace.gz in the out directory that can be
loaded with chrome://tracing. It'll include start and end timings for
make/soong/kati/ninja, and it will import and time-correct the ninja log
files.

Test: m -j; load out/build.trace.gz in chrome://tracing
Test: multiproduct_kati -keep; load out/multiproduct*/build.trace.gz
Change-Id: Ic060fa9515eb88d95dbe16712479dae9dffcf626
2017-02-06 14:05:07 -08:00
Dan Willemsen
c2af0bedc1 Add multiproduct_kati
This is a replacement for build/tools/kati_all_products.sh using the new
Soong ui/build package. It doesn't even attempt to run ninja, and it can
be configured to run only the product config, or only the product config
and Soong.

For AOSP on my machine:
 -only-config  1.4s
 -only-soong   1m20s
 <none>        13m

Test: multiproduct_kati
Change-Id: Ie3e6e7bdf692e46a8b8eb828f437190f8003500b
2017-02-06 14:05:07 -08:00
Dan Willemsen
1e70446251 Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.

There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.

The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.

We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.

Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
  (also importing ninja traces into the same stream). I had this working
  in a previous version of this patch, but removed it to keep the size
  down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
  wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
  can always save the verbose log even if we're not printing it out to
  the console

Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2017-02-06 14:05:07 -08:00