Commit graph

55 commits

Author SHA1 Message Date
Cole Faust
9a346f6da3 Remove SymlinkOutputs
symlink_outputs was added so bazel could run ninja files, but we
abanoned that approach in roboleaf, and then roboleaf was cancelled
entirely. Remove this feature so we're more compatible with upstream
ninja / n2.

Bug: 160568334
Test: Presubmits
Change-Id: Ic368c48dd01b68e51c471c3fe90d0c02c55956e9
2024-01-18 12:43:24 -08:00
Cole Faust
e842a6c874 Revert "Add out/partitions_were_clean_at_start_of_build.txt"
Revert submission 2773149-partition_file_list_check

Reason for revert: b/305103467, b/305103783, b/305103468
Reverted changes: /q/submissionid:2773149-partition_file_list_check

Change-Id: I4116fe908cc914f41dc1bf7e950acb32bc546995
2023-10-13 02:51:08 +00:00
Cole Faust
39ec9276a7 Add out/partitions_were_clean_at_start_of_build.txt
Test: Presubmits
Change-Id: I1c7025f2c7594ae84b740ccbae4dcbc933b8cdf6
2023-10-05 13:26:16 -07:00
Cole Faust
583dfb426f Generate build_number.txt only once
build_number.txt was generated in runKati, and runKati is run multiple
times throughout the build. If a build number is not set, then one will
be generated using the current timestamp, meaning that multiple
different build numbers will be used in different phases of the build.

Instead, generate build_number.txt during SetupOutDir(), so that we
only have 1 build number.

This is a resubmission with a change to ensure out/soong is created
before writing the build number file.

Bug: 297269187
Test: edit a bzl file to force bazel to rerun, change the code so that use_fixed_timestamp=true during partition builds, m installclean, m bazel_sandwich, and observe that the bazel/make partitions are byte-for-byte identical
Change-Id: I47abcca166c701bb66a6a7731aecad5818279244
2023-09-28 17:56:43 -07:00
Roy Luo
f852b72a22 Revert "Generate build_number.txt only once"
This reverts commit ab6e524709.

Reason for revert: DroidMonitor-triggered revert due to breakage https://android-build.corp.google.com/builds/quarterdeck?branch=aosp-main&target=ndk&lkgb=10879153&lkbb=10879224&fkbb=10879224, bug b/302608661.

Change-Id: I07d47a92ea9a9f5d329e813859316decfd75cde1
BUG: 302608661
2023-09-29 00:28:32 +00:00
Cole Faust
ab6e524709 Generate build_number.txt only once
build_number.txt was generated in runKati, and runKati is run multiple
times throughout the build. If a build number is not set, then one will
be generated using the current timestamp, meaning that multiple
different build numbers will be used in different phases of the build.

Instead, generate build_number.txt during SetupOutDir(), so that we
only have 1 build number.

Bug: 297269187
Test: edit a bzl file to force bazel to rerun, change the code so that use_fixed_timestamp=true during partition builds, m installclean, m bazel_sandwich, and observe that the bazel/make partitions are byte-for-byte identical
Change-Id: I5bf3bb6c78e7529749ca9275f67db3f2f9e66af2
2023-09-28 14:11:43 -07:00
Jeongik Cha
f2af38d2cc Unset BUILD_NUMBER/HOSTNAME in kati.go
BUILD_NUMBER/HOSTNAME is passed by a file, not env variable.
To avoid kati recreation, unset BUILD_NUMBER/HOSTNAME after
writing it into a file

Bug: 278060169
Test: BUILD_NUMBER=1 m && BUILD_NUMBER=2 m (check there is no kati
re-run)

Change-Id: I5a31461dcf1e4b0634974bcb48a0d7482d42852a
2023-05-23 10:35:23 +09:00
Jeongik Cha
4e49bbde93 BUILD_NUMBER and BUILD_HOSTNAME doesn't affect kati regeneration
* Extract BUILD_NUMBER, BUILD_HOSTNAME to file to avoid kati change
* Handle FILE_NAME_TAG_PLACEHOLDER string in dist in build/make/packaging/distdir.mk

Test: check if kati isn't invoked even though BUILD_NUMBER, BUILD_HOSTNAME
  is changed
Test: m && m, and check if the second m is no-op
Bug: 278060169
Change-Id: I65eefc6bb86d4076098a1bf8b317b4cb88201499
2023-05-22 12:22:18 +09:00
Colin Cross
f3bdbcbea3 Fix propagating EMPTY_NINJA_FILE from multiproduct_kati to minibp
I6dca478f356f56a8aee1e457d71439272351390b replaced calling
build/blueprint/bootstrap.bash with running minibp directly,
but didn't propagate the EMPTY_NINJA_FILE environment variable
to minibp.  Since everything that uses EMPTY_NINJA_FILE is
executed directly from soong_ui now, replace the EMPTY_NINJA_FILE
environment variable with a config bool and pass it to minibp.

Bug: 189148777
Test: build/soong/build_test.bash --dist --incremental --shard-count=300 --shard=1 && du out/multiproduct
Change-Id: I4d64275ce02c5d68948012f71ac4dc3795af9e85
2021-06-01 11:43:55 -07:00
Colin Cross
41ad6b632f Revert "Skip old file cleanup for non-full builds."
This reverts commit 6e49493dac.

FULL_BUILD is not set during the config stage, so this never calls
cleanOldInstalledFiles.

Bug: 168105598
Bug: 182008128
Test: manual
Change-Id: I58a54f167763cdb77b3c1bd4f18f519fcaf8751a
2021-03-09 12:04:17 -08:00
Jaewoong Jung
6e49493dac Skip old file cleanup for non-full builds.
There are few build targets that don't generate an installed file list,
e.g. product-graph. Skip the old installed file cleanup step so that the
func doesn't complain about a lack of the list file.

Fixes: 168105598
Test: m product-graph
Change-Id: Ib7dce6b801979bb565b74d6355143bac23b84fe6
2021-01-05 16:47:04 -08:00
Jingwen Chen
b1d30d63c5 soong_ui kati.go: write more comments.
This CL also removes --warn_real_to_phony and --warn_phony_looks_real,
since --werror_real_to_phony and --wrror_phony_looks_real are already
specified.

Test: treehugger
Bug: 173474588

Change-Id: Icbb0f53182d1b912c62f8044f52bb79d985bf868
2020-11-18 22:16:08 -05:00
Jingwen Chen
ce679d29ec Add symlink_outputs support to Soong.
This CL adds symlink_outputs to various locations in Soong that creates
actions that creates symlink outputs, and explicitly mark them as such.

Test: m
Bug: 160568334
Change-Id: I322751bada52a9f49011c74731d84761586e03e7
2020-10-19 01:26:50 -04:00
Dan Willemsen
9f43597ff7 Remove obsolete PDK build functionality
This hasn't worked for a couple years, and continues to bitrot. Just
remove it.

Adds a bpfix rule so that we can eventually remove the
product_variables.pdk definition, which is now always a no-op.

Test: treehugger
Change-Id: I830b54d419b59f6db1d4617b45e61a78234f57a7
Merged-In: I830b54d419b59f6db1d4617b45e61a78234f57a7
2020-08-11 01:11:44 +00:00
Colin Cross
8ba7d47bba Dist build.ninja and Android.bp.list
Debugging issues on the build servers can be difficult because the
intermediate files are not visible.  Gzip ninja file and Makefiles
generated by Soong and the ninja files generated by Kati to the dist
directory, and also copy all of the finder output files.

Bug: 157656545
Test: m dist nothing
Change-Id: I48d75305e551ccae81c7a55721981cf58acd838b
2020-06-25 12:50:00 -07:00
Dan Willemsen
6587beddfa Turn on ninja phony outputs
This replaces the _kati_always_build_ hack with a ninja feature so that
ninja can actually understand what's happening. This means that we can
turn on some more options and checks around expected output filenames:

* Remove the output file(s) before the command executes
* Error if the output file(s) don't exist after the execution
* Error if the output is a directory

They're turned on immediately during the soong bootstrap runs, as those
run a limited number of rules. The main ninja execution does not remove
the output files, and prints warnings instead of errors for the others.
I'll turn them on as we understand how often those warnings are seen.

Test: m (check for new warnings)
Test: treehugger (check for new warnings)
Change-Id: I7f5c1a361dd1498eb54a2c07a918f3b0aa086e4c
2020-04-18 20:25:59 -07:00
Dan Willemsen
a17ac9632d Improve error handling around username/hostname
If this couldn't read the current user, it would print a generic
message, then proceed to dereference the invalid user struct :(

Provide somewhat reasonable defaults, and better error messages instead.

Test: m nothing
Test: run docker with misconfigured user
Change-Id: I21af77c7d8d1d2d27cb04546667eb1094c62a7a1
2020-03-10 15:45:47 -07:00
Dan Willemsen
1e775d7619 Remove files that should no longer be installed
Make now exports (partial) lists of what should be installed. Those
lists are incomplete, but if something is removed from the list, we can
remove it from the filesystem so that incremental builds work better.

Test: built-in tests (m blueprint_tools)
Test: remove a module from PRODUCT_PACKAGES, see the print and file removed
Test: change the name of a cts test, see the old one removed from cts
Change-Id: I6bc14c6a5af63383ea265e97ed8b875fa51af0ed
2020-01-03 16:46:15 -08:00
Dan Willemsen
7842826f49 Remove obsolete copy headers
When LOCAL_COPY_HEADER lines are removed, we just removed the copy rule,
we didn't actually remove the old header, so C/C++ files could still
include the old headers during incremental builds.

While we do consider LOCAL_COPY_HEADERS as near-obsolet, and it'll
disappear as we move everything over to Soong, this could produce some
unfortunate incremental build results while people are attempting to
remove them.

Fix this by ensuring that only the files currently listed in
LOCAL_COPY_HEADERS exist in TARGET_OUT_HEADERS after we run Kati.

Test: Remove a LOCAL_COPY_HEADERS entry; see the header removed
Change-Id: I817305703a6996d50490d552623d7df019b608c9
2019-12-13 19:38:23 -08:00
Colin Cross
9016b91290 Set default pool when RBE or goma is enabled
Pass --default_pool=local_pool to kati when RBE or goma is enabled
to put most rules into the local_pool.  Specific rules that support
RBE or goma will set .KATI_NINJA_POOL := none to remove themselves
from the local_pool.  Passing --default_pool will also disable the
hack in kati that sets the pool based on the presence of the string
"/gomacc" in the command line.

Fixes: 143938974
Test: inspect pools in build-${TARGET-PRODUCT}.ninja for m USE_RBE=true
Test: inspect pools in build-${TARGET-PRODUCT}.ninja for m USE_GOMA=true
Change-Id: I839b2488383fcd63fffd613e25b0b9abcb72b567
2019-11-11 16:08:42 -08:00
Dan Willemsen
70c1ff8dac Always set ASAN_SYMBOLIZER_PATH appropriately
Otherwise tools built with ASAN can't symbolize their dumps, as
llvm-symbolizer is not allowed to be used from $PATH.

Fixes: 139825736
Test: cherry-pick https://android-review.googlesource.com/c/platform/system/tools/aidl/+/1106532
  mmma ASAN_OPTIONS="" SANITIZE_HOST=address system/tools/aidl
Change-Id: Ib1eeec15ce6694d7bde54c7201b3280e38df3a83
2019-08-21 22:34:11 +00:00
Dan Willemsen
ce41e943be Remove support for ONE_SHOT_MAKEFILE
Test: treehugger
Change-Id: Icdbe62bcb7bc4717228c5d974962b939d8eafee6
2019-07-31 18:11:27 -07:00
Dan Willemsen
6097746769 Obsolete BUILD_BROKEN_PHONY_TARGETS
There are no remaining users.

Also mark BUILD_BROKEN_ANDROIDMK_EXPORTS and BUILD_BROKEN_ENG_DEBUG_TAGS
as deprecated in scripts/build_broken_logs.go

Test: treehugger
Change-Id: If7892bef1b9001f12a99565f886b395cf1985e70
2019-04-19 11:16:52 -07:00
Dan Willemsen
dcac078c91 Turn on more warnings during CleanSpec processing
We don't want these to be warnings, may as well be errors instead of
turning off the warnings.

Bug: 123583617
Test: treehugger
Change-Id: I6ca518c9647e712426952cb88bdb044d933b23f2
2019-02-04 20:58:25 -08:00
Dan Willemsen
cc62890866 Add --top_level_phony to catch more real-to-phony problems
As more phony rules move either to the packaging kati run, or soong, the
existing real-to-phony check failed to notice that they were phony
rules.

So I added --top_level_phony to Make that assumes that all leaf nodes
(source files, or nodes generated by another ninja file generator) that
do not have a '/' in them are phony targets.

The existing phony-looks-real check in Kati should prevent these from
being real generated files, and our source tree shouldn't have any
source files used at the root level.

Test: build_test on downstream branches
Change-Id: I77cff84b536c6cd6402307a9aa5a9c273e72c71f
2019-01-24 15:53:06 -08:00
Dan Willemsen
71edc8b848 Add BUILD_USERNAME and BUILD_HOSTNAME
As part of a future change to sandbox the build on Linux, the real
username will be switching to "nobody", and the hostname will be
switching to "android-build".

The USER environment variable will reflect the sandboxed value, so for
the build properties that want the external USER, they'll need to use
BUILD_USERNAME.

Similarly, BUILD_HOSTNAME will reflect the real value, while the
`hostname` tool will return "android-build"

Bug: 122270019
Test: check build.prop
Change-Id: I99604b9488732a63690b256dc4dd7894d369a32c
2019-01-07 23:00:54 +00:00
Nan Zhang
17f2767724 Generate build timing metrics to proto format file
Test: Dumped the text formated based metrics file to out dir,
and checked the file.
Bug: b/63815990

Change-Id: Iff476f72a0be74eb53b6b26ef468d11c0f24a404
2019-01-04 15:54:01 -08:00
Dan Willemsen
f99915f569 Speed up build_test
Stop writing out ninja files, as they become very large, especially when
multiplied by the number of defined products.

Test: treehugger
Change-Id: Id9529d14040acb72a0188e58b5db2911f142071e
2018-10-29 10:43:37 -07:00
Dan Willemsen
2d31a44b8a Stop using DIST_DIR in Soong
We're only using it to distribute files in case of failure, which isn't
well supported currently, but can be handled for now by using the
DIST_DIR environment variable during the command execution.

This was at least one cause that we'd be re-running Soong during every
build server build, as the DIST_DIR values are unique.

Test: m dist
Change-Id: Ibd5e6b6c46695350de80b745bfb6a6aa685033a0
2018-10-20 21:33:41 -07:00
Dan Willemsen
fb1271a52b Add a Kati-based packaging step
The idea is that we'd move the installation and packaging tasks over to
it, using data from Soong & the Kati reading Android.mk files.

This would allow us to make more fundamental changes about how we
package things without having to adjust makefiles throughout the tree.
Possible use cases:

* Moving some information from Soong's Android.mk output to a file read
  by the packaging step may allow us to read the Android.mk files less
  often, speeding up builds.

* Refactoring our current two-stage ASAN builds to run the Kati build
  step twice, writing into different object directories, then have a
  single packaging step that reads both outputs. Soong already has the
  capability of writing out a single ninja file with all the asan
  combinations.

* Running two build steps, one building the system-related modules
  using a "generic" device configuration, and one building the vendor
  modules using a specific device configuration. This could enforce a
  GSI/mainline system vs vendor split in a single build invocation.

* If all installation is through this tool, it will be much easier to
  track what should no longer be installed on an incremental build,
  reducing the need for installclean.

* Changing PRODUCT_PACKAGES should be a much faster operation, which
  means we could keep track of local additions to the images. Then
  `mma` would be more persistent, instead of installing something once,
  then never updating it again.

Eventually we plan on switching from Kati to something Go-based, but
this is a more incremental approach while we clean up everything else.

Currently, this just moves the dist-for-goal handling over to the
packaging step, so that we don't need to read Android.mk files when
DIST_DIR changes, or we switch between dist vs not.

Bug: 116968624
Bug: 117463001
Test: m nothing
Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-10-19 09:55:00 -07:00
Dan Willemsen
2997123f73 Refactor Kati logic
Make the cleanspec & Android.mk runs of kati use much of the same code
and arguments.

Also renames 'Kati' to 'KatiBuild' in many cases, in the the expectation
that we'll have a 'KatiPackage' step in the future.

Use --no_ninja_prelude and move local_pool & _kati_always_build_ into
the combined ninja file. This will reduce the need to re-read makefiles
when Goma is enabled, and it allows us to include more than one
Kati-generated ninja file in the build graph.

Bug: 116968624
Test: build_test on downstream branches
Change-Id: Ibdac689b81f62dc293647fad917d84946f2c3cfa
2018-09-28 23:09:24 -07:00
Dan Willemsen
25a56186d9 Pass --writable to kati
These warnings are usually hit in one of the two conditions:

1. Targets that should be marked .PHONY, but are not.
2. Attempts to define output files in the source tree.

See https://android.googlesource.com/platform/build/+/master/Changes.md#phony_targets
for more information.

Test: multiproduct_kati in all downstream branches
Change-Id: Ibe37cce320612825c22987e94d0740e8c0565b6f
2018-09-03 12:32:34 -07:00
Dan Willemsen
d8aa39d1c3 Support turning phony warnings into errors
These still default to warnings, but if a board decides to set
BOARD_BROKEN_PHONY_TARGETS := false, they'll turn into errors. More
likely I'll just be marking the broken targets as broken, then switching
the logic here to be like the dup rules warnings/errors.

Test: On a build with warnings, try <missing>, "false", and "true"
Change-Id: I041e1ff4618c4114ec43015b6c0ae1b49b36b6f8
2018-08-27 15:01:03 -07:00
Dan Willemsen
f880b58970 Turn on phony checks
--warn_real_to_phony will print a warning any time that a non-PHONY rule
depends on a PHONY rule. Since PHONY rules are always considered dirty
this would force rebuilds on every incremental build. Real rules should
always depend on rules that have real outputs.

--warn_phony_looks_real will print a warning any time that a PHONY rule
has an output that looks like a file (has as / in it). This is can cause
confusion for users if it's not a real file, and it shouldn't be marked
as PHONY if it will be run every time (since that will slow down
incremental builds).

Test: run build_test on downstream branches, see few warnings
Change-Id: Id135c85c991976be33cd48a34eba0caaff1eda2c
2018-07-23 23:09:05 -07:00
Dan Willemsen
b82471ad6d Add a unified status reporting UI
This adds a new status package that merges the running of "actions"
(ninja calls them edges) of multiple tools into one view of the current
state, and gives that to a number of different outputs.

For inputs:

Kati's output parser has been rewritten (and moved) to map onto the
StartAction/FinishAction API. A byproduct of this is that the build
servers should be able to extract errors from Kati better, since they
look like the errors that Ninja used to write.

Ninja is no longer directly connected to the terminal, but its output is
read via the protobuf frontend API, so it's just another tool whose
output becomes merged together.

multiproduct_kati loses its custom status routines, and uses the common
one instead.

For outputs:

The primary output is the ui/terminal.Status type, which along with
ui/terminal.Writer now controls everything about the terminal output.
Today, this doesn't really change any behaviors, but having all terminal
output going through here allows a more complicated (multi-line / full
window) status display in the future.

The tracer acts as an output of the status package, tracing all the
action start / finish events. This replaces reading the .ninja_log file,
so it now properly handles multiple output files from a single action.

A new rotated log file (out/error.log, or out/dist/logs/error.log) just
contains a description of all of the errors that happened during the
current build.

Another new compressed and rotated log file (out/verbose.log.gz, or
out/dist/logs/verbose.log.gz) contains the full verbose (showcommands)
log of every execution run by the build. Since this is now written on
every build, the showcommands argument is now ignored -- if you want to
get the commands run, look at the log file after the build.

Test: m
Test: <built-in tests>
Test: NINJA_ARGS="-t list" m
Test: check the build.trace.gz
Test: check the new log files
Change-Id: If1d8994890d43ef68f65aa10ddd8e6e06dc7013a
2018-07-12 14:15:31 -07:00
Dan Willemsen
2bc456eff9 Switch implicit rules into errors
There are no users of these, other than the PDK, which is already
exempted.

Test: m nothing
Test: Check for warnings in downstream build_test logs
Test: Check for warnings in last two days of continuous build logs
Change-Id: I7c7d0068e928e591355c39d22635af86b8f43f1b
2018-06-21 21:41:35 -07:00
Dan Willemsen
fa42f3c10d Turn implicit rules into warnings
Test: build_test in downstream branches
Change-Id: I853e3fa11793cc76ad568b74f7dc31c3d1c5582b
2018-06-20 16:50:45 -07:00
Dan Willemsen
d368d6ff78 Turn suffix rules into errors
We've turned off suffix rules with .SUFFIXES, but make them explicit
errors.

Test: build_test on downstream branches
Change-Id: I682ee3eb3a8f1451cd7e16f1e1d59afca5d21a26
2018-06-15 21:53:18 -07:00
Dan Willemsen
81759847b2 Add --kati_stats to cleanspec processing
So that we can see where the time is going. Also removes the obsolete
ASAN_OPTIONS, which moved to kati, but I forgot to update this instance.

Test: m nothing; check out/soong.log
Change-Id: I0c4066bad20fc2dc22b389f4c973d10dca554ee3
2018-05-02 17:44:36 +00:00
Dan Willemsen
6ab79db2e6 Pass TARGET_DEVICE_DIR from dumpvars to later kati runs
The find commands used to locate the BoardConfig.mk can't be optimized
by Kati, so we're currently spending ~125ms three times during every
build (dumpvars, cleanspec, and the main kati run). Preserve the value
of TARGET_DEVICE_DIR from the dumpvars run so that we only need to run
the find commands once.

Bug: 78020936
Test: out/build-taimen.ninja is identical
Test: out/soong.log shows that we're not running these finds again
Change-Id: Iee56b454c3661de2b58c161169218ecaf2135398
2018-05-02 00:06:28 -07:00
Dan Willemsen
3d60b115a0 Allow boards to turn overriding commands into an error
Many board still have this problem, but if we can switch some over,
we'll prevent global problems and have the ability to clean boards up
one-by-one.

Bug: 77611511
Test: lunch aosp_arm-eng; m nothing
Test: lunch aosp_marlin-eng; m nothing
Test: build_test on all downstream branches
Change-Id: I78bee44adde2059d01188658b9050927748d2028
2018-04-05 16:23:10 -07:00
Dan Willemsen
d9429e6ff2 Handle Scanner errors in kati output filter
If Kati ever wrote a line over 64k characters, the scanner in soong_ui
would error, and Kati would hang trying to write into the pipe. Now if
the Scanner errors out, fall back to copying directly from the pipe to
the output.

Test: Add $(warning $(shell find frameworks)), does not hang
Change-Id: I86412ad2f53f2fe3cbda901ee673abb904d56d3c
2018-03-06 22:08:27 -08:00
Dan Willemsen
0c51851f08 Apply kati output rewriting to dumpvars
This way we strip out ANSI codes when using dumb terminals. It's likely
overkill to use katiRewriteOutput, but we should rarely see any output
on stderr while dumping variables. This also lets us turn on kati_stats.

Bug: 71729611
Test: lunch missing-eng             (colored error)
Test: lunch missing-eng 2>&1 | cat  (non-colored)
Test: TERM=dumb lunch missing-eng   (non-colored)
Change-Id: Ic63fd42d82a4a64e5c68aecd9ae0f242a0d703f1
2018-01-09 02:24:17 -08:00
Dan Willemsen
75d2c170b4 Always record kati stats into soong log
Pass --dump-stats to ckati, but filter out the '*kati*' lines so that
they only end up in our verbose output. That way we've always got access
to the statistics.

Bug: 36182021
Test: m nothing; cat out/soong.log
Change-Id: Iaf7a814fc67f3e475c913faf69924a7f4e2ae3b3
2017-10-14 00:42:30 +00:00
Dan Willemsen
b2e6c2e571 Switch product configs from make to ckati
This speeds up dumping make variables from ~380ms using make to ~220ms
using ckati. It also means that we're consistently using the same parser
for builds (with the same .KATI_READONLY/etc extensions).

envsetup.sh (lunch) / other scripts still use make, changing those to go
through soong_ui will be a future change.

Test: m clean; m nothing
Test: USE_GOMA=true m nothing
Test: m PRODUCT-aosp_x86-sdk
Test: m APP-Calculator
Test: build/soong/build_test.bash -only-config  (on AOSP and internal master)
Change-Id: I6ca554de8de4955fb869001d06d29969b75751cc
2017-10-12 18:11:08 -07:00
Dan Willemsen
59fdf96fcf Split CleanSpec processing out of main Kati run
Instead of pairing the reading of CleanSpec.mk files with the reading of
Android.mk files, split them into separate, individually cachable steps.
This way we only read Android.mk files once after a clean/sync. We'll
still read the CleanSpec.mk files multiple times, but that's
significantly faster than reading all the Android.mk files.

This adds about 50ms if kati doesn't need to reread the CleanSpec.mk
files. Reading all the CleanSpec.mk files takes about a second.

Bug: 35970961
Test: m clean; m nothing; m nothing
Test: Add CleanSpec.mk line, see it executed.
Change-Id: I83bad15c50709510959d5b8b673a907b8aa7de82
2017-08-23 01:53:33 +00:00
Dan Willemsen
e27ed543bf Stop setting ASAN_OPTIONS for kati
Kati sets detect_leaks=0 by default now.

Test: unset ASAN_OPTIONS; SANITIZE_HOST=address m nothing
Change-Id: I0cbf04ae9ed40de520abca3fb35c65e7f543682d
2017-08-21 20:54:06 -07:00
Dan Willemsen
418420e49d Turn find emulator warnings into errors
Test: build/soong/build_test.bash (multiproduct_kati)
Change-Id: Ieb65a457c8c1ed4bff44a01aa2cc11203172ba9b
Merged-In: Ieb65a457c8c1ed4bff44a01aa2cc11203172ba9b
Merged-In: I70b696525a90c64cc05ad4caa1c3647d57ec9e9e
(cherry picked from commit dc5b70f97c)
(cherry picked from commit e8a25d5577)
(cherry picked from commit 14b3f7f12b)
2017-06-21 12:44:18 -07:00
Dan Willemsen
a3e6c520ac Always run asan ckati on the build servers
We're seeing some rare crashes that appear to be stack overflows. To get
better debugging, use the asan version of ckati when 'dist' is
specified.

Bug: 36182021
Test: m -j
Test: m -j dist
Test: make -j dist (w/o lunch)
Change-Id: Ic3a5590974bfd718bf1929355d344b5933ac1d4f
2017-05-05 16:59:10 -07:00
Dan Willemsen
269a8c78e7 Add exec.Cmd wrapper for logging / sandboxing
Wrap os/exec.Cmd to use our Context and Config interfaces for automatic
logging and error handling. It also simplifies environment modification
based on the Config's environment.

This also adds sandboxing on Macs using sandbox-exec. A simple profile
is provided that only logs on violations, though multiproduct_kati on
AOSP has no violations. This isn't applied to ninja, only make / soong /
kati to start with. I measured <5% time increase in reading all
makefiles, and no noticable difference when kati doesn't regenerate.

I'd like to spin up a process to dump violation logs into our log file,
but the log reporting changed over the range of Mac versions that we
support, so that's going to be more complicated. Opening Console.app
works in all cases if you're local -- just search/filter for sandbox.

Linux sandboxing will be implemented later -- the sandbox definition is
opaque enough to support a different implementation.

Test: multiproduct_kati on AOSP master on Mac
Change-Id: I7046229333d0dcc8f426a493e0f7380828879f17
2017-05-05 16:25:40 -07:00