Make sure the staging directory doesn't have stale file after
installclean.
Test: m installclean && ls $ANDROID_PRODUCT_OUT
Change-Id: Idcd7a883a6cba4b7eb1eb99751f5d03cba365a12
Reverts commit 827aead340
The change being reverted is the result of misunderstanding of how the
status system works. It has to be reverted because it would suppress
_all_ error messages from Make. Achieving what this change purports is
is more involved and requires changing the code to separate progress message
stream from application output stream.
Test: run failing build with ANDROID_QUIET_BUILD=tree and observe errors being output
Change-Id: If9148a7fa773ae32fb0870a448e9470560e53900
Export it so that we have one source of truth,
which can be used by the called tools.
Test: Use the exported value in ART build.
Change-Id: Iaa45b1780ffae11e09790e193ca2bd01e2b732a8
It is useless to run this binary when gcert creds aren't available, so
adding a check for gcert creds before calling this binary. Also modified
metrics to capture missing gcert.
Bug: b/237938234
Change-Id: I94b5aa2d53c4e5bb1a5fdc913a33ef5bae965d32
This is to generally warn Google devs about lack of LOAS creds.
Given the switch to RBE, lack of LOAS creds will start resulting in build
failures, so adding a warning message ahead of time to all builds. Will
convert this to a build failure after a week.
We need LOAS creds to be able to even fetch the CDPush config file to
determine whether to use RBE in a build a not, which is why I'm making
this a broad error message across builds.
Test: ran "m nothing" build with / without gcert and with / without
"stubby" in $PATH and confirmed appropriate error message.
Bug: b/235985591
Change-Id: I15cbaf372b0a7b79f868a06c7d5ede19b49ae687
This log is not super useful to be honest, so avoid printing it to
stdout of the build.
Bug: b/236709761
Change-Id: Ic96130a285160b386df3afc269dd5ef97c7965bf
We had been doing this for actions, but not messages.
Test: m nothing && TERM=dumb m nothing
Bug: 235414673
Change-Id: I5c6c009490abe1d02a4ae2272bd1831bd7ca9f7f
RBE logs directory is currently messy:
1. We use RBE_output_dir variable to specify where rbe_metrics.txt /
rbe_metrics.pb file should go to.
2. We use proxy_log_dir to specify where *.rpl / *.rpi (detailed per
action info log file) should go to.
3. We use RBE_log_dir to specify where reproxy.* / bootstrap.* log files
should go to.
Ideally, all RBE related logs should go to one single directory. In this
CL, I'm creating a temporary log directory under out/soong/.temp/rbe/
where all RBE related log files per build would go to. The log dir
prefix is also being set to the same prefix as the socket address file.
Test: Ran a sample build with `m libc` and ensured that logs are getting
cleared across rebuilds and that `rbe_metrics.pb` file is properly
generated and being copied to the right location (i.e., from
out/soong/.temp/rbe/<rand> to out/ dir)
Bug: b/233382420
Change-Id: I46bd38d50419cb9e54e8202d24222979e47ff5ca
This reverts commit b018b83664.
Reason for revert: Builds will start failing now since the gcert change is about to go in.
Change-Id: I81d36ce6bf4f7a9bc120b34d411f2cb4ec64e5eb
This required the following:
- Adding Platform_base_sdk_extension_version to default soong.variables
- Teaching the symlink tree creation code to understand symlinks
- Making finder.go follow symlinks when requested
Adding yet another knob is unfortunate, but I can't allow that
unconditionally because the Android code base contains a number of
symlinks giving rise to infinite directory trees because they point back
to their parent and this seemed preferable to adding complicated logic
like "follow symlink but if only its fully resolved version does not
point under the source tree".
I could be convinced about the latter, though.
Test: Presubmits.
Change-Id: I453f6b7e5334771f5832c700db00f9d24ed1d82f
Now that we are ready to rollout gcert, adding a warning that builds
will fail if gcert doesn't exist. Doing this only for builds with
ANDROID_BUILD_ENVIRONMENT_CONFIG=googler.
Test: Ran a build without gcert and USE_RBE=true and it printed the
warning.
Bug: b/229743321
Change-Id: I616a4a534ab277873ac7e19baecb78620f07854c
These can take a minute or more of build time, and currently hold up
other processing. We could move to a parallel compression scheme to
speed them up, but it's simpler just to background them and continue the
build, waiting to make sure they've finished before exiting.
Testing this on git_master/flame-userdebug:
This brings a local no-op `m nothing dist` build from 2m1s to 1m39s,
even though most of that time is still spent waiting on these (since
there is very little else happening in a `nothing` build when Soong and
Kati don't need to run).
Running `touch art/Android.mk; m nothing dist` (so the above, but with
Kati) goes from 3m39s to 2m2s. We spent 9 seconds at the end waiting for
these to finish.
CI cases that almost always run Kati plus some number of other actions
are likely to completely hide the time taken to dist these files.
Bug: 229932999
Test: Check $DIST_DIR, files still exist
Change-Id: I1fb78d7c4d5103d72b5d71d9277dea0d452f8968
If we didn't need to run soong_build during the current run, we still
try to load the soong metrics. But in the case of `dist`, that's in a
directory that is not guaranteed to persist between runs.
Make loading the soong metrics optional if the file does not exist. Also
fixes a variable shadowing issue that meant we never passed it into
ctx.Metrics.
Test: treehugger
Change-Id: Ic836282f4d13e91daa0e7241ad7c488de3293d8b
This allows us to track how much time is spent in Clang.
Test: TOOLCHAIN_RUSAGE_OUTPUT=/tmp/rusage.txt m
Change-Id: Ib2961904f363bc59bd9d928bb055a96740cb9f17
Tracks the time to call the expconfigfetcher and whether it updated the
config file or not.
Test:
Ran a build with a "regular" user and verified the metrics indicated
NO_CONFIG and no config file was created.
Ran a build with an "experiment" user and verified the metrics indicated
CONFIG and a googler.json config file was created.
Bug: http://b/227186589
Change-Id: I030e9497c369459ae98c0c4f7765aef6de4c601f
Currently, mk2rbc/rbcrun aren't built in certain phases
of the build, which causes the Starlark product configuration
to fail when using get_build_var if a regular build hasn't
been run before.
Bug: 226677850
Test: Manually
Change-Id: If85608eef63be25fddc4d916c82247c13eb0195f
Metrics are added to a materialized protobuf soong_build_metrics.pb
Recorded events have scoped names and recorded runtimes.
For example, mixedbuilds includes the following events:
mixedbuilds.prepare
mixedbuilds.analyze.write_files
mixedbuild (this is a total of all mixedbuild events)
Test: Manually verified materialized protos for mixedbuild, bp2build,
and legacy build.
Change-Id: I2e72d4502c0373bee0279bb1320b2e4dd586129c
Dumping of RBE metrics should only happen when reproxy was started.
Reproxy is started only when Soong is run in make / build mode and not
in dumpvars mode.
Doc: https://docs.google.com/document/d/1Iqhu5QgozB_rA9fucMHO6X4U6A8e8qFtZyub9j83ICQ/edit#
Test: Ran a build with / without RBE. No metrics dumped during lunch
without RBE and metrics dumping worked as expected with RBE.
Bug: b/197213143
Change-Id: Ic3403331e1f4ac2c7c72fd9169aa7b82a7545c75
This is to ensure that we don't default to using
application_default_creds when it is not set and rather
"RBE_use_google_prod_creds" variable is set. This is necessary to
rollout RBE gcert auth.
Bug: b/224446621
Change-Id: I51e3f1aefadf282240b8f861efafde6b9b61d305
Use EXPERIMENTAL_USE_OPENJDK17_TOOLCHAIN=true to build with OpenJDK 17
while still targeting java language version 11.
Bug: 219098645
Test: m EXPERIMENTAL_USE_OPENJDK17_TOOLCHAIN=true
Change-Id: I89fb04ecde4dbd1e102098f316692624dd67f8a2
Combined with roboleaf CI changes to pass this flag and subsequent
pipeline changes, this will allow basic mkfile metrics to be
dashboarded, which will help track mk2bp progress.
Bug: 217764943
Test: manual; verified mk_metrics.pb is generated iff --mk-metrics is
specified, used printproto on result
Change-Id: I348994b4d7e3479d0dc04de0276b94a702aac95b
To support using google prod credentials for Google developers, we need
to be able to make an RPC call to exchange credentials. This RPC call
needs to be issued via stubby and hence allowlisting the stubby binary.
dwillemsen@ also pointed out that this should be less effective for
other tools running within the build since network access is disabled
for the other tools anyway via sandboxing.
Bug: b/222484146
Change-Id: I29aec9576e86b9ed3051df7943404abd478c549b
Having soong generate a list of makefiles for mk2rbc
to look through is much faster than having mk2rbc search
itself. Profiling the readLinesFromeFile() function that
reads the list of makefiles shows it takes on the order
of 200 microseconds, much faster than the ~5 seconds
it takes for mk2rbc to search the tree itself.
This CL also allows include statements that are prefixed
with a variable. The concern with this was that there
would be a lot of load statemnts emitted for generic
include statements, causing the generated code to look
ugly, and converting and loading all those files could
cause performance issues. On the performance issues
front, there's already a check that it doesn't result
in over 150 potentially included files. We can lower
that number if necessary, but it's probably good for now.
On the generated code front, while it's true that it's
ugly, it's better to have working but ugly generated
code than refusing to generate anything working at all.
To ensure the soong finder step isn't slowed down due
to having to find a bunch of new makefiles, I profiled
the combination of newSourceFinder and FindSources in main.go:
Baseline incremental:
338.011634ms
340.853335ms
348.541762ms
333.229644ms
349.124824ms
Baseline clean:
1.003836419s
1.006203912s
996.193648ms
1.031005604s
1.03691152s
Modified incremental:
349.029285ms
349.264496ms
351.774948ms
337.63187ms
359.425306ms
Modified clean:
1.028238704s
1.053103506s
1.032757506s
1.016631201s
1.04288108s
So we can see the times are barely affected by this change.
Fixes: 213508006
Test: go test
Change-Id: Iab18bfb127ba3b7e63f2c01f69064805a8398764
CI builds are doing m installclean between them;
which is not causing the system_dlkm to clean
it's build.prop and it is stuck to the first CI
build when feature landed causing ota signing
test to fail.
Local reproduction using:
m installclean && m -j96 && m dist
validate_target_files out/dist/*target_files-*.zip
Bug: 217507332
Test: Local verification, TH & Forrest
Signed-off-by: Ramji Jiyani <ramjiyani@google.com>
Change-Id: I97f2011a61252d3cb798af0fab99221b754328ca
The expfetcher binary (added in
https://critique.corp.google.com/cl/424076941), can fetch experiments
configuration from CDPush and load them into Soong as part of its
startup. If the binary exists, then we use it to fetch experiments
config and put them in the out directory so that those env variables can
be loaded by Soong during startup and inturn control whether RBE should
be enabled or not.
Design doc: go/rbe-android-dev-gradual-rollout
Test:
1. time lunch aosp_bramble-userdebug (first lunch when "out" dir doesn't
exist) - https://paste.googleplex.com/5824218777255936 - took ~9s and
ran with RBE enabled.
2. time lunch aosp_bramble-userdebug (when "out" dir already exists) - https://paste.googleplex.com/6485045129773056 -
took ~3.5s and ran with RBE enabled.
3. time lunch aosp_bramble-userdebug (when
"vendor/google/tools/soong/expfetcher" doesn't exist) - https://paste.googleplex.com/6103083353374720 - took 6.5s
and not RBE enabled.
Bug: b/215181607
Change-Id: Ie3c085498c59929119534aa98863566eecb8e4eb
When the environment variable config file is loaded after config object
is initialized, we end up loading the USE_RBE variable in the config
file after the code to start reproxy process is run. This causes a
problem when USE_RBE variable itself is set in the config files. To
avoid this, I've moved the config file loading code to be within the
config package, this makes the main.go file cleaner too.
Test:
Ran a build with USE_RBE:true set in the config json file. The build
fails without this change and succeeds with this change.
Bug: b/209486170
Change-Id: Iab3957b64f4b5456a861057d16ad318f4f78e0cb
For example, new metrics at HEAD highlight that 99 of 1379 total
cc_library_static modules are converted.
Test: m bp2build, printproto
Change-Id: I6cc4227124e9a130b75911f3e40e6585d731d00a
Reproxy should be respecting the -j value set by the user and not be
doing excessive local executions assuming the whole machine is available
to it. The local_resource_fraction flag is a way to ensure reproxy
consumes CPU only upto the -j value specified by the user.
Test:
1. Ran with `USE_RBE=true m -j 10 nothing` and local_resource_fraction
was set to 0.14 on a 72-core workstation.
2. Ran with `USE_RBE=true m nothing` and local_resource_fraction was set
to 1.0 on a 72-core machine.
Bug: b/207296459
Change-Id: I118bc10109f5a55df7a6c1ecd79499320055f2ed
Colin and Jin opined that we shouldn't change production in order to
make debugging easier. I originally thought it's better to have one
fewer knob but I don't have very strong feelings about it and their line
of reasoning makes total sense.
Test: Presubmits.
Change-Id: Id8a914aebbeffec6e8f860157c13f81093194f14
This behavior is enabled by setting the environment variable
SOONG_UNBUFFERED_OUTPUT=1 .
This is not the default behavior because only one process can be running
in the console pool and this would thus result in a loss of parallelism
if multiple soong_build invocations are needed.
Changing the value of this environment variable does not result in a
rebuild, since it does not affect the build outputs.
Test: Presubmits.
Change-Id: I49d15eb145966bd805f0a47b46866285203b122a
[ 0% 0/5 ] generating Soong docs at out/soong/docs/soong_build.html
0:03 analyzing Android.bp files and generating ninja file at out/soong/build.ninja
0:03 converting Android.bp files to BUILD files at out/soong/bp2build
0:03 generating the Soong module graph at out/soong/module-graph.json
0:03 generating the Soong module graph as a Bazel workspace at out/soong/queryview
0:03 generating Soong docs at out/soong/docs/soong_build.html
Test: m soong_docs json-module-graph bp2build queryview nothing
Change-Id: I33abb8828ead9ef999ffa1a6c9c4b016957c6dd1
It's built explicitly by Microfactory but is under out/ and as such, it
would be normally flagged as a file in out/ that doesn't have a
generating rule.
Previously, a no-op rule was added to the Ninja file to prevent this
from happening, but there is no good reason to have two mechanisms to
exempt files under out/ from that check.
Test: Presubmits.
Change-Id: I37fc458ecc5f5734cd81bce2f4894063658df38f
Move cc module installation rules into Soong by overriding
InstallBypassMake. Update the locations that find host tools
to look in the Make installation directory instead of the Soong
installation directory, which will no longer be used.
The methods that find host tools are also used on go binaries,
so update the config methods that tell Blueprint where to install
go binaries to the Make installation directory too.
Bug: 204136549
Test: m checkbuild
Change-Id: Id172592c195e506102982a4af0084f6d9c68a896
The bpglob changes in build/blueprint submitted together with this
change make it necessary.
Test: Presubmits.
Change-Id: I3d19761ec1a199c2f77693a882f8744ce404c6de
It's better not to have state hidden in dotfiles (or rather,
dotdirectories) if we can help.
It's questionable whether the "linux-x86" path segment makes sense since
soong_build only ever runs on one operating system, but I didn't want to
rock the boat now.
Drive-by fixed some quoting fixes in rbc-run. Notably, I didn't wrap
`$@` into double quotes because I don't know whether the lack of double
quotes was intended or not.
Also drive-by fixed the fact that "out/soong" was added twice to the
directory name of bpglob. This turned out not to be a problem because
bpglob doesn't need to be explicitly built: if it's needed, it's
declared as an input of the glob files so it'll be built automatically
(at the cost of the first "null build" not actually being a null build)
Test: Presubmits.
Change-Id: I710d8d16cd8212059a0ca1ee95378505303eed83
ToolDir is going to become unstable when switching between KatiEnabled
and Soong-only builds while the duplication between out/soong/host and
out/host is resolved. bpglob gets executed very early during bootstrap,
before the primary builder has run to update the paths to match the
current configuration. Move it into SoongOutDir() so that its path
is more stable.
Moving bpglob causes incremental build issues, so add a bootstrapEpoch
constant that can be changed when making incompatible changes and will
wipe bootstrap intermediates.
Bug: 204136549
Test: m nothing
Change-Id: I7b1bd1ebfe1209d11db691b3ee00873ef92658cd
It was the same as SoongNinjaFile, except it used
shared.JoinPath instead of filepath.Join, which
doesn't matter here because the second argument
is never an absolute path.
Test: Presubmits
Change-Id: I95e64a8b8971bd168a1dc6670ae80f4d336045b5
1. Don't upload if the uploader binary is not present.
Bug: b/193703183
Test: Run the following tests locally:
1. run "go test ." in build/soong/ui/build
2. create vendor/google/misc/metrics_uploader_prebuilt/metrics_uploader.sh
and run m nothing to ensure metrics_uploader.sh was called
3. remove the file created in step 2 then run m nothing and ensure it
doesn't attempt uploading
Change-Id: I081a5510b3f30480720c3e7dd235623c805fa7a4
This was reverted unnecessarily along with the change that actually
broke the build.
Previously, the sdk snapshot was assumed to be generated for the
current build system. This change adds support for the
SOONG_SDK_SNAPSHOT_TARGET_BUILD_RELEASE environment variable which can
be used to select the target build release in which the generated
snapshot will be used.
At the moment this will only affect the properties that are output but
if/when it becomes necessary it can also be used to control more
aspects of the snapshots such as supported member types or even
members.
This change does not modify any properties that are build release
specific. That will come in following changes. However, it does add a
test that targets build release S which defines a baseline for
properties that will be affected in following changes. That baseline
makes it easier to see the effect of those follow up changes.
Bug: 197842263
Test: m nothing
Change-Id: If4b452237f105382550d2842c8010249afbc7432
Previously, the sdk snapshot was assumed to be generated for the
current build system. This change adds support for the
SOONG_SDK_SNAPSHOT_TARGET_BUILD_RELEASE environment variable which can
be used to select the target build release in which the generated
snapshot will be used.
At the moment this will only affect the properties that are output but
if/when it becomes necessary it can also be used to control more
aspects of the snapshots such as supported member types or even
members.
This change does not modify any properties that are build release
specific. That will come in following changes. However, it does add a
test that targets build release S which defines a baseline for
properties that will be affected in following changes. That baseline
makes it easier to see the effect of those follow up changes.
Bug: 197842263
Test: m nothing
Change-Id: I5cfde5e9e86af522b0d5030d301e116b0233fd62
If SOONG_UI_ANSI_OUTPUT is set to a true value force the simple status
output to keep ANSI codes. This will allow buildbots to collect logs
with ANSI codes in them. Smart status output is not affected as it
always keeps ANSI codes.
Bug: 147310922
Test: manual
Change-Id: I0b78ceebb65125b8e8dafb4787816fb679d3eb3e
Make smart status output automatically disable table mode when the
window size is not available.
Bug: 147310922
Test: status_test.go
Change-Id: Ic57e1efd742e869bb84b6a86b3014335f5a65d39
`m product-graph` runs a "non-full" build, which skips
some steps to improve performance. One of the steps
it skips is making a .installable_files file. Soong
reads this file and removes any files that were removed
from the list in this file. It errored out if the file
didn't exist. Make soong less strict about the presence
of this file, because if it doesn't exist, there shouldn't
be any installed files to remove either.
Fixes: 168105598
Test: rm -rf out/, m product-graph
Change-Id: I366f7b09d87911f9660d4e08c2d2f097cc04800f
- Remove useValidations: whenever it was used, it was turned on, so
there was no point in keeping that knob
- Use the new soong_build invocation name constants
Test: Presubmits.
Change-Id: Ia3e8928d30d3de1e0adaa741111771304e56b139
They were a bit repetitive.
In addition, make the environment checks more correct; they could
probably use an integration test or two to make sure that when the
environment changes, exactly those outputs are rebuilt that need to be,
but for now, this is an improvement already.
Test: Presubmits.
Change-Id: Idd79b81ca6975d57d00e5bf4699d266152505ff8
This makes soong_ui the only place where soong_build is invoked, thus
greatly simplifying the conceptual model of the build.
It comes with the slight limitation that now soong_docs (and queryview
and the JSON module graph) are not Make targets anymore, but I suppose
that's an acceptable loss.
The only place where someone depended on soong_docs from a Makefile is
removed in a separate change.
Test: Presubmits.
Change-Id: I3f9ac327725c15d84de725d05e3cdde1da3dcbe2
Its value is a function of the call site, so it doesn't make a lot of
sense to plumb it through the configuration.
Test: Presubmits.
Change-Id: If928b34de075969fd42932212ce9187808cbdf86
It used to be invoked from out/soong/build.ninja, which required two
soong_build invocations one after the other (ne to generate
out/soong/build.ninja, one to generate the queryview workspace). This
was slower and required some shell-quoted-in-ninja-quoted-in-Go .
Test: Presubmits.
Change-Id: Idda79c067606663b66e9f94626fa24f3b5af4114
This includes the JSON graph generator and bp2build.
Before:
GENERATE_BAZEL_FILES=1 m nothing
GENERATE_JSON_MODULE_GRAPH=1 m nothing
Now:
m json-module-graph
m bp2build
They can now also be combined with other targets or each other.
The longer-term goal is to run "m queryview" and "m soong_docs" using
the same infrastructure. There are two alternate approaches:
1. Call soong_build from within the main Ninja invocation. This requires
two sequential soong_build invocations and is thus slower.
2. Do everything requested in the same soong_build invocation. This
would be faster, but one AFAIU can't tell Ninja that multiple possible
actions can build the same output so that doesn't work.
(1) is somewhat more desirable because soong_docs seems to be built
from build/make/core/main.mk ; I assume that that can be worked around
although I haven't checked where the output of "m soong_docs" goes.
Test: Presubmits.
Change-Id: If5ba36490d9f3f60733e6d6be9286eb2b67c3ff5
The only case for this was Blueprint itself at build/blueprint, but with
that being a part of Soong, this special case is not necessary anymore.
Test: Presubmits.
Change-Id: Icc51cd80bd43f936a97018061cfbf76fc385e4c3
It was confusing because bootstrapping uses two configurations: the
"global" config and the special-cased bootstrap one.
This change merges them.
Test: Presubmits.
Change-Id: I82b482cbe28a343ab6991374b2a28667e1a06b48
These are: TopFile and GeneratingPrimaryBuilder.
Also re-shuffle the list of flags to make a bit more sense and finish
the rename of BuildDir and NinjaBuildDir to SoongOutDir and OutDir,
respectively.
Test: Presubmits.
Change-Id: I103ff5f09f1c0d16f695a7da5dea13b55028e33e
The dangling rules check was running `ninja checkbuild -t targets rule`.
This apparently worked fine with glibc, but is failing when ninja is
compiled with musl. Musl's getopt_long is leaving "checkbuild" in argv
when calling ninja's ToolTargets, resulting in:
ninja: error: unknown target tool mode 'checkbuild'
Remove the extraneous checkbuild argument.
Bug: 190084016
Test: m checkbuild
Change-Id: I61b6475a5f7cb9d3d47232057ecfdc798864dd81
Before, we piggybacked on the implementation of regular soong_build and
wrote a fake build.ninja file to satisfy Ninja.
Now, instead, the JSON module graph is a a separate action in the Ninja
output file. This has the pleasant side effect that one can flip back
and forth between generating the JSON file and regular Soong without
loss of incrementality.
Side cleanup: write .d files in a slightly cleaner way.
Test: Presubmits.
Change-Id: Ia853383567b9dd31c53f3bdf56cfc8d517b498ec
These are just out/ and out/soong/ and the old names were quite
confusing.
Test: Presubmits.
Merged-In: I999ca62162062f27e449f83ffb00fa580d4065b8
Merged-In: I2fcd80a9e376a6842a06aa104a9f6f5c5e16f8c7
Change-Id: Ib481d2aac40df8da1b74174dc8ec576c1cb48be0
The following files are forked:
- build-globs.ninja
- The glob list files
- .d files for the glob list files
- The output file (build.ninja or the bp2build marker file)
This makes bp2build and soong_build not overwrite each other's files
they need for proper incrementality.
Test: Presubmits.
Change-Id: I69d192cbd36ecd9677f46f3fa095dfce6f872227
In particular, Bazel sometimes creates directories whose files cannot be deleted.
Test: m clean used to fail after b build //system/timezone/apex:com.android.tzdata, but now it works
Test: Added integration test which fails without this change
Change-Id: I08c8feed21c31ec187fe40be513f7eb4865c8ac3
This is done by moving SetupSignals() to its own little package.
There are a number of tiny little utility packages for soong_ui we might
be better of merging, but that's for another change (maybe)
Test: Presubmits.
Change-Id: I07b0ca98bfb8884ef4223d665e632183b9896a0d
This serves to not link parts of soong_ui (and eventually soong_build)
into a separate, weird binary. This is in turn good because they contain
any number of global variables and no one really thought about what
happens when two instances are executing at the same time in the same
address space.
This comes with a slight performance hit: 5 aosp_* projects build 152
seconds instead of 146. I suppose this is a price worth paying for a
clean design?
Test: presubmits.
Change-Id: I5623dcab2290f0fc392dd2ede597b9794a3d2a4e
Major operations:
* Fix the go_package entries to be consistent, as the new tool
complains when it doesn't have a '/'.
* Regenerate with the new protoc-gen-go tool.
* github.com/golang/protobuf -> google.golang.org/protobuf
* proto.[Un]MarshalText -> prototext.[Un]Marshal
Change-Id: Ie1147bd2457fafb66ba555461b3bf14f0561a25d
Allows the builder of an sdk snapshot to specify a Soong config
variable that can be used to control whether the snapshot's prebuilts
are used or not.
Bug: 193523070
Test: m nothing
Change-Id: Ib09500ba61befc1202dff61dc06847c730ba9253
Currently within metrics we cannot distinguish between Bazel-involved
configurations:
Bazel as Ninja vs Ninja as Ninja builds
Bazel mixed builds vs non-mixed builds
Add these to the build configuration in order to distinguish between and
track them independently.
Test: go soong tests
Test: USE_BAZEL_ANALYSIS=1 m nothing and verify out/soong_metrics
Bug: 193672397
Change-Id: I94ce56c1a98a76d929598d9fe3070df16ffa1376
Previously we ensured the kati marker exists in kati-enabled builds, but
did not ensure the kati marker didn't exist in non-kati-enabled builds,
resulting in soong thinking kati would run after it when it would not.
Also change the condition to only create the kati_enabled marker if we
don't skip kati or kati ninja.
Bug: 189187214
Test: m nothing && soong_ui --make-mode --soong-only && verify
kati_enabled marker is gone
Change-Id: If196b194b6aee38e9d4135889a4a4048ee5add82
The source tree will eventually be made ReadOnly, and recipes that write
directly to the source tree will fail. Use a pattern-match approach on
the results of stdout/stderr to provide hints to the user in such a
scenario.
If multiple patterns are found in raw output, print error hint
corresponding to first pattern match. first pattern match is chosen
since the failing function will be at the top of the stack, and hence
will be logged first
Test: Wrote a unit test to assert errorhint is added to output.
Wrote an integration test that writes to a file in the source tree
1. When source_tree is RO, the recipe fails and an error hint is printed
to stdout
2. When source tree is RW, the recipe succeeds and no error hint is
printed
Bug: 174726238
Change-Id: Id67b48f8094cdf8a571c239ae469d60464a1e89c
This CL adds generateJsonModuleGraph bazelBuildMode and ensures that it
returns Soong early, before Kati and Soong metrics collection begin
(which causes errors, since they're looking for files that Soong in json
dump mode did not write).
Test: TH
Test: rm -rf out && SOONG_DUMP_JSON_MODULE_GRAPH=/tmp/soong.json m nothing
Change-Id: I264eadb3b3b0cd6b6e7a65adc7b39bb1c01ca136
--skip-config can be combined with --soong-only to skip creating
soong.variables from Make, allowing a custom soong.varibles.
Test: manual
Change-Id: Iee9bfa4efeed801339e7c3c1e4807709485e701a
1. Initialize an empty glob file for .bootstrap/build.ninja. Initializing
in soong_ui (and not soong_build) prevents inadvertently creating the
file inside the source tree
2. Remove soong-build-globs.ninja, which is not used during the build
Bug: 187194795
Test: Ran the following command locally for the target
art-target-arm:git_master-art
```
. ./build/envsetup.sh && lunch armv8-eng &&
art/tools/buildbot-build.sh --target
```
Change-Id: Ibe6eeff65ea1ab25136642299e9878d0da1cac42
The previous --skip-kati flag could be interpreted as "do not run kati
to re-generate ninja file". Add a more specific flag for the "soong
only" build use-case, where we do not load the kati-generated ninja
files at all.
Bug: 189187214
Test: build/soong/soong_ui.bash \
--make-mode \
--soong-only
--skip-soong-tests \
TARGET_PRODUCT=mainline_sdk \
SOONG_ALLOW_MISSING_DEPENDENCIES=true \
SOONG_SDK_SNAPSHOT_VERSION=unversioned \
SOONG_SDK_SNAPSHOT_USE_SRCJAR=true \
out/soong/mainline-sdks/art-module-sdk.zip
Change-Id: I91abbd28af517d4b550ebc6d88fd64947caf9545
Merged-In: I91abbd28af517d4b550ebc6d88fd64947caf9545
(cherry picked from commit 546de4a1f3)
- Rename the "BuildX" variables to "RunX"
- Remove redundant comments
- Inline all the "what to do" based on config in build.go
- Inline some constants only used in one place
Bug: 189187214
Test: m nothing
Test: build/soong/build_test.bash
Change-Id: I111a69e642212d7938d4971283545e0d9acbb01a
Merged-In: I111a69e642212d7938d4971283545e0d9acbb01a
(cherry picked from commit d274ea9196)
The following two Make vars control RO/RW access to the source tree
1. BUILD_BROKEN_SRC_DIR_IS_WRITABLE
2. BUILD_BROKEN_SRC_DIR_RW_ALLOWLIST
By default, (1) will be truthy.
- this ensures that this CL is a non breaking change across all products
- different products can opt in to set is as "false"
Bug: 174726238
Test: from build/soong dir, ran go test ./ui/build
Change-Id: I4d55ac74f02b2a73194d31506a9010162620b25a
I6dca478f356f56a8aee1e457d71439272351390b replaced calling
build/blueprint/bootstrap.bash with running minibp directly,
but didn't propagate the EMPTY_NINJA_FILE environment variable
to minibp. Since everything that uses EMPTY_NINJA_FILE is
executed directly from soong_ui now, replace the EMPTY_NINJA_FILE
environment variable with a config bool and pass it to minibp.
Bug: 189148777
Test: build/soong/build_test.bash --dist --incremental --shard-count=300 --shard=1 && du out/multiproduct
Change-Id: I4d64275ce02c5d68948012f71ac4dc3795af9e85
This CL fixes a typo in writeFakeNinjaFile to correctly write
a fake out/soong/build.ninja and its depfile. It also modifies bootstrap
phase to *not* run the main soong build phase (which takes more than a
minute) if GENERATE_BAZEL_FILES=1.
This change has the side effect that `GENERATE_BAZEL_FILES=1 m nothing`
no longer generates the real build.ninja, which is fine because one
shouldn't be using GENERATE_BAZEL_FILES=1 for that anyway (or, use
USE_BAZEL_ANALYSIS=1). This change has no effect on mixed builds.
Time on a change to Soong or any Android.bp files:
Before: bp2build_workspace_marker (~20 seconds) + build.ninja (1 min)
After: bp2build_workspace_marker (~20 seconds)
Time on the second of two consecutive `b build //bionic/...`: 2.070s
Test: TH
Test: Soong integration tests
Change-Id: I43720641815994caba97b8d165d7c3fc254cbd06
SOONG_SDK_SNAPSHOT_VERSION=current will generate unversioned and
versioned prebuilts and a versioned snapshot module. This is the
default behavior. The zip file containing the generated snapshot will
be <sdk name>-current.zip.
SOONG_SDK_SNAPSHOT_VERSION=unversioned will generate unversioned
prebuilts only and the zip file containing the generated snapshot will
be <sdk name>.zip.
SOONG_SDK_SNAPSHOT_VERSION=<number> will generate versioned prebuilts
and a versioned snapshot module only. The zip file containing the
generated snapshot will be <sdk name>-<number>.zip.
Bug: 157884619
Test: m nothing
m SOONG_SDK_SNAPSHOT_VERSION=current art-module-sdk
- check that the generated Android.bp file has not changed
from the default.
m SOONG_SDK_SNAPSHOT_VERSION=none art-module-sdk
- check that the generated Android.bp file does not contain
versioned modules.
m SOONG_SDK_SNAPSHOT_VERSION=2 art-module-sdk
- check that the generated Android.bp file only contains
version 2 of each module.
Change-Id: I087e9d7d3ad110508a3d6a39bca50cbb46b3ce82
Installclean is used in some build server builds when switching between
different TARGET_BUILD_APPS, remove the symbols directory so it doesn't
accumulate old symbols files.
Bug: 186507256
Test: m && m installclean && m
Change-Id: I316703cd0985ee555959a6dcdb39d5a702ab0332
This refactors bazel-build mode determination logic in soong_ui so it's
clearer which of three possible modes are being used in a given
invocation (NO_BAZEL, GENERATE_BUILD_FILES, or MIXED_BUILDS).
Test: bootstrap tests
Change-Id: I41d2baebf8d560c2cc42db8daa8b936101d453e3
It now handles adding .bp files and changing globs.
In order to do this, depfiles are now written separately from RunBlueprint.
This is necessary due to the confluence of a number of seemingly
unrelated factors:
1. The glob filelist dependencies are discovered in globSingleton
2. Singletons need to be registered because otherwise singleton module
types panic
3. Singletons don't work because they require mutators bp2build does not
run
Due to (1), we would need to run the glob singleton. However, due to (2)
and (3), we can't run singletons and have to run Blueprint with
StopBeforeGeneratingBuildActions, which is when the build actions
writing glob files would be generated. So what happens is:
1. When bp2build is run, the glob singleton is disabled
2. At the end of bp2build, the list of glob files is artifically added
to the depfile of the workspace marker file
3. When build.ninja is generated, the Ninja file containing the glob
list file is written by the now-active glob singleton
Test: Presubmits.
Change-Id: I3c5898d8c57c554a93520276c64a952afc912dbe
This is done by setting the INTEGRATED_BP2BUILD environment variable
when invoking the build.
Even though the name of the marker file insinuates that a Bazel
workspace is already created, this is not the case yet.
An issue that remains is that a .d file is not written for the marker
file so it won't be rebuilt if a .bp file changes. Fixing this requires
delicate surgery because writing the .d file is the result of delicate
interplay between Soong and Blueprint.
There are also a number of semi-related fixes:
- The name of soong.environment.{used,available} is now on the command
line of soong_build (soong_docs is still special cased because its
command line in the Ninja file is taken from the os.Args of
soong_build so it's not trivial to remove the --{available,used}_env
from it
- bp2build writes a separate soong.environment.used file
- I had to call SetAllowMissingDependencies() separately when creating
the android.Context for bp2build so that bp2build runs in the
integration tests (it was not obvious how not to do this)
- Fixed a number of integration tests where a command with an expected
exit code of 1 was used as the last one in a test case, thereby
breaking the test suite
Test: Presubmits.
Change-Id: Ibeb61c26022cf801dcb98505b4039151b3409873
This is so that the way soong_build is invoked is simpler, which is in
turn useful so that it can be invoked multiple times within a single
build, which in turn will be used to invoke bp2build routinely (as
opposed to manually when needed)
Test: Presubmits.
Change-Id: Iddaebb05ff7bcedc0db8273192bb31284b739920
The output from bootstrap would contain the a one-line summary of RBE
download / upload stats.
Bug: b/183007125
Change-Id: I90082a8433504e40bbf92992c3c8d1d656c49429
This variable tells reproxy the number of times an action that
mismatches between local and remote-executions should be run remotely to
determine if the action is inherently non-deterministic.
An action is considered non-deterministic if the number of unique output
digests we get from the remote-run is > 1.
Bug: b/178842439
Change-Id: I0d50eed29b801be67a7ede9b25ce7f4535980f05
It's replaced with a flag in bootstrap.Args .
Test: "m nothing" (presubmits take a full workday these days)
Change-Id: Ia0bfa2091013e389890e583c559006077806af99
This requires linking Blueprint into soong_ui. It lets us avoid the
complicated dance of Ninja files and shell scripts: now the information
as to how soong_build is built is passed directly to Blueprint using a
struct that contains all the information the command line arguments used
to contain.
The ability to run Blueprint from the command line is kept (for now).
Some variables in bootstrap/command.go needed public accessor functions
because soong_build reads them. This will be disentangled by moving the
flag parsing to soong_build.
The presence of the flag definitions in Blueprint means that soong_ui
now also accepts them. This is not a problem in practice because they
are ignored and because soong_ui itself is hidden behind a few layers of
shell scripts.
Test: Presubmits + the new bootstrap_test.sh .
Change-Id: I6dca478f356f56a8aee1e457d71439272351390b
This lets us not leak $PATH to soong_build. It was only needed for
Darwin tests anyway.
Test: Will ask jingwen@ who has a Mac.
Change-Id: I4647e41275b323fe6283580f8f92718c6229f23e
- Incorporate BAZEL_* variables in soong.environment.available
- Set SOONG_DELVE when requested
Test: Manual.
Change-Id: I3ac675f4876220b788cc48d288b402ac6016c37b
This is exclusively used by tests so it's awful, but Darwin builds are
currently broken so let's unbreak them first then start thinking.
Test: "m nothing" on Linux (I don't have access to a mac OS machine)
Change-Id: I1ea9ca71a5aff36a3454282a4c87ad368138baaa
This reverts commit 6e49493dac.
FULL_BUILD is not set during the config stage, so this never calls
cleanOldInstalledFiles.
Bug: 168105598
Bug: 182008128
Test: manual
Change-Id: I58a54f167763cdb77b3c1bd4f18f519fcaf8751a
This sandboxes Soong's tests better.
Setting "TOP" is slightly ugly, but doing it properly would conflict
with aosp/1609218.
Test: Manual ("m nothing")
Change-Id: Ie3d3cd17f116a5fe0ad25c670fc5ed9c33f8123b
This works by setting the SOONG_UI_DELVE= environment variable to the
port on which soong_ui should accept a Delve connection on.
This is achieved by reusing the Delve execution logic between soong_ui
and soong_build.
Test: Manual.
Change-Id: Id2c1d4b6faac1a4a3918c91030ce2239f7daf54f
Rename to get consistent ramdisk directory naming in
out/target/product/<name>:
debug_ramdisk
ramdisk
vendor_debug_ramdisk
vendor_ramdisk
Test: build and inspect out/target/product/<name>
Change-Id: Ib86e6509bd910e8c95bcb6e3f7bb639ebbfe3758
This lets one avoid any decisions as to when to chdir there during its
execution and leads to better sandboxing because the pwd doesn't leak to
init() functions anymore.
Test: Manual.
Change-Id: I1560da8ed3a621249426f9e8908aa890c21e13ba
TestSendLog sometimes hangs on the buildbot, although it has never
been reproduced locally. The cause appears to be a unix domain
packet that was sent but never received. Skip the test in short
mode for now, which will leave it manually runnable but not run
it as part of the build.
Fixes: 120596545
Test: m nothing
Change-Id: I3e6bc9b5f1d9c15dc9a7294d98cbd917d0637c44
Also delete the now-unnecessary soong_env binary.
Test: Manually. Also checked what happens when a used environment
variable changes.
Change-Id: Ib393e7f444e94819198c6cce4bcd8719d9fd9441
bp2build is a Soong mode that returns as soon as the BUILD files are generated. This causes it not generate the build.ninja file, which the Ninja process executing soong_build itself expects to produce as an output. If there isn't an existing build.ninja file generated from a previous build, GENERATE_BAZEL_FILES=true m nothing will fail.
This CL generates the expected files as a workaround, and also makes GENERATE_BAZEL_FILES=true skip the Kati/Ninja steps in soong_ui since they aren't needed.
Test: rm -rf out/ && GENERATE_BAZEL_FILES=true m nothing && m libc
Fixes: 178683777
Change-Id: I2515ef7961682d2be5f096ed24831cc185165a67
This error message it currently being
logged to stdout which goes to build.log file . We want it to be logged to
build_error.log file instead so that downstream systems that need to run
based on the error message will continue to work.
Test: Tested by running `m toybox` and ensuring that FAILED: prefix is
at the beginning of the line.
Bug: b/177221705
Change-Id: Ie0c9acee2739acf149384e33804b5fca3273c24f
Therefore, no need to remove vendor-ramdisk-debug.cpio.lz4
for installclean as it will not be installed to $OUT dir.
Bug: None
Test: make vendorbootimage_debug
Change-Id: Ib7fb9931c55a9bb24e5ca8c4a80bd72f7233c57f
There are few build targets that don't generate an installed file list,
e.g. product-graph. Skip the old installed file cleanup step so that the
func doesn't complain about a lack of the list file.
Fixes: 168105598
Test: m product-graph
Change-Id: Ib7dce6b801979bb565b74d6355143bac23b84fe6
To measure the MaxRSS memory correctly, read the proc status file
under /proc/<pid>/status and extract the MaxRSS value from it. The
implementation is only available for Linux based distributions.
Bug: b/169453825
Test: go test
Change-Id: I32e3068fee7447f9ef5dfb5a8d8dcb6934e0af23
The metrics.go had missing comments and also did minor code refactoring
to be more in golang best practices.
Bug: b/172917718
Test: m nothing.
Change-Id: Id63e4af3f27b582fc13ef3f988ac5c8707fb9adf
No needed to define soong_metrics_proto since it is defined in
the Android.bp file.
Bug: b/172917718
Test: m nothing
Change-Id: Ib5f4cd6d06c08660803cf070da544e9c2d1dce1f
The setup_tools metrics has been missing since day one and is
needed to track down slowness of setting up the build system.
Bug: b/175706305
Test: m nothing and ran the printproto command to check that the
setup_tools field has been set.
Change-Id: Idc8a38fa830e0a906170feeede63435715eaa05b
The event.go code was refactored to follow the best practices
of golang code style.
Bug: b/172918681
Test: go test ./ui/metrics
Change-Id: Ic57d7cd576a9b146530ae37f03f4f5679d6cffef
Bazel is executed several times during the execution of soong_build.
For each bazel execution, generate a profile and save under the
BAZEL_METRICS_DIR which is defined in soong_ui.
Bug: b/174479924
Test: * USE_BAZEL_ANALYSIS=1 USE_BAZEL=1 m nothing and checked
if the cquery and graph build bazel profiles were generated.
* Verified that the generated bazel profiles were uploaded
to the local dev metrics pipeline.
Change-Id: I3d20204484dc6c5a1525a5d3eec1d62cfb33535b
Also get Bazel to write real files there (not symlinks) so that the DIST_DIR can be independent.
Test: Manually using e.g. DIST_DIR=/tmp/foo USE_BAZEL=1 m dist
Change-Id: I39d5219500864c9ecc85f356a028e9b5bf2607f4
The upload functionality now supports directories to be uploaded
for metrics purpose.
Bug: b/174479728
Test: Wrote unit test case.
Change-Id: I6906be4c1b7fd76ddf6ff7b94e48fe1c4209c59d
Code refactoring has been done for logs directory logic code since
the bazel metrics directory depends on the log directory. For builds
that did not specify a dist directory, the log directory is under
out directory. With dist, the logs directory is under <dist dir>/logs.
This matters for Android CI builds where the metrics files are
placed under logs directory. With this change, the bazel metrics
directory and corresponding files will be under <dist dir>/logs
directory for Android CI builds.
Bug: b/174479728
Test: * USE_BAZEL=1 m nothing (bazel_metrics dir in out dir)
* m nothing (bazel_metrics dir deleted)
* DIST_DIR=/tmp/build USE_BAZEL=1 m nothing dist (bazel_metrics
is in /tmp/build/logs directory)
Change-Id: Ic9e1ff49a1964fcaaf801bde2c19f33597ca1db4
The existing --skip-make flag disables both the config step and the kati
step in the build. Add support for a --skip-kati flag that skips just
the kati step, and refactor things so that the logic is shared between
these two.
Bug: 174315599
Test: TARGET_PRODUCT=aosp_arm64 soong_ui --make-mode --skip-kati;
(verify soong.variables is regenerated)
Change-Id: I75b1910fc1c12fcda130e37b7bc4c050131c7b33
On top of improving the comments, I also refactored the status checker
to simplify and clarify its use case to be a ninja stuckness checker,
since it doesn't appear to have other purposes.
Test: TH presubmit
Bug: b/173474588
Change-Id: I2cf51a1ebf16071a24a1c13c06c7b1adf60256de
This CL also removes --warn_real_to_phony and --warn_phony_looks_real,
since --werror_real_to_phony and --wrror_phony_looks_real are already
specified.
Test: treehugger
Bug: 173474588
Change-Id: Icbb0f53182d1b912c62f8044f52bb79d985bf868
This removes the need to source bazelenv.sh for USE_BAZEL_ANALYSIS, and
unifies mixed builds to use the checked in tools/bazel and bazelrc.
It also unifies all bazel-related output to be in out/bazel.
Without aosp/1502095, this change still requires toplevel_output_directories to be an empty
list, otherwise there'll be this error:
ERROR: Directories specified with toplevel_output_directories should be
ignored and can not be used as sources.
Test: With aosp/1441774: rm -rf out/ && lunch aosp_cf_x86_auto && USE_BAZEL_ANALYSIS=1 m libc && prebuilts/build-tools/linux-x86/bin/ninja -f out/combined-aosp_cf_x86_auto.ninja -t commands libc | grep bazel-out | wc -l # 2 results
Change-Id: I69b217ec88da531415792bb6e04b6a194ca4718d
This patch adds information behind the Bazel USE_BAZEL=1 execution
pipeline.
Test: m
Bug: 173474588
Change-Id: I54bdd26fb14c8a371de0922d193419112e8c3ece
Ensures that Bazel and Ninja are using the same binaries from PATH, such
as `sort` from the linked bug.
Test: USE_BAZEL=1 m
Test: rm out/soong/apex/depsinfo/new-allowed-deps.txt && USE_BAZEL=1 m
Bug: 161074854
Fixes: 161074854
Change-Id: Icc3e717e6d3cfcecfe90f0fb3a94a86485a11918
break and default statements are not needed inside of a switch
statement.
Bug: b/173028918
Test: USE_BAZEL m nothing
Change-Id: I14aa8cbae3a9b3c859f147c5134600f1e75c6a89
Added a tracer named "bazel" in runBazel function to capture topline
metrics when running bazel.
Bug: b/173028918
Test: * USE_BAZEL=1 m nothing and checked soong_metrics file using
printproto command to verify that bazel topline metrics
were captured.
Change-Id: I5cb0728a4da48545dc701cc79627c318d5d9a29d
Generate the bazel profile file that contains the metrics of
executing the ninja build phase of Soong.
Bug: b/173028918
Test: * m nothing and check out dir that there is no bazel_metrics dir.
* USE_BAZEL=1 m nothing and checked
out/bazel_metrics/build_bazel_profile.gz existed.
* m nothing and checked out/bazel_metrics does not exist.
Change-Id: I797d1eecf5c2c260ade8b55dc69c237bfd0fcbd4
$OUT/bazel_metrics directory is created in order to dump the
set of bazel profile files being created from a single build
execution. The directory is created from soong_ui.
Bug: b/173028918
Test: * m nothing and check out dir that there is no bazel_metrics dir.
* USE_BAZEL=1 and checked out/bazel_metrics was created.
* Ran m nothing and verified that out/bazel_metrics directory is
deleted to ensure no metrics is uploaded or collected from the
pipeline.
Change-Id: I1d3c2471065fae6931f35cef7f0f35fd84f76bef
The UseBazel() function will be used in several places to perform
different functionality such as setting up the metrics path.
Bug: b/173028918
Test: m nothing
Change-Id: Ie94073b8f0552ec7528e1cc2f3be74b693dc3135
This ensures that the lunch repo rule always reads the right Ninja file.
Test: USE_BAZEL=1 m
Test: m
Change-Id: I7f670dd538f46c4dae5ea537b857ff04520b60e7
This ensures that USE_BAZEL=1 mode has the same directory structure
under out/ as it does without USE_BAZEL=1.
Test: Manually verified building USE_BAZEL=1 aosp_flame with droid and
libc
Change-Id: I27e810772d1bf04bd7930ea122217b93d2fd48b4
Collect the resource information of each executed process in soong_ui
for metrics analysis.
Bug: b/169453825
Test: m nothing; "lunch 1" && m
Change-Id: I8c7fe019111921c4c59174d133b85439ec825885
This was caught while running the soong_build docs action under Bazel as
ninja executor. Since the config file is not an explicit input to the
action, it's not present in Bazel's execution root during action
execution time, leading to a missing file error. To fix it, add the file
as an input to the action, if it's configured in product variables.
New soong_build statement:
build out/soong/docs/soong_build.html: s.writedocs.soongDocs $
out/.module_paths/Android.bp.list out/soong/soong.variables $
out/soong/dexpreopt.config | out/soong/.bootstrap/bin/soong_build
outDir = out/soong/docs
Test: m
Test: USE_BAZEL=1 m
Bug: 161958189
Fixes: 161958189
Change-Id: I2bf264e848bb29ac91f4196fbe9c4ecf3cea8a2c
There is no need for multiproduct_kati to run the tests for
every product, they don't vary by product config. --skip-soong-tests
can also be used for local development to run soong_build even if
the tests don't pass.
Bug: 156428456
Test: m --skip-soong-tests nothing
Change-Id: I9c00e3d1b6e51d17bb290339c3f124d4d1c9e69f
We need to capture the user build command to understand what additional
arguments is passed in to Soong. This data is useful to know which
targets are being executed and have statistics on top build targets
are specified.
Bug: b/169453712
Test: ran "m blueprint_tools", "m nothing", "lunch 1" and checked
in soong_metrics proto.
Change-Id: I7ab94dcf037211f1abffe6afd012b03ffbf99756
Several structs and files are named time which records the total
execution of a traceable event. Soon, resource information of an
executed process is collected during a build event so a more generic
name is applied.
Bug: b/169453825
Test: go test and m nothing. Checked the soong_metrics protobuf.
Change-Id: I65a782603530d51018d6cc1192c18cfeef7566a0
We would like to collect the resource usage of processes such as
kati, soong_build and ninja on the resource usage such as memory
and context switching. This changelist contains the defined fields
in the protobuf file of the resource information to be collected
from interested processes.
Bug: b/169453825
Test: N/A
Change-Id: I4b3cbbc3e7b78900875c10d34d574425f8b11730
Test:
m USE_GOMA=true GOMA_DIR=$(goma_ctl goma_dir) toybox
m FORCE_USE_GOMA=true GOMA_DIR=$(goma_ctl goma_dir) toybox
both result in error message being printed.
Change-Id: I94ccaf217dba71aca990d88d205bad669a49a2f5
This CL adds symlink_outputs to various locations in Soong that creates
actions that creates symlink outputs, and explicitly mark them as such.
Test: m
Bug: 160568334
Change-Id: I322751bada52a9f49011c74731d84761586e03e7
Propagate the subprocess stats from ninja to build.trace.gz.
There is too much data here to put into separate counters for each
statistic, so put them into args on each duration event for now.
We would like to track max RSS for each subprocess, but the Linux
kernel inherits the max RSS of the ninja process in each subprocess,
which sets a lower bound on the measurable max RSS to the size of
the ninja process. The ninja process is large due to the multi-GB
build.ninja files.
Bug: 170701554
Test: examine build.trace.gz
Change-Id: I8aaaafe627a57f1a500af098c097c6381c583ba5
Clean the odm_dlkm and vendor_dlkm staging directories in installclean
to get consistent results in incremental builds on the build servers.
Test: treehugger
Change-Id: Id431d7457d85e0be68437a2034fd544d74cc3709
Collect the total number of CPU cores and available RAM for each build
as part of the system resource info. This is to know what kind of
build machines that the developers are using.
Bug: b/169453825
Test: m nothing and ran printproto on soong_metrics to validate the
data.
Change-Id: I8617bdb9490b37d7689abd27d349af1a651533db
This retriggers soong_build whenever a new bzl, WORKSPACE, or
BUILD.bazel file is changed or added.
Test: Manually verified on bionic/libc genrules with manual changes to
related BUILD/bzl/WORKSPACE files -- these all retrigger builds.
Test: Updated finder_test.go
Change-Id: I634384f88781a6b6db32f5d6bf9c07e179e14c39
Added additional instruction in the regen.sh script to build the
blueprint_tools and aprotoc in case the regen.sh script fails.
Bug: b/169395325
Test: ./regen.sh
Change-Id: Icb38c907fb47febc17eef1067d06d0588dd0b7ba
The hostname is collected during a build to determine where the user
is build from (from office workstation, cloudtop, etc...). This is a
useful metrics for RBE related builds.
Bug: b/169395325
Test: m nothing, ran printproto to verify that soong_metrics has the
hostname.
Change-Id: I3e5d64f948ae43c5dda60de1d4a52b187e36dff8
Tradefed packages are created in out/host/common/obj/PACKAGING
and should be removed during installclean so that they get an
accurate embedded build number.
Fixes: 170308394
Test: treehugger
Change-Id: Ib7a57377b5c9512584bc93f10da316eadf786979
This reverts commit 323dc60712.
Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests
Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
Prints the following warning messsage:
WARNING:
Setting OUT_DIR to a path other than out may result in slow RBE builds.
See http://go/android_rbe_out_dir for a workaround.
Bug: b/169676232
Change-Id: Id1ad06c4dc672c24373642e0b624833eb6a0dbcf
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.
Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
FORCE_USE_GOMA is needed to be collected as a metric for the
deprecation of GOMA on Android builds.
Bug: b/169423400
Test: m nothing, ran printproto command
Change-Id: I12300389f1f27239d8f21875d1b6e4ad069d95d7
We should override log-dir with the environment variables if its set.
This is needed in our integration tests and generally to allow for
user-overrides.
Test: RBE_log_dir=`pwd`/rbelogs mmma external/boringssl
Change-Id: I3d316a2d60122c42f5c3b5dfede4037362a7b5f1
This is an initial CL that collects the environment variables
USE_GOMA and USE_RBE. This is a needed metric to determine how
many users are currently using GOMA so we can help them migrate
to RBE.
Bug: b/163598127
Test: m nothing and checked soong_metrics
Change-Id: I39536eabad01aafbb3bd7182515d72230b36f658
This hasn't worked for a couple years, and continues to bitrot. Just
remove it.
Adds a bpfix rule so that we can eventually remove the
product_variables.pdk definition, which is now always a no-op.
Test: treehugger
Change-Id: I830b54d419b59f6db1d4617b45e61a78234f57a7
Merged-In: I830b54d419b59f6db1d4617b45e61a78234f57a7
There are cases where the developer is not building a product so
the metadata information is not being set. For now, set the OS
name so we can track which OS the user is using it.
Fixes: b/163118212
Test: m nothing and checked the blueprint
Change-Id: I9b521eee2d266b50625aa8e916ddaacb4e2feaea
"exec format error" showed up on Mac when running the rbe_test.go.
The generated bash scripts did not have a newline at the end of
the file.
Bug: b/162600461
Fixes: b/162600461
Test: "m blueprint_tools" on Android Mac builder.
Change-Id: I425fa5f428733b6569c3fcdaa312023f9761d020
Disable the error test cases to allow developers to build
AOSP on Mac.
Bug: b/162600461
Test: m nothing
Change-Id: Ic8c6fe04743a8e4fe197a1bb71259036fb749edc
Disable the error test cases to allow developers to build
AOSP on Mac.
Bug: b/162600461
Test: m nothing
Change-Id: Iac4373dcb10710d8a2325eeb3ee20a6eeca6143f
Pass -n to blueprint to make it use validation dependencies for tests,
which will let them run in parallel with soong_build.
Bug: 156428456
Test: m nothing
Change-Id: I706c14330543e2a7728e5dc385976900c67cb2f5
Currently, the build timestamp saved to the build_date_timestamp
field in the MetricsBase metrics protobuf usually comes from a text
file named build_date.txt. It may not be accurate for metrics purposes.
Instead, use the build start time from Soong main UI for a better
timestamp.
Bug: b/140638454
Test: Ran and checked the build_date_timestamp by running the printproto
command.
Change-Id: I4d893fbbf9830ba21911e56ae13bc5272be47ae6
From several dashboards, there are high spikes from several
builds where the real time of executing a part of the build
system took many many hours. This is due to using the wall
time clock when calculating the duration. For metrics, the
monotonic clock should be used for more accurate measurement.
Also, cleaned up the time.go file to use the proper proto
functions when marshalling perf protobuf and deleted
unnecessary functions.
Bug: b/143492984
Test: Wrote and ran unit test case.
Change-Id: I1540cfbf37915e3045ecedaa2595c1f9429bcd62
From aosp/1329396, the RBE metrics protobuf file is part of the
metrics uploading process. The RBE metrics protobuf file is
generated by running the bootstrap shutdown command. A new function
named DumpRBEMetrics was written in order to generate the RBE metrics
protobuf file before sending to the uploading process.
Bug: b/140638454
Test: * Unit test cases
* Ran RBE build on my local host and verified the
metrics protobuf file is created.
* Ran non-RBE build after RBE build and verified that
the previous metrics protobuf file was deleted.
Change-Id: I4b8068905cb67c4b8c2d94793917b98974fed707
Help debugging finder issues by disting module_paths/files.db for
later analysis.
Bug: 157656545
Test: treehugger
Change-Id: Idfa4cffe7ef724b399a2667efd487f9c3c76bb6b
Specifically, make Android.bp.list and soong.variables explicit inputs
While this is not a comprehensive list of all inputs
of this action (as the action depends on all blueprint
files in the source tree), this is closer to the truth.
This is a rollforward CL, which was originally rolled
back, as path validation failed for when OUT_DIR was
an absolute path. Validation has now been relaxed.
Test: Manually verified ninja output, checkbuild approved validation
for aosp-crosshatch, and manually ran frameworks/rs/build_rs.py with
patch fix
Change-Id: I4eb0d517f57336dd54eaa4bd31f46df9e93e6da2
Debugging issues on the build servers can be difficult because the
intermediate files are not visible. Gzip ninja file and Makefiles
generated by Soong and the ninja files generated by Kati to the dist
directory, and also copy all of the finder output files.
Bug: 157656545
Test: m dist nothing
Change-Id: I48d75305e551ccae81c7a55721981cf58acd838b
While this is not a comprehensive list of all inputs of this action (as the action
depends on all blueprint files in the source tree), this closer to the truth.
Test: Manually verified ninja output, checkbuild approved validation for aosp-crosshatch
Change-Id: I77a0f612afb025232bee7d3ac31257b808991829
The metrics uploader was currently running on foreground where it
would copy the metrics files in a separate directory and then forked
into the background for the upload process. As a result, running the
lunch command would take a second longer to run since each metrics
uploader run had an average of half a second.
Bug: 140638454
Test: * Wrote and updated unit test cases.
* Set ANDROID_ENABLE_METRICS_UPLOAD to point to the latest
metrics_uploader bash script. Executed the "lunch 1" command
and measured the running time. Executed "m nothing" command
and checked that the metrics were uploaded.
* Ran "lunch 1" and "m nothing" with
ANDROID_ENABLE_METRICS_UPLOAD=""
* Removed oauth from metrics_uploader and ran "m nothing" and
"lunch 1". The oauth Message appeared only to "m nothing"
Change-Id: I13c61e666c8f44613dee291a704cef6a27335188
External applications such as metrics uploader depends on the TMPDIR.
DumpMakeVars is changing the TMPDIR environment variable value. Once
the work is done, TMPDIR directory is deleted. Additional flow of
the tool or external applications can panic since TMPDIR directory
does not exist. Since this is an isolated case, pass the TMPDIR
directory to the command of dumping the makefile variables.
Bug: b/140638454
Test: * Ran lunch command with ANDROID_ENABLE_METRICS_UPLOAD set
and metrics_uploader did not crash.
* m nothing
Change-Id: Ib78a3f8cfb5da48f65bd9bce1511e9abdf3d9cca
Soong now supports the ability to upload metrics to another location
by setting the ANDROID_ENABLE_METRICS_UPLOAD to an uploader that
accepts the upload.proto proto buffer message. When the environment
variable is set, a set of build metrics files (soong_metrics,
rbe_metrics.pb and build_error) is uploaded.
Bug: 140638454
Test: * Wrote unit test cases
* Setup the uploader, built a succcessful and failed aosp_arm-eng
target and monitor the uploading of the metrics.
Change-Id: I76a65739c557dc90345e098ca03119a950ece2d2
Drop down to a single high-mem task when we've got <=16GB, as the system
probably isn't completely free RAM, ninja will be using a couple gigs,
along with whatever "normal" actions will be running concurrently. So
it's unlikely that we can handle two 6-8GB actions along with everything
else.
Also print warnings when we detect <=16GB total RAM, or when we're
running more parallel jobs than than we have GB RAM. These both notify
the user and suggest lowering the `-j` value if they run into problems.
Test: fake totalRAM to [0.5,8,16]GB, checking warning
Test: fake totalRAM to 17GB, `m -j4 nothing` has no warning
Test: `m -j187 nothing` on a 188GB system
Test: `m -j188 nothing` on a 188GB system
Change-Id: Ieb008e9f462d5f40fb65781d94cf116b1caf8446
The stats output will now use the new "DEBUG" message type, which we can
always redirect to verbose.log.gz.
Test: m aprotoc (look in verbose.log.gz)
Change-Id: Ie1b58f12c008ff7d29f11ff7a9807488dba8a504
misc_info.txt's build rule does not contain inputs to tell it to
refresh on an incremental build, so incremental builds (even after
installclean) were reusing this file.
With this change, misc_info.txt is removed by installclean so that an
incremental build generates the file fresh.
Test: # misc_info.txt exists; m installclean; # misc_info.txt gone
Bug: 155930200
Change-Id: Ia319925dad26f98049a988ee1ceb1e20db621353
This was a temporary migration tool, and has been broken ever since the
ninja environment started being restricted. Remove it, since it has
lasted two releases.
Test: TEMPORARY_DISABLE_PATH_RESTRICTIONS=true m
Change-Id: I2e3709525598030f5c24d12fa966e531dfed33ae
A user was trying to use OUT_DIR=/tmp/...-out DIST_DIR=/tmp/...-dist
Test: OUT_DIR=/tmp/test-out DIST_DIR=/tmp/test-dist m nothing
Change-Id: I811cc8e1bda8d766406603df47309a8637cef48c
By the time we do argument parsing, we've already written to $OUT_DIR or
the default, so don't allow overriding the output directory on the
command line.
Test: m OUT_DIR=newdir
Change-Id: I6f8c07cbef3a71d393b766dc5a01ae9c3df8631c
Includes the exit code, real time, user time, system time, and maxrss.
Test: m nothing; check out/soong.log
Change-Id: I93dea9fc1c7fb892150e16e11cae09f2372f4429
This replaces the _kati_always_build_ hack with a ninja feature so that
ninja can actually understand what's happening. This means that we can
turn on some more options and checks around expected output filenames:
* Remove the output file(s) before the command executes
* Error if the output file(s) don't exist after the execution
* Error if the output is a directory
They're turned on immediately during the soong bootstrap runs, as those
run a limited number of rules. The main ninja execution does not remove
the output files, and prints warnings instead of errors for the others.
I'll turn them on as we understand how often those warnings are seen.
Test: m (check for new warnings)
Test: treehugger (check for new warnings)
Change-Id: I7f5c1a361dd1498eb54a2c07a918f3b0aa086e4c
This CL adds a remoteexec package that allows adding a configurable RBE
prefix to the template.
Test: built aosp crosshatch userdebug with and without RBE_CXX_LINKS.
Change-Id: Ica920c3d7f79f2996210b9cbd448126451c1707c
A new protobuf message named Upload is defined to send build metrics
to the Android Build Server API after a build is completed. These
metrics are collected from googlers only, as described in
go/local-developers-metrics-wfh.
Bug: b/140638454
Test: m nothing
Change-Id: I5e49b5ace053913be2b75673cb1375f4cef37825
The dist dir is created late in the build process. If a
soong sandbox attempts to mount it before it's created
the sandbox will create an empty file in its place.
Test: lunch aosp_cf_x86_phone-userdebug && make -j dist
Change-Id: Ie6513bf702de6e6322c78753d406d70ea3ccc04d
It's safer to disallow writing to the root filesystem of the sandbox.
Still allow write access to the source, tmp, out and dist directories.
Test: build aosp_cf_x86_64_phone-userdebug
Change-Id: Idf0d3a420669fbf6c3aba24e058f09a314ec6d0e
buld_error protobuf file is created the moment a build action failed.
This is for external system to detect immediately what caused the
build failure while monitoring the build_progress.pb file.
Bug: b/150401146
Test: * m nothing
* Built successfully aosp_arm-eng and checked build_error file
was not created.
* Modified a cpp file to cause a build failed. Ran m and build
failed. Checked immediately during the build that the
build_error file was created.
* m clean; m and during build, checked if build_error was
removed.
Change-Id: I778616ae8cf242b49ad263b79c93321959112caa
For successful builds, a blank build_error proto file is created.
This slows down the data processing part where it reads the build_error
file and there are no errors to be processed.
Bug: 142277430
Test: Did a successful build and checked if the build_error file
was not available. Changed a source file to make the build
failed and checked if the build_error file was available.
Change-Id: I300dffe32fb7f8bf984fa20ae368bdd02bf12992
Soong UI generates a proto file named build_progress.pb in $(OUT_DIR)
output directory that contains build action numbers (how many are executing,
finished and total) during the course of a build. This is for external
systems that invokes the Platform Build Systems and would like to know
the completion status.
Bug: b/150401146
Test: Wrote a bash script that continuously read the
build_progress.pb file and computed the build completed percentage
while building the aosp_arm-eng target. Compared the percentage between
the Soong output console and the one reported by the bash script.
Change-Id: I7c7347bc8e41958093892d8e2731c4f4169937dd
This makes sure that we don't keep old vendor_boot artifacts around
after a clean.
Signed-off-by: Will McVicker <willmcvicker@google.com>
Fixes: 151957122
Bug: 151949371
Test: touch $OUT/vendor-ramdisk/test-file && make installclean
Change-Id: I357929583f234212a998dceaa31506fce42e0255
Soong UI will generate a new proto file in the out directory for
other systems to read the build completion status during a course
of a build. The proto file contains build action counters where
total_actions: represents the total build actions of a build (will
increase and may decrease during a course of the build)
finished_actions: represents the number of executed build actions.
finished_actions never decreases and finished_actions <= total_actions.
current_actions: represents the number of build actions being executed
and current_actions + finished_actions <= total_actions.
Bug: 150401146
Test: executed regen.sh script.
Change-Id: I04f4c07855723d0684b141d88cb3529ab5d9fccd
If this couldn't read the current user, it would print a generic
message, then proceed to dereference the invalid user struct :(
Provide somewhat reasonable defaults, and better error messages instead.
Test: m nothing
Test: run docker with misconfigured user
Change-Id: I21af77c7d8d1d2d27cb04546667eb1094c62a7a1
We have known problems (docker; distros w/o user namespaces) which mean that we won't be turning this into a fatal error anytime soon. Also remove the bug report link, it's not useful to continue getting the same reports.
Change-Id: I271871d68150417ac938f074d3730cad4518e327
Test: treehugger
Collect the number of modules and variants and some basic statistics
on memory usage inside soong_build by writing out a proto that is
read back in by soong_ui.
Test: examine soong.log
Change-Id: I6926876377a4f6229cf41fdbf166ae03c885ea55
Trying to override the default directory for ccache by using the
CCACHE_DIR environment variable fails unless it is in the allowed
list.
Bug: 149670916
Test: manual
Change-Id: I8e7eea7a5c25d7ea5f0956fafc70d62522f3c4fc
Apparently PackageManager gets confused when the apk disappears, but the
directory still exists.
Test: lunch aosp_arm-eng; m BasicDreams
<remove BasicDreams from PRODUCT_PACKAGES>
m nothing
ls out/target/product/generic/system/app
Change-Id: I486c0ddadde55fad226049d3fb0f09af88117da0
These variables control whether we accept cached-results / not and
whether we should update the cache with locally executed results or not.
I need accpet-cached whitelisted to purge the the RBE cache for the
invalid cache entry we have set for the failure we say yesterday.
Bug: b/148387048
Change-Id: I7344fc083f82e0b7bc11084376a267d19cf30bb8
This CL adds RBE support to javac, r8, and d8 rules which is only
enabled if respective environment variables are set.
Test: an aosp_crosshatch build with and without the new variables.
Change-Id: Ic82f3627944f6a5ee7b9f3228170c2709b1bfcb8
Create a highmem pool based on the total RAM and the number of CPUs,
with an override via the NINJA_HIGHMEM_NUM_JOBS variable. Put
metalava into the highmem pool.
Ninja does not support nested pools, and when goma or RBE is enabled
the maximum ninja parallelism is set very high with local jobs in a
local pool. When both the local pool and highmem pool are enabled,
the total number of local jobs will be as high as the sum of the sizes
of the two pools. Keep the highmem pool limited to 1/16th of the
local pool when remote builds are enabled to try to minimize the
effect while still limiting highmem jobs.
Fixes: 142644983
Test: m nothing, examine pools
Test: m USE_GOMA=true nothing, examine pools
Change-Id: Id79f11f44948992960ac34ecf831dacbe21bd332
Determining the total build time from the existing metrics is
tricky because some of the metrics are overlapping. For example,
the "soong" metric includes "blueprint bootstrap",
"environment check", "minibp", "bpglob", "minibootstrap", and
"bootstrap". Create a new singleton "total" metric that
covers the interesting parts of the build.
Test: m nothing
Change-Id: Iedcf5c9cf0d27252b694d35e6ff66ca1fcf734ac
This relands I12a0f907753fefd1997ab8b4ea2ac331234093cf along with
a fix to blueprint for absolute paths.
Store the current working directory and then change to the root
directory so that all file accesses must go through helpers in
the android package that properly track dependencies.
Change-Id: I24ac485677aa102eec1a2521d16820da6ee1ae77
Fixes: 146437378
Test: m checkbuild
Test: m OUT_DIR=/tmp/out nothing
Store the current working directory and then change to the root
directory so that all file accesses must go through helpers in
the android package that properly track dependencies.
Fixes: 146437378
Test: m checkbuild
Change-Id: I12a0f907753fefd1997ab8b4ea2ac331234093cf
This is used by the art apex test.
Bug: 147197813
Test: m EMMA_INSTRUMENT=true EMMA_INSTRUMENT_FRAMEWORK=true art-check-debug-apex-gen
Change-Id: Id185cd35f16131f2c9a8afeba2e5b87834b0e620
We're passing in ASAN_OPTIONS=detect_leaks=0 on the command line, and
the build started failing after we limited the environment.
Test: forrest of aosp-master/aosp_x86_64-eng
Change-Id: I42c91897c7677e1a249412e5a8bc7bb1edb2f881
Make now exports (partial) lists of what should be installed. Those
lists are incomplete, but if something is removed from the list, we can
remove it from the filesystem so that incremental builds work better.
Test: built-in tests (m blueprint_tools)
Test: remove a module from PRODUCT_PACKAGES, see the print and file removed
Test: change the name of a cts test, see the old one removed from cts
Change-Id: I6bc14c6a5af63383ea265e97ed8b875fa51af0ed
See the change that added ALLOW_NINJA_ENV for more information.
Test: m nothing; check out/soong.log for small list
Test: ALLOW_NINJA_ENV=true m nothing; check out/soong.log
Change-Id: I7761c6a07a7f8b0acee107e9c27c7739dd4b63ab
Ninja does not track changes in environment variables, so we get
potentially incorrect builds when environment variables change during
incremental builds if some action was using one of them.
Add a variable to limit exposure of these variables to ninja, and thus,
all actions run by ninja. Kati and Soong can still read environment
variables, they explicitly track which ones they read so that we can
re-run them appropriately.
This list is just the beginning, there's no good way to detect which
environment variables are currently being used and to pass them through.
So this initial change won't have a behavioral change, and we'll flip
the switch and see what fails or who complains, flipping it off and on
and adding to the list until we can make this always happen.
Also adds a board-specific `BUILD_BROKEN_NINJA_USES_ENV_VARS := ...`
list so that we can temporarily allow board-specific variables until
they're fixed.
Test: check out/soong.log
Test: ALLOW_NINJA_ENV=false m nothing; check out/soong.log
Test: set BUILD_BROKEN_NINJA_USES_ENV_VARS := OLDPWD
ALLOW_NINJA_ENV=false m nothing; check out/soong.log
Change-Id: I08e4834ce12100a577ef7d6a9a21b9e9d345cb93
Colin originally wrote this for the highmem pool:
https://android-review.googlesource.com/c/platform/build/soong/+/1168271
But while that's a larger change, this is useful to just be in the logs,
and for use in multiproduct_kati to better limit the number of
concurrent jobs.
Test: check soong.log on linux and mac
Change-Id: I0518d303a220d775f8d78dba9f660b2954e68e3e
(Although I was the one who added the gzcat reference, I don't believe
we actually use it. And neither my Debian host nor toybox has gzcat --
as opposed to zcat -- anyway. So just quietly drop that.)
Test: treehugger
Change-Id: I95d94dab1daf72faedfa170770e8ebfaa37b70c6
When LOCAL_COPY_HEADER lines are removed, we just removed the copy rule,
we didn't actually remove the old header, so C/C++ files could still
include the old headers during incremental builds.
While we do consider LOCAL_COPY_HEADERS as near-obsolet, and it'll
disappear as we move everything over to Soong, this could produce some
unfortunate incremental build results while people are attempting to
remove them.
Fix this by ensuring that only the files currently listed in
LOCAL_COPY_HEADERS exist in TARGET_OUT_HEADERS after we run Kati.
Test: Remove a LOCAL_COPY_HEADERS entry; see the header removed
Change-Id: I817305703a6996d50490d552623d7df019b608c9
