This also shaves off a couple seconds from the build.
Test: `time m nothing` 2 times with and without the change
Bug: 243699430
Change-Id: I5cc5cba2b98c845ddd6ba7c3b31ad678451afe41
Per aosp/2183398, out/soong/soong.variables provide some useful information for debugging purpose.
This change enables copying out/soong/soong.variables to out/dist/soong_ui/soong/soong.variables in dist mode, similar to build.ninja.gz in the same directory.
Test confirmed that the content of the out/soong/soong.variables is
equal to that of out/dist/soong_ui/soong/soong.variables.
Test: m dist $target &&
diff out/soong/soong.variables out/dist/soong_ui/soong/soong.variables
Change-Id: I1bfa53a8e116c49fda2cc9a7252f4da0c59ce5b5
Currently, running lunch writes multiple lines including less
informative ones.
This change removes unnecessary variables and enables the user to focus
on more useful information when running the lunch command.
Bug: 204196816
Test: m
Change-Id: Ifbbe01abf03a1701158881febe5982489f96dc65
Make sure the staging directory doesn't have stale file after
installclean.
Test: m installclean && ls $ANDROID_PRODUCT_OUT
Change-Id: Idcd7a883a6cba4b7eb1eb99751f5d03cba365a12
Export it so that we have one source of truth,
which can be used by the called tools.
Test: Use the exported value in ART build.
Change-Id: Iaa45b1780ffae11e09790e193ca2bd01e2b732a8
It is useless to run this binary when gcert creds aren't available, so
adding a check for gcert creds before calling this binary. Also modified
metrics to capture missing gcert.
Bug: b/237938234
Change-Id: I94b5aa2d53c4e5bb1a5fdc913a33ef5bae965d32
This is to generally warn Google devs about lack of LOAS creds.
Given the switch to RBE, lack of LOAS creds will start resulting in build
failures, so adding a warning message ahead of time to all builds. Will
convert this to a build failure after a week.
We need LOAS creds to be able to even fetch the CDPush config file to
determine whether to use RBE in a build a not, which is why I'm making
this a broad error message across builds.
Test: ran "m nothing" build with / without gcert and with / without
"stubby" in $PATH and confirmed appropriate error message.
Bug: b/235985591
Change-Id: I15cbaf372b0a7b79f868a06c7d5ede19b49ae687
This log is not super useful to be honest, so avoid printing it to
stdout of the build.
Bug: b/236709761
Change-Id: Ic96130a285160b386df3afc269dd5ef97c7965bf
RBE logs directory is currently messy:
1. We use RBE_output_dir variable to specify where rbe_metrics.txt /
rbe_metrics.pb file should go to.
2. We use proxy_log_dir to specify where *.rpl / *.rpi (detailed per
action info log file) should go to.
3. We use RBE_log_dir to specify where reproxy.* / bootstrap.* log files
should go to.
Ideally, all RBE related logs should go to one single directory. In this
CL, I'm creating a temporary log directory under out/soong/.temp/rbe/
where all RBE related log files per build would go to. The log dir
prefix is also being set to the same prefix as the socket address file.
Test: Ran a sample build with `m libc` and ensured that logs are getting
cleared across rebuilds and that `rbe_metrics.pb` file is properly
generated and being copied to the right location (i.e., from
out/soong/.temp/rbe/<rand> to out/ dir)
Bug: b/233382420
Change-Id: I46bd38d50419cb9e54e8202d24222979e47ff5ca
This reverts commit b018b83664.
Reason for revert: Builds will start failing now since the gcert change is about to go in.
Change-Id: I81d36ce6bf4f7a9bc120b34d411f2cb4ec64e5eb
This required the following:
- Adding Platform_base_sdk_extension_version to default soong.variables
- Teaching the symlink tree creation code to understand symlinks
- Making finder.go follow symlinks when requested
Adding yet another knob is unfortunate, but I can't allow that
unconditionally because the Android code base contains a number of
symlinks giving rise to infinite directory trees because they point back
to their parent and this seemed preferable to adding complicated logic
like "follow symlink but if only its fully resolved version does not
point under the source tree".
I could be convinced about the latter, though.
Test: Presubmits.
Change-Id: I453f6b7e5334771f5832c700db00f9d24ed1d82f
Now that we are ready to rollout gcert, adding a warning that builds
will fail if gcert doesn't exist. Doing this only for builds with
ANDROID_BUILD_ENVIRONMENT_CONFIG=googler.
Test: Ran a build without gcert and USE_RBE=true and it printed the
warning.
Bug: b/229743321
Change-Id: I616a4a534ab277873ac7e19baecb78620f07854c
These can take a minute or more of build time, and currently hold up
other processing. We could move to a parallel compression scheme to
speed them up, but it's simpler just to background them and continue the
build, waiting to make sure they've finished before exiting.
Testing this on git_master/flame-userdebug:
This brings a local no-op `m nothing dist` build from 2m1s to 1m39s,
even though most of that time is still spent waiting on these (since
there is very little else happening in a `nothing` build when Soong and
Kati don't need to run).
Running `touch art/Android.mk; m nothing dist` (so the above, but with
Kati) goes from 3m39s to 2m2s. We spent 9 seconds at the end waiting for
these to finish.
CI cases that almost always run Kati plus some number of other actions
are likely to completely hide the time taken to dist these files.
Bug: 229932999
Test: Check $DIST_DIR, files still exist
Change-Id: I1fb78d7c4d5103d72b5d71d9277dea0d452f8968
If we didn't need to run soong_build during the current run, we still
try to load the soong metrics. But in the case of `dist`, that's in a
directory that is not guaranteed to persist between runs.
Make loading the soong metrics optional if the file does not exist. Also
fixes a variable shadowing issue that meant we never passed it into
ctx.Metrics.
Test: treehugger
Change-Id: Ic836282f4d13e91daa0e7241ad7c488de3293d8b
This allows us to track how much time is spent in Clang.
Test: TOOLCHAIN_RUSAGE_OUTPUT=/tmp/rusage.txt m
Change-Id: Ib2961904f363bc59bd9d928bb055a96740cb9f17
Tracks the time to call the expconfigfetcher and whether it updated the
config file or not.
Test:
Ran a build with a "regular" user and verified the metrics indicated
NO_CONFIG and no config file was created.
Ran a build with an "experiment" user and verified the metrics indicated
CONFIG and a googler.json config file was created.
Bug: http://b/227186589
Change-Id: I030e9497c369459ae98c0c4f7765aef6de4c601f
Currently, mk2rbc/rbcrun aren't built in certain phases
of the build, which causes the Starlark product configuration
to fail when using get_build_var if a regular build hasn't
been run before.
Bug: 226677850
Test: Manually
Change-Id: If85608eef63be25fddc4d916c82247c13eb0195f
Metrics are added to a materialized protobuf soong_build_metrics.pb
Recorded events have scoped names and recorded runtimes.
For example, mixedbuilds includes the following events:
mixedbuilds.prepare
mixedbuilds.analyze.write_files
mixedbuild (this is a total of all mixedbuild events)
Test: Manually verified materialized protos for mixedbuild, bp2build,
and legacy build.
Change-Id: I2e72d4502c0373bee0279bb1320b2e4dd586129c
Dumping of RBE metrics should only happen when reproxy was started.
Reproxy is started only when Soong is run in make / build mode and not
in dumpvars mode.
Doc: https://docs.google.com/document/d/1Iqhu5QgozB_rA9fucMHO6X4U6A8e8qFtZyub9j83ICQ/edit#
Test: Ran a build with / without RBE. No metrics dumped during lunch
without RBE and metrics dumping worked as expected with RBE.
Bug: b/197213143
Change-Id: Ic3403331e1f4ac2c7c72fd9169aa7b82a7545c75
This is to ensure that we don't default to using
application_default_creds when it is not set and rather
"RBE_use_google_prod_creds" variable is set. This is necessary to
rollout RBE gcert auth.
Bug: b/224446621
Change-Id: I51e3f1aefadf282240b8f861efafde6b9b61d305
Use EXPERIMENTAL_USE_OPENJDK17_TOOLCHAIN=true to build with OpenJDK 17
while still targeting java language version 11.
Bug: 219098645
Test: m EXPERIMENTAL_USE_OPENJDK17_TOOLCHAIN=true
Change-Id: I89fb04ecde4dbd1e102098f316692624dd67f8a2
Combined with roboleaf CI changes to pass this flag and subsequent
pipeline changes, this will allow basic mkfile metrics to be
dashboarded, which will help track mk2bp progress.
Bug: 217764943
Test: manual; verified mk_metrics.pb is generated iff --mk-metrics is
specified, used printproto on result
Change-Id: I348994b4d7e3479d0dc04de0276b94a702aac95b
To support using google prod credentials for Google developers, we need
to be able to make an RPC call to exchange credentials. This RPC call
needs to be issued via stubby and hence allowlisting the stubby binary.
dwillemsen@ also pointed out that this should be less effective for
other tools running within the build since network access is disabled
for the other tools anyway via sandboxing.
Bug: b/222484146
Change-Id: I29aec9576e86b9ed3051df7943404abd478c549b
Having soong generate a list of makefiles for mk2rbc
to look through is much faster than having mk2rbc search
itself. Profiling the readLinesFromeFile() function that
reads the list of makefiles shows it takes on the order
of 200 microseconds, much faster than the ~5 seconds
it takes for mk2rbc to search the tree itself.
This CL also allows include statements that are prefixed
with a variable. The concern with this was that there
would be a lot of load statemnts emitted for generic
include statements, causing the generated code to look
ugly, and converting and loading all those files could
cause performance issues. On the performance issues
front, there's already a check that it doesn't result
in over 150 potentially included files. We can lower
that number if necessary, but it's probably good for now.
On the generated code front, while it's true that it's
ugly, it's better to have working but ugly generated
code than refusing to generate anything working at all.
To ensure the soong finder step isn't slowed down due
to having to find a bunch of new makefiles, I profiled
the combination of newSourceFinder and FindSources in main.go:
Baseline incremental:
338.011634ms
340.853335ms
348.541762ms
333.229644ms
349.124824ms
Baseline clean:
1.003836419s
1.006203912s
996.193648ms
1.031005604s
1.03691152s
Modified incremental:
349.029285ms
349.264496ms
351.774948ms
337.63187ms
359.425306ms
Modified clean:
1.028238704s
1.053103506s
1.032757506s
1.016631201s
1.04288108s
So we can see the times are barely affected by this change.
Fixes: 213508006
Test: go test
Change-Id: Iab18bfb127ba3b7e63f2c01f69064805a8398764
CI builds are doing m installclean between them;
which is not causing the system_dlkm to clean
it's build.prop and it is stuck to the first CI
build when feature landed causing ota signing
test to fail.
Local reproduction using:
m installclean && m -j96 && m dist
validate_target_files out/dist/*target_files-*.zip
Bug: 217507332
Test: Local verification, TH & Forrest
Signed-off-by: Ramji Jiyani <ramjiyani@google.com>
Change-Id: I97f2011a61252d3cb798af0fab99221b754328ca
The expfetcher binary (added in
https://critique.corp.google.com/cl/424076941), can fetch experiments
configuration from CDPush and load them into Soong as part of its
startup. If the binary exists, then we use it to fetch experiments
config and put them in the out directory so that those env variables can
be loaded by Soong during startup and inturn control whether RBE should
be enabled or not.
Design doc: go/rbe-android-dev-gradual-rollout
Test:
1. time lunch aosp_bramble-userdebug (first lunch when "out" dir doesn't
exist) - https://paste.googleplex.com/5824218777255936 - took ~9s and
ran with RBE enabled.
2. time lunch aosp_bramble-userdebug (when "out" dir already exists) - https://paste.googleplex.com/6485045129773056 -
took ~3.5s and ran with RBE enabled.
3. time lunch aosp_bramble-userdebug (when
"vendor/google/tools/soong/expfetcher" doesn't exist) - https://paste.googleplex.com/6103083353374720 - took 6.5s
and not RBE enabled.
Bug: b/215181607
Change-Id: Ie3c085498c59929119534aa98863566eecb8e4eb
When the environment variable config file is loaded after config object
is initialized, we end up loading the USE_RBE variable in the config
file after the code to start reproxy process is run. This causes a
problem when USE_RBE variable itself is set in the config files. To
avoid this, I've moved the config file loading code to be within the
config package, this makes the main.go file cleaner too.
Test:
Ran a build with USE_RBE:true set in the config json file. The build
fails without this change and succeeds with this change.
Bug: b/209486170
Change-Id: Iab3957b64f4b5456a861057d16ad318f4f78e0cb
Reproxy should be respecting the -j value set by the user and not be
doing excessive local executions assuming the whole machine is available
to it. The local_resource_fraction flag is a way to ensure reproxy
consumes CPU only upto the -j value specified by the user.
Test:
1. Ran with `USE_RBE=true m -j 10 nothing` and local_resource_fraction
was set to 0.14 on a 72-core workstation.
2. Ran with `USE_RBE=true m nothing` and local_resource_fraction was set
to 1.0 on a 72-core machine.
Bug: b/207296459
Change-Id: I118bc10109f5a55df7a6c1ecd79499320055f2ed
Colin and Jin opined that we shouldn't change production in order to
make debugging easier. I originally thought it's better to have one
fewer knob but I don't have very strong feelings about it and their line
of reasoning makes total sense.
Test: Presubmits.
Change-Id: Id8a914aebbeffec6e8f860157c13f81093194f14
