Commit graph

765 commits

Author SHA1 Message Date
Jeongik Cha
f2ecf7659e Enable NINJA_LOG as weight list source
Bug: 273947040
Test: m
Change-Id: I1a803f5c1da75235395fc594e4f9d17f48c55db8
2023-05-19 14:03:45 +09:00
Chris Parsons
fddeaff4a6 Revert "Turn on HINT_FROM_SOONG by default"
This reverts commit f5c872f36b.

Revert reason: The original commit seems to have a performance regression for a number of benchmarked performance CUJs.

Bug: 283143307

Test: Benchmarking results. See attached bug.
Change-Id: Ib9463c8d30a5ba61640993424696a84e2e03040a
2023-05-17 22:30:34 +00:00
Treehugger Robot
abc2b2adb3 Merge "Turn on HINT_FROM_SOONG by default" 2023-05-15 14:36:05 +00:00
Joe Onorato
8ef13dbb0c Merge "TARGET_RELEASE is unavailable here, so don't print it." 2023-05-12 01:44:46 +00:00
Mark Dacek
923a40777b Merge "Refactor upload-only to process bazel metrics outside of upload.go" 2023-05-10 17:52:40 +00:00
Treehugger Robot
f47760461c Merge "Change bp2build converter of module "package"." 2023-05-10 17:38:19 +00:00
Joe Onorato
98e986542b TARGET_RELEASE is unavailable here, so don't print it.
Test: m
Change-Id: Iacb00db54882ed2578ca1aa1d8e71d5732fbe247
2023-05-10 05:56:57 -07:00
Wei Li
2c9e8d6128 Change bp2build converter of module "package".
1) Use attribute name "default_package_metadata" instead of
   "default_applicable_licenses" in packages to better describe its use.
2) Add a filegroup "default_metadata_file" in packages to search for
   METADATA file in each package.
3) Include "default_metadata_file" in each package's
   "default_package_metadata" attribute.

Bug: 275472038
Test: CIs
Change-Id: I645c013c39e3190fd96c4a549d39a331aced16bd
2023-05-09 17:37:04 -07:00
MarkDacek
733b77c2de Refactor upload-only to process bazel metrics outside of upload.go
Bug: 279987768
Test: m nothing
Test: b build libcore:all (with prints to verify similiar data)
Change-Id: I01d960a7215be30728c9e90f52c2455fd21cf1b8
2023-05-09 20:19:20 +00:00
Mark Dacek
3e63067b11 Merge "Add --bazel-exit-code parameter to soong_ui." 2023-05-09 16:26:46 +00:00
Jeff Hamilton
d27fedf8e9 Add TARGET_RELEASE to the banner printout
Test: run lunch with and without a release and confirmed it's printed
Change-Id: I67e86aa1328ae464a1bb10dfdcd8e024e72d77a8
2023-05-08 03:17:06 +00:00
MarkDacek
d33c2fd5a2 Add --bazel-exit-code parameter to soong_ui.
This is for use in tracking bazel exit codes from b invocations.

Bug: 279754118
Test: b build libcore:all (with prints to verify)
Change-Id: If36b5b73ff93cd15647e5c1c1f676e95137210ff
2023-05-04 21:27:34 +00:00
Cole Faust
0428f71b30 Remove "dumping rbe metrics" message
The issue where RBE was taking a long time has been fixed, so this
message isn't that important anymore, and it was overriding the last
ephemeral status message that soong output.

Test: Presubmits
Change-Id: I019f351a3307176ec7b4786d4387fde9061ba08b
2023-05-03 18:21:05 -07:00
MarkDacek
ed2253e8df Process bazel metrics prior to potentially exiting upload.go
This is a necessary step for running b on CI with metrics collection.

Bug: 280491043
Test: b build libcore:all
Change-Id: I37d621c38e3f891dd63e5977befa6ce6604487ea
2023-05-02 21:35:39 +00:00
Anton Hansson
c8cea365f3 Merge "Remove --skip-kati" 2023-04-27 11:26:50 +00:00
Anton Hansson
50151a7489 Remove --skip-kati
Not used anymore.

Test: No
Change-Id: Ide0d29b773583d6a8faff5b41a82d650cc2f9bcb
2023-04-26 18:09:02 +00:00
MarkDacek
f47e142ffd Refactor MixedBuildsEnabled and add --ensure-allowlist-integrity.
Currently, there is little verification around allowlisted modules
actually being mixed-built. This flag would allow us to verify
that a module allowlisted is mixed-built for at least one variant.

Bug: 278910100
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
--bazel-force-enabled-modules=com.google.android.neuralnetworks (This
fails, as expected)
Test: build/soong/test/mixed_mode_test.sh

Change-Id: Icd5976f4f44f1a8caca1e5247d986642f7995f97
2023-04-25 20:55:07 +00:00
Jeongik Cha
f5c872f36b Turn on HINT_FROM_SOONG by default
In most case, turning on HINT_FROM_SOONG improves build performance even
though a model is not so accurate. At least, it doesn't cause
performance regression (very likely)

Bug: 275458825
Test: m
Change-Id: Ie4ff69e68a93f6f0db3d060980657ae623816b2b
2023-04-14 00:53:24 +09:00
Ramy Medhat
c8f6cc24f9 Enable reclient cache directory.
The cache directory stores files that should persist between builds.
Currently it will contain reproxy.creds, which caches the authentication
mechanism and token used to make subsequent builds faster.

Test: m nothing build takes 7s
Change-Id: Iaaa20b92630405918e5e91e09eca6c798b8fc872
2023-04-03 09:56:10 -04:00
Treehugger Robot
bc2906efef Merge "Modify Soong to utilize from-text android.jar in build" 2023-03-29 20:31:20 +00:00
Treehugger Robot
998fdd7e37 Merge "support PRODUCT_SOURCE_ROOT_DIRS product variable" 2023-03-29 17:04:37 +00:00
Jihoon Kang
1bff0349d4 Modify Soong to utilize from-text android.jar in build
Context
- from-text android.jar files are built using Metalava, and these can be
  utilized in `decodeSdkDep` so that any modules that depends on APIs
  can be compiled using from-text android.jars
- This change removes dependency on source java files when compiling
  stub android.jar files

Implementation
- Modify java_api_library module to create system modules using the
  generated android.jar
- Replace modules in decodeSdkDep to link against java_api_library
  modules
- Add --build-from-text-stub flag to hide the feature behind a flag

Test: m --build-from-text-stub
Bug: 271154441
Change-Id: I104df595edc65c0006820d5ae5b15f1fb167e190
2023-03-28 21:53:45 +00:00
Jeongik Cha
56df091391 Merge changes from topic "b/273282046"
* changes:
  Add HINT_FROM_SOONG option for ninja weight list
  Add EXTERNAL_FILE option for ninja weight list
2023-03-28 03:51:20 +00:00
Sam Delmerico
98a7329d59 support PRODUCT_SOURCE_ROOT_DIRS product variable
Soong analyzes the entire source tree even though not every lunch target
needs to know about every module. For example, OEM sources can be
ignored for cuttlefish products. This functionality allows blueprint to
ignore a list of undesired directories.

Bug: 269457150
Change-Id: I1eec5d7b6a268cae4c633d8d89ed485598ebca45
2023-03-27 14:42:36 -04:00
Jeongik Cha
e114e60615 Add HINT_FROM_SOONG option for ninja weight list
If this option is set, Soong generates ninja weight list including
modules in HugeModulesList in allowlists.go

Test: m --ninja_weight_source=soong
Bug: 273282046
Change-Id: Id92b7f9f9e8152c1c46ae071c5821a479cf47bce
2023-03-23 01:45:27 +09:00
Jeongik Cha
518f3ea184 Add EXTERNAL_FILE option for ninja weight list
Test: m --ninja_weight_source=file,<file path>
Bug: 271527305
Change-Id: Ibeae4c757dff281be69486a9758dbee3584d9dec
2023-03-22 21:45:49 +09:00
LaMont Jones
80f7035dac ui: fix logic sense for multitree build
Bug: b/266730267
Test: treehugger, manual
Change-Id: I059a9b90ecb70de5965cb2fe3f8a1e16de46c116
2023-03-20 19:58:25 +00:00
Treehugger Robot
f4f034356e Merge "Resolve metrics_proto warning" 2023-03-17 21:46:55 +00:00
Jeongik Cha
8d63d56ac6 Resolve metrics_proto warning
Bug: 271526845
Test: manually invoke internal protobuf presubmit
Change-Id: I79d1cc027e422fa1ca6106f5b45ef11045f6d486
2023-03-17 03:52:13 +09:00
Jeongik Cha
96d6227999 minor refactor ui/build/ninja.go
1. extract .ninja_log as const
2. log an error during reading .ninja_log

Test: m nothing
Bug: 271527305
Change-Id: I395dd8419620bfa9fad3af23c96e5a22ca44e2fb
2023-03-17 01:53:11 +09:00
Jeongik Cha
0cf44d5e8d Add --ninja_weight_source option
It has 3 options for now.
* ninja_log: uses ninja log file for data source
* evenly_distributed: pass empty list for ninja to consider every work as the same
* not_used: do not use critical path logic in ninja

In addition, I added the option in the metrics to track

Bug: 271527305
Test: m --ninja_weight_source=ninja_log|empty|not_used
Change-Id: Ib4c812c20606a34b17d3f0edb71057b477c4f90e
2023-03-16 19:52:27 +09:00
Treehugger Robot
da4a78bf40 Merge "Add --multitree-build flag" 2023-03-16 02:34:22 +00:00
LaMont Jones
52a7243387 Add --multitree-build flag
Add a flag that enables logic specific to multitree-build builds.

Bug: b/266730267
Test: treehugger, manual
Change-Id: Ia6f392525faa9d66bb6e0692b2a2e9c3b8493b0d
2023-03-15 23:08:34 +00:00
Kevin Dagostino
24de5f2485 Merge "Cache GoogleProdCredsExist() to save ~800ms during builds." 2023-03-15 21:44:26 +00:00
Jeongik Cha
28c1fe5bc8 Add critical path infomation into metrics
To improve build efficiency, the metrics for critical path and
parallelism ratio is necessary. That information has been included in
soong.log, so added it into metrics as well.

Bug: 271526845
Test: build and check if metrics pb has critical path info
Change-Id: I14e1a78c13d400b792d3b05df18604da48759ade
2023-03-14 17:59:22 +09:00
Kevin Dagostino
096ab2f5af Cache GoogleProdCredsExist() to save ~800ms during builds.
During a m nothing call; GoogleProdCredsExist check is called 3 times.
Each call is ~400ms so caching during the run will save ~800ms.

Test: time m nothing
Change-Id: I8a92c8fa48b22c7fb56be0f3c9cb5bf8787da6a9
2023-03-13 05:58:04 +00:00
Treehugger Robot
91108ad92a Merge "Add a post-build step for dist builds that records what changed in the build." 2023-03-02 18:53:38 +00:00
Christopher Parsons
7720f5704c Merge "Add a persistent bazel server between builds" 2023-03-02 14:22:34 +00:00
Chris Parsons
9402ca881c Add a persistent bazel server between builds
This feature is toggled on with USE_PERSISTENT_BAZEL, which is off by
default. Those that opt-in will have a bazel server running between
builds (with a 3hr default TTL) which will greatly improve analysis on
subsequent builds. (As Bazel maintains a cache of analysis results).

Bug: 266983462
Test: Manual `m nothing` runs (timing with and without the feature)
Test: New integration test
Test: Presubmits
Change-Id: I3af4948baa0c490e9b87c48ffdbe9f67732586c7
2023-03-01 23:30:19 -05:00
Treehugger Robot
732592c151 Merge "Create a ninja.environment file" 2023-03-01 03:22:31 +00:00
Spandan Das
2db59da973 Create a ninja.environment file
This contains the environment variables set by soong_ui during the ninja
execution. This file will be unused in single-tree builds, but will be
useful for multi-tree. The orchestrator will use this file to set the
correct environment for each inner tree build in the combined ninja
execution.

Test: m nothing compiles
Change-Id: I0570e34f51f426448464fb80030d4bea1cd52acb
2023-02-28 23:49:22 +00:00
Joe Onorato
7f29a66586 Add a post-build step for dist builds that records what changed in the build.
This will be used to skip tests when related files (or nothing) has changed.

Also minor refactoring of evaluateWhatToRun to put the logic all in one place.

Change-Id: Ia56b251716f6b6640bc5526a597cf7c066231e99
Test: go test, m (with soong tests), manually verify
2023-02-28 15:02:22 -08:00
Cole Faust
b1fbc79206 Add BUILD_BROKEN_PYTHON_IS_PYTHON2
This provides an escape hatch to undo the effects of
aosp/2417672. It's an enviornment variable instead of
a product config variable because python is often used
during product config.

Bug: 203436762
Test: m nothing and get_build_var with aosp/2458935
Change-Id: I0dd0f7c221427ea487fd7993aef68ce803eaff21
2023-02-27 13:02:02 -08:00
Nathan Egge
7b067fb753 Add JAVAC to the list of env variables to unset.
Some linux environments export JAVAC for legacy compatibility and this
 variable must be unset to ensure Soong and Make JAVAC values match.

Test: manual

Change-Id: Iae19fcb91b0a7651cbe00fe50f6daed54b4d7879
2023-02-17 18:07:04 +00:00
Cole Faust
64955b32bf Add status messages for RBE and product config
RBE adds a lot of time to m nothing builds,
so add status messages for it so that it's not misattributed
to some other part of the build.

Bug: 269644490
Test: m nothing
Change-Id: Ifff11bfc88f43b559526f3e3d3022c1388e42156
2023-02-16 21:06:19 +00:00
MarkDacek
00e3152aca Write bazel metrics file from soong.
Test: b build libcore:all
Change-Id: I372d48a9c7132bc57ecc1a8ba06eb098d28cbad7
2023-01-25 22:43:14 +00:00
Kousik Kumar
c881833bda Fix config file fetching
1. Make config file fetching run in background again if a file already
   exists - this behavior was changed in
   https://android-review.git.corp.google.com/c/platform/build/soong/+/2059887
   by accident.
2. Always try to load a config file even if fetching a config file fails
   - this behavior was changed in
     https://android-review.git.corp.google.com/c/platform/build/soong/+/2380792
     iiuc.

Bug: b/207551843
Test: Ran builds with no config file present, with config file present
and confirmed that config file fetching happens in the background now.

Change-Id: I6321684a0e17df3371f3f415602e7fa33cfba454
2023-01-16 17:20:19 +00:00
MarkDacek
7901e585bd Fix upload-only to retrieve the METRICS_UPLOADER variable
in a timely manner.

There should not be harm in bypassing fetchEnvConfig for this case
as the metrics uploader should not be present for invalid cases.

Bug: b/264905338
Test: b build libcore:all
Test: printfs to verify that upload.go has the variable set.
Change-Id: Ia7d03f25e74d4ec2d6cb83793b793a23b47f26de
2023-01-10 20:17:22 +00:00
Jason Wu
205d1489fb Merge "Stop embedding soong_build_metrics in soong_metrics" 2023-01-10 02:53:16 +00:00
Jason Wu
41886f27aa Stop embedding soong_build_metrics in soong_metrics
Test: Run `m --bazel-mode-dev nothing` and query the database to make sure
soong_build_metrics will only update to its standalone column
Run `bp2build` and `b build` to make sure soong_build_metrics and
soong_metrics.soong_build_metrics will be null.

Bug: 262280865
Change-Id: Ice5115c468bda350b742c5a1d03fff66ad346cd6
2023-01-09 17:48:48 -05:00