Commit graph

856 commits

Author SHA1 Message Date
MarkDacek
d33c2fd5a2 Add --bazel-exit-code parameter to soong_ui.
This is for use in tracking bazel exit codes from b invocations.

Bug: 279754118
Test: b build libcore:all (with prints to verify)
Change-Id: If36b5b73ff93cd15647e5c1c1f676e95137210ff
2023-05-04 21:27:34 +00:00
Cole Faust
0428f71b30 Remove "dumping rbe metrics" message
The issue where RBE was taking a long time has been fixed, so this
message isn't that important anymore, and it was overriding the last
ephemeral status message that soong output.

Test: Presubmits
Change-Id: I019f351a3307176ec7b4786d4387fde9061ba08b
2023-05-03 18:21:05 -07:00
MarkDacek
ed2253e8df Process bazel metrics prior to potentially exiting upload.go
This is a necessary step for running b on CI with metrics collection.

Bug: 280491043
Test: b build libcore:all
Change-Id: I37d621c38e3f891dd63e5977befa6ce6604487ea
2023-05-02 21:35:39 +00:00
Mark Dacek
4188a899dc Merge "Add bazel exit code to bazel metrics proto." 2023-04-28 19:40:52 +00:00
MarkDacek
18ed8a912b Add bazel exit code to bazel metrics proto.
Bug: 279754118
Test: N/A
Change-Id: I52558843f40bf54b75a4255b0a9d96e9dcaa81bc
2023-04-28 14:50:32 +00:00
Anton Hansson
c8cea365f3 Merge "Remove --skip-kati" 2023-04-27 11:26:50 +00:00
Anton Hansson
50151a7489 Remove --skip-kati
Not used anymore.

Test: No
Change-Id: Ide0d29b773583d6a8faff5b41a82d650cc2f9bcb
2023-04-26 18:09:02 +00:00
MarkDacek
f47e142ffd Refactor MixedBuildsEnabled and add --ensure-allowlist-integrity.
Currently, there is little verification around allowlisted modules
actually being mixed-built. This flag would allow us to verify
that a module allowlisted is mixed-built for at least one variant.

Bug: 278910100
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
--bazel-force-enabled-modules=com.google.android.neuralnetworks (This
fails, as expected)
Test: build/soong/test/mixed_mode_test.sh

Change-Id: Icd5976f4f44f1a8caca1e5247d986642f7995f97
2023-04-25 20:55:07 +00:00
Ramy Medhat
c8f6cc24f9 Enable reclient cache directory.
The cache directory stores files that should persist between builds.
Currently it will contain reproxy.creds, which caches the authentication
mechanism and token used to make subsequent builds faster.

Test: m nothing build takes 7s
Change-Id: Iaaa20b92630405918e5e91e09eca6c798b8fc872
2023-04-03 09:56:10 -04:00
Treehugger Robot
bc2906efef Merge "Modify Soong to utilize from-text android.jar in build" 2023-03-29 20:31:20 +00:00
Treehugger Robot
998fdd7e37 Merge "support PRODUCT_SOURCE_ROOT_DIRS product variable" 2023-03-29 17:04:37 +00:00
Jihoon Kang
1bff0349d4 Modify Soong to utilize from-text android.jar in build
Context
- from-text android.jar files are built using Metalava, and these can be
  utilized in `decodeSdkDep` so that any modules that depends on APIs
  can be compiled using from-text android.jars
- This change removes dependency on source java files when compiling
  stub android.jar files

Implementation
- Modify java_api_library module to create system modules using the
  generated android.jar
- Replace modules in decodeSdkDep to link against java_api_library
  modules
- Add --build-from-text-stub flag to hide the feature behind a flag

Test: m --build-from-text-stub
Bug: 271154441
Change-Id: I104df595edc65c0006820d5ae5b15f1fb167e190
2023-03-28 21:53:45 +00:00
Jeongik Cha
56df091391 Merge changes from topic "b/273282046"
* changes:
  Add HINT_FROM_SOONG option for ninja weight list
  Add EXTERNAL_FILE option for ninja weight list
2023-03-28 03:51:20 +00:00
Sam Delmerico
98a7329d59 support PRODUCT_SOURCE_ROOT_DIRS product variable
Soong analyzes the entire source tree even though not every lunch target
needs to know about every module. For example, OEM sources can be
ignored for cuttlefish products. This functionality allows blueprint to
ignore a list of undesired directories.

Bug: 269457150
Change-Id: I1eec5d7b6a268cae4c633d8d89ed485598ebca45
2023-03-27 14:42:36 -04:00
Jeongik Cha
e114e60615 Add HINT_FROM_SOONG option for ninja weight list
If this option is set, Soong generates ninja weight list including
modules in HugeModulesList in allowlists.go

Test: m --ninja_weight_source=soong
Bug: 273282046
Change-Id: Id92b7f9f9e8152c1c46ae071c5821a479cf47bce
2023-03-23 01:45:27 +09:00
Jeongik Cha
518f3ea184 Add EXTERNAL_FILE option for ninja weight list
Test: m --ninja_weight_source=file,<file path>
Bug: 271527305
Change-Id: Ibeae4c757dff281be69486a9758dbee3584d9dec
2023-03-22 21:45:49 +09:00
LaMont Jones
80f7035dac ui: fix logic sense for multitree build
Bug: b/266730267
Test: treehugger, manual
Change-Id: I059a9b90ecb70de5965cb2fe3f8a1e16de46c116
2023-03-20 19:58:25 +00:00
Treehugger Robot
f4f034356e Merge "Resolve metrics_proto warning" 2023-03-17 21:46:55 +00:00
Usta (Tsering) Shrestha
9741785c4c Merge "python aprotoc instead of jq and textproto" 2023-03-17 14:13:18 +00:00
Usta Shrestha
47541bb1cf python aprotoc instead of jq and textproto
Bug: 273996308
Test: incremental_build.sh -c clean
Change-Id: I5b6c32667003010ccac5d3a46ac888f26788b3c5
2023-03-16 21:28:21 +00:00
Jeongik Cha
8d63d56ac6 Resolve metrics_proto warning
Bug: 271526845
Test: manually invoke internal protobuf presubmit
Change-Id: I79d1cc027e422fa1ca6106f5b45ef11045f6d486
2023-03-17 03:52:13 +09:00
Jeongik Cha
96d6227999 minor refactor ui/build/ninja.go
1. extract .ninja_log as const
2. log an error during reading .ninja_log

Test: m nothing
Bug: 271527305
Change-Id: I395dd8419620bfa9fad3af23c96e5a22ca44e2fb
2023-03-17 01:53:11 +09:00
Jeongik Cha
0cf44d5e8d Add --ninja_weight_source option
It has 3 options for now.
* ninja_log: uses ninja log file for data source
* evenly_distributed: pass empty list for ninja to consider every work as the same
* not_used: do not use critical path logic in ninja

In addition, I added the option in the metrics to track

Bug: 271527305
Test: m --ninja_weight_source=ninja_log|empty|not_used
Change-Id: Ib4c812c20606a34b17d3f0edb71057b477c4f90e
2023-03-16 19:52:27 +09:00
Jeongik Cha
767ce714bb Add time unit for CriticalPathInfo's fields' name
Bug: 271526845
Test: check metrics
Change-Id: Ie7e133033371906b29caad16d0b6bcd238b9f8db
2023-03-16 18:29:55 +09:00
Treehugger Robot
da4a78bf40 Merge "Add --multitree-build flag" 2023-03-16 02:34:22 +00:00
LaMont Jones
52a7243387 Add --multitree-build flag
Add a flag that enables logic specific to multitree-build builds.

Bug: b/266730267
Test: treehugger, manual
Change-Id: Ia6f392525faa9d66bb6e0692b2a2e9c3b8493b0d
2023-03-15 23:08:34 +00:00
Kevin Dagostino
24de5f2485 Merge "Cache GoogleProdCredsExist() to save ~800ms during builds." 2023-03-15 21:44:26 +00:00
Treehugger Robot
cd7db25738 Merge changes I7ec5a9f7,I76807f9b
* changes:
  Add docstrings for CriticalPathInfo
  Add long running jobs info
2023-03-15 07:26:40 +00:00
Jeongik Cha
cf833776a4 Add docstrings for CriticalPathInfo
Bug: 271526845
Test: n/a
Change-Id: I7ec5a9f7b39fd7441f1aac6556a77046a8210ae8
2023-03-15 12:54:14 +09:00
Jeongik Cha
4199d4796c Add long running jobs info
For the data to prioritize long running tasks in ninja, I added the
information about that.

Bug: 271526845
Test: build and check metrics
Change-Id: I76807f9b0d6a50d940b5f35dd6a9ef3cf693eec5
2023-03-15 12:46:58 +09:00
Jeongik Cha
58fa5a3f36 Merge "Add critical path infomation into metrics" 2023-03-15 03:40:11 +00:00
Jeongik Cha
28c1fe5bc8 Add critical path infomation into metrics
To improve build efficiency, the metrics for critical path and
parallelism ratio is necessary. That information has been included in
soong.log, so added it into metrics as well.

Bug: 271526845
Test: build and check if metrics pb has critical path info
Change-Id: I14e1a78c13d400b792d3b05df18604da48759ade
2023-03-14 17:59:22 +09:00
Kevin Dagostino
096ab2f5af Cache GoogleProdCredsExist() to save ~800ms during builds.
During a m nothing call; GoogleProdCredsExist check is called 3 times.
Each call is ~400ms so caching during the run will save ~800ms.

Test: time m nothing
Change-Id: I8a92c8fa48b22c7fb56be0f3c9cb5bf8787da6a9
2023-03-13 05:58:04 +00:00
Treehugger Robot
bb7c61ff61 Merge "Use Sec as well as Usec in ProcResInfo" 2023-03-10 22:19:33 +00:00
MarkDacek
a18ba22ba9 Add branch and manifest info to soong_metrics.
Bug: 259303480
Test: N/A
Change-Id: I434e24250ff640fddfca93ecb91fdfa49710e2f2
2023-03-07 21:55:26 +00:00
Jeongik Cha
bd465cca8a Use Sec as well as Usec in ProcResInfo
Timeval has two fields: Sec and Usec, but in previous implementation, it
only uses Usec, so Sec data is abandoned.

Bug: 271526845
Test: check metric, and if seconds data is reserved.
Change-Id: I911467f77b41995f6c833099648fe62fbc9909eb
2023-03-06 11:44:45 +09:00
Treehugger Robot
91108ad92a Merge "Add a post-build step for dist builds that records what changed in the build." 2023-03-02 18:53:38 +00:00
Christopher Parsons
7720f5704c Merge "Add a persistent bazel server between builds" 2023-03-02 14:22:34 +00:00
Chris Parsons
9402ca881c Add a persistent bazel server between builds
This feature is toggled on with USE_PERSISTENT_BAZEL, which is off by
default. Those that opt-in will have a bazel server running between
builds (with a 3hr default TTL) which will greatly improve analysis on
subsequent builds. (As Bazel maintains a cache of analysis results).

Bug: 266983462
Test: Manual `m nothing` runs (timing with and without the feature)
Test: New integration test
Test: Presubmits
Change-Id: I3af4948baa0c490e9b87c48ffdbe9f67732586c7
2023-03-01 23:30:19 -05:00
Treehugger Robot
732592c151 Merge "Create a ninja.environment file" 2023-03-01 03:22:31 +00:00
Spandan Das
2db59da973 Create a ninja.environment file
This contains the environment variables set by soong_ui during the ninja
execution. This file will be unused in single-tree builds, but will be
useful for multi-tree. The orchestrator will use this file to set the
correct environment for each inner tree build in the combined ninja
execution.

Test: m nothing compiles
Change-Id: I0570e34f51f426448464fb80030d4bea1cd52acb
2023-02-28 23:49:22 +00:00
Joe Onorato
7f29a66586 Add a post-build step for dist builds that records what changed in the build.
This will be used to skip tests when related files (or nothing) has changed.

Also minor refactoring of evaluateWhatToRun to put the logic all in one place.

Change-Id: Ia56b251716f6b6640bc5526a597cf7c066231e99
Test: go test, m (with soong tests), manually verify
2023-02-28 15:02:22 -08:00
Cole Faust
b1fbc79206 Add BUILD_BROKEN_PYTHON_IS_PYTHON2
This provides an escape hatch to undo the effects of
aosp/2417672. It's an enviornment variable instead of
a product config variable because python is often used
during product config.

Bug: 203436762
Test: m nothing and get_build_var with aosp/2458935
Change-Id: I0dd0f7c221427ea487fd7993aef68ce803eaff21
2023-02-27 13:02:02 -08:00
Nathan Egge
7b067fb753 Add JAVAC to the list of env variables to unset.
Some linux environments export JAVAC for legacy compatibility and this
 variable must be unset to ensure Soong and Make JAVAC values match.

Test: manual

Change-Id: Iae19fcb91b0a7651cbe00fe50f6daed54b4d7879
2023-02-17 18:07:04 +00:00
Cole Faust
64955b32bf Add status messages for RBE and product config
RBE adds a lot of time to m nothing builds,
so add status messages for it so that it's not misattributed
to some other part of the build.

Bug: 269644490
Test: m nothing
Change-Id: Ifff11bfc88f43b559526f3e3d3022c1388e42156
2023-02-16 21:06:19 +00:00
MarkDacek
00e3152aca Write bazel metrics file from soong.
Test: b build libcore:all
Change-Id: I372d48a9c7132bc57ecc1a8ba06eb098d28cbad7
2023-01-25 22:43:14 +00:00
Liz Kammer
7c7409d57d Merge "Change visibility of bp2build proto for dir rename" 2023-01-25 14:26:19 +00:00
Treehugger Robot
a413e4d28a Merge "Fix config file fetching" 2023-01-17 19:29:03 +00:00
Kousik Kumar
c881833bda Fix config file fetching
1. Make config file fetching run in background again if a file already
   exists - this behavior was changed in
   https://android-review.git.corp.google.com/c/platform/build/soong/+/2059887
   by accident.
2. Always try to load a config file even if fetching a config file fails
   - this behavior was changed in
     https://android-review.git.corp.google.com/c/platform/build/soong/+/2380792
     iiuc.

Bug: b/207551843
Test: Ran builds with no config file present, with config file present
and confirmed that config file fetching happens in the background now.

Change-Id: I6321684a0e17df3371f3f415602e7fa33cfba454
2023-01-16 17:20:19 +00:00
Usta Shrestha
da15c61add metrics: count symlink/mkdir in symlink forest
Bug: 260029212
Test: inspect out/bp2build_metrics.pb
Change-Id: Ia84c095a8d7b129cca629256b6c626c726cbfab1
2023-01-12 14:52:56 -05:00