Commit graph

843 commits

Author SHA1 Message Date
Jeongik Cha
56df091391 Merge changes from topic "b/273282046"
* changes:
  Add HINT_FROM_SOONG option for ninja weight list
  Add EXTERNAL_FILE option for ninja weight list
2023-03-28 03:51:20 +00:00
Jeongik Cha
e114e60615 Add HINT_FROM_SOONG option for ninja weight list
If this option is set, Soong generates ninja weight list including
modules in HugeModulesList in allowlists.go

Test: m --ninja_weight_source=soong
Bug: 273282046
Change-Id: Id92b7f9f9e8152c1c46ae071c5821a479cf47bce
2023-03-23 01:45:27 +09:00
Jeongik Cha
518f3ea184 Add EXTERNAL_FILE option for ninja weight list
Test: m --ninja_weight_source=file,<file path>
Bug: 271527305
Change-Id: Ibeae4c757dff281be69486a9758dbee3584d9dec
2023-03-22 21:45:49 +09:00
LaMont Jones
80f7035dac ui: fix logic sense for multitree build
Bug: b/266730267
Test: treehugger, manual
Change-Id: I059a9b90ecb70de5965cb2fe3f8a1e16de46c116
2023-03-20 19:58:25 +00:00
Treehugger Robot
f4f034356e Merge "Resolve metrics_proto warning" 2023-03-17 21:46:55 +00:00
Usta (Tsering) Shrestha
9741785c4c Merge "python aprotoc instead of jq and textproto" 2023-03-17 14:13:18 +00:00
Usta Shrestha
47541bb1cf python aprotoc instead of jq and textproto
Bug: 273996308
Test: incremental_build.sh -c clean
Change-Id: I5b6c32667003010ccac5d3a46ac888f26788b3c5
2023-03-16 21:28:21 +00:00
Jeongik Cha
8d63d56ac6 Resolve metrics_proto warning
Bug: 271526845
Test: manually invoke internal protobuf presubmit
Change-Id: I79d1cc027e422fa1ca6106f5b45ef11045f6d486
2023-03-17 03:52:13 +09:00
Jeongik Cha
96d6227999 minor refactor ui/build/ninja.go
1. extract .ninja_log as const
2. log an error during reading .ninja_log

Test: m nothing
Bug: 271527305
Change-Id: I395dd8419620bfa9fad3af23c96e5a22ca44e2fb
2023-03-17 01:53:11 +09:00
Jeongik Cha
0cf44d5e8d Add --ninja_weight_source option
It has 3 options for now.
* ninja_log: uses ninja log file for data source
* evenly_distributed: pass empty list for ninja to consider every work as the same
* not_used: do not use critical path logic in ninja

In addition, I added the option in the metrics to track

Bug: 271527305
Test: m --ninja_weight_source=ninja_log|empty|not_used
Change-Id: Ib4c812c20606a34b17d3f0edb71057b477c4f90e
2023-03-16 19:52:27 +09:00
Jeongik Cha
767ce714bb Add time unit for CriticalPathInfo's fields' name
Bug: 271526845
Test: check metrics
Change-Id: Ie7e133033371906b29caad16d0b6bcd238b9f8db
2023-03-16 18:29:55 +09:00
Treehugger Robot
da4a78bf40 Merge "Add --multitree-build flag" 2023-03-16 02:34:22 +00:00
LaMont Jones
52a7243387 Add --multitree-build flag
Add a flag that enables logic specific to multitree-build builds.

Bug: b/266730267
Test: treehugger, manual
Change-Id: Ia6f392525faa9d66bb6e0692b2a2e9c3b8493b0d
2023-03-15 23:08:34 +00:00
Kevin Dagostino
24de5f2485 Merge "Cache GoogleProdCredsExist() to save ~800ms during builds." 2023-03-15 21:44:26 +00:00
Treehugger Robot
cd7db25738 Merge changes I7ec5a9f7,I76807f9b
* changes:
  Add docstrings for CriticalPathInfo
  Add long running jobs info
2023-03-15 07:26:40 +00:00
Jeongik Cha
cf833776a4 Add docstrings for CriticalPathInfo
Bug: 271526845
Test: n/a
Change-Id: I7ec5a9f7b39fd7441f1aac6556a77046a8210ae8
2023-03-15 12:54:14 +09:00
Jeongik Cha
4199d4796c Add long running jobs info
For the data to prioritize long running tasks in ninja, I added the
information about that.

Bug: 271526845
Test: build and check metrics
Change-Id: I76807f9b0d6a50d940b5f35dd6a9ef3cf693eec5
2023-03-15 12:46:58 +09:00
Jeongik Cha
58fa5a3f36 Merge "Add critical path infomation into metrics" 2023-03-15 03:40:11 +00:00
Jeongik Cha
28c1fe5bc8 Add critical path infomation into metrics
To improve build efficiency, the metrics for critical path and
parallelism ratio is necessary. That information has been included in
soong.log, so added it into metrics as well.

Bug: 271526845
Test: build and check if metrics pb has critical path info
Change-Id: I14e1a78c13d400b792d3b05df18604da48759ade
2023-03-14 17:59:22 +09:00
Kevin Dagostino
096ab2f5af Cache GoogleProdCredsExist() to save ~800ms during builds.
During a m nothing call; GoogleProdCredsExist check is called 3 times.
Each call is ~400ms so caching during the run will save ~800ms.

Test: time m nothing
Change-Id: I8a92c8fa48b22c7fb56be0f3c9cb5bf8787da6a9
2023-03-13 05:58:04 +00:00
Treehugger Robot
bb7c61ff61 Merge "Use Sec as well as Usec in ProcResInfo" 2023-03-10 22:19:33 +00:00
MarkDacek
a18ba22ba9 Add branch and manifest info to soong_metrics.
Bug: 259303480
Test: N/A
Change-Id: I434e24250ff640fddfca93ecb91fdfa49710e2f2
2023-03-07 21:55:26 +00:00
Jeongik Cha
bd465cca8a Use Sec as well as Usec in ProcResInfo
Timeval has two fields: Sec and Usec, but in previous implementation, it
only uses Usec, so Sec data is abandoned.

Bug: 271526845
Test: check metric, and if seconds data is reserved.
Change-Id: I911467f77b41995f6c833099648fe62fbc9909eb
2023-03-06 11:44:45 +09:00
Treehugger Robot
91108ad92a Merge "Add a post-build step for dist builds that records what changed in the build." 2023-03-02 18:53:38 +00:00
Christopher Parsons
7720f5704c Merge "Add a persistent bazel server between builds" 2023-03-02 14:22:34 +00:00
Chris Parsons
9402ca881c Add a persistent bazel server between builds
This feature is toggled on with USE_PERSISTENT_BAZEL, which is off by
default. Those that opt-in will have a bazel server running between
builds (with a 3hr default TTL) which will greatly improve analysis on
subsequent builds. (As Bazel maintains a cache of analysis results).

Bug: 266983462
Test: Manual `m nothing` runs (timing with and without the feature)
Test: New integration test
Test: Presubmits
Change-Id: I3af4948baa0c490e9b87c48ffdbe9f67732586c7
2023-03-01 23:30:19 -05:00
Treehugger Robot
732592c151 Merge "Create a ninja.environment file" 2023-03-01 03:22:31 +00:00
Spandan Das
2db59da973 Create a ninja.environment file
This contains the environment variables set by soong_ui during the ninja
execution. This file will be unused in single-tree builds, but will be
useful for multi-tree. The orchestrator will use this file to set the
correct environment for each inner tree build in the combined ninja
execution.

Test: m nothing compiles
Change-Id: I0570e34f51f426448464fb80030d4bea1cd52acb
2023-02-28 23:49:22 +00:00
Joe Onorato
7f29a66586 Add a post-build step for dist builds that records what changed in the build.
This will be used to skip tests when related files (or nothing) has changed.

Also minor refactoring of evaluateWhatToRun to put the logic all in one place.

Change-Id: Ia56b251716f6b6640bc5526a597cf7c066231e99
Test: go test, m (with soong tests), manually verify
2023-02-28 15:02:22 -08:00
Cole Faust
b1fbc79206 Add BUILD_BROKEN_PYTHON_IS_PYTHON2
This provides an escape hatch to undo the effects of
aosp/2417672. It's an enviornment variable instead of
a product config variable because python is often used
during product config.

Bug: 203436762
Test: m nothing and get_build_var with aosp/2458935
Change-Id: I0dd0f7c221427ea487fd7993aef68ce803eaff21
2023-02-27 13:02:02 -08:00
Nathan Egge
7b067fb753 Add JAVAC to the list of env variables to unset.
Some linux environments export JAVAC for legacy compatibility and this
 variable must be unset to ensure Soong and Make JAVAC values match.

Test: manual

Change-Id: Iae19fcb91b0a7651cbe00fe50f6daed54b4d7879
2023-02-17 18:07:04 +00:00
Cole Faust
64955b32bf Add status messages for RBE and product config
RBE adds a lot of time to m nothing builds,
so add status messages for it so that it's not misattributed
to some other part of the build.

Bug: 269644490
Test: m nothing
Change-Id: Ifff11bfc88f43b559526f3e3d3022c1388e42156
2023-02-16 21:06:19 +00:00
MarkDacek
00e3152aca Write bazel metrics file from soong.
Test: b build libcore:all
Change-Id: I372d48a9c7132bc57ecc1a8ba06eb098d28cbad7
2023-01-25 22:43:14 +00:00
Liz Kammer
7c7409d57d Merge "Change visibility of bp2build proto for dir rename" 2023-01-25 14:26:19 +00:00
Treehugger Robot
a413e4d28a Merge "Fix config file fetching" 2023-01-17 19:29:03 +00:00
Kousik Kumar
c881833bda Fix config file fetching
1. Make config file fetching run in background again if a file already
   exists - this behavior was changed in
   https://android-review.git.corp.google.com/c/platform/build/soong/+/2059887
   by accident.
2. Always try to load a config file even if fetching a config file fails
   - this behavior was changed in
     https://android-review.git.corp.google.com/c/platform/build/soong/+/2380792
     iiuc.

Bug: b/207551843
Test: Ran builds with no config file present, with config file present
and confirmed that config file fetching happens in the background now.

Change-Id: I6321684a0e17df3371f3f415602e7fa33cfba454
2023-01-16 17:20:19 +00:00
Usta Shrestha
da15c61add metrics: count symlink/mkdir in symlink forest
Bug: 260029212
Test: inspect out/bp2build_metrics.pb
Change-Id: Ia84c095a8d7b129cca629256b6c626c726cbfab1
2023-01-12 14:52:56 -05:00
MarkDacek
7901e585bd Fix upload-only to retrieve the METRICS_UPLOADER variable
in a timely manner.

There should not be harm in bypassing fetchEnvConfig for this case
as the metrics uploader should not be present for invalid cases.

Bug: b/264905338
Test: b build libcore:all
Test: printfs to verify that upload.go has the variable set.
Change-Id: Ia7d03f25e74d4ec2d6cb83793b793a23b47f26de
2023-01-10 20:17:22 +00:00
Jason Wu
205d1489fb Merge "Stop embedding soong_build_metrics in soong_metrics" 2023-01-10 02:53:16 +00:00
Jason Wu
41886f27aa Stop embedding soong_build_metrics in soong_metrics
Test: Run `m --bazel-mode-dev nothing` and query the database to make sure
soong_build_metrics will only update to its standalone column
Run `bp2build` and `b build` to make sure soong_build_metrics and
soong_metrics.soong_build_metrics will be null.

Bug: 262280865
Change-Id: Ice5115c468bda350b742c5a1d03fff66ad346cd6
2023-01-09 17:48:48 -05:00
MarkDacek
a3a62a37b4 Rename proto field.
Test: N/A
Change-Id: I97dd3bc26afcbef4ad69cb7a3cbad73bf34b5fe6
2023-01-04 22:04:44 +00:00
MarkDacek
6614d9cba3 Add upload-only mode and manual build-started-time flag.
This involved refactoring much of the main method in soong_ui/main.

Test: b build libcore:all
Test: use the build-time started flag and verify via printf that
it's the same across upload.go and metrics.SetBuildDateTimestamp()

Change-Id: Id7fe256337e8ee6c40542eba662c0eadb38e9674
2022-12-22 20:13:56 +00:00
MarkDacek
2ee2db3bf4 Add bazel metrics proto.
This is to marshal Bazel metrics for the sake of uploading to the
Android metrics pipeline.
Test: m nothing
Test: run the regen script

Change-Id: Ia4eca879cff07425c389e30a385418f9865159d6
2022-12-20 18:34:22 +00:00
Paul Crowley
e7e9e09128 Merge "Fix go link given when stubby is absent." 2022-12-15 18:42:59 +00:00
Jason Wu
f0ec631046 Upload soong_build_metrics
Test: m nothing
Test: m --bazel-mode-dev
Test: Verified soong_build_metrics is available in local metrics tables from builds that include this CL

Bug: 257537925
Change-Id: Ic5419ecb203c9b62457e1c553d592ba593a9bfff
2022-12-09 11:48:26 -05:00
MarkDacek
d0e7cd3d96 Add skip-metrics-upload flag to soong_ui.
Bug: b/261203925
Test: m libcore --skip-metrics-upload
Change-Id: I0b10027a0f359e682116c2ebb884808c65430dd0
2022-12-07 19:12:02 +00:00
Liz Kammer
798e96cc2c Change visibility of bp2build proto for dir rename
Test: CI
Change-Id: I3ba505c2e9fd23590531735ba6ae3aa003423cef
2022-12-06 09:51:17 -05:00
Spandan Das
c576383336 Create a new product variable to gate blueprint files
Users can use this feature by
1. Setting PRODUCT_INCLUDE_TAGS += <val> in their product mk files
2. Set
```
blueprint_packge_includes {
  match_al: ["<val>"],
}

other_module_type {name: foo}
other_module_type {name: bar}
```

bar and foo will be included if and only if <val> is set

Test: Unit tests in blueprint
Test: TH

Change-Id: I32eed4e3b5ac47fb565c62d13d8881fa984c86f4
2022-12-05 19:15:06 +00:00
MarkDacek
d06db5dd5b Add command line flag for extra bazel-built modules
Test: m libcore --bazel-mode-staging --bazel-force-enabled-modules=libcore and find out/bazel/ -name libcore*

Change-Id: I065696d06ce07e05300a41e133867a6e7a891b5e
2022-12-02 21:16:11 +00:00
Mark Dacek
ea80b16cd9 Merge "Add build-command arg to soong_ui." 2022-12-02 19:16:02 +00:00