Commit graph

866 commits

Author SHA1 Message Date
Treehugger Robot
abc2b2adb3 Merge "Turn on HINT_FROM_SOONG by default" 2023-05-15 14:36:05 +00:00
Joe Onorato
8ef13dbb0c Merge "TARGET_RELEASE is unavailable here, so don't print it." 2023-05-12 01:44:46 +00:00
Mark Dacek
923a40777b Merge "Refactor upload-only to process bazel metrics outside of upload.go" 2023-05-10 17:52:40 +00:00
Treehugger Robot
f47760461c Merge "Change bp2build converter of module "package"." 2023-05-10 17:38:19 +00:00
Joe Onorato
98e986542b TARGET_RELEASE is unavailable here, so don't print it.
Test: m
Change-Id: Iacb00db54882ed2578ca1aa1d8e71d5732fbe247
2023-05-10 05:56:57 -07:00
Wei Li
2c9e8d6128 Change bp2build converter of module "package".
1) Use attribute name "default_package_metadata" instead of
   "default_applicable_licenses" in packages to better describe its use.
2) Add a filegroup "default_metadata_file" in packages to search for
   METADATA file in each package.
3) Include "default_metadata_file" in each package's
   "default_package_metadata" attribute.

Bug: 275472038
Test: CIs
Change-Id: I645c013c39e3190fd96c4a549d39a331aced16bd
2023-05-09 17:37:04 -07:00
MarkDacek
733b77c2de Refactor upload-only to process bazel metrics outside of upload.go
Bug: 279987768
Test: m nothing
Test: b build libcore:all (with prints to verify similiar data)
Change-Id: I01d960a7215be30728c9e90f52c2455fd21cf1b8
2023-05-09 20:19:20 +00:00
Mark Dacek
3e63067b11 Merge "Add --bazel-exit-code parameter to soong_ui." 2023-05-09 16:26:46 +00:00
Jeff Hamilton
d27fedf8e9 Add TARGET_RELEASE to the banner printout
Test: run lunch with and without a release and confirmed it's printed
Change-Id: I67e86aa1328ae464a1bb10dfdcd8e024e72d77a8
2023-05-08 03:17:06 +00:00
MarkDacek
d33c2fd5a2 Add --bazel-exit-code parameter to soong_ui.
This is for use in tracking bazel exit codes from b invocations.

Bug: 279754118
Test: b build libcore:all (with prints to verify)
Change-Id: If36b5b73ff93cd15647e5c1c1f676e95137210ff
2023-05-04 21:27:34 +00:00
Cole Faust
0428f71b30 Remove "dumping rbe metrics" message
The issue where RBE was taking a long time has been fixed, so this
message isn't that important anymore, and it was overriding the last
ephemeral status message that soong output.

Test: Presubmits
Change-Id: I019f351a3307176ec7b4786d4387fde9061ba08b
2023-05-03 18:21:05 -07:00
MarkDacek
ed2253e8df Process bazel metrics prior to potentially exiting upload.go
This is a necessary step for running b on CI with metrics collection.

Bug: 280491043
Test: b build libcore:all
Change-Id: I37d621c38e3f891dd63e5977befa6ce6604487ea
2023-05-02 21:35:39 +00:00
Mark Dacek
4188a899dc Merge "Add bazel exit code to bazel metrics proto." 2023-04-28 19:40:52 +00:00
MarkDacek
18ed8a912b Add bazel exit code to bazel metrics proto.
Bug: 279754118
Test: N/A
Change-Id: I52558843f40bf54b75a4255b0a9d96e9dcaa81bc
2023-04-28 14:50:32 +00:00
Anton Hansson
c8cea365f3 Merge "Remove --skip-kati" 2023-04-27 11:26:50 +00:00
Anton Hansson
50151a7489 Remove --skip-kati
Not used anymore.

Test: No
Change-Id: Ide0d29b773583d6a8faff5b41a82d650cc2f9bcb
2023-04-26 18:09:02 +00:00
MarkDacek
f47e142ffd Refactor MixedBuildsEnabled and add --ensure-allowlist-integrity.
Currently, there is little verification around allowlisted modules
actually being mixed-built. This flag would allow us to verify
that a module allowlisted is mixed-built for at least one variant.

Bug: 278910100
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
--bazel-force-enabled-modules=com.google.android.neuralnetworks (This
fails, as expected)
Test: build/soong/test/mixed_mode_test.sh

Change-Id: Icd5976f4f44f1a8caca1e5247d986642f7995f97
2023-04-25 20:55:07 +00:00
Jeongik Cha
f5c872f36b Turn on HINT_FROM_SOONG by default
In most case, turning on HINT_FROM_SOONG improves build performance even
though a model is not so accurate. At least, it doesn't cause
performance regression (very likely)

Bug: 275458825
Test: m
Change-Id: Ie4ff69e68a93f6f0db3d060980657ae623816b2b
2023-04-14 00:53:24 +09:00
Ramy Medhat
c8f6cc24f9 Enable reclient cache directory.
The cache directory stores files that should persist between builds.
Currently it will contain reproxy.creds, which caches the authentication
mechanism and token used to make subsequent builds faster.

Test: m nothing build takes 7s
Change-Id: Iaaa20b92630405918e5e91e09eca6c798b8fc872
2023-04-03 09:56:10 -04:00
Treehugger Robot
bc2906efef Merge "Modify Soong to utilize from-text android.jar in build" 2023-03-29 20:31:20 +00:00
Treehugger Robot
998fdd7e37 Merge "support PRODUCT_SOURCE_ROOT_DIRS product variable" 2023-03-29 17:04:37 +00:00
Jihoon Kang
1bff0349d4 Modify Soong to utilize from-text android.jar in build
Context
- from-text android.jar files are built using Metalava, and these can be
  utilized in `decodeSdkDep` so that any modules that depends on APIs
  can be compiled using from-text android.jars
- This change removes dependency on source java files when compiling
  stub android.jar files

Implementation
- Modify java_api_library module to create system modules using the
  generated android.jar
- Replace modules in decodeSdkDep to link against java_api_library
  modules
- Add --build-from-text-stub flag to hide the feature behind a flag

Test: m --build-from-text-stub
Bug: 271154441
Change-Id: I104df595edc65c0006820d5ae5b15f1fb167e190
2023-03-28 21:53:45 +00:00
Jeongik Cha
56df091391 Merge changes from topic "b/273282046"
* changes:
  Add HINT_FROM_SOONG option for ninja weight list
  Add EXTERNAL_FILE option for ninja weight list
2023-03-28 03:51:20 +00:00
Sam Delmerico
98a7329d59 support PRODUCT_SOURCE_ROOT_DIRS product variable
Soong analyzes the entire source tree even though not every lunch target
needs to know about every module. For example, OEM sources can be
ignored for cuttlefish products. This functionality allows blueprint to
ignore a list of undesired directories.

Bug: 269457150
Change-Id: I1eec5d7b6a268cae4c633d8d89ed485598ebca45
2023-03-27 14:42:36 -04:00
Jeongik Cha
e114e60615 Add HINT_FROM_SOONG option for ninja weight list
If this option is set, Soong generates ninja weight list including
modules in HugeModulesList in allowlists.go

Test: m --ninja_weight_source=soong
Bug: 273282046
Change-Id: Id92b7f9f9e8152c1c46ae071c5821a479cf47bce
2023-03-23 01:45:27 +09:00
Jeongik Cha
518f3ea184 Add EXTERNAL_FILE option for ninja weight list
Test: m --ninja_weight_source=file,<file path>
Bug: 271527305
Change-Id: Ibeae4c757dff281be69486a9758dbee3584d9dec
2023-03-22 21:45:49 +09:00
LaMont Jones
80f7035dac ui: fix logic sense for multitree build
Bug: b/266730267
Test: treehugger, manual
Change-Id: I059a9b90ecb70de5965cb2fe3f8a1e16de46c116
2023-03-20 19:58:25 +00:00
Treehugger Robot
f4f034356e Merge "Resolve metrics_proto warning" 2023-03-17 21:46:55 +00:00
Usta (Tsering) Shrestha
9741785c4c Merge "python aprotoc instead of jq and textproto" 2023-03-17 14:13:18 +00:00
Usta Shrestha
47541bb1cf python aprotoc instead of jq and textproto
Bug: 273996308
Test: incremental_build.sh -c clean
Change-Id: I5b6c32667003010ccac5d3a46ac888f26788b3c5
2023-03-16 21:28:21 +00:00
Jeongik Cha
8d63d56ac6 Resolve metrics_proto warning
Bug: 271526845
Test: manually invoke internal protobuf presubmit
Change-Id: I79d1cc027e422fa1ca6106f5b45ef11045f6d486
2023-03-17 03:52:13 +09:00
Jeongik Cha
96d6227999 minor refactor ui/build/ninja.go
1. extract .ninja_log as const
2. log an error during reading .ninja_log

Test: m nothing
Bug: 271527305
Change-Id: I395dd8419620bfa9fad3af23c96e5a22ca44e2fb
2023-03-17 01:53:11 +09:00
Jeongik Cha
0cf44d5e8d Add --ninja_weight_source option
It has 3 options for now.
* ninja_log: uses ninja log file for data source
* evenly_distributed: pass empty list for ninja to consider every work as the same
* not_used: do not use critical path logic in ninja

In addition, I added the option in the metrics to track

Bug: 271527305
Test: m --ninja_weight_source=ninja_log|empty|not_used
Change-Id: Ib4c812c20606a34b17d3f0edb71057b477c4f90e
2023-03-16 19:52:27 +09:00
Jeongik Cha
767ce714bb Add time unit for CriticalPathInfo's fields' name
Bug: 271526845
Test: check metrics
Change-Id: Ie7e133033371906b29caad16d0b6bcd238b9f8db
2023-03-16 18:29:55 +09:00
Treehugger Robot
da4a78bf40 Merge "Add --multitree-build flag" 2023-03-16 02:34:22 +00:00
LaMont Jones
52a7243387 Add --multitree-build flag
Add a flag that enables logic specific to multitree-build builds.

Bug: b/266730267
Test: treehugger, manual
Change-Id: Ia6f392525faa9d66bb6e0692b2a2e9c3b8493b0d
2023-03-15 23:08:34 +00:00
Kevin Dagostino
24de5f2485 Merge "Cache GoogleProdCredsExist() to save ~800ms during builds." 2023-03-15 21:44:26 +00:00
Treehugger Robot
cd7db25738 Merge changes I7ec5a9f7,I76807f9b
* changes:
  Add docstrings for CriticalPathInfo
  Add long running jobs info
2023-03-15 07:26:40 +00:00
Jeongik Cha
cf833776a4 Add docstrings for CriticalPathInfo
Bug: 271526845
Test: n/a
Change-Id: I7ec5a9f7b39fd7441f1aac6556a77046a8210ae8
2023-03-15 12:54:14 +09:00
Jeongik Cha
4199d4796c Add long running jobs info
For the data to prioritize long running tasks in ninja, I added the
information about that.

Bug: 271526845
Test: build and check metrics
Change-Id: I76807f9b0d6a50d940b5f35dd6a9ef3cf693eec5
2023-03-15 12:46:58 +09:00
Jeongik Cha
58fa5a3f36 Merge "Add critical path infomation into metrics" 2023-03-15 03:40:11 +00:00
Jeongik Cha
28c1fe5bc8 Add critical path infomation into metrics
To improve build efficiency, the metrics for critical path and
parallelism ratio is necessary. That information has been included in
soong.log, so added it into metrics as well.

Bug: 271526845
Test: build and check if metrics pb has critical path info
Change-Id: I14e1a78c13d400b792d3b05df18604da48759ade
2023-03-14 17:59:22 +09:00
Kevin Dagostino
096ab2f5af Cache GoogleProdCredsExist() to save ~800ms during builds.
During a m nothing call; GoogleProdCredsExist check is called 3 times.
Each call is ~400ms so caching during the run will save ~800ms.

Test: time m nothing
Change-Id: I8a92c8fa48b22c7fb56be0f3c9cb5bf8787da6a9
2023-03-13 05:58:04 +00:00
Treehugger Robot
bb7c61ff61 Merge "Use Sec as well as Usec in ProcResInfo" 2023-03-10 22:19:33 +00:00
MarkDacek
a18ba22ba9 Add branch and manifest info to soong_metrics.
Bug: 259303480
Test: N/A
Change-Id: I434e24250ff640fddfca93ecb91fdfa49710e2f2
2023-03-07 21:55:26 +00:00
Jeongik Cha
bd465cca8a Use Sec as well as Usec in ProcResInfo
Timeval has two fields: Sec and Usec, but in previous implementation, it
only uses Usec, so Sec data is abandoned.

Bug: 271526845
Test: check metric, and if seconds data is reserved.
Change-Id: I911467f77b41995f6c833099648fe62fbc9909eb
2023-03-06 11:44:45 +09:00
Treehugger Robot
91108ad92a Merge "Add a post-build step for dist builds that records what changed in the build." 2023-03-02 18:53:38 +00:00
Christopher Parsons
7720f5704c Merge "Add a persistent bazel server between builds" 2023-03-02 14:22:34 +00:00
Chris Parsons
9402ca881c Add a persistent bazel server between builds
This feature is toggled on with USE_PERSISTENT_BAZEL, which is off by
default. Those that opt-in will have a bazel server running between
builds (with a 3hr default TTL) which will greatly improve analysis on
subsequent builds. (As Bazel maintains a cache of analysis results).

Bug: 266983462
Test: Manual `m nothing` runs (timing with and without the feature)
Test: New integration test
Test: Presubmits
Change-Id: I3af4948baa0c490e9b87c48ffdbe9f67732586c7
2023-03-01 23:30:19 -05:00
Treehugger Robot
732592c151 Merge "Create a ninja.environment file" 2023-03-01 03:22:31 +00:00