Commit graph

823 commits

Author SHA1 Message Date
Treehugger Robot
bb7c61ff61 Merge "Use Sec as well as Usec in ProcResInfo" 2023-03-10 22:19:33 +00:00
MarkDacek
a18ba22ba9 Add branch and manifest info to soong_metrics.
Bug: 259303480
Test: N/A
Change-Id: I434e24250ff640fddfca93ecb91fdfa49710e2f2
2023-03-07 21:55:26 +00:00
Jeongik Cha
bd465cca8a Use Sec as well as Usec in ProcResInfo
Timeval has two fields: Sec and Usec, but in previous implementation, it
only uses Usec, so Sec data is abandoned.

Bug: 271526845
Test: check metric, and if seconds data is reserved.
Change-Id: I911467f77b41995f6c833099648fe62fbc9909eb
2023-03-06 11:44:45 +09:00
Treehugger Robot
91108ad92a Merge "Add a post-build step for dist builds that records what changed in the build." 2023-03-02 18:53:38 +00:00
Christopher Parsons
7720f5704c Merge "Add a persistent bazel server between builds" 2023-03-02 14:22:34 +00:00
Chris Parsons
9402ca881c Add a persistent bazel server between builds
This feature is toggled on with USE_PERSISTENT_BAZEL, which is off by
default. Those that opt-in will have a bazel server running between
builds (with a 3hr default TTL) which will greatly improve analysis on
subsequent builds. (As Bazel maintains a cache of analysis results).

Bug: 266983462
Test: Manual `m nothing` runs (timing with and without the feature)
Test: New integration test
Test: Presubmits
Change-Id: I3af4948baa0c490e9b87c48ffdbe9f67732586c7
2023-03-01 23:30:19 -05:00
Treehugger Robot
732592c151 Merge "Create a ninja.environment file" 2023-03-01 03:22:31 +00:00
Spandan Das
2db59da973 Create a ninja.environment file
This contains the environment variables set by soong_ui during the ninja
execution. This file will be unused in single-tree builds, but will be
useful for multi-tree. The orchestrator will use this file to set the
correct environment for each inner tree build in the combined ninja
execution.

Test: m nothing compiles
Change-Id: I0570e34f51f426448464fb80030d4bea1cd52acb
2023-02-28 23:49:22 +00:00
Joe Onorato
7f29a66586 Add a post-build step for dist builds that records what changed in the build.
This will be used to skip tests when related files (or nothing) has changed.

Also minor refactoring of evaluateWhatToRun to put the logic all in one place.

Change-Id: Ia56b251716f6b6640bc5526a597cf7c066231e99
Test: go test, m (with soong tests), manually verify
2023-02-28 15:02:22 -08:00
Cole Faust
b1fbc79206 Add BUILD_BROKEN_PYTHON_IS_PYTHON2
This provides an escape hatch to undo the effects of
aosp/2417672. It's an enviornment variable instead of
a product config variable because python is often used
during product config.

Bug: 203436762
Test: m nothing and get_build_var with aosp/2458935
Change-Id: I0dd0f7c221427ea487fd7993aef68ce803eaff21
2023-02-27 13:02:02 -08:00
Nathan Egge
7b067fb753 Add JAVAC to the list of env variables to unset.
Some linux environments export JAVAC for legacy compatibility and this
 variable must be unset to ensure Soong and Make JAVAC values match.

Test: manual

Change-Id: Iae19fcb91b0a7651cbe00fe50f6daed54b4d7879
2023-02-17 18:07:04 +00:00
Cole Faust
64955b32bf Add status messages for RBE and product config
RBE adds a lot of time to m nothing builds,
so add status messages for it so that it's not misattributed
to some other part of the build.

Bug: 269644490
Test: m nothing
Change-Id: Ifff11bfc88f43b559526f3e3d3022c1388e42156
2023-02-16 21:06:19 +00:00
MarkDacek
00e3152aca Write bazel metrics file from soong.
Test: b build libcore:all
Change-Id: I372d48a9c7132bc57ecc1a8ba06eb098d28cbad7
2023-01-25 22:43:14 +00:00
Liz Kammer
7c7409d57d Merge "Change visibility of bp2build proto for dir rename" 2023-01-25 14:26:19 +00:00
Treehugger Robot
a413e4d28a Merge "Fix config file fetching" 2023-01-17 19:29:03 +00:00
Kousik Kumar
c881833bda Fix config file fetching
1. Make config file fetching run in background again if a file already
   exists - this behavior was changed in
   https://android-review.git.corp.google.com/c/platform/build/soong/+/2059887
   by accident.
2. Always try to load a config file even if fetching a config file fails
   - this behavior was changed in
     https://android-review.git.corp.google.com/c/platform/build/soong/+/2380792
     iiuc.

Bug: b/207551843
Test: Ran builds with no config file present, with config file present
and confirmed that config file fetching happens in the background now.

Change-Id: I6321684a0e17df3371f3f415602e7fa33cfba454
2023-01-16 17:20:19 +00:00
Usta Shrestha
da15c61add metrics: count symlink/mkdir in symlink forest
Bug: 260029212
Test: inspect out/bp2build_metrics.pb
Change-Id: Ia84c095a8d7b129cca629256b6c626c726cbfab1
2023-01-12 14:52:56 -05:00
MarkDacek
7901e585bd Fix upload-only to retrieve the METRICS_UPLOADER variable
in a timely manner.

There should not be harm in bypassing fetchEnvConfig for this case
as the metrics uploader should not be present for invalid cases.

Bug: b/264905338
Test: b build libcore:all
Test: printfs to verify that upload.go has the variable set.
Change-Id: Ia7d03f25e74d4ec2d6cb83793b793a23b47f26de
2023-01-10 20:17:22 +00:00
Jason Wu
205d1489fb Merge "Stop embedding soong_build_metrics in soong_metrics" 2023-01-10 02:53:16 +00:00
Jason Wu
41886f27aa Stop embedding soong_build_metrics in soong_metrics
Test: Run `m --bazel-mode-dev nothing` and query the database to make sure
soong_build_metrics will only update to its standalone column
Run `bp2build` and `b build` to make sure soong_build_metrics and
soong_metrics.soong_build_metrics will be null.

Bug: 262280865
Change-Id: Ice5115c468bda350b742c5a1d03fff66ad346cd6
2023-01-09 17:48:48 -05:00
MarkDacek
a3a62a37b4 Rename proto field.
Test: N/A
Change-Id: I97dd3bc26afcbef4ad69cb7a3cbad73bf34b5fe6
2023-01-04 22:04:44 +00:00
MarkDacek
6614d9cba3 Add upload-only mode and manual build-started-time flag.
This involved refactoring much of the main method in soong_ui/main.

Test: b build libcore:all
Test: use the build-time started flag and verify via printf that
it's the same across upload.go and metrics.SetBuildDateTimestamp()

Change-Id: Id7fe256337e8ee6c40542eba662c0eadb38e9674
2022-12-22 20:13:56 +00:00
MarkDacek
2ee2db3bf4 Add bazel metrics proto.
This is to marshal Bazel metrics for the sake of uploading to the
Android metrics pipeline.
Test: m nothing
Test: run the regen script

Change-Id: Ia4eca879cff07425c389e30a385418f9865159d6
2022-12-20 18:34:22 +00:00
Paul Crowley
e7e9e09128 Merge "Fix go link given when stubby is absent." 2022-12-15 18:42:59 +00:00
Jason Wu
f0ec631046 Upload soong_build_metrics
Test: m nothing
Test: m --bazel-mode-dev
Test: Verified soong_build_metrics is available in local metrics tables from builds that include this CL

Bug: 257537925
Change-Id: Ic5419ecb203c9b62457e1c553d592ba593a9bfff
2022-12-09 11:48:26 -05:00
MarkDacek
d0e7cd3d96 Add skip-metrics-upload flag to soong_ui.
Bug: b/261203925
Test: m libcore --skip-metrics-upload
Change-Id: I0b10027a0f359e682116c2ebb884808c65430dd0
2022-12-07 19:12:02 +00:00
Liz Kammer
798e96cc2c Change visibility of bp2build proto for dir rename
Test: CI
Change-Id: I3ba505c2e9fd23590531735ba6ae3aa003423cef
2022-12-06 09:51:17 -05:00
Spandan Das
c576383336 Create a new product variable to gate blueprint files
Users can use this feature by
1. Setting PRODUCT_INCLUDE_TAGS += <val> in their product mk files
2. Set
```
blueprint_packge_includes {
  match_al: ["<val>"],
}

other_module_type {name: foo}
other_module_type {name: bar}
```

bar and foo will be included if and only if <val> is set

Test: Unit tests in blueprint
Test: TH

Change-Id: I32eed4e3b5ac47fb565c62d13d8881fa984c86f4
2022-12-05 19:15:06 +00:00
MarkDacek
d06db5dd5b Add command line flag for extra bazel-built modules
Test: m libcore --bazel-mode-staging --bazel-force-enabled-modules=libcore and find out/bazel/ -name libcore*

Change-Id: I065696d06ce07e05300a41e133867a6e7a891b5e
2022-12-02 21:16:11 +00:00
Mark Dacek
ea80b16cd9 Merge "Add build-command arg to soong_ui." 2022-12-02 19:16:02 +00:00
Sasha Smundak
4cbe83ad5e Allow choosing which soong_build invocations to run in the debugger
When `SOONG_DELVE_STEPS` enviromnment variable is set in addition
to `SOONG_DELVE`, only given soong_build invocations ('steps') are
run in the debugger.
Also, make bootstrapBlueprint code table-driven.

Test: treehugger
Change-Id: Ia4016240ca9e88c2a85c6d06851c5bba30e7b2b5
2022-12-01 21:12:11 -08:00
MarkDacek
b96561eed1 Add build-command arg to soong_ui.
This is a duplicate of aosp/2313558 - this failed due to
a double-commit occurring somewhere in the chain.

Test: m nothing
Test: b build libcore:all
Change-Id: Ie5f1029429fa31170d1051ca3e7fc038ef637702
2022-12-02 04:34:43 +00:00
Paul Crowley
4965dc0cc3 Fix go link given when stubby is absent.
The link here named a heading that has been changed, and so no longer works.
Instead use a dedicated go link for this purpose.

Test: none
Change-Id: Icea2fe330d8c777593802d561afd5aa0f76c9ed0
2022-11-29 13:08:06 -08:00
Alexander Smundak
2f213d0946 Merge "Allow profiling Soong from the full build" 2022-11-22 18:52:25 +00:00
Paul Duffin
032734ca9f Merge "Remove support for unused env vars from sdk snapshot generation" 2022-11-22 15:56:04 +00:00
Sasha Smundak
faa97b7eb9 Allow profiling Soong from the full build
Test: treehugger
Change-Id: I5112364fff635b856a55744457f547d3eba91391
2022-11-21 10:56:51 -08:00
Treehugger Robot
bd12239355 Merge "remove dead Bazel code from soong_ui" 2022-11-16 15:27:42 +00:00
Sam Delmerico
d9a34353cf remove dead Bazel code from soong_ui
Test: CI
Change-Id: Ie9b56c59440ab63d421d19b382cf4f669a5ea215
2022-11-15 17:29:27 -05:00
Spandan Das
ed9729727c Merge changes from topics "nested-nsjail", "ro-api-surfaces-dir"
* changes:
  Special-case Soong finder to look in out/api_surfaces
  nsjail support verification should respect BUILD_BROKEN* flag for SrcDir
2022-11-15 19:56:23 +00:00
Usta (Tsering) Shrestha
4b67bb3013 Merge "re-trigger symlink-forest if bazel.list changes" 2022-11-15 16:09:56 +00:00
usta
40a8159a06 re-trigger symlink-forest if bazel.list changes
Bug: b/258870895
Test: run `m --bazel-mode nothing && touch bionic/docs/BUILD.bazel && m --bazel-mode nothing` Then verify symlink_forest reran and `out/soong/workspace/bionic/docs/BUILD.bazel` does NOT exist
Change-Id: I49797fe52b3e0fb6ffe17ed57b8da0b69628fd7e
2022-11-14 17:50:08 -05:00
Spandan Das
394aa32e8b Special-case Soong finder to look in out/api_surfaces
Add a new argument in soong_ui that will be used to gate this behavior.

This approach is expected to solve incrementality issues in multi-tree.
As part of soong's bootstrap process, all source dirs are added to deps
of out/soong/build.ninja (via globs). Since multitree_build writes to the
"source" api_surfaces directory, it changes its mtime and causes a
recompilation of out/soong/build.ninja in the subsequent invocation.

Test: TH (for single-tree)
Test: Inspected ninja files (for multi-tree)
(Run a full build)
touch out/api_surfaces/vendorapi/libc/.../math.h
orchestrator/prebuilts/build-tools/linux-x86/bin/nsjail --config
out/trees/vendor_aosp_cf_arm64_phone/nsjail.cfg --
prebuilts/build-tools/linux-x86/bin/ninja -f out/soong/bootstrap.ninja
-d explain -n out/soong/build.ninja
(ninja: no work to do)

Change-Id: Ib823163ec1153344a2f593daa8d7156c24ff5bc3
2022-11-14 21:10:38 +00:00
Spandan Das
2d997046ba nsjail support verification should respect BUILD_BROKEN* flag for SrcDir
This ensures that soong_ui is successful in setting up its own nsjail in
workflows that externally make the source tree ReadOnly (e.g. a nested
nsjail in multitree).

Test: TH
Change-Id: I6d0ec4a9fffda1d4e5996f475da611e1deb0888d
2022-11-14 19:50:40 +00:00
Liz Kammer
2af5ea85b0 Ignore bazel-generated paths in dangling rule test
Bazel is expected to generate some files and symlinks that will not be
created by aquery, we explicitly ignore files that are in bazel output
paths but are not in a generated file directory.

Test: $ m --bazel-mode-staging checkbuild and verify it gets past
   dangling rule test
Fixes: 258396112
Change-Id: I6dc45ea2613c9e1488ad3eda2313033c717217ff
2022-11-14 13:41:02 -05:00
Paul Duffin
82d75ade2a Remove support for unused env vars from sdk snapshot generation
The following environment variables are no longer used when generating
sdk snapshots:

* SOONG_SDK_SNAPSHOT_PREFER - we can control the modules that are
  preferred at build time now.

* SOONG_SDK_SNAPSHOT_USE_SOURCE_CONFIG_VAR - this is done as a
  post-processing step by the mainline_module_sdks.sh script to avoid
  having to regenerate the ninja files for every module.

* SOONG_SDK_SNAPSHOT_VERSION - support for this was removed a long time
  ago but was still referenced from dumpvars.go.

Bug: 259095197
Test: m nothing
      # Searched for uses of these variables but could not find any
      # outside the build.
Change-Id: Id2b32b29ff6f09faffea43292f549ba13acd956d
2022-11-14 17:45:07 +00:00
Chris Parsons
035e03affe Flip on Bazel mixed builds by default
This turns on Bazel-assisted builds for all Soong users.

If this CL breaks your build or your workflow, you may temporarily
disable Bazel mode by adding BUILD_BROKEN_DISABLE_BAZEL=1 to your
environment.

Test: TH
Test: Various off-band post-submit verification with "bazel staging
mode", which is an opt-in version of this feature.

Change-Id: Ifa44a79437ce76ea739ed661d4e2d567e0f4a4d5
2022-11-01 19:27:50 +00:00
Treehugger Robot
8238ae869e Merge "Add groundwork to easily all-enable bazel-mode" 2022-11-01 18:24:40 +00:00
Chris Parsons
b6e9690776 Add groundwork to easily all-enable bazel-mode
This will make it easy to change a "false" boolean to "true" to push
`--bazel-mode` by default for all users.

Users may disable bazel-by-default with BUILD_BROKEN_DISABLE_BAZEL, and
bazel is disabled by default on Darwin hosts (due to lack of test
coverage).

Bug: 254628592
Bug: 254084584
Test: Presubmits
Test: Presubmits for aosp/2263623
Change-Id: Iec712119e06ca8ec93028207d88277f879184cc2
2022-11-01 09:53:46 -04:00
usta
7f56eaa748 Fix collection of stale soong build metrics
Bug: b/256037411
Test: run mixed build twice and ensure soong_build_metric.pb file is absent on second run.
Change-Id: Ia11d850e1ad318e6383d8b392f20420259bc2f4d
2022-10-31 14:10:59 -04:00
usta
4f5d2c1e97 include symlink metrics in bp2build_metrics.pb
Bug: b/256212479
Test: Prior to thi CL bp2build.symlink_forest event was missing in bp2build_metrics.pb after a clean mixed build
Change-Id: I53bfc4114a383c0d1f9c4c7945e7d4c69bc50b0c
2022-10-31 11:38:10 -04:00