Commit graph

813 commits

Author SHA1 Message Date
Nathan Egge
7b067fb753 Add JAVAC to the list of env variables to unset.
Some linux environments export JAVAC for legacy compatibility and this
 variable must be unset to ensure Soong and Make JAVAC values match.

Test: manual

Change-Id: Iae19fcb91b0a7651cbe00fe50f6daed54b4d7879
2023-02-17 18:07:04 +00:00
Cole Faust
64955b32bf Add status messages for RBE and product config
RBE adds a lot of time to m nothing builds,
so add status messages for it so that it's not misattributed
to some other part of the build.

Bug: 269644490
Test: m nothing
Change-Id: Ifff11bfc88f43b559526f3e3d3022c1388e42156
2023-02-16 21:06:19 +00:00
MarkDacek
00e3152aca Write bazel metrics file from soong.
Test: b build libcore:all
Change-Id: I372d48a9c7132bc57ecc1a8ba06eb098d28cbad7
2023-01-25 22:43:14 +00:00
Liz Kammer
7c7409d57d Merge "Change visibility of bp2build proto for dir rename" 2023-01-25 14:26:19 +00:00
Treehugger Robot
a413e4d28a Merge "Fix config file fetching" 2023-01-17 19:29:03 +00:00
Kousik Kumar
c881833bda Fix config file fetching
1. Make config file fetching run in background again if a file already
   exists - this behavior was changed in
   https://android-review.git.corp.google.com/c/platform/build/soong/+/2059887
   by accident.
2. Always try to load a config file even if fetching a config file fails
   - this behavior was changed in
     https://android-review.git.corp.google.com/c/platform/build/soong/+/2380792
     iiuc.

Bug: b/207551843
Test: Ran builds with no config file present, with config file present
and confirmed that config file fetching happens in the background now.

Change-Id: I6321684a0e17df3371f3f415602e7fa33cfba454
2023-01-16 17:20:19 +00:00
Usta Shrestha
da15c61add metrics: count symlink/mkdir in symlink forest
Bug: 260029212
Test: inspect out/bp2build_metrics.pb
Change-Id: Ia84c095a8d7b129cca629256b6c626c726cbfab1
2023-01-12 14:52:56 -05:00
MarkDacek
7901e585bd Fix upload-only to retrieve the METRICS_UPLOADER variable
in a timely manner.

There should not be harm in bypassing fetchEnvConfig for this case
as the metrics uploader should not be present for invalid cases.

Bug: b/264905338
Test: b build libcore:all
Test: printfs to verify that upload.go has the variable set.
Change-Id: Ia7d03f25e74d4ec2d6cb83793b793a23b47f26de
2023-01-10 20:17:22 +00:00
Jason Wu
205d1489fb Merge "Stop embedding soong_build_metrics in soong_metrics" 2023-01-10 02:53:16 +00:00
Jason Wu
41886f27aa Stop embedding soong_build_metrics in soong_metrics
Test: Run `m --bazel-mode-dev nothing` and query the database to make sure
soong_build_metrics will only update to its standalone column
Run `bp2build` and `b build` to make sure soong_build_metrics and
soong_metrics.soong_build_metrics will be null.

Bug: 262280865
Change-Id: Ice5115c468bda350b742c5a1d03fff66ad346cd6
2023-01-09 17:48:48 -05:00
MarkDacek
a3a62a37b4 Rename proto field.
Test: N/A
Change-Id: I97dd3bc26afcbef4ad69cb7a3cbad73bf34b5fe6
2023-01-04 22:04:44 +00:00
MarkDacek
6614d9cba3 Add upload-only mode and manual build-started-time flag.
This involved refactoring much of the main method in soong_ui/main.

Test: b build libcore:all
Test: use the build-time started flag and verify via printf that
it's the same across upload.go and metrics.SetBuildDateTimestamp()

Change-Id: Id7fe256337e8ee6c40542eba662c0eadb38e9674
2022-12-22 20:13:56 +00:00
MarkDacek
2ee2db3bf4 Add bazel metrics proto.
This is to marshal Bazel metrics for the sake of uploading to the
Android metrics pipeline.
Test: m nothing
Test: run the regen script

Change-Id: Ia4eca879cff07425c389e30a385418f9865159d6
2022-12-20 18:34:22 +00:00
Paul Crowley
e7e9e09128 Merge "Fix go link given when stubby is absent." 2022-12-15 18:42:59 +00:00
Jason Wu
f0ec631046 Upload soong_build_metrics
Test: m nothing
Test: m --bazel-mode-dev
Test: Verified soong_build_metrics is available in local metrics tables from builds that include this CL

Bug: 257537925
Change-Id: Ic5419ecb203c9b62457e1c553d592ba593a9bfff
2022-12-09 11:48:26 -05:00
MarkDacek
d0e7cd3d96 Add skip-metrics-upload flag to soong_ui.
Bug: b/261203925
Test: m libcore --skip-metrics-upload
Change-Id: I0b10027a0f359e682116c2ebb884808c65430dd0
2022-12-07 19:12:02 +00:00
Liz Kammer
798e96cc2c Change visibility of bp2build proto for dir rename
Test: CI
Change-Id: I3ba505c2e9fd23590531735ba6ae3aa003423cef
2022-12-06 09:51:17 -05:00
Spandan Das
c576383336 Create a new product variable to gate blueprint files
Users can use this feature by
1. Setting PRODUCT_INCLUDE_TAGS += <val> in their product mk files
2. Set
```
blueprint_packge_includes {
  match_al: ["<val>"],
}

other_module_type {name: foo}
other_module_type {name: bar}
```

bar and foo will be included if and only if <val> is set

Test: Unit tests in blueprint
Test: TH

Change-Id: I32eed4e3b5ac47fb565c62d13d8881fa984c86f4
2022-12-05 19:15:06 +00:00
MarkDacek
d06db5dd5b Add command line flag for extra bazel-built modules
Test: m libcore --bazel-mode-staging --bazel-force-enabled-modules=libcore and find out/bazel/ -name libcore*

Change-Id: I065696d06ce07e05300a41e133867a6e7a891b5e
2022-12-02 21:16:11 +00:00
Mark Dacek
ea80b16cd9 Merge "Add build-command arg to soong_ui." 2022-12-02 19:16:02 +00:00
Sasha Smundak
4cbe83ad5e Allow choosing which soong_build invocations to run in the debugger
When `SOONG_DELVE_STEPS` enviromnment variable is set in addition
to `SOONG_DELVE`, only given soong_build invocations ('steps') are
run in the debugger.
Also, make bootstrapBlueprint code table-driven.

Test: treehugger
Change-Id: Ia4016240ca9e88c2a85c6d06851c5bba30e7b2b5
2022-12-01 21:12:11 -08:00
MarkDacek
b96561eed1 Add build-command arg to soong_ui.
This is a duplicate of aosp/2313558 - this failed due to
a double-commit occurring somewhere in the chain.

Test: m nothing
Test: b build libcore:all
Change-Id: Ie5f1029429fa31170d1051ca3e7fc038ef637702
2022-12-02 04:34:43 +00:00
Paul Crowley
4965dc0cc3 Fix go link given when stubby is absent.
The link here named a heading that has been changed, and so no longer works.
Instead use a dedicated go link for this purpose.

Test: none
Change-Id: Icea2fe330d8c777593802d561afd5aa0f76c9ed0
2022-11-29 13:08:06 -08:00
Alexander Smundak
2f213d0946 Merge "Allow profiling Soong from the full build" 2022-11-22 18:52:25 +00:00
Paul Duffin
032734ca9f Merge "Remove support for unused env vars from sdk snapshot generation" 2022-11-22 15:56:04 +00:00
Sasha Smundak
faa97b7eb9 Allow profiling Soong from the full build
Test: treehugger
Change-Id: I5112364fff635b856a55744457f547d3eba91391
2022-11-21 10:56:51 -08:00
Treehugger Robot
bd12239355 Merge "remove dead Bazel code from soong_ui" 2022-11-16 15:27:42 +00:00
Sam Delmerico
d9a34353cf remove dead Bazel code from soong_ui
Test: CI
Change-Id: Ie9b56c59440ab63d421d19b382cf4f669a5ea215
2022-11-15 17:29:27 -05:00
Spandan Das
ed9729727c Merge changes from topics "nested-nsjail", "ro-api-surfaces-dir"
* changes:
  Special-case Soong finder to look in out/api_surfaces
  nsjail support verification should respect BUILD_BROKEN* flag for SrcDir
2022-11-15 19:56:23 +00:00
Usta (Tsering) Shrestha
4b67bb3013 Merge "re-trigger symlink-forest if bazel.list changes" 2022-11-15 16:09:56 +00:00
usta
40a8159a06 re-trigger symlink-forest if bazel.list changes
Bug: b/258870895
Test: run `m --bazel-mode nothing && touch bionic/docs/BUILD.bazel && m --bazel-mode nothing` Then verify symlink_forest reran and `out/soong/workspace/bionic/docs/BUILD.bazel` does NOT exist
Change-Id: I49797fe52b3e0fb6ffe17ed57b8da0b69628fd7e
2022-11-14 17:50:08 -05:00
Spandan Das
394aa32e8b Special-case Soong finder to look in out/api_surfaces
Add a new argument in soong_ui that will be used to gate this behavior.

This approach is expected to solve incrementality issues in multi-tree.
As part of soong's bootstrap process, all source dirs are added to deps
of out/soong/build.ninja (via globs). Since multitree_build writes to the
"source" api_surfaces directory, it changes its mtime and causes a
recompilation of out/soong/build.ninja in the subsequent invocation.

Test: TH (for single-tree)
Test: Inspected ninja files (for multi-tree)
(Run a full build)
touch out/api_surfaces/vendorapi/libc/.../math.h
orchestrator/prebuilts/build-tools/linux-x86/bin/nsjail --config
out/trees/vendor_aosp_cf_arm64_phone/nsjail.cfg --
prebuilts/build-tools/linux-x86/bin/ninja -f out/soong/bootstrap.ninja
-d explain -n out/soong/build.ninja
(ninja: no work to do)

Change-Id: Ib823163ec1153344a2f593daa8d7156c24ff5bc3
2022-11-14 21:10:38 +00:00
Spandan Das
2d997046ba nsjail support verification should respect BUILD_BROKEN* flag for SrcDir
This ensures that soong_ui is successful in setting up its own nsjail in
workflows that externally make the source tree ReadOnly (e.g. a nested
nsjail in multitree).

Test: TH
Change-Id: I6d0ec4a9fffda1d4e5996f475da611e1deb0888d
2022-11-14 19:50:40 +00:00
Liz Kammer
2af5ea85b0 Ignore bazel-generated paths in dangling rule test
Bazel is expected to generate some files and symlinks that will not be
created by aquery, we explicitly ignore files that are in bazel output
paths but are not in a generated file directory.

Test: $ m --bazel-mode-staging checkbuild and verify it gets past
   dangling rule test
Fixes: 258396112
Change-Id: I6dc45ea2613c9e1488ad3eda2313033c717217ff
2022-11-14 13:41:02 -05:00
Paul Duffin
82d75ade2a Remove support for unused env vars from sdk snapshot generation
The following environment variables are no longer used when generating
sdk snapshots:

* SOONG_SDK_SNAPSHOT_PREFER - we can control the modules that are
  preferred at build time now.

* SOONG_SDK_SNAPSHOT_USE_SOURCE_CONFIG_VAR - this is done as a
  post-processing step by the mainline_module_sdks.sh script to avoid
  having to regenerate the ninja files for every module.

* SOONG_SDK_SNAPSHOT_VERSION - support for this was removed a long time
  ago but was still referenced from dumpvars.go.

Bug: 259095197
Test: m nothing
      # Searched for uses of these variables but could not find any
      # outside the build.
Change-Id: Id2b32b29ff6f09faffea43292f549ba13acd956d
2022-11-14 17:45:07 +00:00
Chris Parsons
035e03affe Flip on Bazel mixed builds by default
This turns on Bazel-assisted builds for all Soong users.

If this CL breaks your build or your workflow, you may temporarily
disable Bazel mode by adding BUILD_BROKEN_DISABLE_BAZEL=1 to your
environment.

Test: TH
Test: Various off-band post-submit verification with "bazel staging
mode", which is an opt-in version of this feature.

Change-Id: Ifa44a79437ce76ea739ed661d4e2d567e0f4a4d5
2022-11-01 19:27:50 +00:00
Treehugger Robot
8238ae869e Merge "Add groundwork to easily all-enable bazel-mode" 2022-11-01 18:24:40 +00:00
Chris Parsons
b6e9690776 Add groundwork to easily all-enable bazel-mode
This will make it easy to change a "false" boolean to "true" to push
`--bazel-mode` by default for all users.

Users may disable bazel-by-default with BUILD_BROKEN_DISABLE_BAZEL, and
bazel is disabled by default on Darwin hosts (due to lack of test
coverage).

Bug: 254628592
Bug: 254084584
Test: Presubmits
Test: Presubmits for aosp/2263623
Change-Id: Iec712119e06ca8ec93028207d88277f879184cc2
2022-11-01 09:53:46 -04:00
usta
7f56eaa748 Fix collection of stale soong build metrics
Bug: b/256037411
Test: run mixed build twice and ensure soong_build_metric.pb file is absent on second run.
Change-Id: Ia11d850e1ad318e6383d8b392f20420259bc2f4d
2022-10-31 14:10:59 -04:00
usta
4f5d2c1e97 include symlink metrics in bp2build_metrics.pb
Bug: b/256212479
Test: Prior to thi CL bp2build.symlink_forest event was missing in bp2build_metrics.pb after a clean mixed build
Change-Id: I53bfc4114a383c0d1f9c4c7945e7d4c69bc50b0c
2022-10-31 11:38:10 -04:00
Lukács T. Berki
4bb30b59a9 Merge changes from topic "separate-symlink-forest-invocation"
* changes:
  Build the symlink tree on multiple threads.
  Create Bazel symlink forest in a separate process.
2022-10-28 07:05:00 +00:00
Lukacs T. Berki
c541cd27fa Create Bazel symlink forest in a separate process.
This helps with incrementality a lot: the symlink forest must depend on
almost every directory in the source tree so that if a new file is added
or removed from *anywhere*, it is regenerated.

Previously, we couldn't do this without invoking bp2build, which is
quite wasteful because bp2build takes way more time than the symlink
forest creation, even though we do the latter in a very suboptimal way
at the moment.

This means that if a source file is added or removed (which does not
affect globs), we don't pay the cost of bp2build anymore.

Also refactored symlink_forest.go on the side. Too much state was being
passed around in arguments.

This change reimplements aosp/2263423 ; the semantics of not touching an
output file is the exact same as order-only inputs and the latter is a
bit fewer lines of code.

Test: Presubmits.
Change-Id: I565c580df8a01bacf175d56747c3f50743d4a4d4
2022-10-27 08:08:45 +00:00
Romain Jobredeaux
0a7529b02c Log whether user explicitly disabled mixed builds in Soong metrics.
Test: unit tests
Bug: 254649740
Change-Id: I2d4080955e10fe4b50e335c97c79dddbdd22c170
2022-10-26 15:06:33 -04:00
Joe Onorato
934977cbbe Merge "Move bazel from being a shell function to being a script at build/bazel/bin/bazel" 2022-10-25 17:56:58 +00:00
Liz Kammer
f2a80c6396 Add non-zero exit + error message to soong metrics
Improves some error messages that would have given little/no information

Bug: 254650145
Test: relevant unit tests
Test: Induce errors and check metrics proto
Change-Id: Ife6116af74af6e62c2f8ae8774e53c28178fb8d0
2022-10-24 10:51:56 -04:00
Joe Onorato
ba29f380d3 Move bazel from being a shell function to being a script at build/bazel/bin/bazel
Bug: 254713799
Test: b build //packages/modules/adb/apex:com.android.adbd
Change-Id: I6a0a89438a46d8252c3e0c24319038b0cf2dbd59
2022-10-24 06:38:11 -07:00
Romain Jobredeaux
7c4ef0718c Merge "Add Soong metric variable to indicate whether bazel mixed build was force-disabled." 2022-10-21 18:22:02 +00:00
Mark Dacek
99db041a14 Merge "Add bazel staging mode to soong build." 2022-10-21 17:38:43 +00:00
Romain Jobredeaux
ea098efcd8 Add Soong metric variable to indicate whether bazel mixed build was force-disabled.
Test: unit tests
Change-Id: I17b96edd28ace5a29a6889d6f003b6a38dbeb68e
Bug: 254649740
2022-10-21 15:50:01 +00:00
Jingwen Chen
3b13b61d9d Don't rely on TOP+OUT_DIR as the location for OUT_DIR.
OUT_DIR may not be in TOP (where the source tree is).

Test: presubmits
Test: set OUT_DIR=$HOME/out1; m --bazel-mode nothing

Fixes: 254022328
Change-Id: I8cf9fe2da3cdc0ba5489dfbe41d9d32f06ace83d
2022-10-21 09:39:12 +00:00