Commit graph

725 commits

Author SHA1 Message Date
MarkDacek
d33c2fd5a2 Add --bazel-exit-code parameter to soong_ui.
This is for use in tracking bazel exit codes from b invocations.

Bug: 279754118
Test: b build libcore:all (with prints to verify)
Change-Id: If36b5b73ff93cd15647e5c1c1f676e95137210ff
2023-05-04 21:27:34 +00:00
Mark Dacek
c0bdca9110 Merge "Fix instantiation of config.bazelForceEnabledModules" 2023-05-04 18:20:00 +00:00
Jeongik Cha
73d491196b Measure 'ninja_hint' time
Bug: 273282046
Test: m nothing and check out/soong_build_metrics.pb
Change-Id: I6440c3279b141c1f057145b668f8b96c45eaa75d
2023-05-04 18:16:11 +09:00
Jeongik Cha
291cc5f9d9 Merge "Use module type, size of srcs or deps to prioritize module" 2023-05-04 06:10:32 +00:00
MarkDacek
1de78f3174 Fix instantiation of config.bazelForceEnabledModules
Currently, uninstantiated, the field will be set to [""]
instead of an empty slice.

Test: m nothing --ensure-allowlist-integrity
Test: m nothing --ensure-allowlist-integrity
--bazel-force-enabled-modules=com.google.android.neuralnetworks (this
fails, as expected)

Change-Id: Ib4cd26f0cb0d40714b8c3f263ee2d22093ee15ef
2023-05-03 03:48:27 +00:00
MarkDacek
6f6b962bd0 Modify --ensure-allowlist-integrity call to avoid spurious errors.
We now ensure that the module is disabled and _not_ enabled before
failing the build.

Bug: 280439299
Test: m nothing --ensure-allowlist-integrity --bazel-mode-staging (on
master and aosp)

Change-Id: I59c969a27b064f07913a511cb29bc06d23daa371
2023-05-02 21:07:36 +00:00
Cole Faust
c9508aac4c Load starlark files from soong
There are a number of instances where we are exporting information
from soong to bazel via soong_injection. This could be more bazel-centric
if the information was instead held in bzl files, and both bazel and
soong read it from there.

Add a starlark package that will run
//build/bazel/constants_exported_to_soong.bzl at initialization time,
and then results can be retreived with GetStarlarkValue.

Since changes to the starlark files mean that soong has to rerun,
add them as ninja deps.

Unfortunately, the starlark code has to be run at runtime rather than
pregenerating their results, because tests run from intellij wouldn't
go through any pregeneration steps. This means that starlark is run
multiple times during the build, once per test package and once per
primary builder invocation. (currently 3, could be reduced to 2 if we
made the symlink forest generation into its own standalone tool) The
starlark code we have so far in this cl is very fast, roughly half a
millisecond, so it's not a big deal for now, but something to keep an
eye on as we add more starlark constants.

Bug: 279095899
Test: go test
Change-Id: I1e7ca1df1d8d67333cbfc46e8396e229820e4476
2023-04-26 17:18:19 -07:00
MarkDacek
f47e142ffd Refactor MixedBuildsEnabled and add --ensure-allowlist-integrity.
Currently, there is little verification around allowlisted modules
actually being mixed-built. This flag would allow us to verify
that a module allowlisted is mixed-built for at least one variant.

Bug: 278910100
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
--bazel-force-enabled-modules=com.google.android.neuralnetworks (This
fails, as expected)
Test: build/soong/test/mixed_mode_test.sh

Change-Id: Icd5976f4f44f1a8caca1e5247d986642f7995f97
2023-04-25 20:55:07 +00:00
Alix
c6a918aa11 add support for escaping for zip2zip
Change-Id: I6c2f39e24247752de065fd3116f670639322b943
Bug: 270896651
Test: zip2zip tests
2023-04-17 14:58:02 +00:00
Jeongik Cha
b745e2e205 Use module type, size of srcs or deps to prioritize module
Instead of listing up module names, use another information from soong
such as module type, or size of inputs.

I focused on recall, not precision or accuracy, because false negative
module can slow down whole build

Bug: 273282046
Test: m --ninja_weight_source=ninja_log
Change-Id: I8fcc26f1312fda36197c787d240bc3d7b5346481
2023-04-15 00:58:15 +09:00
Steven Moreland
97f5966f49 Merge "sbox: print more errors" 2023-03-30 18:25:59 +00:00
Treehugger Robot
bc2906efef Merge "Modify Soong to utilize from-text android.jar in build" 2023-03-29 20:31:20 +00:00
Treehugger Robot
998fdd7e37 Merge "support PRODUCT_SOURCE_ROOT_DIRS product variable" 2023-03-29 17:04:37 +00:00
Steven Moreland
ee975ee3da sbox: print more errors
The output directory is deleted, so when I want to
see the files there, it can really be helpful to
see more.

Bug: N/A
Test: N/A
Change-Id: Ie40c412fe1bebbf87db00a4dbce107696ab2805e
2023-03-28 22:39:00 +00:00
Jihoon Kang
1bff0349d4 Modify Soong to utilize from-text android.jar in build
Context
- from-text android.jar files are built using Metalava, and these can be
  utilized in `decodeSdkDep` so that any modules that depends on APIs
  can be compiled using from-text android.jars
- This change removes dependency on source java files when compiling
  stub android.jar files

Implementation
- Modify java_api_library module to create system modules using the
  generated android.jar
- Replace modules in decodeSdkDep to link against java_api_library
  modules
- Add --build-from-text-stub flag to hide the feature behind a flag

Test: m --build-from-text-stub
Bug: 271154441
Change-Id: I104df595edc65c0006820d5ae5b15f1fb167e190
2023-03-28 21:53:45 +00:00
Jeongik Cha
56df091391 Merge changes from topic "b/273282046"
* changes:
  Add HINT_FROM_SOONG option for ninja weight list
  Add EXTERNAL_FILE option for ninja weight list
2023-03-28 03:51:20 +00:00
Sam Delmerico
98a7329d59 support PRODUCT_SOURCE_ROOT_DIRS product variable
Soong analyzes the entire source tree even though not every lunch target
needs to know about every module. For example, OEM sources can be
ignored for cuttlefish products. This functionality allows blueprint to
ignore a list of undesired directories.

Bug: 269457150
Change-Id: I1eec5d7b6a268cae4c633d8d89ed485598ebca45
2023-03-27 14:42:36 -04:00
Treehugger Robot
401a1cf0c3 Merge "Fix possible race condition" 2023-03-23 15:35:00 +00:00
Jeongik Cha
036b5a3ed3 Fix possible race condition
Test: m
Bug: 271526845
Change-Id: I8b8d57c45d9be3b538422588521bcfd85b61aa56
2023-03-23 17:34:24 +09:00
Jeongik Cha
e114e60615 Add HINT_FROM_SOONG option for ninja weight list
If this option is set, Soong generates ninja weight list including
modules in HugeModulesList in allowlists.go

Test: m --ninja_weight_source=soong
Bug: 273282046
Change-Id: Id92b7f9f9e8152c1c46ae071c5821a479cf47bce
2023-03-23 01:45:27 +09:00
Pranav Gupta
51645ff23e Add skip-sdk-check to extract_apks
Add skip-sdk-check to skip checking the SDK version when extracting an
APK/APEX from an App Set Bundle. This can be used when the platform SDK
version is not defined and the APEXs/APKs use SHA based SDK versions.
This check should not be set to true for non Beta dessert releases

Bug: 274518686
Test: # Add SHA targeting modules to platform
m SOONG_SKIP_APPSET_SDK_CHECK=true  #Build Passes
m SOONG_SKIP_APPSET_SDK_CHECK=false #Build Fails
m #No config supplied, build fails

Change-Id: I1919437d3410f09c991e1de39031bd88e1f8246a
2023-03-21 08:13:25 -07:00
LaMont Jones
52a7243387 Add --multitree-build flag
Add a flag that enables logic specific to multitree-build builds.

Bug: b/266730267
Test: treehugger, manual
Change-Id: Ia6f392525faa9d66bb6e0692b2a2e9c3b8493b0d
2023-03-15 23:08:34 +00:00
Jeongik Cha
58fa5a3f36 Merge "Add critical path infomation into metrics" 2023-03-15 03:40:11 +00:00
Jeongik Cha
28c1fe5bc8 Add critical path infomation into metrics
To improve build efficiency, the metrics for critical path and
parallelism ratio is necessary. That information has been included in
soong.log, so added it into metrics as well.

Bug: 271526845
Test: build and check if metrics pb has critical path info
Change-Id: I14e1a78c13d400b792d3b05df18604da48759ade
2023-03-14 17:59:22 +09:00
Spandan Das
98cb85624c Generate a BUILD file for every Android.bp file in api_bp2build
workspace.

This is necessary to solve bazel package boundary issues where the api
file might exist in a different directory

Test: m api_bp2build && build/bazel/bin/bazel build --config=android
--config=api_bp2build //build/orchestrator/apis:system
Test: multitree_build system/nothing (in multitree)

Change-Id: Id64085d65a1943bdb394ea80c875db96ca373839
2023-03-09 23:15:34 +00:00
Jingwen Chen
cf9e2c8b23 Don't create bp2build_all_srcs filegroups in queryview.
These are not necessary for queryview and can lead to inaccurate query results.

Test: m queryview; grep bp2build_all_srcs
Change-Id: Ifaf65b4ec5828718c2becb39b4507cf033ea4546
2023-03-08 13:32:35 +00:00
Jason Wu
074c8c3b61 Merge "Upload soong_build_metrics when user builds" 2023-03-02 20:42:00 +00:00
Chris Parsons
9402ca881c Add a persistent bazel server between builds
This feature is toggled on with USE_PERSISTENT_BAZEL, which is off by
default. Those that opt-in will have a bazel server running between
builds (with a 3hr default TTL) which will greatly improve analysis on
subsequent builds. (As Bazel maintains a cache of analysis results).

Bug: 266983462
Test: Manual `m nothing` runs (timing with and without the feature)
Test: New integration test
Test: Presubmits
Change-Id: I3af4948baa0c490e9b87c48ffdbe9f67732586c7
2023-03-01 23:30:19 -05:00
Jason Wu
d125440aff Upload soong_build_metrics when user builds
Test: m nothing and check the events in database
Bug: 269751664
Change-Id: Ic07a6e1797b11f5413d7ecd78969dec64e9eeac9
2023-03-01 20:26:30 -05:00
Cole Faust
81f170f6bc Disable RBE in multiproduct_kati
RBE has signifigant performance overhead during
analysis time, in theory to reduce execution time.
But multiproduct_kati only tests analysis, and does
not run any execution, so RBE is just slowing us
down.

Bug: 229863781
Bug: 265603584
Test: time ./build/soong/build_test.bash  --products aosp_arm64 on stage_aosp_master
Change-Id: Icbb2f001fee03a75bad37c14d08a203be20ac6cf
2023-02-17 11:36:35 -08:00
Cole Faust
aafdf54f70 Merge "Export apex_available_baseline to soong_injection" 2023-02-09 18:39:27 +00:00
Cole Faust
9e384e2e6b Export apex_available_baseline to soong_injection
So that it can be used in the bazel implementation.

Bug: 268006095
Test: m nothing, check that it's in out/soong/soong_injection
Change-Id: I1520dd874076dee7fa083648d0790b060d658e5a
2023-02-08 17:43:09 -08:00
Jason Wu
51d0ad7c30 Upload soong_metrics when mixed build fails
Test:  `m --bazel-mode-dev` and intentionally break the build.
We can see soong_metrics in the output directory and metrics_uploader
executed.

Bug: 268228148
Change-Id: I79e3b11468fca446673ebf752697a99eaff01b0f
2023-02-08 18:13:43 -05:00
Usta (Tsering) Shrestha
3a49e9a068 Merge "reduce forest generation to be incremental" 2023-02-08 18:46:16 +00:00
Jason Wu
7796b11726 Temporary fix to make soong_metrics files get uploaded
Test: ```m nothing``` and verify the data in local builds table
Bug: 268228148

Change-Id: Idb4b5305f19cf4a69a60896555357623a1ce4532
2023-02-08 01:34:20 +00:00
Usta (Tsering) Shrestha
c4c07b12b6 reduce forest generation to be incremental
Previously, symlink forest generation involved removing the entire
symlink forest and recreating it from scratch. With this change,
a) symlinks which need not change are untouched,
b) symlinks pointing to the wrong location are fixed, and
c) symlinks which should no longer exist are removed.

On AOSP on my local machine, this reduces the symlink forest generation
step from 2.5s to 1.1s clean, and 0.6s when a single file is added to
a source directory.

Bug: 257528847
Test: m bp2build, touch `fakefile` under the forest, remove a file
from the source tree, rerun m bp2build. Manually verify the new forest
does not retain the link to the deleted source file, and that fakefile
no longer exists in the forest.

Change-Id: I481371ae487e9419af6a3a4370c552578b07d650
2023-02-07 06:23:43 +00:00
Cole Faust
5c503d1c43 Precompile python sources
This signifigantly improves the startup time of soong-built
python binaries. For example, running
`m apexer && time out/host/linux-x86/bin/apexer` gives
0.734s before this cl, and 0.094s after.

Fixes: 259718110
Test: Presubmits
Change-Id: Ib19e83e2c60c39a849525be117279c318de3afa7
2023-01-27 15:43:38 -08:00
MarkDacek
00e3152aca Write bazel metrics file from soong.
Test: b build libcore:all
Change-Id: I372d48a9c7132bc57ecc1a8ba06eb098d28cbad7
2023-01-25 22:43:14 +00:00
Usta (Tsering) Shrestha
5c1b0ba692 Merge "metrics: count symlink/mkdir in symlink forest" 2023-01-13 04:01:55 +00:00
Usta Shrestha
da15c61add metrics: count symlink/mkdir in symlink forest
Bug: 260029212
Test: inspect out/bp2build_metrics.pb
Change-Id: Ia84c095a8d7b129cca629256b6c626c726cbfab1
2023-01-12 14:52:56 -05:00
Cole Faust
2b4bc2fddb Merge "Increase ram usage estimate in multiproduct_kati" 2023-01-12 17:33:57 +00:00
Cole Faust
36b259f1e9 Increase ram usage estimate in multiproduct_kati
multiproduct_kati assumed each build would take ~30GB of RAM,
but it seems it's closer to 40GB now.

The current build_test builds use 7 jobs, so this would bring
us down to 5, which should still keep us under the 6 hour
build time limit as currently these tests are at ~3 hours.

Bug: 262629589
Test: Presubmits
Change-Id: I51af364cc602a3bf7ba85fd8f92c4dd57585abbd
2023-01-11 15:06:50 -08:00
Spandan Das
255648c6d5 Create a smoke test for api_bp2build workspace
This test builds an "empty" filegroup. Running it can be useful to
verify that the api_bp2build workspace has been setup correctly.

Test: TH
Change-Id: I78639a2dcdeb659b2d6fdf6abc85d85c4fb697bf
2023-01-11 03:21:18 +00:00
Spandan Das
83e787e837 Add a new CreateSoongInjectionDirFiles function
- This is a wrapper function for all files in soong_injection directory.
This should prevent an error in the workspace generated for
api_bp2build.
- Rename the existing CreateSoongInjectionFiles fn and make it package
  private to prevent confusion

(The subsequent CL in this stack should contain a smoke test for
api_bp2build)

Test: b build //:empty --config=api_bp2build --config=android
Test: TH
Change-Id: Iddb0aa1aff2f709826edd587aa99fccddf80f08f
2023-01-11 03:21:00 +00:00
Cole Faust
b85d1a15cc Bp2build support for multiple product configs
Create a
build/bazel/product_config/generated/products/<product_name>/BUILD
file that contains the platform definitions needed for
a particular product. Currently we just create it for the
current lunch target, but the idea is that eventually when
all product config is in starlark, all the products will
have their platform definitions in the tree at once.

Bug: 249685973
Test: Presubmits
Change-Id: I08c82ff28dcf62f09d3b1d2e3186a6b961e12f6e
2023-01-05 12:49:14 -08:00
Usta (Tsering) Shrestha
253171b016 Merge "Use comma not pipe in switch-case" 2023-01-04 15:43:57 +00:00
Usta Shrestha
7fae695336 Use comma not pipe in switch-case
brings down time to complete by 40%

Bug: NA
Test: m queryview and m soong_docs
Change-Id: I3681e2f5015fc1f5cabaf0462d4d52a4fe9db56c
2023-01-03 21:24:18 +00:00
MarkDacek
6614d9cba3 Add upload-only mode and manual build-started-time flag.
This involved refactoring much of the main method in soong_ui/main.

Test: b build libcore:all
Test: use the build-time started flag and verify via printf that
it's the same across upload.go and metrics.SetBuildDateTimestamp()

Change-Id: Id7fe256337e8ee6c40542eba662c0eadb38e9674
2022-12-22 20:13:56 +00:00
Jason Wu
1c6ca94935 Merge "Delete Stale Metrics Files" 2022-12-20 15:56:21 +00:00
Jason Wu
cc166a77bd Delete Stale Metrics Files
Test: Run `m --bazle-mode-dev nothing` again  after `m --bazel-mode-dev nothing`.
bp2build_metrics.pb no longer exists, the rest metrics files
have the timestamp of the latest creation time.

Bug: 256617990
Change-Id: I869d1b9c3abcb517744e66b21e5c12d6b5b1e97b
2022-12-19 21:16:50 +00:00
Sasha Smundak
1845f42085 Simplify soong_build top-level code
* `android.Context` holds `android.Config`, so provide an accessor to it
  and avoid passing them separately to a lot of functions.
* factor common code in `readBazelPaths`/`getExistingBazelRelatedFiles`
  into `readFileLines`
* refactor check-error-and-quit into `maybeQuit`
* use switch instead of if-elseif-... where appropriate
* rearrange the code in `runApiBp2build`, `runSymlinkForestCreation`

Test: treehugger
Change-Id: I639929c21ec4999cdfd737c07136f32df2d488bc
2022-12-16 11:03:17 -08:00
Sasha Smundak
af5ca926be Keep all Soong command line arguments in a single structure
Define a structure android.CmdArgs and keep all soong_build command line
arguments used to build a configuration in it. Some of them related to BazelMode
were previously kept in Blueprint's bootstrap.Args where they do not belong.
This simplifies the code and lays the gound work for future simplifying changes.

Test: treehugger
Change-Id: I995871f4e6b0e95cc9df65696ad2020df221d8c8
2022-12-16 11:02:12 -08:00
Jason Wu
f0ec631046 Upload soong_build_metrics
Test: m nothing
Test: m --bazel-mode-dev
Test: Verified soong_build_metrics is available in local metrics tables from builds that include this CL

Bug: 257537925
Change-Id: Ic5419ecb203c9b62457e1c553d592ba593a9bfff
2022-12-09 11:48:26 -05:00
Mark Dacek
a366e48bc3 Merge "Add skip-metrics-upload flag to soong_ui." 2022-12-07 19:18:12 +00:00
MarkDacek
d0e7cd3d96 Add skip-metrics-upload flag to soong_ui.
Bug: b/261203925
Test: m libcore --skip-metrics-upload
Change-Id: I0b10027a0f359e682116c2ebb884808c65430dd0
2022-12-07 19:12:02 +00:00
Treehugger Robot
a0206f3d29 Merge "Create soong_injection files in ReadWrite mode" 2022-12-07 17:17:12 +00:00
Spandan Das
067210f70d Create soong_injection files in ReadWrite mode
The synthetic soong_injection workspace is shared between api_bp2build
(used in API export) and bp2build (used in API domain analysis which
runs mixed builds by default). The former creates them in ReadOnly mode,
but the latter creates them in ReadWrite mode (to allow users to
edit/experiment). To prevent errors in the end-to-end builds in
multittree, make api_bp2build create these files in ReadWrite mode as
well.

Test: mulittree_build locally
Change-Id: I8766d88951a3b625739e4dc12d468450ad7322ea
2022-12-07 01:25:03 +00:00
Spandan Das
c576383336 Create a new product variable to gate blueprint files
Users can use this feature by
1. Setting PRODUCT_INCLUDE_TAGS += <val> in their product mk files
2. Set
```
blueprint_packge_includes {
  match_al: ["<val>"],
}

other_module_type {name: foo}
other_module_type {name: bar}
```

bar and foo will be included if and only if <val> is set

Test: Unit tests in blueprint
Test: TH

Change-Id: I32eed4e3b5ac47fb565c62d13d8881fa984c86f4
2022-12-05 19:15:06 +00:00
MarkDacek
d06db5dd5b Add command line flag for extra bazel-built modules
Test: m libcore --bazel-mode-staging --bazel-force-enabled-modules=libcore and find out/bazel/ -name libcore*

Change-Id: I065696d06ce07e05300a41e133867a6e7a891b5e
2022-12-02 21:16:11 +00:00
Usta (Tsering) Shrestha
93b2a9b99f metric namespace for symlink_forest
Change-Id: Ifaa734a222bf37bccc3a74724f597f2d55200d6b
Test: run a build and inspeck event keys in bp2build_metrics.pb
Bug: 260936908
2022-12-01 06:00:27 +00:00
Treehugger Robot
b4bac94eb5 Merge "Copy bazelrc files into the queryview workspace" 2022-11-30 18:36:48 +00:00
Treehugger Robot
e0229e6855 Merge "Add the symlinks of the external repository in api_bp2build workspace" 2022-11-30 17:38:09 +00:00
Treehugger Robot
9b2e24c318 Merge "Exclude prebuilts/clang/host/linux-x86/clang-dev/BUILD from the symlink forest." 2022-11-30 11:51:59 +00:00
Jingwen Chen
bcfadce598 Exclude prebuilts/clang/host/linux-x86/clang-dev/BUILD from the symlink forest.
Background: toolchain/llvm_android/test_compiler.py generates clang-dev, a directory
symlink pointing to out/install/linux-x86/clang-dev, which itself
contains a BUILD file. This BUILD file should be excluded from the
symlink forest because it's not an allowlisted BUILD file.

Problem: However, our current symlink forest logic and finder logic doesn't
handle this situation. It's not in bazel.list:

```
$ less out/.module_paths/bazel.list | grep prebuilts/clang
prebuilts/clang/host/linux-x86/BUILD.bazel
prebuilts/clang/host/linux-x86/cc_toolchain_config.bzl
prebuilts/clang/host/linux-x86/cc_toolchain_constants.bzl
prebuilts/clang/host/linux-x86/cc_toolchain_features.bzl
prebuilts/clang/host/linux-x86/clang-r450784d/BUILD.bazel
prebuilts/clang/host/linux-x86/clang-r450784e/BUILD.bazel
prebuilts/clang/host/linux-x86/clang-r458507/BUILD.bazel
prebuilts/clang/host/linux-x86/clang-r468909/BUILD.bazel
prebuilts/clang/host/linux-x86/clang-r468909b/BUILD.bazel
prebuilts/clang/host/linux-x86/clang-r475365/BUILD.bazel
```

The symlink forest logic uses this bazel.list file and matches it
against the keepExistingBuildFile list to exclude BUILD files from the
symlink forest. Since clang-dev/BUILD.bazel is not in bazel.list, it's
symlinked into the forest.

All of these gymnastics could be avoided if we added custom BUILD file
name support to Bazel, and only symlinked (e.g.) Android.bazel and
ignored all other BUILD files in the tree. It would be very clear which
checked-in BUILD files we want to use in the symlink forest.

I think we should pursue custom BUILD file names instead of adding more
complexities, to say, the finder logic, to support such a use case. It's
also why I decided to add this hardcoded exclusion to keep the
workaround simple.

Bug: 260809113
Test: presubmits
Test: $ DIST_DIR=/tmp/dist prebuilts/python/linux-x86/bin/python3 toolchain/llvm_android/test_compiler.py --build-only --target aosp_raven-userdebug --no-clean-built-target --module dist --module droid --module tidy-soong_subset --with-tidy ./
Change-Id: I415371543585c1c5e8e00e6958105f65ea5978ee
2022-11-30 08:44:05 +00:00
Spandan Das
6b91a38b51 Add the symlinks of the external repository in api_bp2build workspace
Unlike bp2build, api_bp2build depends on a well-defined subset of checked-in
BUILD/bzl files, and we create symlinks for that subsset in
out/soong/api_bp2build. Since we now cd into the workspace after
aosp/2308141, we need to add the sylimks of the external repositories as
well

Test: bazel build --config=api_bp2build //:empty
Change-Id: Ia7ecc5543226893a3878bc0fd35e025a37c80413
2022-11-30 02:22:57 +00:00
Cole Faust
6d51e731a6 Copy bazelrc files into the queryview workspace
They are still read when using bazel with queryview, so they
need to be in the workspace so that they can be found.

Bug: 234449134
Test: m queryview && ./build/bazel/bin/bazel query --config=queryview //...
Change-Id: I1f2187dffd5a0e6cfb44ebe0c6738fa9bbef509d
2022-11-29 12:46:48 -08:00
Jason Wu
47d7ccfffd Merge "Upload bp2build_metrics file" 2022-11-29 18:50:32 +00:00
Jason Wu
c4a03138e0 Upload bp2build_metrics file
Test: m nothing
Test: m --bazel-mode-dev
Test: Verified bp2build_metrics is available in local metrics tables from builds that include this CL

Bug: 257537925

Change-Id: I09617a7850f1471f18192a0418396126fd4c98ef
2022-11-29 16:18:45 +00:00
Anton Hansson
c5cee23d16 Merge "extract_apks matches APKs with >= 1 matching ABI" 2022-11-23 16:29:31 +00:00
Sam Delmerico
b48d57bdcd extract_apks matches APKs with >= 1 matching ABI
Prior to this change, the bundletool and extract_apks tools require that
all ABIs that an APEX or APK provides must be compatible with the
TargetConfig. Instead, this change allows an APK to be selected if it
has at least one compatible ABI with the TargetConfig.

Bug: 260115309
Test: go test .
Change-Id: If67ce8128099611257a834862295a2bf5fa427d3
2022-11-22 17:53:26 -05:00
Sasha Smundak
4975c82666 Provide Bazel timing breakdown for the mixed builds
Instead of single event in the build metrics
```
| soong_build.mixed_build.bazel          | 1:17.1 |
```
it will now show:
```
| soong_build.mixed_build.bazel          | 1:17.1 |
| soong_build.mixed_build.bazel.cquery   |   51.9 |
| soong_build.mixed_build.bazel.aquery   |   23.1 |
| soong_build.mixed_build.bazel.symlinks |    2.1 |
```

Test: treehugger
Change-Id: I84a7dda6e3122860da9aaa98bfa6afe33d392dcf
2022-11-16 18:44:08 -08:00
Spandan Das
ed9729727c Merge changes from topics "nested-nsjail", "ro-api-surfaces-dir"
* changes:
  Special-case Soong finder to look in out/api_surfaces
  nsjail support verification should respect BUILD_BROKEN* flag for SrcDir
2022-11-15 19:56:23 +00:00
Paul Duffin
c467a98bc0 Merge "Build mode function should return output file" 2022-11-15 09:37:52 +00:00
Paul Duffin
54616f20b0 Merge "Stop creating unnecessary Context objects" 2022-11-14 22:30:53 +00:00
Paul Duffin
2335544ef2 Merge "Avoid writing soong_build_metrics.pb multiple times" 2022-11-14 22:30:41 +00:00
Paul Duffin
3d12f4e997 Merge "Group mixed mode and normal soong build" 2022-11-14 22:30:26 +00:00
Spandan Das
394aa32e8b Special-case Soong finder to look in out/api_surfaces
Add a new argument in soong_ui that will be used to gate this behavior.

This approach is expected to solve incrementality issues in multi-tree.
As part of soong's bootstrap process, all source dirs are added to deps
of out/soong/build.ninja (via globs). Since multitree_build writes to the
"source" api_surfaces directory, it changes its mtime and causes a
recompilation of out/soong/build.ninja in the subsequent invocation.

Test: TH (for single-tree)
Test: Inspected ninja files (for multi-tree)
(Run a full build)
touch out/api_surfaces/vendorapi/libc/.../math.h
orchestrator/prebuilts/build-tools/linux-x86/bin/nsjail --config
out/trees/vendor_aosp_cf_arm64_phone/nsjail.cfg --
prebuilts/build-tools/linux-x86/bin/ninja -f out/soong/bootstrap.ninja
-d explain -n out/soong/build.ninja
(ninja: no work to do)

Change-Id: Ib823163ec1153344a2f593daa8d7156c24ff5bc3
2022-11-14 21:10:38 +00:00
Paul Duffin
3f7bf9fa0d Allow namespace config to be tested properly
Previously, the NewTextContext created a context that always called
NewNameResolver with a export filter that always returned true.

This change fixes that by:
1. Changing NewNameResolver to take a Config parameter instead of a
   filter parameter and pushing the code to create the filter from the
   Config from newNameResolver() in cmd/soong_build/main.go into the
   NewNameResolver function.

2. Extracting a newTestContextForFixture that does not create a
   NameResolver or set it on the context. That avoids creating a
   NameResolver before the test has prepared the config.

3. Modify the fixture to create and set the NameResolver in the Context
   after the config has been prepared by the test.

4. Added test to verify that it all works.

Bug: 234825639
Test: m nothing
Change-Id: Ie4b13f18093dc01a0ab65a9ecfd143585d843b55
2022-11-08 17:38:56 +00:00
Paul Duffin
b713ddfcc1 Build mode function should return output file
Each build mode has to provide an output file, which also has a
corresponding dependency file (<output file>.d). Previously, some of
the build mode functions returned that file, others just returned and
relied on the doChosenActivity() method to return the file.

That was confusing as the same file name was referenced from two
separate places. This change makes all the build mode functions return
their output file explicitly and the doChosenActivity()just returns
what it receives. It also moves the writeDepsFile call to adjacent to
the return.

Bug: 257650737
Test: m nothing
Change-Id: I35f96ab404d9ffb1da747a3ecb8fc4e1feb9c026
2022-11-08 10:42:55 +00:00
Paul Duffin
b4e8d91d72 Stop creating unnecessary Context objects
Previously, the Context object created in the main() method was
initialized (ctx.Register()) for the main soong build. Build modes
that did not want that build modes that did not want that (symlink,
bp2build, apiBp2build) had to create their own context, or in the
case of runSymlinkForestCreation its own EventHandler. That was
very confusing.

This change avoids that by simply pushing the call to ctx.Register()
into the build modes that require it and that allows the other build
modes to use the main context and its EventHandler.

One point to note is that to ensure the exact same initialization as
before the runApiBp2build has to explicitly call the following:
    ctx.SetAllowMissingDependencies(false)
That is because the main context could have that set to true.

Bug: 257650737
Test: m nothing
Change-Id: Iee239fb87edf443fed65156fa14b8a30c89a2328
2022-11-08 10:42:55 +00:00
Paul Duffin
39eae8fb90 Avoid writing soong_build_metrics.pb multiple times
Previously, running `m nothing`	would write the	soong_build_metrics.pb
file 3 times:
1. For bp2build - it contains a single `soong_build` event as the rest
   is written to the `bp2build_metrics.pb` file.
2. For Bazel symlink forrest - it too contains a single `soong_build`
   event as the rest is appended to the `bp2build_metrics.pb` file.
3. For soong_build proper (including mixed builds) - it contains useful
   information.

This change avoids that by pushing the code to write the
soong_build_metrics.pb file from the top level where it applies to all
build modes into the build mode specific function that needs it.

Bug: 257590265
Test: m nothing
      # Run with some logging to make sure the file is only created once.
Change-Id: Iee239fb87edf443fed65156fa14b8a30c89a2328

Change-Id: I09724f6143352de6b7d6fb29eaf3fbed5a1abd21
2022-11-08 10:42:55 +00:00
Paul Duffin
0c09a43cef Group mixed mode and normal soong build
Mixed mode is a special form of the soong build. They both share common
initialization and metrics so they should be grouped together to avoid
duplicating that common behavior.

Bug: 257650737
Test: m nothing
Change-Id: I939b6264f94b69825735dce895132f33401238b6
2022-11-08 10:41:55 +00:00
Paul Duffin
f490656a0f Stop LOG_DIR changes causing bp2build to rerun
Retrieving an environment var through Config.Getenv() automatically
adds it to a soong.environment.used* file along with its current value.
Changes to its current value will trigger a rebuild of the appropriate
target. The LOG_DIR is a sub directory of DIST_DIR and on build servers
DIST_DIR encodes the build number making it unique every time. So,
using Config.Getenv("LOG_DIR") will mean the CI builds will never
re-use the previous output.

Previously, bp2build and symlink forrest both added LOG_DIR to their
respective soong.environment.used* file. This change avoids that by
passing through the value from the main() function which retrieves it
using availableEnv["LOG_DIR"] which avoids it being added to that
file.

Bug: 257590265
Test: grep LOG_DIR out/soong/soong.environment.used*
      # Before this change it includes LOG_DIR, after it does not.
Change-Id: Iac76aca2d5dc04f47b0924ab5afedef5014bc8b6
2022-11-07 14:12:11 +00:00
Paul Duffin
780a185ede Avoid loss of ninja_deps events
Previously, writeDepFile was passed a copy of the main EventHandler to
which it then added an event. Adding an event requires appending
information to slices in the EventHandler structure. As the slices are
a copy the addition only affects the copy and are not present in the
EventHandler which is written out as part of the metrics.

This change fixes that by passing a pointer to an EventHandler into
writeDepFile. For consistency it also replaces a few other uses of
EventHandler with *EventHandler instead.

Bug: 257590265
Test: m nothing
      # Check generated text metrics to make sure they don't contain ninja_deps.
      # Apply this change.
      m nothing
      # Check generated text metrics to make sure they do contain  ninja_deps.
Change-Id: I0fa0f2b08844cb7c7b4e99b37602703deb43ad20
2022-11-07 14:12:11 +00:00
Paul Duffin
c639059fd8 Make CodegenContext reference *android.Context
Previously, it stored an android.Context (no pointer) which while it
worked (because the current contents are themselves pointers) it is
inconsistent with how the rest of the code references it and could
cause weird behavior if additional fields were added to the Context.

Test: m nothing
      m bp2build
Change-Id: I9c70f925dde85e5b15431cc232b0c2eb6371ec6d
2022-11-05 17:43:12 +00:00
Spandan Das
7b6d5331bd Use CreateCodegenMetrics in api_bp2build as well
The internals of CodegenMetrics was updated in aosp/2276671. Use the
new API to initialize an empty object to prevent a null pointer
exception in api_bp2uild.

Test: m api_bp2build
Change-Id: Iad9c38b9881da896171f1c6d4e49d4875acfaab8
2022-11-01 17:37:07 +00:00
Jingwen Chen
e647db8186 Fix --bazel-mode-staging flag.
It was assigning --bazel-mode-staging to cmdlineArgs.BazelMode, which
looks wrong.

Test: presubmits
Change-Id: I24b087977fc6861eef3a1c30e58d9bf42a344c52
2022-11-01 11:28:29 +00:00
usta
4f5d2c1e97 include symlink metrics in bp2build_metrics.pb
Bug: b/256212479
Test: Prior to thi CL bp2build.symlink_forest event was missing in bp2build_metrics.pb after a clean mixed build
Change-Id: I53bfc4114a383c0d1f9c4c7945e7d4c69bc50b0c
2022-10-31 11:38:10 -04:00
Mark Dacek
df02a8c258 Merge "Add bazelMode arguments to multiproduct_kati." 2022-10-28 15:22:02 +00:00
MarkDacek
c9d5bd8816 Add bazelMode arguments to multiproduct_kati.
These are needed to add staging-mode to some throttled builds. See go/roboleaf-launch-tests for details.

Test: m nothing
Test: build/soong/build_test.bash --bazel-mode-staging

Change-Id: I456ef80fb9c50579a28fba3dbd184d4e4653dde7
2022-10-27 21:24:13 +00:00
Lukacs T. Berki
c541cd27fa Create Bazel symlink forest in a separate process.
This helps with incrementality a lot: the symlink forest must depend on
almost every directory in the source tree so that if a new file is added
or removed from *anywhere*, it is regenerated.

Previously, we couldn't do this without invoking bp2build, which is
quite wasteful because bp2build takes way more time than the symlink
forest creation, even though we do the latter in a very suboptimal way
at the moment.

This means that if a source file is added or removed (which does not
affect globs), we don't pay the cost of bp2build anymore.

Also refactored symlink_forest.go on the side. Too much state was being
passed around in arguments.

This change reimplements aosp/2263423 ; the semantics of not touching an
output file is the exact same as order-only inputs and the latter is a
bit fewer lines of code.

Test: Presubmits.
Change-Id: I565c580df8a01bacf175d56747c3f50743d4a4d4
2022-10-27 08:08:45 +00:00
Usta (Tsering) Shrestha
39f17ad8b8 Merge "don't touch bp2build_workspace_marker file if it exists" 2022-10-25 13:40:59 +00:00
Usta (Tsering) Shrestha
5689aa36f6 don't touch bp2build_workspace_marker file if it exists
- the generating rule has `restat=true`, so its output need not have a newer timestamp to prevent it from rerunning on subsequent ninja invocations
- its dependents (so far only bootstrap.ninja's "build build.ninja") already have proper dependencies in the depfile, namely
  1. Android.bp.list,
  2. entries in Android.bp.list,
  3. bazel.list and
  4. entries in bazel.list

This change prevents symlink related changes from spuriously retriggering build.ninja

Change-Id: I93f1fea7054dfbfc7c13ece34d2d1f07a81bbe07
Test: manually verifying bazel analysis is triggered only under the right set of CUJs using build/bazel/ci/incremental_build.py
Bug: b/239044236
2022-10-25 13:40:34 +00:00
Lukacs T. Berki
a3b9688859 Do not add .bp files to the .d file of bp2build.
This is okay because Blueprint already adds the module files to the
.d file of its output, so until now, they were duplicated.

Test: presubmits
Change-Id: I1bb75c5124aadd1703f3d150948dd7540ba772be
2022-10-25 09:26:06 +00:00
Treehugger Robot
63108dc2be Merge "write soong.environment.used.* file only if changed" 2022-10-24 21:36:06 +00:00
Usta Shrestha
2ba28a34bd write soong.environment.used.* file only if changed
Bug: b/255309129
Test: `lunch aosp_x86-userdebug` and `export NINJA_ARGS="-d explain"`
      Then do the following:
        1) rm -rf out && m bp2build
        2) touch bionic/libc && m bp2build
        3) ALLOW_MISSING_DEPENDENCIES=true m bp2build
      After each step above observe the following two
        a)`stat out/soong/soong.environment.used.bp2build | grep Modify`
        b)`stat out/soong/bp2build_workspace_marker | grep Modify`
      Verified:
       step 2 changes only (b) - prior to this CL, it'd change (a) as well
       step 3 still changes both (as expected)
Change-Id: I98a94878a14e19043f448b0904fc67d5d1dc9733
2022-10-24 17:55:06 +00:00
Liz Kammer
f2a80c6396 Add non-zero exit + error message to soong metrics
Improves some error messages that would have given little/no information

Bug: 254650145
Test: relevant unit tests
Test: Induce errors and check metrics proto
Change-Id: Ife6116af74af6e62c2f8ae8774e53c28178fb8d0
2022-10-24 10:51:56 -04:00
Mark Dacek
99db041a14 Merge "Add bazel staging mode to soong build." 2022-10-21 17:38:43 +00:00
Dennis Shen
ff34e291d8 cherrypick of ag/20211286
update and create bundle config pb python library

update config.pb file to the source of truth:
google3/third_party/java_src/android_appbundle/bundletool/schemas/proto/config.proto

current config.pb is stale and needs update.

Also, add a python library host soong module config_proto, so that it
can be used by other python code.

BUG: b/240288941
TETS: tested along with primary_train_build_worker that reuse
BundleConfig.pb

Change-Id: I426a5cb7ec42573ed2110494c50813abc0a96296
2022-10-20 17:02:42 +00:00
MarkDacek
b78465de1d Add bazel staging mode to soong build.
This is to use bazel to build targets that are being prepared for an
incipient release to the prod mode allowlist.

Bug: 254265047
Test: m nothing
Test: m nothing --bazel-mode-dev
Test: m nothing --bazel-mode-staging
Change-Id: Ic78a59cf51dba83ef1ac26483586560ea9b24aaf
2022-10-19 15:57:18 +00:00