Add a flag that enables logic specific to multitree-build builds.
Bug: b/266730267
Test: treehugger, manual
Change-Id: Ia6f392525faa9d66bb6e0692b2a2e9c3b8493b0d
To improve build efficiency, the metrics for critical path and
parallelism ratio is necessary. That information has been included in
soong.log, so added it into metrics as well.
Bug: 271526845
Test: build and check if metrics pb has critical path info
Change-Id: I14e1a78c13d400b792d3b05df18604da48759ade
workspace.
This is necessary to solve bazel package boundary issues where the api
file might exist in a different directory
Test: m api_bp2build && build/bazel/bin/bazel build --config=android
--config=api_bp2build //build/orchestrator/apis:system
Test: multitree_build system/nothing (in multitree)
Change-Id: Id64085d65a1943bdb394ea80c875db96ca373839
These are not necessary for queryview and can lead to inaccurate query results.
Test: m queryview; grep bp2build_all_srcs
Change-Id: Ifaf65b4ec5828718c2becb39b4507cf033ea4546
This feature is toggled on with USE_PERSISTENT_BAZEL, which is off by
default. Those that opt-in will have a bazel server running between
builds (with a 3hr default TTL) which will greatly improve analysis on
subsequent builds. (As Bazel maintains a cache of analysis results).
Bug: 266983462
Test: Manual `m nothing` runs (timing with and without the feature)
Test: New integration test
Test: Presubmits
Change-Id: I3af4948baa0c490e9b87c48ffdbe9f67732586c7
RBE has signifigant performance overhead during
analysis time, in theory to reduce execution time.
But multiproduct_kati only tests analysis, and does
not run any execution, so RBE is just slowing us
down.
Bug: 229863781
Bug: 265603584
Test: time ./build/soong/build_test.bash --products aosp_arm64 on stage_aosp_master
Change-Id: Icbb2f001fee03a75bad37c14d08a203be20ac6cf
So that it can be used in the bazel implementation.
Bug: 268006095
Test: m nothing, check that it's in out/soong/soong_injection
Change-Id: I1520dd874076dee7fa083648d0790b060d658e5a
Test: `m --bazel-mode-dev` and intentionally break the build.
We can see soong_metrics in the output directory and metrics_uploader
executed.
Bug: 268228148
Change-Id: I79e3b11468fca446673ebf752697a99eaff01b0f
Previously, symlink forest generation involved removing the entire
symlink forest and recreating it from scratch. With this change,
a) symlinks which need not change are untouched,
b) symlinks pointing to the wrong location are fixed, and
c) symlinks which should no longer exist are removed.
On AOSP on my local machine, this reduces the symlink forest generation
step from 2.5s to 1.1s clean, and 0.6s when a single file is added to
a source directory.
Bug: 257528847
Test: m bp2build, touch `fakefile` under the forest, remove a file
from the source tree, rerun m bp2build. Manually verify the new forest
does not retain the link to the deleted source file, and that fakefile
no longer exists in the forest.
Change-Id: I481371ae487e9419af6a3a4370c552578b07d650
This signifigantly improves the startup time of soong-built
python binaries. For example, running
`m apexer && time out/host/linux-x86/bin/apexer` gives
0.734s before this cl, and 0.094s after.
Fixes: 259718110
Test: Presubmits
Change-Id: Ib19e83e2c60c39a849525be117279c318de3afa7
multiproduct_kati assumed each build would take ~30GB of RAM,
but it seems it's closer to 40GB now.
The current build_test builds use 7 jobs, so this would bring
us down to 5, which should still keep us under the 6 hour
build time limit as currently these tests are at ~3 hours.
Bug: 262629589
Test: Presubmits
Change-Id: I51af364cc602a3bf7ba85fd8f92c4dd57585abbd
This test builds an "empty" filegroup. Running it can be useful to
verify that the api_bp2build workspace has been setup correctly.
Test: TH
Change-Id: I78639a2dcdeb659b2d6fdf6abc85d85c4fb697bf
- This is a wrapper function for all files in soong_injection directory.
This should prevent an error in the workspace generated for
api_bp2build.
- Rename the existing CreateSoongInjectionFiles fn and make it package
private to prevent confusion
(The subsequent CL in this stack should contain a smoke test for
api_bp2build)
Test: b build //:empty --config=api_bp2build --config=android
Test: TH
Change-Id: Iddb0aa1aff2f709826edd587aa99fccddf80f08f
Create a
build/bazel/product_config/generated/products/<product_name>/BUILD
file that contains the platform definitions needed for
a particular product. Currently we just create it for the
current lunch target, but the idea is that eventually when
all product config is in starlark, all the products will
have their platform definitions in the tree at once.
Bug: 249685973
Test: Presubmits
Change-Id: I08c82ff28dcf62f09d3b1d2e3186a6b961e12f6e
This involved refactoring much of the main method in soong_ui/main.
Test: b build libcore:all
Test: use the build-time started flag and verify via printf that
it's the same across upload.go and metrics.SetBuildDateTimestamp()
Change-Id: Id7fe256337e8ee6c40542eba662c0eadb38e9674
Test: Run `m --bazle-mode-dev nothing` again after `m --bazel-mode-dev nothing`.
bp2build_metrics.pb no longer exists, the rest metrics files
have the timestamp of the latest creation time.
Bug: 256617990
Change-Id: I869d1b9c3abcb517744e66b21e5c12d6b5b1e97b
* `android.Context` holds `android.Config`, so provide an accessor to it
and avoid passing them separately to a lot of functions.
* factor common code in `readBazelPaths`/`getExistingBazelRelatedFiles`
into `readFileLines`
* refactor check-error-and-quit into `maybeQuit`
* use switch instead of if-elseif-... where appropriate
* rearrange the code in `runApiBp2build`, `runSymlinkForestCreation`
Test: treehugger
Change-Id: I639929c21ec4999cdfd737c07136f32df2d488bc
Define a structure android.CmdArgs and keep all soong_build command line
arguments used to build a configuration in it. Some of them related to BazelMode
were previously kept in Blueprint's bootstrap.Args where they do not belong.
This simplifies the code and lays the gound work for future simplifying changes.
Test: treehugger
Change-Id: I995871f4e6b0e95cc9df65696ad2020df221d8c8
Test: m nothing
Test: m --bazel-mode-dev
Test: Verified soong_build_metrics is available in local metrics tables from builds that include this CL
Bug: 257537925
Change-Id: Ic5419ecb203c9b62457e1c553d592ba593a9bfff
The synthetic soong_injection workspace is shared between api_bp2build
(used in API export) and bp2build (used in API domain analysis which
runs mixed builds by default). The former creates them in ReadOnly mode,
but the latter creates them in ReadWrite mode (to allow users to
edit/experiment). To prevent errors in the end-to-end builds in
multittree, make api_bp2build create these files in ReadWrite mode as
well.
Test: mulittree_build locally
Change-Id: I8766d88951a3b625739e4dc12d468450ad7322ea
Users can use this feature by
1. Setting PRODUCT_INCLUDE_TAGS += <val> in their product mk files
2. Set
```
blueprint_packge_includes {
match_al: ["<val>"],
}
other_module_type {name: foo}
other_module_type {name: bar}
```
bar and foo will be included if and only if <val> is set
Test: Unit tests in blueprint
Test: TH
Change-Id: I32eed4e3b5ac47fb565c62d13d8881fa984c86f4
Background: toolchain/llvm_android/test_compiler.py generates clang-dev, a directory
symlink pointing to out/install/linux-x86/clang-dev, which itself
contains a BUILD file. This BUILD file should be excluded from the
symlink forest because it's not an allowlisted BUILD file.
Problem: However, our current symlink forest logic and finder logic doesn't
handle this situation. It's not in bazel.list:
```
$ less out/.module_paths/bazel.list | grep prebuilts/clang
prebuilts/clang/host/linux-x86/BUILD.bazel
prebuilts/clang/host/linux-x86/cc_toolchain_config.bzl
prebuilts/clang/host/linux-x86/cc_toolchain_constants.bzl
prebuilts/clang/host/linux-x86/cc_toolchain_features.bzl
prebuilts/clang/host/linux-x86/clang-r450784d/BUILD.bazel
prebuilts/clang/host/linux-x86/clang-r450784e/BUILD.bazel
prebuilts/clang/host/linux-x86/clang-r458507/BUILD.bazel
prebuilts/clang/host/linux-x86/clang-r468909/BUILD.bazel
prebuilts/clang/host/linux-x86/clang-r468909b/BUILD.bazel
prebuilts/clang/host/linux-x86/clang-r475365/BUILD.bazel
```
The symlink forest logic uses this bazel.list file and matches it
against the keepExistingBuildFile list to exclude BUILD files from the
symlink forest. Since clang-dev/BUILD.bazel is not in bazel.list, it's
symlinked into the forest.
All of these gymnastics could be avoided if we added custom BUILD file
name support to Bazel, and only symlinked (e.g.) Android.bazel and
ignored all other BUILD files in the tree. It would be very clear which
checked-in BUILD files we want to use in the symlink forest.
I think we should pursue custom BUILD file names instead of adding more
complexities, to say, the finder logic, to support such a use case. It's
also why I decided to add this hardcoded exclusion to keep the
workaround simple.
Bug: 260809113
Test: presubmits
Test: $ DIST_DIR=/tmp/dist prebuilts/python/linux-x86/bin/python3 toolchain/llvm_android/test_compiler.py --build-only --target aosp_raven-userdebug --no-clean-built-target --module dist --module droid --module tidy-soong_subset --with-tidy ./
Change-Id: I415371543585c1c5e8e00e6958105f65ea5978ee
Unlike bp2build, api_bp2build depends on a well-defined subset of checked-in
BUILD/bzl files, and we create symlinks for that subsset in
out/soong/api_bp2build. Since we now cd into the workspace after
aosp/2308141, we need to add the sylimks of the external repositories as
well
Test: bazel build --config=api_bp2build //:empty
Change-Id: Ia7ecc5543226893a3878bc0fd35e025a37c80413
They are still read when using bazel with queryview, so they
need to be in the workspace so that they can be found.
Bug: 234449134
Test: m queryview && ./build/bazel/bin/bazel query --config=queryview //...
Change-Id: I1f2187dffd5a0e6cfb44ebe0c6738fa9bbef509d
Test: m nothing
Test: m --bazel-mode-dev
Test: Verified bp2build_metrics is available in local metrics tables from builds that include this CL
Bug: 257537925
Change-Id: I09617a7850f1471f18192a0418396126fd4c98ef
Prior to this change, the bundletool and extract_apks tools require that
all ABIs that an APEX or APK provides must be compatible with the
TargetConfig. Instead, this change allows an APK to be selected if it
has at least one compatible ABI with the TargetConfig.
Bug: 260115309
Test: go test .
Change-Id: If67ce8128099611257a834862295a2bf5fa427d3
