This allows "bazel mixed builds prod mode", in additional to reworking
the mechanism in which mixed builds dev mode is enabled.
As a followup, CI scripts will be migrated to use the new flags, as
USE_BAZEL_ANALYSIS=1 is deprecated.
Test: Manually ran --bazel-mode with an allowlist verifying that the
module alone was enabled
Test: Manually verified --bazel-mode and --bazel-mode-dev cause a build
failure
Change-Id: If0d34360e60452f428b05828f4ec7596b7cb619a
This refactoring prepares for introduction of bazel prod mode, an
alternative mechanism for mixed builds allowlist handling.
* Decide bazel-mode as close to soong_build main as possible
* BazelContext itself decides whether a module is allowlisted
* Separate bp2build and mixed build allowlist
Test: m nothing, manually verified all modules are mixed build disabled
(via metrics)
Test: USE_BAZEL_ANALYSIS=1 m nothing, manually verified that mixed build
disabled/enabled modules are identical before and after change.
Change-Id: I0f55d8b85000cb4a871a099edc6d7d868d7df509
This was accidentally suppressed in aosp/2124255, so this reverts that
bug.
Test: USE_BAZEL_ANALYSIS=1 m nothing && ls out/soong/soong_injection/cquery.out
Change-Id: I5331526a766100bb399aac93238919abd1ff5ca2
Post deps still supports creating variants, while final deps does not,
this move will ensure all variants are created prior to queuing Bazel
requests.
Test: CI
Change-Id: I92caae009ea3a8769f3f11a9b1ab80de6aac17f3
* Add ApexCqueryInfo to obtain apex artifacts used by the makefile
generator and downstream modules
* Refactor code common to GenerateAndroidBuildActions and ProcessBazelQueryResponse
* Implement android.MixedBuildBuildable for modules
* Enable mixed build for apex modules with payload_type:"image"
The first take 6a2b7c40b was setting compressed APEX suffix incorrectly, and
was reverted in 8a3c91494.
Fixes: 239925080 239695521 232085015
Test: treehugger
Change-Id: I1720f8db3c7cc773183d25a815d9b7eeaf7c73ad
* Add ApexCqueryInfo to obtain apex artifacts used by the makefile
generator and downstream modules
* Refactor code common to GenerateAndroidBuildActions and ProcessBazelQueryResponse
* Implement android.MixedBuildBuildable for modules
* Enable mixed build for apex modules with payload_type:"image"
Bug: 232085015
Test: treehugger
Change-Id: I9f1e03e5e7a5b5dde35a5db10c253069543ac973
Allowlist manual BUILD files in prebuilts/bazel for
the toolchain setup.
Test: b build build/bazel/examples/java/...
Change-Id: Id649dce631731e5396da4a9fe224e33f8949dd7d
Bug: 196441639
For the generic target (i.e, arch == "common" and os == Device), the config
string should be "target|common", not "x86_64|common".
Also renamed local variable from `os`.
Bug: 232085015
Test: treehugger
Change-Id: I36e696203f207dd295ed7f109b7c07b576ebf3c2
This large refactoring has both immense performance implications and
improves mixed builds complexity / usability. Summary:
1. Queueing calls to Bazel is done in a new mutator instead of a full
soong_build pass. Normal soong_build flow is interrupted (via a
functional hook in blueprint) to invoke bazel and parse its response.
2. Implementing mixed build support for additional modules is as simple
as implementing MixedBuildsBuildable. In this interface, define the
request that must be queued to Bazel, and then subsequently define
how to handle the returned bazel cquery metadata.
3. Mixed builds consists of only a single pass. This greatly
improves mixed build performance.
Result:
A 33% runtime improvement on soong analysis phase with mixed builds.
Caveats:
C++ BazelHandler handling still remains a bit of a mess; I did what
I could within this CL's scope, but this may require additional cleanup.
Test: Treehugger
Test: Verified that aosp_arm ninja file is bit-for-bit identical with or
without this change.
Change-Id: I412d9c94d429105f4ebfafc84100d546069e6621
This change constitutes a number of fixes which cause mixed builds to
have deterministic ninja file output:
1. Depsets are identified based on a hash of their contents instead of
an arbitrary ID integer from Bazel
2. Depset definitions in the ninja file are sorted by the above hashes
3. BuildStatements (action information from Bazel's aquery) are sorted
by their contents
Test: Ran `USE_BAZEL_ANALYSIS=1 m nothing` three times and verified the
md5sum of out/soong/build.ninja was identical all three runs.
Test: mixed_droid
Change-Id: Iffdf6cc62c31d76fbbfa78726827497516171f4f
Each depset now corresponds to a phony rule which depends on other
depsets or on full paths; thus, bazel's depset structure is preserved in
the form of phony rules of name bazel_depset_{id}.
Previously, flattening and recopying large lists of file path strings
was quite inefficient. This was particularly evident as we enumerated
hundreds of clang headers for each cc compile action.
This reduces soong_build analysis time by about 30% for mixed builds.
It also reduces ninja file size by ~750MB.
Fixes: 229405615
Test: Unit tests, manually verified metrics, mixed_droid CI
Change-Id: I78df152ac1488ae0c6807afdde4b4ad5e6d26287
This involves handling of linux-* platforms (identifying them
appropriately as host), disabling unsupported Windows platform, and
denylisting of still-problematic modules.
Test: mixed_droid, with verification that shared library artifacts
originate from bazel-out
Change-Id: Ib52db49a2d2a3c1ff9b76af23fd4f22cfc9182d0
This also removes the special-case filegroup from mixed builds
buildroot; no special handling is required. Since we're currently
hardcoding linux_x86_64 as our host platform, it should be fine
to continue assumping that hardcoded host for now.
Test: USE_BAZEL_ANALYSIS=1 m adbd
Change-Id: I35509f4eb33ba7a243fab4c34b35958f6f2fceab
target.kind was a field that existed for only a few months, and have
been removed from Bazel. This is needed to update to a newer Bazel version.
Also, this reverts filegroups to be queried for a specific arch again,
but leaving file targets in the common arch (which is correct, anyway).
Fixes: b/199363072
Test: mixed_libc
Test: presubmits
Change-Id: I3b8e5c43a39516d977d327a44a75750b2f251be3
This will be the list of targets directly being requested + additional
targets necessary for mixed_root. This reduces the size of the cquery
file by ~300Kb. Additionally, it removes unnecessary targets, making it
ever so slightly easier to find what one is looking for in cquery.out
for debugging purposes.
Test: build/bazel/ci/mixed_droid.sh
Change-Id: I6d0e337c9ab7c13388459dee16415be0ab3c0506
This should fix incrementality issues in cases where actions don't wipe
the old output. (Looking at you, llvm-ar)
Test: mixed_libc.sh
Change-Id: I4b6639e74c72921a5a703d73fd36f766f121c6ad
This builds cap_names.list.h, which uses an eponymous filegroup
"generate_cap_names_list.awk" in Soong, but uses the file target
directly in Bazel.
This also improve filegroup support for mixed builds, by issuing a
cquery call _without_ arch. Filegroups in Soong don't have configurable
properties, so don't generate Bazel filegroups into buildroot's
config_nodes (which was x86_64 by default).
The mixed_build_root now looks like this:
```
config_node(...)
config_node(...)
config_node(...)
config_node(...)
...
filegroup(name = "common",
srcs = ["@//bionic/linker:linker_sources_x86",
"@//bionic/libc:kernel_input_headers",
"@//system/timezone/apex:com.android.tzdata-androidManifest",
"@//external/libcap:generate_cap_names_list.awk",
"@//bionic/linker:linker_sources_arm64",
"@//bionic/linker:linker_sources",
"@//bionic/libc:libc_sources_shared_arm",
"@//bionic/linker:linker_sources_x86_64",
"@//bionic/libc:all_kernel_uapi_headers",
"@//build/bazel/examples/apex/minimal:build.bazel.examples.apex.minimal-file_contexts",
"@//system/core/libcutils:android_filesystem_config_header",
"@//bionic/libc:libc_sources_static",
"@//bionic/linker:linker_sources_arm",
"@//bionic/libc/tools:bionic-gensyscalls",
"@//bionic/tools:bionic-generate-version-script",
"@//bionic/libc:libc_sources_shared"],
)
mixed_build_root(name = "buildroot",
deps = [":x86",
":arm64",
":arm",
":common",
":x86_64"],
)
```
Test: CI
Fixes: 198595323
Fixes: 198235838
Change-Id: I6df9a14da556cf358d96e6a99b514f66a2638295
If a valid JAVA_HOME is set in the environment when running
bazel with --experimental_repository_disable_download, bazel
will unsuccessfully attempt to download rules_java. The above
flag has the practical effect of preventing the download attempt
(some clues as to why in the doc at go/aosp-bazel-jdk).
Note that this also prevents building java targets in mixed mode.
Eventually, we will pull in the java_rules into AOSP so they don't
need to be downloaded.
Test: build/bazel/ci/mixed_libc.sh
Change-Id: Icdb186c18a3334f4caa17243a8620a1cc8c1db0e
These are just out/ and out/soong/ and the old names were quite
confusing.
Test: Presubmits.
Merged-In: I999ca62162062f27e449f83ffb00fa580d4065b8
Merged-In: I2fcd80a9e376a6842a06aa104a9f6f5c5e16f8c7
Change-Id: Ib481d2aac40df8da1b74174dc8ec576c1cb48be0
They used to be defined in Blueprint but since it doesn't have a
separate existence anymore, we can consolidate all command line
arguments here.
Note that Blueprint is called directly from soong_ui, but that happens
by a simple Go function call and not by invoking a separate binary, so
command line arguments are not needed there.
Test: Presubmits.
Change-Id: Ifa2c101ddbe424b76fc5a508d3d41c329e7353ca
Add a new request type with its own StarLark function
Hook it up via GetPythonBinary
Add to MockBazelContext a LabelToPythonBinary
Add a test for the new request type
Test: request_type_test.go:TestGetPythonBinaryParseResults
Change-Id: I05f6506adfbbdac8b3f40475509ed02ab8e844e5
Because it's commonly useful and not cc/ specific
Also export GenerateBazelBuildActions and update uses
Test: Existing tests pass
Change-Id: Ibc6858bb1129afba181a7686dda432defe33b00d
Rather than specifying a specific arch/os, use the android_target
platform that incorporates arch/os based on product config.
Test: build/bazel/ci/mixed_libc.sh
Change-Id: I2b30cfb9778444acbfd7648af2d331303cab416a
In mixed builds, Bazel operates in <root>/out/soong/workspace, but the real
OUT_DIR is just <root>out/, not <root>/out/soong/workspace/out. This CL
ensures that the correct OUT_DIR is passed to mixed builds's bazel by
making OUT_DIR absolute.
Also add STANDALONE_BAZEL to build_test's Bazel smoke test.
Test: TH
Change-Id: If8b4058d43bf821f5196016f9b0ad38edfe18716
