This is the bulk of the "allowlist v2" feature. It will disable bp2build
generation for modules which have transitive dependencies without a
bazel build definition.
This CL includes this mutator, but doesn't register it as a bp2build
mutator (outside of a few unit tests). This allows us to easily iterate
on completion of this feature and ensure there are no launch blockers
before we finalize the change in AOSP.
Bug: 285631638
Test: Unit tests
Change-Id: Ifb0a079c409ca19b02cafa3fab2efa0d3deebc50
- Allowlist an aidl module by package, not name (to support a small
module name change)
- Implement some unit test framework changes which facilitate better
aidl bp2build testing
- Support a convenience function to add a load hook for registering a
module as "has a bazel definition of a given target name"
Bug: 301676937
Test: m bp2build, verified the aidl target was generated before/after
this CL.
Test: Presubmits
Test: Ran bp2build progress and ensured that aidl_interface targets
under frameworks/ continued to appear converted
Change-Id: I62412057d6f61a2ce2bc39488c75af793eb14c94
This should be no-op, as the underlying mutator has not changed yet.
Some other refactoring is required and done in this CL:
- Delete some old, dead ApiBp2build code
- Fix casting to TopDownMutator when it's not necessary
This change is required to prepare for allowlist v2 work, as only
BottomUp mutators can AddDependency.
Bug: 285631638
Test: m nothing
Test: presubmits
Change-Id: I5212a5f5634cc13056195783e6df37ff8eb000da
In other words, if, in bp2build, module "foo" would generate "foo",
and "foo_two", and "foo_two" already exists in a build file,
bp2build should label "foo" as being unconvertible.
Fixes: 301321658
Fixes: 301312582
Bug: 285631638
Test: Unit tests
Test: Verified that `m bp2build` results in bit-for-bit identical
contents for out/soong/bp2build before and after this change.
Change-Id: Icbbdd69fce83579ec9b172d04b2bf1f294698f70
This no-op refactoring facilitates some upcoming functional changes for
"bp2build allowlist v2". The work requires that the bp2build conversion
mutator be changed from a TopDown mutator to a BottomUp mutator.
Refactoring all bp2build-related methods so that they use Bp2buildMutatorContext
makes it easier to make this functional change without touching tens of
files and multiple projects.
Bug: 285631638
Test: m bp2build
Change-Id: I3d1ef3064146e959c6f0dc315350fc9764bf2bd2
This feature is obsolete.
This makes a large number of codepaths "dead code" (such as
module-specific implementations of ApiBp2build functionality). These
will be deleted in a followup CL.
Bug: 284029211
Test: Presubmits
Change-Id: Ib53b99f1fe8c24380d219caf44e9bb3b96724fa0
There are three types of tests, deviceless tests, device-driven tests
and host-driven device tests. But currently we don't have information
to get the type of a test and can't generate test targets on desired
types.
Test: b test //platform_testing/tests/example/native:hello_world_test
Test: b test //packages/modules/adb:adbd_test
Bug: 296312548
Change-Id: I3f022ef769636d508e055477623a4d1a6a1d9044
This function is called in getOtherModuleLabel, and is used to mark the
otherModule as an unconvertedDep. This meant that if a soong module `A`
depends on soong_zip, it would add soong_zip as unconverted dep of `A`.
All go modules have been converted in bp2build, so this CL special
cases go_package and go_binary in this function.
Bug: 294098662
Test: printf metrics in build_conversion.go# GenerateBazelTargets, and
made sure that the string "has unconverted.*soong_zip" does not appear
Change-Id: I8d1d0876c581f9b2eb72dafcb3a28bd5577bbb4c
Fixes the error
```
fmt.Errorf call has arguments but no formatting directives
```
Test: go test ./android
Change-Id: Ic589d6133ccdc88e77dedd4b8a5a56166f62efa9
This also changes the expectation of ConvertWithBp2build. Each
implementation must either create one or more Bazel target modules, or
mark the module as unconvertible (with a specific reason).
Manually verified no runtime hit in AOSP
In AOSP, the metrics file size increases from 252K to 1.6M
This changes some effective module counts in bp2build metrics:
- Removes "package" modules from the module count list in
metrics, as these will not be converted like regular modules.
- Counts Handcrafted modules as being "unconverted", as bp2build is not
responsible for them.
Bug: 285631638
Test: Verified generated BUILD.bazel files are bit-for-bit identical
with this change
Test: Manually verified one case of each implemented reasonType
Change-Id: I308dd451d8f28379b15671dae9f931bd0446f5c1
The parsed BUILD files will be scanned for obvious definitions of BUILD
targets which have Android.bp counterparts. In such cases, bp2build will
automatically omit conversion of these defined modules (to prevent
collisions). With this change, we no longer need one-off denylisting of
modules which have BUILD file definitions.
This has a 0.03s to 0.2s slowdown for bp2build with current state. This
impact is identical on a heavier test branch, as well. I also ran an
experiment that applied BUILD scanning to all source BUILD files
(regardless of allowlisting), and this had a 2 second slowdown.
We may want to look into parallelizing or improving the performance of
this evaluation, but it's probably not worth the effort at this time,
since the current performance hit is small.
Test: New integration test
Test: Removed libprotobuf-python from denylist and tested building the
package
Test: Treehugger
Change-Id: Ibde3bab12cd4a8fed642ad46e5344a56953bec91
Bazel will fail on queries with missing dependencies. Instead, we check
for missing dependencies in mixed builds and we fall back to Soong when
we are aware of missing dependencies in a module.
Test: go test soong tests
Change-Id: I4f83752704970d8b43650d5b55ff35799c7bc625
Prevoiusly, we were partially correcting for override modules in
bp2build/mixed builds in some but not all places. Now we always check
for override modules and ensure that Bazel_module properties are
propagated properly for override modules.
Bug: 279609939
Test: go test soong tests
Change-Id: I5445aa71f4c8013315415a2ca9ab9c6b3be6bce0
Currently, there is little verification around allowlisted modules
actually being mixed-built. This flag would allow us to verify
that a module allowlisted is mixed-built for at least one variant.
Bug: 278910100
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
--bazel-force-enabled-modules=com.google.android.neuralnetworks (This
fails, as expected)
Test: build/soong/test/mixed_mode_test.sh
Change-Id: Icd5976f4f44f1a8caca1e5247d986642f7995f97
Bug: 240424572
Test: Manual tests:
1. m --dev-mode-staging com.android.adbd com.android.media.swcodec.
2. verify the DCLA libs from the two apexes have the same size and
sha1sum, and also match the libs in bazel-out.
3. empty the DCLA libs list in allowlist.go and repeat step 1
4. repeat step 2 and verify the opposite result
5. build git_master: mainline_modules_bundles-userdebug in ABTD
with the cl, then follow go/build-sideload-dcla-locally to
download the adbd and swcodec aab files, run the DCLA trimming
workflow locally, and verify the symlinks in the two trimmed
apexes are identical and also match the lib path in the DCLA
apex that was created by the workflow.
Change-Id: Ib2f8a29126a54829c0e10eba17b256a79930fd70
Previously in mixed builds, only the tidy files for the boundary module
were built, whereas all of its transitive dependencies' tidy files were
not being built. Instead we should export the list of transitive tidy
files for a module so that we can run clang-tidy for the boundary module
as well as its dependencies.
Bug: 195029134
Test: WITH_TIDY=1 DISABLE_ARTIFACT_PATH_REQUIREMENTS=true mss tidy-packages-modules-NeuralNetworks --bazel-mode-dev
Change-Id: I463646d2ae1fc4aa075a54c264e1c34571c3fd5c
* Remove `Bp2buildCcLibraryStaticOnlyList` it is never set
* Remove the rest of the sutff related to it
* Streamline MixedBuildsEnabled
* Rename BazelContext.BazelAllowlisted to BazelContext.NameFilter
to reflect its action
* Rename bazelContext to mixedBuildBazelContext
* Fix minor warnings from Go static analyzer
Test: treehugger
Change-Id: Ie245c992f94bbfd5c7c23dd06917026200a28263
This product is experimental and unsupported for Bazel at this time
Test: lunch aosp_riscv64-userdebug && m nothing
Test: lunch aosp_arm64-userdebug && m nothing
Change-Id: I761afb9524f1008f9d5472d2de19bb84342e9e29
instead of scanning every entry in the map for prefix-match, search each ancestor of the dir
Bug: N/A
Test: bazel_test.go#TestShouldKeepExistingBuildFileForDir()
Change-Id: I7c9afa456cca5a70624ae1e9c9b96c43b0110c1e
Also change behavior of the recursive check - the more specific
directory will be chosen now, for competing recursives.
This is part of an effort to enable mixed_libc and mixed_droid on
internal master - many denies are required and this would simplify the
effort.
Test: m nothing
Change-Id: I7de05337644d8e1804078c66669dd0b2d65f3e71
The generated Bazel workspace will only contain api specific targets.
This is feasible since these targets do not have any cross dependencies
with the targets in the bp2build workspace
The advantages of a new mode are
1. Does not pollute bp2build workspace with api targets
2. Does not block api targets with the current allowlist conversion
mechansims in bp2build
(In the future we might want to combine these two workspaces)
A Soong module type will generate a Bazel target if it implements
ApiProvider interface
Test: m apigen
Test: m nothing
Change-Id: I69c57ca6539f932e0ad554ce84a87fb7936fdba0
It's not needed anymore since aosp/2197837,
it should've been removed in that cl but I forgot.
Bug: 234167862
Test: ./build/bazel/ci/bp2build.sh
Change-Id: I3d67a6e1894ad401525070ad37d3158708898306
Previous behavior:
- Packge not listed in bp2buildKeepExistingBuildFile:
- Use bp2build generated build file
- Package listed in bp2buildKeepExistingBuildFile:
- Use handcrafted build file even if there were allowlisted bp2build
modules in the same package.
- Package listed in bp2buildKeepExistingBuildFile and a soong module has
a bp2build: { label } attribute:
- Merge the handcrafted and bp2build generated build files
New behavior:
- Packge not listed in bp2buildKeepExistingBuildFile:
- Use bp2build generated build file
- Package listed in bp2buildKeepExistingBuildFile:
- Merge with bp2build generated build file.
Bug: 234167862
Test: ./build/bazel/ci/bp2build.sh
Change-Id: Ifbaf4f8f0f5158b5b2bd6d534eb2311e2e5f399b
This allows "bazel mixed builds prod mode", in additional to reworking
the mechanism in which mixed builds dev mode is enabled.
As a followup, CI scripts will be migrated to use the new flags, as
USE_BAZEL_ANALYSIS=1 is deprecated.
Test: Manually ran --bazel-mode with an allowlist verifying that the
module alone was enabled
Test: Manually verified --bazel-mode and --bazel-mode-dev cause a build
failure
Change-Id: If0d34360e60452f428b05828f4ec7596b7cb619a
This refactoring prepares for introduction of bazel prod mode, an
alternative mechanism for mixed builds allowlist handling.
* Decide bazel-mode as close to soong_build main as possible
* BazelContext itself decides whether a module is allowlisted
* Separate bp2build and mixed build allowlist
Test: m nothing, manually verified all modules are mixed build disabled
(via metrics)
Test: USE_BAZEL_ANALYSIS=1 m nothing, manually verified that mixed build
disabled/enabled modules are identical before and after change.
Change-Id: I0f55d8b85000cb4a871a099edc6d7d868d7df509
Bazel conversion state needs to be cloned during module cloning, so it
should be registered via module.AddProperties. Until now it was done
implicitly as the modules converted so far added the whole
base.commonProperties field (and thus transitively all its subfields).
However, adding CommonProperties also adds its fields to the list of
the module's attributes, which is undesirable. The problem surfaced
while implementing Bazel conversion to the `license` rule.
Bug: 190817312
Test: treehugger
Change-Id: Id3de4ede8df81b21f00065a3a1bdc2d707391c3a
This large refactoring has both immense performance implications and
improves mixed builds complexity / usability. Summary:
1. Queueing calls to Bazel is done in a new mutator instead of a full
soong_build pass. Normal soong_build flow is interrupted (via a
functional hook in blueprint) to invoke bazel and parse its response.
2. Implementing mixed build support for additional modules is as simple
as implementing MixedBuildsBuildable. In this interface, define the
request that must be queued to Bazel, and then subsequently define
how to handle the returned bazel cquery metadata.
3. Mixed builds consists of only a single pass. This greatly
improves mixed build performance.
Result:
A 33% runtime improvement on soong analysis phase with mixed builds.
Caveats:
C++ BazelHandler handling still remains a bit of a mess; I did what
I could within this CL's scope, but this may require additional cleanup.
Test: Treehugger
Test: Verified that aosp_arm ninja file is bit-for-bit identical with or
without this change.
Change-Id: I412d9c94d429105f4ebfafc84100d546069e6621
To make testing easier, refactor existing module-global variables into a
struct that can be mocked.
Test: build/bazel/bp2build.go
Change-Id: I9d177677644ea743641a745b1839a3a8b29f902a
