Mutator contexts are created for every module, and in the case of
transition mutator contexts for every dependency of every module.
Unlike the Blueprint mutator contexts that they wrap, the Soong
mutator contexts can be relatively large. Add global pools for all
of them that can avoid the repeated allocations.
Test: SOONG_PROFILE_MEM=/tmp/mem.pprof m nothing
Change-Id: I64a5f3c91292cff6352300f99c11ac50c713f96d
Converting coverageMutator to a TransitionMutator requires adding
the ctx.Device() and ctx.DeviceConfig() methods.
Bug: 319288033
Test: builds
Change-Id: I697b48eb89bc23800d2d3c62d68358769f0d1075
This allows vendors (like google) to specify which team owns the test
module and code.
Team is a commonProperty on modules and points to the designate "team"
module. The DepsMutator adds the dependency on the "team" module and
"GenerateBuildActions" write the team data to intermediate files.
A new singleton rule, all_teams visits all modules and writes out
the proto containing the team for each module.
If a module doesn't have a team, then it finds the package in the
blueprint file and parent directory blueprint files that have a
default_team and uses that team.
Test: m all_teams
Test: go test ./python ./java ./cc ./rust ./android
Test: added team to HelloWorldHostTest and built the new asciiproto target
Test: added package default_team and checkout output proto.
Change-Id: I5c07bf489de460a04fc540f5fff0394f39f574a7
Using generics for the providers API allows a type to be associated
with a ProviderKey, resulting in a type-safe API without that doesn't
require runtime type assertions by every caller.
Unfortunately, Go does not allow generic types in methods, only in
functions [1]. This prevents a type-safe API on ModuleContext, and
requires moving the API to be functions that take a ModuleContext as
a parameter.
This CL creates the new API, but doesn't convert all of the callers.
[1] https://go.googlesource.com/proposal/+/refs/heads/master/design/43651-type-parameters.md#no-parameterized-methods)
Bug: 316410648
Test: builds
Change-Id: I3e30d68b966b730efd968166a38a25cc144bd6de
This prevents bp2buld conversion of modules which have transitive deps
that are not converted.
This does not change most allowlist semantics -- that change is still to
come. As a result, this effectively removes conversion of a few modules
which were previously converted under old semantics, however, these
modules are not currently used in any meaningful bazel builds, and will
be fixed at a later time.
Test: bp2build.sh
Test: m nothing
Test: manually spotchecked allowlisted modules in metrics to ensure the
diffs were minor
Test: manually verified bp2build performance regresses by about 0.4s
Change-Id: Id5c44fa5394917b28a3e707a81555b9e467d6621
This is the bulk of the "allowlist v2" feature. It will disable bp2build
generation for modules which have transitive dependencies without a
bazel build definition.
This CL includes this mutator, but doesn't register it as a bp2build
mutator (outside of a few unit tests). This allows us to easily iterate
on completion of this feature and ensure there are no launch blockers
before we finalize the change in AOSP.
Bug: 285631638
Test: Unit tests
Change-Id: Ifb0a079c409ca19b02cafa3fab2efa0d3deebc50
This should be no-op, as the underlying mutator has not changed yet.
Some other refactoring is required and done in this CL:
- Delete some old, dead ApiBp2build code
- Fix casting to TopDownMutator when it's not necessary
This change is required to prepare for allowlist v2 work, as only
BottomUp mutators can AddDependency.
Bug: 285631638
Test: m nothing
Test: presubmits
Change-Id: I5212a5f5634cc13056195783e6df37ff8eb000da
This no-op refactoring facilitates some upcoming functional changes for
"bp2build allowlist v2". The work requires that the bp2build conversion
mutator be changed from a TopDown mutator to a BottomUp mutator.
Refactoring all bp2build-related methods so that they use Bp2buildMutatorContext
makes it easier to make this functional change without touching tens of
files and multiple projects.
Bug: 285631638
Test: m bp2build
Change-Id: I3d1ef3064146e959c6f0dc315350fc9764bf2bd2
The fixes are
- Dedupe the dir list. Since we partition the proto srcs per pacakge and
then iterate the map, a single include directory was being listed
multiple times
- Drop target_compatible_with from these proto_library targets. The
compatibility will be enforced at the top-level <lang>_proto_library.
Test: Added a unit test
Change-Id: Ia18c0f8c495585010fd4e372092afe80c5d8290c
The current API restricts creation of targets to the directory of the
visited soong module. This CL proposes adding a `Dir` property in
`CommonAttributes` that can be used to create a bazel target in
a specific dir. The use case for this is to dynamically create
additional targets for proto_library that are adjacent to .proto files
(Bazel poses a strict requirement about proto_library being in the
same package as the .proto file, but Soong does not)
Usage is restricted to dirs that have an existing Android.bp file. There
are some places in bp2build where we use existence of Android.bp/BUILD
on filesystem to curate a compatible fully qualified path (e.g. headers).
If we use `CommonAttributes.Dir` to arbritraily create BUILD
files, then it might render those curated labels incompatible.
Test: go test ./bp2build
Change-Id: If9446700457eddfb389be9d9bde39087f67daa60
This also changes the expectation of ConvertWithBp2build. Each
implementation must either create one or more Bazel target modules, or
mark the module as unconvertible (with a specific reason).
Manually verified no runtime hit in AOSP
In AOSP, the metrics file size increases from 252K to 1.6M
This changes some effective module counts in bp2build metrics:
- Removes "package" modules from the module count list in
metrics, as these will not be converted like regular modules.
- Counts Handcrafted modules as being "unconverted", as bp2build is not
responsible for them.
Bug: 285631638
Test: Verified generated BUILD.bazel files are bit-for-bit identical
with this change
Test: Manually verified one case of each implemented reasonType
Change-Id: I308dd451d8f28379b15671dae9f931bd0446f5c1
Bazel will fail on queries with missing dependencies. Instead, we check
for missing dependencies in mixed builds and we fall back to Soong when
we are aware of missing dependencies in a module.
Test: go test soong tests
Change-Id: I4f83752704970d8b43650d5b55ff35799c7bc625
The use case for this is creating config_setting(s) specific to an apex
variant and selecting stub/impl in that config_setting. We likely need
only a handful of such config_setting(s), but determining that list
requires iterating the build graph.
Test: go test ./bp2build
Change-Id: I9aa552e3d0bcf67513023c3a7d4bbf8fae464ee4
Soong does not enforce apex_available on the contents of test apex. To
prevent special-casing test apexes in the apex validation aspect in
Bazel, drop the test apexes from the tags altogether.
( The core problem I am trying to solve is making sure that stub
libraries in Bazel have a single apex available. apex validation happens
to be a nice side benefit)
Bug: 277651159
Test: go test ./bp2build
Change-Id: Ibb3cfedb5c0f2cda0464bf3758c70b67cb5885d1
The core problem I am trying to solve is making sure that stub
libraries in Bazel have a single apex available. In Soong, this will be
enforced using some graph walk which is not easy to port to Bazel.
However, we might need to revisit this when we build the enforcement
mechanism in Bazel. We likely need a `test_for` on the top level
apex_test Bazel rule so that the test apex is _allowed_ to link against impl
of the library it is trying to test.
(This CL retricts this to cc_library, I can expand this to other modules
if this is the right approach.)
Bug: 277651159
Change-Id: Iaeec22c5626df79a33785c766ed29102b1da403e
If apex_available is missing from Android.bp files, ApexAvailable
returns [//apex_available:platform], which is the default. To avoid
BUILD file verbosity, remove the build system default value from the
generated BUILD files.
Bug: 277651159
Test: go test ./bp2build
Change-Id: I4129c2a93ac28578f46ebeed49baa23ce727aa1f
The expected use case for this is to create aliases for stub libraries
in the @api_surfaces repository in build/bazel/api_surfaces.
This restricts the scope to just aliases. If we have a use case for
generating actual Bazel targets in another directory, a workaround could
be to generate the targets in the current directory (via
CreateBazelTargetModule) and aliases to it in the other directory
Test: go test ./bp2build
Change-Id: I6b63d9d018618d447fc7c260a2a94aaa00e57a4d
This is a more flexible version of ApexAvailableTags
that can be used for the aidl modules.
Bug: 260694842
Test: go test
Change-Id: Iddb15e737c19e95977f74e8c627d6044d8875746
This isn't benchmarked or anything, but should give
minor performance improvements.
Test: Verified ninja files didn't change on aosp_arm64-userdebug
Change-Id: I030991f91c326669b7b7e73b8171da4b2a68cbb4
The tags will then be read by an aspect applied from apex attributes to
validate that the deps are explicitly opt-in to the appropriate apex.
The semantics of this is the same as the Soong apex_available property.
This also adds the "base_apex_name" attr to override_apex conversion so
that apex_available checks continue to work on the non-Google and non-Go
apex names.
Test: presubmits
Fixes: 218841706
Change-Id: I19b3637da9aa47573b252d74f62b601fbdc3784d
The generated Bazel workspace will only contain api specific targets.
This is feasible since these targets do not have any cross dependencies
with the targets in the bp2build workspace
The advantages of a new mode are
1. Does not pollute bp2build workspace with api targets
2. Does not block api targets with the current allowlist conversion
mechansims in bp2build
(In the future we might want to combine these two workspaces)
A Soong module type will generate a Bazel target if it implements
ApiProvider interface
Test: m apigen
Test: m nothing
Change-Id: I69c57ca6539f932e0ad554ce84a87fb7936fdba0
The logic is not 100% provably the same since HEAD was quite
confusing at some points, but I did make an effort to preserve
functional equivalence.
In case that effort was not enough, it should be pretty easy to
tweak the logic at HEAD since it's still quite malleable.
Bug: 231370928
Test: Presubmits.
Change-Id: I17b2efbfb5c4d0aedd922caed54ff8d857e578df
Previously we ran mutators in bp2build mode to add dependencies, now we
look up modules by name directly. Remove workarounds to allow bp2build
mode to not fail when adding/handling dependencies.
Test: m bp2build
Change-Id: Ibf6fd905150cac306e5c395902ef28f609f4df2a
These contained duplicate code, but evolved separately in the effort to
identify module type of those created with CreateModule. This refactors
to share a common implementation between the two.
Test: m json-module-graph and verify Type fields
Test: CI
Change-Id: Ifdb9a006d9b1bef7411f9ce3a4384797693b4bfc
Also fix some bugs pertaining to configurable attribute handling of bool
attributes and label sttributes, so that they may support values across
multiple different axes at the same time.
Test: unit tests for bp2build
Test: mixed_droid
Change-Id: I411efcfddf02d55dbc0775962068a11348a8bb2c
Each conversion required defining a separate mutator, which will each
operate on _all_ modules and requires each to repeat checks whether the
mutator should operator. Instead, we introduce a single mutator and
modules can define a ConvertWithBp2build to implement bp2build
conversion for that module.
Test: bp2build.sh
Bug: 183079158
Change-Id: I99d4b51f441c2903879092c5b56313d606d4338d
Introduce `commonAttributes` & `fillCommonBp2BuildModuleAttrs used in
CreateBazelTargetModule
Adapt `bp2BuildInfo` to use `commonAttrs` instead of `Name`.
And thus also all downstream users of `CreateBazelTargetModule`.
As initial user, the Soong `required` property will be
translated to Bazel's `data`.
Bug: 198146582, 196091467
Test: build_converstion_test.go:TestCommonBp2BuildModuleAttrs
Test: go test
Test: mixed_{libc,droid}.sh
Change-Id: Ib500e40f7e2cb48c459f1ebe3188962fc41ec124
Support three options for converting modules with unconverted
dependencies
1. (default) Warn when converting a module if it has unconverted deps.
2. Error when encountering a module with unconverted deps. (not hooked
up yet)
Test: build/bazel/ci/bp2build.sh
Test: build/bazel/ci/mixed_libc.sh
Test: BP2BUILD_ERROR_UNCONVERTED=1 build/bazel/ci/bp2build.sh with
unconverted deps -- get appropriate error
Bug: 181155349
Change-Id: Ifaabf0cd2e43e963366dc137159c705294165c3d
A performance improvement for bp2build as Blueprint/Soong no longer have
the overhead of additional modules. The creation of these modules
results in:
* traversal of additional modules for each subsequent mutator
* synchronization over a go channel to collect newly created modules:
https://cs.android.com/android/platform/superproject/+/master:build/blueprint/context.go;l=2594,2600;drc=1602226f23181b8c3fbfcaf3358f0297e839d7d3
We avoid both of these by storing the information directly in the
underlying module.
Also as a fringe benefit, removes some necessary boilerplate for
conversion.
For benchmarks, reduces runtime ~1% for 1% converted, ~24% for 100%
converted. See more: go/benchmarks-for-https:-r.android.com-1792714
Test: ran benchmarks/tests in bp2build
Test: build/bazel/ci/bp2build.sh
Change-Id: Ie9273b8cbab5bc6edac1728067ce184382feb211
Refactor bp2build to retrieve modules directly by name, instead of via
DirectDeps. This functions properly as bp2build has no need for variant
information of the blueprint graph.
Test: USE_BAZEL_ANALYSIS=1 m fmtlib
Change-Id: Ief4b67bc56f24929871af772f3a742f07085bf8c
Adds PrepareForTestWithNeverallowRules to make it easy to test
neverallow rules. Avoid exporting any unnecessary neverallow related
methods from the android package.
Bug: 181070625
Test: m nothing
Change-Id: Idfc6955cb23f1a4d1790be7879388154b03f3980
cc_object crtbrand sets product_variable.platform_sdk_version.asflag
and will not compile correctly within mixed builds without it.
Only handles product_variables that expand product variables.
Bug: 181794963
Test: ~/aosp/build/bazel/scripts/milestone-2/demo.sh full
Change-Id: I293fcb18032aa51f63bb7b3de94abd6d1ec38180
This CL adds the support for specifying lists of directories in
build/soong/android/bazel.go, which are then written into
out/soong/bp2build/MANIFEST. Using this configuration,
modules/directories can either default to bp2build_available: true or
false, while still retaining the ability to opt-in or out at the module level.
It also ensures that ConvertWithBp2Build returns true iff the module
type has a registered bp2build converter.
Test: go tests
Test: demo.sh full
Test: TreeHugger presubmits for bp2build and mixed builds.
Change-Id: I0e0f6f4b1b2ec045f2f1c338f7084defc5d23a55
