* changes:
Handle proto.include_dirs in bp2build for py
Add proto.local_include_dirs support in bp2build
Add manual tag to proto_library at the root package
If a python_library uses a pkg_path foo/bar, then the proto srcs in
that libray need to import the dep .proto as foo/bar/proto.proto.
This behavior is restricted to python modules. To implement this is in
bp2build, this CL creates a new interface with a single method
`PkgPath`. Only python module structs implement this interface, and
this method is only available during bp2build
Test: Added a bp2build unit test
Test: TH
Change-Id: If8d207c0b321f75337a053795826b283a5eaaf46
This is a followup to aosp/2711093 which added support for
proto.include_dirs. local_include_dirs is simlar, except that is
relative to the module directory.
Test: go test ./bp2build
Bug: 285140726
Change-Id: I32ddc7371048672d6935f827c8aee9d767305e2c
This is a fix similar to aosp/2707793. This adds manual tags to the
top-level proto_library created in the root package, plus any
dynamically created proto_library targets for proto.include_dirs. These
proto_library targets might not specify their deps correctly and might
be unbuildable.
(We need to keep these targets because they provide a ProtoInfo for
cc/java/py source gen)
Test: go test ./bp2build
Change-Id: Ic00f05186327fcfcc8d33a0a2c0891ed619b7acb
The config constant is consumed by rust toolchain implemented in aosp/2709668.
Bug: 295918553
Test: go test
Change-Id: I0a28a802ad28f6ee24dfb1b5bdab72094534ae04
Since we no longer support "zip" APEX, we don't need "host" support.
For example, we don't need go/python binary support.
Bug: 279835185
Test: m
Change-Id: I6e8d2b205e42662f31866dc9ac7507524effd144
expandSrcsForBazel always prefixed : in OriginalModuleName. The
exceptions to this are filegroups that appear in a different soong
namespace. For these cases, we were not correctly substituting the soong
module name with the equivalent bazel label.
Test: go test ./bp2build
Change-Id: If090f3f8819835177c1f4d191b3eef6bb6e30ace
There are three types of tests, deviceless tests, device-driven tests
and host-driven device tests. But currently we don't have information
to get the type of a test and can't generate test targets on desired
types.
Test: b test //platform_testing/tests/example/native:hello_world_test
Test: b test //packages/modules/adb:adbd_test
Bug: 296312548
Change-Id: I3f022ef769636d508e055477623a4d1a6a1d9044
Soong's proto.include_dirs becomes the -I path for aprotoc cpp code
generation. This does not translate well to Bazel because it runs this
action in a sandbox. Even if we construct an -I path, the .proto files
will not be there. This CL attempts to handle this kind of dependency
automatically via bp2build.
For this hypothetical example
```
foo.proto # contains `import bar.proto"
Android.bp # cc_library {srcs:"foo.proto", p.include_dirs:["subdir"]},
subdir/bar.proto
```
Implementation details for CcProtoGen of foo
- Glob the labels of .proto files for each includeDir, and create a
proto_library that encapsulates them.
- Since Bazel poses a contraint that proto_library target needs to be
in the same pacakge as the .proto file, the proto_library might be created
in a subdirectory with an import_prefix
- Add bar's proto_library as transitive deps of foo's cc_proto_library.
This will be added to -I during aprotoc. We cannot add them to `deps`,
otherwise bar's symbols will be statically linked into foo's .a
file.
Implementation details for clang compile of foo
At the end of CcProtoGen, we have converted foo.proto
to .cpp/.h files. To compile them to .a files, we need the .h files
generated from bar.proto. Soong specifies this at the
top-level cc_library soong module, so add those deps as the
implementation deps for clang compile.
(Will add support for java in a follow-up CL)
Test: go test ./bp2build
Test: built some internal modules that were previously blocked by this
Bug: 285140726
Change-Id: I7e1f9f0d1b1ba916a7ba8278f6cfb342b381d695
Mostly so that genrules can have their command qualified on a product
variable.
Bug: 295910468
Test: m nothing
Change-Id: I28cc18a1b3b00368f4768a1326259a656ce970ec
This is a fix for aosp/2693190 that handled .proto files that end up in
different bazel packages. It did it by creating proto_library targets in
the correct bazel package.
Changing the granularity causes issues if the the new proto_library in
the subpackage imports a .proto file from a parent package or a
different package. e.g.
```
tmp
├── foo.proto
└── subdir/import_foo.proto # contains an `import "foo.proto"`
└── subdir/Android.bp # package boundary
├── Android.bp # contains a cc_library with foo.proto and
# subdir/import_foo.proto
```
At ToT, the ProtoInfo we provide to CcProtoGen is correct, but the
proto_library in subdir/BUILD will not compile because it does not have
a dep on the proto_library in ./BUILD
This CL creates a workaround by adding `manual` to the proto_library
targets. This CL is based on the assumption that the buildable unit in
bp2build is cc_library_*, and not proto_library necessarily (atleast
till we do a manual/automated cleanup)
Test: Created an integration test in build/bazel
Test: go test ./bp2build
Bug: 292583584
Bug: 246997908
Change-Id: I73120be2411967cb144f37ed4417f76ecf1a6ffa
So that users can use soong config variables / product variables
to adjust a genrule's command.
Bug: 295910468
Test: m nothing
Change-Id: I9fedf8d5d52e515c3fdb913411ce1b3fecb7ba81
Protos in another directory were using import prefix, which was
prepending the repository-relative path with the value, instead, we want
to strip the prefix of the directory of the module the protos were used
in such that they can be referenced at the appropriate relative path.
Reference on import_prefix:
https://bazel.build/reference/be/protocol-buffer#proto_library.import_prefix
Test: b build //packages/modules/adb:libfastdeploy_host
Change-Id: If050b0f5fc5103bd9cc5a99703bd604325aa4204
In this Android.bp file
```
my_cc_defaults {
enabled: false,
soong_config_variables: {
my_bool_variable: {
conditions_default: {enabled: false},
}
}
}
```
The inner enabled: false is a no-op because the top-level enabled is
false. Currently, bp2build will raise an exception for this Android.bp
file.
However, it does not need to. `productVariableConfigEnableLabels` runs
only if the top-level enabled is false. If it sees enabled: false via
conditions_default, it should just ignore it since it is a no-op.
Test: go test ./bp2build
Bug: 210546943
Change-Id: I816f209eaf21de65ddfbc2893e5255be94bcaa11
with 4000X clones of adbd, bp2build.Codegen time goes from 36s to 7s
Test: m bp2build and verify same BUILD.bazel files
Bug: 292281398
Change-Id: Ibd3b328a917fe5d1bdad67c67116da336f9033cb
If enabled does not appear inside `soong_config_vars`, we can ignore it.
Bug: 210546943
Test: go test ./bp2build
Change-Id: I9e4d51c3b683f262921449634f827915ce87dc8d
* changes:
Handle .proto files in different package for filegroups
Handle .proto files that end up in a different package
Allow creation of BazelTargets in a different directory
Followup to aosp/2693190, this CL adds the support to filegroups.
<mod>__bp2build_converted is now an alias to a proto_library
target <mod>_proto. This proto_library will be created in a different
package if the .proto file exists in a different package.
Test: bp2build unit tests
Test: TH
Bug: 292583584
Change-Id: I8ca452aacf1a86dfc9e218464e38aab89afa5a29
Bazel poses a strict requirement that .proto files and proto_library
must be in the same package. This CL handles this automatically by
creating the proto_library in a separate dir/package if necessary
Implementation details
- Partition the `srcs` by package. `srcs` has been computed using
`transformSubpackagePath`, so the information about packages is
available at this point
- Create a proto_library in each package by using
`CommonAttributes.Dir`. Collect all these additional libraries
and put them in `info.Proto_libraries` so that they get added as deps
of (cc|python|...)_proto_library
- Add an import_prefix to the proto_library in subpackages relative to
the current directory. This relies on the assumption that every src is
beneath the current directory (Soong will complain if a path in
Android.bp contains ../)
filegroup module type uses a separate code-path to create proto_library.
This will be handled in the next CL in stack.
Test: bp2build unit tests
Test: TH
Test: Built the failing internal module mentioned in
b/292583584#comment1
Bug: 292583584
Change-Id: I437fc89092321b26c5f0511387cde9e84084d6f9
The current API restricts creation of targets to the directory of the
visited soong module. This CL proposes adding a `Dir` property in
`CommonAttributes` that can be used to create a bazel target in
a specific dir. The use case for this is to dynamically create
additional targets for proto_library that are adjacent to .proto files
(Bazel poses a strict requirement about proto_library being in the
same package as the .proto file, but Soong does not)
Usage is restricted to dirs that have an existing Android.bp file. There
are some places in bp2build where we use existence of Android.bp/BUILD
on filesystem to curate a compatible fully qualified path (e.g. headers).
If we use `CommonAttributes.Dir` to arbritraily create BUILD
files, then it might render those curated labels incompatible.
Test: go test ./bp2build
Change-Id: If9446700457eddfb389be9d9bde39087f67daa60
That build doesn't set platform_sdk_version.
Bug: 269577299
Test: ./prebuilts/kernel-build-tools/build-prebuilts.sh on the kernel-build-tools branch
Change-Id: I68eb33bfb80770cef9c3a9c5e01f1ed746a2d069
If `srcs` contains a gensrcs/genrule module, the current bp2build module
will put it in the catch-all `srcs` attribute. This is reserved for .cpp
sources, so if the genrule produces a .proto/.aidl/... file, this will
fail.
This handles genrules that produce .proto files. To implement this, this
creates an additional partition that detects if the other module is a
genrule/gensrc that produces .proto files. If true, it will append it to
the proto partition.
This CL does not handle
- genrule that produce .c/.aidl/.yacc/.... files. They will continue to
be partitioned into the catch-all partition
- java modules
Test: unit tests
Test: TH
Bug: 293205700
Change-Id: Ib720fdf3a053ff5dd256e6faa632e3fa7776966d
...instead of based on constraint settings.
Bug: 269577299
Test: m nothing and ./build/bazel/ci/bp2build.sh
Change-Id: Ib9caec79c92b8fd304e46be841de5612bd1637e3
Since these modules have been converted, we should tag them as converted
in the dashboard.
Test: Put a print statement for `metrics` and made sure that the go
modules appear there
Bug: 294098662
Change-Id: I6282100111030a94f15f330916eaf41fcfc16e1a
Revert submission 2651299-bazel_sandwich
Reason for revert: b/293883239, checking if this breaks the build.
Reverted changes: /q/submissionid:2651299-bazel_sandwich
Change-Id: I5f1d4bb13d21e6599f5c353dcaba2375f5ec234d
The "bazel sandwich" is a mechanism for bazel to depend on make/soong
outputs. The name comes from the fact that bazel is now at the top
and bottom of the build graph. This is intended to allow us to work
on converting the partition builds to bazel while not all of the
dependencies of the partition have been converted.
It works by adding the bazel_sandwich_import_file rule, which emits a
dangling symlink that starts with bazel_sandwich:, and includes
information that the aquery handler in soong reads. The aquery handler
rewrites the symlink so that it points to a file generated by
make/soong, and adds a ninja dependency from the symlink to the file
it's targeting.
This allows us to depend on make-built files from bazel, but notably
it doesn't allow us to depend on analysis-time information from make.
This shouldn't be a problem for the partitions, but limits the use of
the bazel sandwich to similar, less complicated types of builds.
go/roboleaf-bazel-sandwich
Bug: 265127181
Test: m bazel_sandwich
Change-Id: Ic41bae7be0b55f251d04a6a95f846c50ce897adc
If we have a soong_config_module_type with target.android_<arch>.*
properties, we should only panic if a soong_config_var is used to select
on those properties. We already call `IsZero` in an upper function, but
we need to call it in `AddSoongConfigPropertiesFromTargetStruct` as well
because conditions_default gets its own struct.
Test: unit tests
Test: repro'd cmd in b/293206386#comment1, it passes now
Bug: 293206386
Change-Id: I23f14f83cc9be66487625e292e26b204dfea4f9a
These tests cannot be run with b since it uses functions that should not be
used outside tests, and it enforces this by looking at os.Args. Since
the pattern (`-test`) is missing in os.Args, these tests panic in b.
A module can be a dep of soong_build via two mechanisms
1. Listed in deps explicitly
2. Via plugin_for to create a reverse dep
This CL denylists (1) explicitly. e.g. soong-android. Another option
would have been to denylist by using Deps() of soong-build, but that
would denylist `golang-protobuf-proto` as well
This CL denylists (2) programmatically.
Note that this CL only denylists creation of the `go_test` target. We
will still create `go_binary` and `go_library` for soong since there
might be tools that depend on soong.
e.g.
```
--> depends on
dexpreot_gen --> soong-dexpreopt --> soong-android (test failure)
```
(The above example means that we might need to convert _some_ of soong
after all, but we can refactor the common bits to common utils. Keeping
this out of scope for now).
Test: b test //build/soong/... //system/tools/...
too)
Bug: 284483729
Change-Id: I809012bdd383febca30da95e991edfde5fd4d9b9
For go modules with non-empty testSrcs, we will create an additional
go_test target. To build a standalone test executable, we need to
include the source files in the compilation unit. This will be done
using the `embed` attribute
1. For tests of go_library, this is straightforward. It will embed the
go_library and "inherit" its .go files and deps
2. For tetss of go_binary, we need to create an additional go_source
target since go_binary cannot be embedded inside a go_test
Using `b test` for these tests revealed that certain tests are not
hermitic and rely on `testdata` files checked into the tree. This CL
introduces an allowlist to skip generating go_test targets for them.
Test: bp2build unit test
Test: TH
Bug: 284483729
Bug: 288491147
Change-Id: Ic736d655babc2f6067e4da75384900b7b8bdc2ed
cc_test bazel macro will default `gtest` to True (similar to Soong). So
we can skip explicitly setting this in the generated BUILD files. The
benefit will be that it will make the generated files less verbose,
without loss of information.
This will be implemented by changing its datatype to *bool from bool.
Test: go test ./bp2build
Test: TH
Change-Id: I284e10f1d58c8e7893b170209827f7d5084ca95e
This property implicitly adds `libgtest_isolated_main` to the static
libs of the test. bp2build will make this and `liblog` explicit in BUILD
files
Since the deps are made explicit in BUILD files, `isolated` becomes a
no-op for cc_test. Remove this property from cc_test.
Test: unit tests
Bug: 244432609
Change-Id: I189a7b6b62d9064f4b2abad49ac4975468046498
Soong defaults to false, while bp2build defaults to true. This CL fixes
bp2build's default to false to match Soong.
To prevent future differences, bp2build now uses the `isolated()`
function used by Soong. The type of the context object has been changed
to EarlyModuleContext so that the ctx object does not depend on
mutations that might not run in bp2build.
Test: TH
Bug: 244432609
Change-Id: I15346107896312961e8d12270cf2f9a2a48827e1
Embedding multiple includes from a genrule may be difficult to read,
instead we generate a header library that contains the headers and all
include dirs that Soong generates.
Test: go test bp2build tests
Change-Id: I590c74c133f015f27cccf5a2fd916153ad9c125e
This is currently implicitly added by the cc_test macro. This can cause
duplicate issues if users write a BUILD file as
```
cc_test (
name = "mytest",
deps = ["libgtest"],
)
```
Making this dep explicit in BUILD files prevents this issue.
The property `gtest` will not be removed as part of this CL. cc_test
macro will use this property to add some gtest specific copts
Test: unit tests
Bug: 249374572
Change-Id: Ife931e26367fd4aab302bedc709cf907e2c01245
This allows us to set product variables as build settings instead
of loading them from a target's provider, which further allows us
to read product config variables in transitions.
Bug: 287539062
Bug: 269577299
Test: Presubmits
Change-Id: I8497703f706162572ceb3486240e1eb02a37f5f6
* changes:
Respect package boundaries in bp2build conversion of go modules
Create a temporary denylist for go binaries used in mixed builds
Partial bp2build conversion of blueprint_go_binary
Partial bp2build conversion of bootstratp_go_package
bp2build's codegen context does not implement
BazelPathConversionContext. To reuse the utility function
transformPackagePaths, update its signature
(Also make deps of go_library unique to make the conversion resilient)
Test: go test ./bp2build
Change-Id: I126b1057d2b26bc6c7d3be2780f1b62d28323cf0
This allows us to rollout building _some_ go targets using rules_go
without affecting mixed builds.
Test: Presubmit
Bug: 284483729
Change-Id: I0ccb4c9b90614369147a380f44f7ae372ef9396e
This module type does not implement android.Module, and therefore we
cannot write a conventional bp2build converter for this module type.
Instead this has been special-cased inside bp2build/build_conversion.go.
There is one major deviation between Soong and Bazel's
go_binary/go_library. Soong
collects the deps in the transitve closure and puts them on compile/link
paths. Bazel OTOH, requires the direct imports to be listed in deps of the binary
explicitly (AFAIK). Since bp2build cannot determine the list of direct
imports from the list of transitive deps, put all the transitive deps in
`deps`
Test: unit tests
Test: TH
Bug: 284483729
Change-Id: I004aaf8607fef1697a0d9e7d018ad657b67778ac
This module type does not implement android.Module, and therefore we
cannot write a conventional bp2build converter for this module type.
Instead this has been special-cased inside bp2build/build_conversion.go.
Because of the above, we also do not have access to useful functions
available in the ctx object of ConvertWithBp2build. This includes
1. Finding the package (directory) of a dep. This requires getting a
handle of the underlying module from its name (string). To solve, we
do a pre-visit to collect this information. This did not increase the
wall time. On my machine, `m bp2build --skip-soong-tests` takes ~14s
before and after this CL
2. Converting srcs to labels. This requires glob and package boundary
resolution. This CL introduces a partial implementation for this
function. (glob patterns are not used in go tools)
For (1), I considered creating a `ModuleFromName` on
`blueprint.Context` instead of a pre-run, but this increased the time to ~27s.
Test: unit tests
Test: TH
Bug: 284483729
Change-Id: Ifeb029103d14947352556dba295591dd7038b090
Bazel does not support using .rscrip/.fs files as cc srcs.
For a module foo with foo srcs, Bp2build will create a bazel
target of rule class rscript_to_cpp with name foo_renderscript
which will have only the renderscript files as srcs.
Bp2build will also create a target foo with the expected cc rule class.
The foo target will have all other src files as srcs and will also
have the target foo_renderscript as another src.
Bug: 210509914
Test: bp2build testing & b build target rstest-latency
Change-Id: Ifdc55051a3595f5fcf54eab8b59e11e9351e141c
This involves some minor changes to testing infrastructure.
Bug: 285631638
Fixes: 286545783
Test: unit test
Change-Id: If64ba29308d99e63a1cc5526feaf077f2cb14478
In Soong, CFI being explicitly set to false overrides presence in
CFI_INCLUDE_PATHS, so this change causes Bazel to mimic the same
precedence.
Bug: 278789968
Test: Unit tests
Change-Id: I827b3d4272eac2002031970f1e67c9321979b14b
This also changes the expectation of ConvertWithBp2build. Each
implementation must either create one or more Bazel target modules, or
mark the module as unconvertible (with a specific reason).
Manually verified no runtime hit in AOSP
In AOSP, the metrics file size increases from 252K to 1.6M
This changes some effective module counts in bp2build metrics:
- Removes "package" modules from the module count list in
metrics, as these will not be converted like regular modules.
- Counts Handcrafted modules as being "unconverted", as bp2build is not
responsible for them.
Bug: 285631638
Test: Verified generated BUILD.bazel files are bit-for-bit identical
with this change
Test: Manually verified one case of each implemented reasonType
Change-Id: I308dd451d8f28379b15671dae9f931bd0446f5c1
So that it's easier to maintain by being closer to the actual long-term
usage.
Bug: 279095899
Test: m nothing
Change-Id: I0b7cfa2eeb48595dd3081b458bbd1156ec2184a3
