`statslog.cpp` with cmd `"$(location stats-log-api-gen) --cpp $(genDir)/statslog.cpp"` produces `out/.intermediates/frameworks/proto_logging/stats/stats_log_api_gen/statslog.cpp/gen/statslog.cpp`. Hence, $(genDir) is equivalent to `<package-dir>/<module-name>/gen`.
Currently, bp2buld converts `$(genDir)` to `$(GENDIR)`. In Bazel, `$(GENDIR)` is only the base of the generated code (e.g. `bazel-bin`).
```
genrule(
name = "statslog.cpp",
cmd = "$(location :stats-log-api-gen) --cpp $(GENDIR)/statslog.cpp",
outs = ["statslog.cpp"],
tools = [":stats-log-api-gen"],
)
```
produces `bazel-bin/statslog.cpp` but expects to have `bazel-bin/frameworks/proto_logging/stats/stats_log_api_gen/statslog.cpp`.
By converting to `$(RULEDIR)` which is `bazel-bin/frameworks/proto_logging/stats/stats_log_api_gen`.
There had not been any genrule module allowlisted with genDir
yet. So this should not cause any issue with modules converted before
this CL.
Bug: 247536535
Test: go tests
Test: presubmit builds and tests
Change-Id: I65c6aafadab6b180b7ef700427e041547ae7e98a
'$' cannot be included in the genrule source file name as it is an
invalid character.
One workaround to include a file with '$' in the filename is to use
glob(*) and match pattern.
However, shell currently evaluates the '$' sign and leads to unexpected behavior.
This change fixes the issue by shell-escaping the filepath in generating
build actions.
Test: m
Bug: b/194980152
Change-Id: I6fd919c568b5b6526e4de5155104a08ecadab307
All existing gensrsc modulePartners who use the property can use BUILD_BROKEN_DEP_FILE to bypass the error
Test: CI
Bug: 179452413
Fix: 179452413
Change-Id: I7cd39484b43eba693d79188b9a374f192198f90f
This also introduces a workaround for the fact that
apexer depends on aapt2, but aapt2 doesn't build
with bp2build yet. Aapt2 is removed from apexer's
requirements during bp2build.
Bug: 204244290
Test: ./build/bazel/ci/bp2build.sh
Change-Id: I837597ce035c7d5c06e1a3957166583a7a94b5c7
This large refactoring has both immense performance implications and
improves mixed builds complexity / usability. Summary:
1. Queueing calls to Bazel is done in a new mutator instead of a full
soong_build pass. Normal soong_build flow is interrupted (via a
functional hook in blueprint) to invoke bazel and parse its response.
2. Implementing mixed build support for additional modules is as simple
as implementing MixedBuildsBuildable. In this interface, define the
request that must be queued to Bazel, and then subsequently define
how to handle the returned bazel cquery metadata.
3. Mixed builds consists of only a single pass. This greatly
improves mixed build performance.
Result:
A 33% runtime improvement on soong analysis phase with mixed builds.
Caveats:
C++ BazelHandler handling still remains a bit of a mess; I did what
I could within this CL's scope, but this may require additional cleanup.
Test: Treehugger
Test: Verified that aosp_arm ninja file is bit-for-bit identical with or
without this change.
Change-Id: I412d9c94d429105f4ebfafc84100d546069e6621
While Bazel genrules will allow genrules to accept a directory as input,
the results can be unexpected to a user as changes to the contents of
the directory may not trigger a rebuild as expected. Restricting this
in Soong ensures that converted targets will behave as expected.
Test: CI
Change-Id: I8616f58d1df267005e6c0ca3f4730d06de52c0d9
Point the user toward checking their $(location) label is mentioned
elsewhere in the build rule (as opposed to a typo of the module name,
for example).
Test: genrule_test.go
Change-Id: Icc19740938e2b56eef24540534f9cc5bfa9420b8
Bp2build currently supports building genrules for cc modules, but does
not support building java_genrule* modules. This commit adds this
functionality.
Bug: 213480907
Test: go test ./bp2build
Change-Id: I473196c5bcf5582ba0c8faa65b5005f81ac973a4
Since the genrule tool dep mutator runs after the prebuilt mutators,
this adds a helper function android.PrebuiltGetPreferred to resolve the
source or prebuilt as appropriate.
Test: m SOONG_CONFIG_art_module_source_build=false droid
in internal
Bug: 214292395
Change-Id: I1a208fd048b998f9f19ad1f45d8389decda2cb9e
Each conversion required defining a separate mutator, which will each
operate on _all_ modules and requires each to repeat checks whether the
mutator should operator. Instead, we introduce a single mutator and
modules can define a ConvertWithBp2build to implement bp2build
conversion for that module.
Test: bp2build.sh
Bug: 183079158
Change-Id: I99d4b51f441c2903879092c5b56313d606d4338d
Add annotations to dependency tags that are dynamic or classpath
linkage.
Bug: 207445310
Test: m checkbuild
Change-Id: Ife89b8f234aa40c380c721eda7dd18cab697fbb3
Add PathForGoBinary that uses pathForInstall to return the install
path of a GoBinaryTool. This will replace various places that used
PathForOutput to reconstruct a path to a Go tool, and will support
moving Go tools to the Make install directory outside of the
PathForOutput directory in a future patch.
Bug: 204136549
Test: m checkbuild
Change-Id: I83a3be9f5c621975540f5ed601a0b9e2611c98b9
This also removes the special-case filegroup from mixed builds
buildroot; no special handling is required. Since we're currently
hardcoding linux_x86_64 as our host platform, it should be fine
to continue assumping that hardcoded host for now.
Test: USE_BAZEL_ANALYSIS=1 m adbd
Change-Id: I35509f4eb33ba7a243fab4c34b35958f6f2fceab
The previous error message did not really give a good indication
of what the problems with my genrule was.
Test: Make a broken genrule
Change-Id: I4bb67f2792ca5a910f70354f10d0f81f330b5c96
Introduce `commonAttributes` & `fillCommonBp2BuildModuleAttrs used in
CreateBazelTargetModule
Adapt `bp2BuildInfo` to use `commonAttrs` instead of `Name`.
And thus also all downstream users of `CreateBazelTargetModule`.
As initial user, the Soong `required` property will be
translated to Bazel's `data`.
Bug: 198146582, 196091467
Test: build_converstion_test.go:TestCommonBp2BuildModuleAttrs
Test: go test
Test: mixed_{libc,droid}.sh
Change-Id: Ib500e40f7e2cb48c459f1ebe3188962fc41ec124
Pass the architecture, mulitlib type and native bridge state to
each variant of a cc_genrule rule as environment variables.
Bug: 200872604
Test: TestCmdPrefix
Change-Id: I39c4c2d5bbd4f4cc72a4777715db1df049345b37
A performance improvement for bp2build as Blueprint/Soong no longer have
the overhead of additional modules. The creation of these modules
results in:
* traversal of additional modules for each subsequent mutator
* synchronization over a go channel to collect newly created modules:
https://cs.android.com/android/platform/superproject/+/master:build/blueprint/context.go;l=2594,2600;drc=1602226f23181b8c3fbfcaf3358f0297e839d7d3
We avoid both of these by storing the information directly in the
underlying module.
Also as a fringe benefit, removes some necessary boilerplate for
conversion.
For benchmarks, reduces runtime ~1% for 1% converted, ~24% for 100%
converted. See more: go/benchmarks-for-https:-r.android.com-1792714
Test: ran benchmarks/tests in bp2build
Test: build/bazel/ci/bp2build.sh
Change-Id: Ie9273b8cbab5bc6edac1728067ce184382feb211
Because it's commonly useful and not cc/ specific
Also export GenerateBazelBuildActions and update uses
Test: Existing tests pass
Change-Id: Ibc6858bb1129afba181a7686dda432defe33b00d
Refactor bp2build to retrieve modules directly by name, instead of via
DirectDeps. This functions properly as bp2build has no need for variant
information of the blueprint graph.
Test: USE_BAZEL_ANALYSIS=1 m fmtlib
Change-Id: Ief4b67bc56f24929871af772f3a742f07085bf8c
And when genrule generates multiple output files, tag can be used to
choose a single output file.
Bug: 192200378
Test: soong test
Change-Id: I3e44c137ad95468616ab883d3b277593cd82d1e8
A module will be installed to /debug_ramdisk if debug_ramdisk is set to
true.
This is a reland of f84e9c05e2, with a fix
that removes /first_stage_ramdisk.
Bug: 184004542
Test: soong test
Change-Id: I739de63cfec6b0fec5a90f7c4741fc4d884d209c
This reverts commit f84e9c05e2.
Because this breaks the usage of boot-debug.img and
vendor_boot-debug.img
Bug: 185970130
Bug: 185990198
Test: make bootimage_debug
Change-Id: I7886c971982faae1d5bc34688643de8d94d6e201
OriginalModuleName is a clearer name for what the field represents.
Also document it.
Follow-up from aosp/1675466.
Test: TH
Change-Id: Ie1152b5ae63f388164582be70e193a91ef96c89c
This allows us to, for certain modules, enable bp2build, but disable
mixed builds.
This facilitates easier iteration on these two features, as modules can
be incrementally "fixed" for bp2build, even if they are not ready for
mixed builds integration.
Test: USE_BAZEL_ANALYSIS=1 m libc
Test: Manually verified that note_* modules exist in bionic/libc/BUILD
Test: bazel build //bionic/....
Change-Id: I3462c8398680140cfc953443d96196143b8ad0a3
A module will be installed to debug_ramdisk (or
debug_ramdisk/first_stage_ramdisk if recovery as boot is true) if
debug_ramdisk is set to true.
Bug: 184004542
Test: soong test
Change-Id: Ic5a4d27407e506fffa462de2149e0785f11b2ac7
Rather than having a single storage container, make multiple so that it
is easier to unit test building modules with bazel with MockBazelContext
Test: go test genrule_test
Change-Id: I1da85d28f096d5102ad889b9518fdda6914342b1
Previously, genrules which were used as generated_headers only
propagated the old Soong gen directory as an include path, even if the
genrule was managed by Bazel. With this chance, the gen directory of the
Baze output files are used in mixed-build mode.
Test: USE_BAZEL_ANALYSIS=1 m libc
Test: m libc
Change-Id: I3913c4e5b0354421ba1fe43d1f6ef4d1e729a058
Use an interface instead of a string to retrieve the value for
$(location) or $(locations) expansions in genrules to allow
delaying the evaluation until the RuleBuilderCommand is available.
This allows using helpers like RuleBuilderCommand.PathForInputs
to properly rewrite the values for sandboxing.
Also remove the standalone SboxPathFor* methods that don't operate
on a specific RuleBuilderCommand that are now unnecessary.
Test: genrule_test.go
Change-Id: I8bb2647332ef118204a216cead23d062517e2b8c
This CL adds the support for specifying lists of directories in
build/soong/android/bazel.go, which are then written into
out/soong/bp2build/MANIFEST. Using this configuration,
modules/directories can either default to bp2build_available: true or
false, while still retaining the ability to opt-in or out at the module level.
It also ensures that ConvertWithBp2Build returns true iff the module
type has a registered bp2build converter.
Test: go tests
Test: demo.sh full
Test: TreeHugger presubmits for bp2build and mixed builds.
Change-Id: I0e0f6f4b1b2ec045f2f1c338f7084defc5d23a55
If a genrule depends on a module that is disabled, in this case because
it is a device module in a host-only build, it can cause panics when
getPathsFromModuleDep retrieves a nil Path from the disabled module.
Treat disabled modules as missing dependencies.
Test: TestGenruleAllowMissingDependencies
Change-Id: I3c689c6b5505b21eaf7ae7cb93c00f96f438ac17
These packages have already been migrated to use per test build
directory so have no need for a FixtureFactory.
Bug: 183235980
Test: m nothing
Change-Id: I667d1d992caaf0f615de91f89efdae11c44986c2
This excludes the bp2build mutator from running for cc_genrule or
java_genrule, which may require alternative implementations.
Test: bp2build; bazel build //bionic/...
Change-Id: Ie14dd6dfa5e806fd6ac6e8f9989356f3880c79bf
Using $out.rsp as the rsp file adds extra complexity around keeping
the $ unescaped. Make callers to FlagWithRspFileInputList provide
an explicit path for the rsp file instead.
Bug: 182612695
Test: rule_builder_test.go
Change-Id: I3f531d80c1efa8a9d09aac0a63790c5b11a9f0c6
modules converted with bp2build_available are will also be available to
be used in mixed builds.
Test: build/bazel/scripts/milestone-2/demo.sh full
Test: go tests
Change-Id: I49f16ec3ba5bb11dfed8066af069c27eb04371fb
This allows more direct access to the bazel label and whether the module
is bp2build available.
Test: go test *
Change-Id: I23375835d20fa53d7d94127b2dc2d5bb20487bfb
This eliminates the need to remove quotes, delete attributes, and
re-checking that name has correct prefix. Additionally, this allows
assignment directly to the BazelTargetModuleProperties struct, which
allows defaulting unused fields and clarity of which field is being set.
Test: go test soong tests
Test: ran ./build/bazel/scripts/milestone-2/demo.sh
Change-Id: Ia9bfcce76234c793a4ddd5f29a661150f83341c9
This CL replaces the "__bp2build__" name prefix boilerplate with a props
creation function, and centralizes the prefixing in there.
Test: TH
Test: soong tests
Change-Id: Ic963199ab60dcce0d3361abff111cfa9acd4c21b
This CL adds a per-target allowlist to instruct bp2build on which modules it should generate Bazel targets for.
Test: soong tests
Change-Id: I869e66fce405c2c6689b381569b8cc0118cbcf76
This CL refactors the CreateBazelTargetModule API to minimize boilerplate, and to establish a well defined function signature for the expected metadata about a BazelTargetModule.
Test: soong tests
Test: TH
Change-Id: I474ff5a2b0db8deeed49ba4ca73b416ccb494fdd
This currently expands all globs, still need to support converting glob
syntax.
Test: go build_conversion_test
Test: GENERATE_BAZEL_FILES=true m nothing
Test: m nothing
Bug: 165114590
Change-Id: If7b26e8e663d17566fad9614ca87a8da1f095284
The previous implementation relied on the implicit registration of Bp2Build mutators, resulting in test non-hermeticity. Refactor bp2build tests to explicitly specify the bp2build mutators under test.
Test: Soong tests
Test: TH
Change-Id: I9b9674bad1ea533b3bd31b07077a9e02c99b4c1d
Results of `cquery //bionic/...`: https://paste.googleplex.com/5608885982330880?raw
Note that this CL does not contain the global module name mapping to
fully qualified label lookup mechanism, so while the BUILD targets are
queryable, they are not buildable (yet).
Test: GENERATE_BAZEL_FILES=true m nothing && ./build/bazel/scripts/bp2build-sync.sh write && bazel cquery //bionic/...
Change-Id: I6f2316222723d8b612d9b041ca82a188cd05bac9
This relands I38393900677c5dbe8e72fe06a7dd7d71f3c09f82 after
I8038ed5c6f05c989ac21ec06c4552fb3136b9a7a, which makes the ASAN
variants of libraries have PackagingSpecs so they can be copied
into sandboxed genrules.
Copy tools used by genrules into the sandbox directory. This
ensures correct dependencies on all files used by tools, and
is one step closer to enabling genrules inside unselected
namespaces.
Bug: 124313442
Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I57c0d5fc8bba216fac4deb972d0d2098593e8963
This effectively moves execution of Bazel actions outside of soong_build
and alongside ninja execution of the actual ninja files, whether that be
by ninja or by Bazel itself.
This almost allows for mixed builds and Bazel-as-Ninja-executor to
coexist, but requires hacks explained in b/175307058.
Test: Treehugger
Test: lunch aosp_flame && USE_BAZEL_ANALYSIS=1 m libc
Test: lunch aosp_flame && USE_BAZEL=1 USE_BAZEL_ANALYSIS=1 m libc,
though this requires a hack of the main BUILD file. See b/175307058
Change-Id: Ia2f6b0f1057e8cea3809de66d8287f13d84b510c
Current HWASan build on master fails if OUT_DIR is ~17 characters,
passes with 3. Halve defaultShardSize to avoid this.
Bug: 170327170
Test: build with long OUT_DIR
Change-Id: I04e3cae91118891c0e28d895108670dd0d373a3f
Enable the RuleBuilder and RuleBuilderCommand methods to access
the BuilderContext by passing it to NewRuleBuilder instead of
RuleBuilder.Build.
Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I63e6597e19167393876dc2259d6f521363b7dabc
gensrcs modules run the same command once for each input file to
produce each output file. Each command needs its own depfile instead
of reusing one per shard. The depfiles will be merged together
by RuleBuilder into one depfile per shard.
Test: TestGenSrcs
Change-Id: Iaf4f2cf9f5592c20e32944ddf34e0a61aff17ba8
Relands Ic22603a5c0718b5a21686672a7471f952b4d1017 with a minor
change to track libc++ dependencies for python hosts and after
a fix to an internal genrule that depended on transitively
installed java libraries (ag/13068670).
Soong currently assumes that installed files should depend on
installed files of all transitive dependencies, which results
in extra installed file dependencies through genrules, static
libs, etc.
Annotate dependency tags for dependencies for which the
installed files are necessary such as shared libraries
and JNI libraries.
This avoids extra installed files, and is also a first step
towards genrules using their own copy of tools instead of
the installed copy.
Bug: 124313442
Test: m checkbuild
Test: java.TestBinary
Test: cc.TestInstallSharedLibs
Test: deptag_test.go
Change-Id: I725871249d561428e6f67bba6a7c65b580012b72
This relands I3b918a6643cea77199fd39577ef71e34cdeacdb1 with a fix
to create the directory for the output depfile if doesn't exist.
In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.
Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I8af00c8c0b25a92f55a5032fcb525715ae8297c2
The depfile was intended to be placed alongside the per-shard
output file, but was in the final directory instead. Move
it into the per-shard directory.
Also clean up and comment how shards and shard directories
work.
Test: m framework-cppstream-protos
Test: TestGenSrcs
Change-Id: Ia526ae8997314998814ed8e35598cc2a5bc4aabd
In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.
Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I3b918a6643cea77199fd39577ef71e34cdeacdb1
In preparation for more complicated sandboxing support in sbox, use
a single implementation of the sbox sandboxing by moving genrule to
use RuleBuilder's sbox support instead of creating an sbox rule
directly.
Also move genrule's input list hash support into RuleBuilder.
Test: genrule_test.go
Test: rule_builder_test.go
Change-Id: I292184d02743c7e6887ebbcd232ba565db2ab0cc
Soong currently assumes that installed files should depend on
installed files of all transitive dependencies, which results
in extra installed file dependencies through genrules, static
libs, etc.
Annotate dependency tags for dependencies for which the
installed files are necessary such as shared libraries
and JNI libraries.
This avoids extra installed files, and is also a first step
towards genrules using their own copy of tools instead of
the installed copy.
Bug: 124313442
Test: m checkbuild
Test: java.TestBinary
Test: cc.TestInstallSharedLibs
Test: deptag_test.go
Change-Id: Ic22603a5c0718b5a21686672a7471f952b4d1017
Prepare for using Config when adding singletons by passing
Config to NewTestContext and NewContext instead of to ctx.Register.
This will enable a followup change to store SingletonMakeVarsProviders
registered on the Context in the Config, which is necessary to run
multiple tests in parallel without data races.
Test: all soong tests
Change-Id: Id229629a4e42ff4487d317241673837726c075fc
Makes it possible for make modules to depend on genrules via
constructs like $(call intermediates-dir-for,ETC,foo)/foo.
Test: presubmit
Change-Id: Ic9a014f55ea503f1c0c8167f351ecd1407083562
Add vendor_ramdisk_available and vendor_ramdisk attribute to
various rules. When a vendor_ramdisk variant of a module is
generated, it is installed to $OUT/vendor-ramdisk.
It is similar to a ramdisk image.
Test: m nothing -j
Change-Id: Ib2d16459f3094dbe21c3bdb7c016cb4b2bf62765
This reverts commit 323dc60712.
Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests
Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
With this change, bazel_module is a specifiable property on
genrule module definitions. With bazel-enabled mode, soong_build will
defer to Bazel for information on these modules.
source build/soong/bazelenv.sh to enter bazel-enabled mode.
Test: Manually verified on bionic/libc genrules using aosp_cf_x86_phone-userdebug
Change-Id: I3619848186d50be7273a5eba31c79989b981d408
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.
Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
