This excludes the bp2build mutator from running for cc_genrule or
java_genrule, which may require alternative implementations.
Test: bp2build; bazel build //bionic/...
Change-Id: Ie14dd6dfa5e806fd6ac6e8f9989356f3880c79bf
Using $out.rsp as the rsp file adds extra complexity around keeping
the $ unescaped. Make callers to FlagWithRspFileInputList provide
an explicit path for the rsp file instead.
Bug: 182612695
Test: rule_builder_test.go
Change-Id: I3f531d80c1efa8a9d09aac0a63790c5b11a9f0c6
modules converted with bp2build_available are will also be available to
be used in mixed builds.
Test: build/bazel/scripts/milestone-2/demo.sh full
Test: go tests
Change-Id: I49f16ec3ba5bb11dfed8066af069c27eb04371fb
This allows more direct access to the bazel label and whether the module
is bp2build available.
Test: go test *
Change-Id: I23375835d20fa53d7d94127b2dc2d5bb20487bfb
This eliminates the need to remove quotes, delete attributes, and
re-checking that name has correct prefix. Additionally, this allows
assignment directly to the BazelTargetModuleProperties struct, which
allows defaulting unused fields and clarity of which field is being set.
Test: go test soong tests
Test: ran ./build/bazel/scripts/milestone-2/demo.sh
Change-Id: Ia9bfcce76234c793a4ddd5f29a661150f83341c9
This CL replaces the "__bp2build__" name prefix boilerplate with a props
creation function, and centralizes the prefixing in there.
Test: TH
Test: soong tests
Change-Id: Ic963199ab60dcce0d3361abff111cfa9acd4c21b
This CL adds a per-target allowlist to instruct bp2build on which modules it should generate Bazel targets for.
Test: soong tests
Change-Id: I869e66fce405c2c6689b381569b8cc0118cbcf76
This CL refactors the CreateBazelTargetModule API to minimize boilerplate, and to establish a well defined function signature for the expected metadata about a BazelTargetModule.
Test: soong tests
Test: TH
Change-Id: I474ff5a2b0db8deeed49ba4ca73b416ccb494fdd
This currently expands all globs, still need to support converting glob
syntax.
Test: go build_conversion_test
Test: GENERATE_BAZEL_FILES=true m nothing
Test: m nothing
Bug: 165114590
Change-Id: If7b26e8e663d17566fad9614ca87a8da1f095284
The previous implementation relied on the implicit registration of Bp2Build mutators, resulting in test non-hermeticity. Refactor bp2build tests to explicitly specify the bp2build mutators under test.
Test: Soong tests
Test: TH
Change-Id: I9b9674bad1ea533b3bd31b07077a9e02c99b4c1d
Results of `cquery //bionic/...`: https://paste.googleplex.com/5608885982330880?raw
Note that this CL does not contain the global module name mapping to
fully qualified label lookup mechanism, so while the BUILD targets are
queryable, they are not buildable (yet).
Test: GENERATE_BAZEL_FILES=true m nothing && ./build/bazel/scripts/bp2build-sync.sh write && bazel cquery //bionic/...
Change-Id: I6f2316222723d8b612d9b041ca82a188cd05bac9
This relands I38393900677c5dbe8e72fe06a7dd7d71f3c09f82 after
I8038ed5c6f05c989ac21ec06c4552fb3136b9a7a, which makes the ASAN
variants of libraries have PackagingSpecs so they can be copied
into sandboxed genrules.
Copy tools used by genrules into the sandbox directory. This
ensures correct dependencies on all files used by tools, and
is one step closer to enabling genrules inside unselected
namespaces.
Bug: 124313442
Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I57c0d5fc8bba216fac4deb972d0d2098593e8963
This effectively moves execution of Bazel actions outside of soong_build
and alongside ninja execution of the actual ninja files, whether that be
by ninja or by Bazel itself.
This almost allows for mixed builds and Bazel-as-Ninja-executor to
coexist, but requires hacks explained in b/175307058.
Test: Treehugger
Test: lunch aosp_flame && USE_BAZEL_ANALYSIS=1 m libc
Test: lunch aosp_flame && USE_BAZEL=1 USE_BAZEL_ANALYSIS=1 m libc,
though this requires a hack of the main BUILD file. See b/175307058
Change-Id: Ia2f6b0f1057e8cea3809de66d8287f13d84b510c
Current HWASan build on master fails if OUT_DIR is ~17 characters,
passes with 3. Halve defaultShardSize to avoid this.
Bug: 170327170
Test: build with long OUT_DIR
Change-Id: I04e3cae91118891c0e28d895108670dd0d373a3f
Enable the RuleBuilder and RuleBuilderCommand methods to access
the BuilderContext by passing it to NewRuleBuilder instead of
RuleBuilder.Build.
Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I63e6597e19167393876dc2259d6f521363b7dabc
gensrcs modules run the same command once for each input file to
produce each output file. Each command needs its own depfile instead
of reusing one per shard. The depfiles will be merged together
by RuleBuilder into one depfile per shard.
Test: TestGenSrcs
Change-Id: Iaf4f2cf9f5592c20e32944ddf34e0a61aff17ba8
Relands Ic22603a5c0718b5a21686672a7471f952b4d1017 with a minor
change to track libc++ dependencies for python hosts and after
a fix to an internal genrule that depended on transitively
installed java libraries (ag/13068670).
Soong currently assumes that installed files should depend on
installed files of all transitive dependencies, which results
in extra installed file dependencies through genrules, static
libs, etc.
Annotate dependency tags for dependencies for which the
installed files are necessary such as shared libraries
and JNI libraries.
This avoids extra installed files, and is also a first step
towards genrules using their own copy of tools instead of
the installed copy.
Bug: 124313442
Test: m checkbuild
Test: java.TestBinary
Test: cc.TestInstallSharedLibs
Test: deptag_test.go
Change-Id: I725871249d561428e6f67bba6a7c65b580012b72
This relands I3b918a6643cea77199fd39577ef71e34cdeacdb1 with a fix
to create the directory for the output depfile if doesn't exist.
In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.
Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I8af00c8c0b25a92f55a5032fcb525715ae8297c2
The depfile was intended to be placed alongside the per-shard
output file, but was in the final directory instead. Move
it into the per-shard directory.
Also clean up and comment how shards and shard directories
work.
Test: m framework-cppstream-protos
Test: TestGenSrcs
Change-Id: Ia526ae8997314998814ed8e35598cc2a5bc4aabd
In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.
Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I3b918a6643cea77199fd39577ef71e34cdeacdb1
In preparation for more complicated sandboxing support in sbox, use
a single implementation of the sbox sandboxing by moving genrule to
use RuleBuilder's sbox support instead of creating an sbox rule
directly.
Also move genrule's input list hash support into RuleBuilder.
Test: genrule_test.go
Test: rule_builder_test.go
Change-Id: I292184d02743c7e6887ebbcd232ba565db2ab0cc
Soong currently assumes that installed files should depend on
installed files of all transitive dependencies, which results
in extra installed file dependencies through genrules, static
libs, etc.
Annotate dependency tags for dependencies for which the
installed files are necessary such as shared libraries
and JNI libraries.
This avoids extra installed files, and is also a first step
towards genrules using their own copy of tools instead of
the installed copy.
Bug: 124313442
Test: m checkbuild
Test: java.TestBinary
Test: cc.TestInstallSharedLibs
Test: deptag_test.go
Change-Id: Ic22603a5c0718b5a21686672a7471f952b4d1017
Prepare for using Config when adding singletons by passing
Config to NewTestContext and NewContext instead of to ctx.Register.
This will enable a followup change to store SingletonMakeVarsProviders
registered on the Context in the Config, which is necessary to run
multiple tests in parallel without data races.
Test: all soong tests
Change-Id: Id229629a4e42ff4487d317241673837726c075fc
Makes it possible for make modules to depend on genrules via
constructs like $(call intermediates-dir-for,ETC,foo)/foo.
Test: presubmit
Change-Id: Ic9a014f55ea503f1c0c8167f351ecd1407083562
Add vendor_ramdisk_available and vendor_ramdisk attribute to
various rules. When a vendor_ramdisk variant of a module is
generated, it is installed to $OUT/vendor-ramdisk.
It is similar to a ramdisk image.
Test: m nothing -j
Change-Id: Ib2d16459f3094dbe21c3bdb7c016cb4b2bf62765
This reverts commit 323dc60712.
Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests
Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
With this change, bazel_module is a specifiable property on
genrule module definitions. With bazel-enabled mode, soong_build will
defer to Bazel for information on these modules.
source build/soong/bazelenv.sh to enter bazel-enabled mode.
Test: Manually verified on bionic/libc genrules using aosp_cf_x86_phone-userdebug
Change-Id: I3619848186d50be7273a5eba31c79989b981d408
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.
Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
Enforce min_sdk_version for every payload dependency of updatable
APEX/APKs.
android.CheckMinSdkVersion() calls ApexModule.ShouldSupportSdkVersion
for every transitive dependency from APEX/APK modules to see if it
meets the min_sdk_version requirements.
The common implementation for apex/android_app is provided in
android/apex.go.
Bug: 145796956
Bug: 152655956
Bug: 153333044
Test: m nothing
Change-Id: I4a947dc94026df7cebd552b6e8ccdb4cc1f67170
Give prebuilt_etc and sh_binary their own packages and split the
gigantic main Android.bp up to small, per-package ones.
Test: m nothing, TreeHugger
Bug: 156980228
Change-Id: I7b00cd344b9f16861f1ff39edf0029f016b853d0
This change automates one of the best practices for
custom tools. The best practice says "Prefer to list
input files on the command line, otherwise we may not
know to re-run your command when a new input file is
added." [1]
Normally you'd reference these inputs with $(in) or
one of the forms of the $(location*) substituions on
the command line for the custom tool. However, if the
custom tool does not accept the list of inputs on the
command line, the build system could fail to re-run
the custom tool if the list changes.
This change adds a hash of the list of input names to
the sbox command that wraps the custom tool. If the
list of inputs change, the hash will change, and
therefore the sbox command will change, causing ninja
to re-run the custom tool. The hash is visible to
(but ignored by) the sbox command, and hidden from
your custom tool.
[1] https://android.googlesource.com/platform/build/soong/+/refs/heads/master/docs/best_practices.md#custom-build-tools
Test: TestGenruleHashInputs
Bug: 149397658
Change-Id: I18b547ea3c4296ee15bd6150a4778a8f376d80b7
This is for use to add dependencies between specific module variants,
without getting the dependencies potentially duplicated as more
variants are created.
Moved genrule tool dependency registration to this phase, to avoid
potential splitting of its dependencies, and for consistency with the
same kind of dependency on dex2oat in https://r.android.com/1205730.
Test: m nothing
Bug: 145934348
Change-Id: I1f9e75354e359cef5f41d14e5016b0a67d7655d7
The filesystem object was available through ModuleContext.Fs(), but
gives too much access to the filesystem without enforicing correct
dependencies. In order to support sandboxing the soong_build
process move the filesystem into the Config. The next change will
make it private.
Bug: 146437378
Test: all Soong tests
Change-Id: I5d3ae9108f120fd335b21efd612aefa078378813
Avoid having to pass ModuleFactoryAdaptor to every call to
RegisterModuleType in a test by wrapping RegisterModuleType.
Test: all soong tests
Change-Id: If8847d16487de0479cc3020b728256922b3cadba
Prepare for making the image mutator available to all modules and
moving it between the os and arch mutators by moving it into the
android package and using an interface implemented by the module
types to control it.
Bug: 142286466
Test: No unexpected changes to out/soong/build.ninja
Change-Id: I0dcc9c7b5ec80edffade340c367f6ae4da34151b
The arch variants are hardcoded in every module type. Refactor
them out into a Target.Variations() method in preparation for
splitting the arch mutator into two, which will require using
different variations.
Test: m checkbuild
Change-Id: I28ef7cd5168095ac888fe77f04e27f9ad81978c0
gensrcs modules use a single command to convert all of the sources,
which can lead to command line length limits, long critical path
times, and excessive rebuilds. Shard them into multiple rules,
defaulting to 100 commands in each.
Test: TestGenSrcs
Test: m
Fixes: 70221552
Fixes: 138438756
Fixes: 138829091
Change-Id: I8409e43011a4754e095ad1b368570a2ba8d75e50
Instead of using android.Phony, which is a "real" rule, use
blueprint.Phony, which is ninja's built-in "phony" rule. That's desired
in this case, since we only want this to be an alias to the inputs.
Test: cd system/bt; mma; mma (second run had nothing to do)
Test: treehugger
Change-Id: Ib0b630e23653c05a12cf617350f8b4c6d6d6cdff
For remote builds, we actually need to know that we depend on all
outputs from a genrule, not just the first one. But having every user
depend on every genrule output increases the ninja file size by >40%.
So to work around the increase in file size, use a ninja phony rule to
produce a single alias to all of the files. Unlike make, where phony
rules are always dirty, ninja does not attempt to stat phony rules as
long as they have inputs, so they act as an alias. (If they don't have
inputs, it will stat the file an only consider it dirty if it doesn't
exist.)
My remote build tooling can then see that it's a phony rule with inputs,
and collapse all of these inputs onto the actual rule.
This only applies to genrules that have >6 outputs, in order to keep the
graph simpler and easier to read for the common case. That's ~10% of the
genrules in AOSP, and it increases the ninja file size by ~1.3%.
Test: manual ninja file inspection
Test: treehugger
Change-Id: I1a26a02fe983f0c92ce726ada3133707223e662e
RuleBuilder does its own ninja escaping, so values that will be
passed to RuleBuilder must not be pre-escaped. Add a new
android.ExpandNinjaEscaped method that explicitly handles ninja
escaping. Some of the expansion functions return ninja values
(like "${in}") that need to stay unescaped, so add a bool return
value to the expansion function in android.ExpandNinjaEscaped.
Test: expand_test.go
Change-Id: Ib03d4db38b5e3e5bffbd87acf14f55e276a53d04
* changes:
Consolidate *MutatorContext and ModuleContext into BaseModuleContext
Add test for missing defaults modules with AllowMissingDependencies
Capture missing dependency error rules
Share buildDir for android/soong/android tests
Reimplement AddMissingDependencies
Add GenerateAndroidBuildActions to DefaultsModuleBase
genrule modules were not mutated by the apex mutator. As a result, a
module that is mutated for an apex have depended on the genrule modules
host_bionic_linker_{asm|flags} that are not mutated for the APEX.
This in turn caused an inconsistency problem because the
host_bionic_linker_* genrules use the dynamic linker that isn't mutated
for APEX as an input, while the outputs of the genrules are used by
modules mutated for APEX. The inconsistency caused b/135008828.
Fixing this issue by correctly let genrule to be mutated by the apex
mutator.
Bug: 135008828
Test: checkout master-art-host
ALLOW_MISSING_DEPENDENCIES=true DIST_DIR=out/dist
./art/tools/dist_linux_bionic.sh -j50 showcommands
com.android.runtime.host
is passing
Change-Id: I92d90400c61d3951f6202d932832d0cb63c95a7d
Add an empty GenerateAndroidBuildActiosn to DefaultsModuleBase
so that every defaults module doesn't need to provide one. This
will also allow adding an implementation in the next patch.
Test: m checkbuild
Change-Id: I13554bdb3a287c2f18e1efab74d4f08a1ba8620c
Set the location label for missing srcs and tools to avoid
nonsensical errors when parsing the command.
Test: genrule_test.go
Test: paths_test.go
Test: unbundled branch with missing framework-res module needed by robolectric genrule
Change-Id: I9c1f1cd82a80f048c0e903b8e93910b1ae34b0b1
Add a proto.plugin property to allow specifying a custom protoc
plugin to generate the code.
Fixes: 70706119
Test: m am StreamingProtoTest
Change-Id: I1ecdd346284b42bbcc8297019d98d2cd564eb94c
Export genrules to Make, which will allow it to support
dist-for-goals.
Test: m PRODUCT-sdk_phone_armv7-win_sdk sdk_repo dist
Bug: 80494441
Bug: 128878015
Change-Id: Idb8dfcec2be5bfab94d17395826ef451b0dd9b63
Move the logic from ctx.ExpandSources into android.PathsForModuleSrc
and ctx.ExpandSource into android.PathForModuleSrc, and deprecate
them. When combined with the pathDepsMutator this will let all
properties that take source paths also take filegroups or genrule
outputs, as long as they are tagged with `android:"path"`.
Test: All soong tests
Change-Id: I01625e76b5da19240e9649bf26a014eeeafcab8f
Add `android:"path"` to all properties that take paths to source
files, and remove the calls to ExtractSource(s)Deps, the
pathsDepsMutator will add the necessary SourceDepTag dependency.
Test: All soong tests
Change-Id: I488ba1a5d680aaa50b04fc38acf693e23c6d4d6d
Follow the change to blueprint to make *Escape take and return a string
and add *EscapeList that take and return slices of strings. Fix up
a few places that were unnecessarily converting a string to a slice
and back to a string.
Test: m nothing
Change-Id: I3fa87de175522205f36544ef76aa2f04aef1b936
Add an empty DepsMutator to ModuleBase so it doesn't have to be
implemented on every module that doesn't need it.
Test: all soong tests
Change-Id: I545a832a0dbf27386d3080377a75ea482cd9ce59
Add arch_variant to the genrule srcs property so that cc_genrule
modules can vary the srcs and outs by architecture.
Test: cc/genrule_test.go
Change-Id: Idb17013e137d7ca21ca4cfc3e4c1fa7d89977043
There was no way to select a single source file from a genrule that
has multiple source files. Make Soong's genrule closer to Bazel
by supporting $(location) for input and outputs. Change the
label used for tools referenced through filegroups to the name
of the module instead of the name of the file, which means it
matches the string used in the tools property.
Also support :module format in the tools property in preparation
for deprecating the tool_files property and putting both files
and modules in the tools property.
Bug: 117354232
Test: genrule_test.go
Change-Id: Id31a4ac4ff7064076a576c1d8ffeb7c19fc6b9a4
Merged-In: Id31a4ac4ff7064076a576c1d8ffeb7c19fc6b9a4
(cherry picked from commit 098d22b5f6)
- Register a singleton and implement GenerateBuildActions func in java/jdeps.go.
- Declare a interface and a struct to collect info in android/module.go.
- Implement IDEInfo for Library & Import module in java/jdeps.go.
- Implement IDEInfo for Genrule module in genrule/genrule.go.
- Implement IDEInfo for fileGroup module in android/filegroup.go.
- Test codes for jdeps.go in java/jdeps_test.go.
Bug: 111044346
Test: export SOONG_COLLECT_JAVA_DEPS=1;mmm packages/apps/Settings
out/soong/module_bp_java_deps.json will be generated
Change-Id: If61da77b4d7614c2c5da438b6af4c725ceccc5c3
Report errors inside the android.Expand lambda and don't return an
error so that processing can continue and multiple errors can be
reported. Check for errors (for example incorrect syntax of variables)
immediately after Expand returns. Avoids a misleading error message:
specified depfile=true but did not include a reference to '${depfile}' in cmd
Bug: 111219250
Test: m checkbuild
Change-Id: Id9a16c1609f5fd9345bfa1a2191261cff72fd382
SourceDepTag is going to become a set of tags of the same type
instead of a single tag, remove references to it outside the
android module.
Bug: 80144045
Test: soong tests
Change-Id: I00b2ea5040e4fc95dfbfdd79e21579853c478fcb
Bug: http://b/64121881
Bug: http://b/78188880
- Allow filegroup's properties to be extended by a LoadHook
- Support a filegroup (':module') in a prebuilt's 'Srcs' property to
export files from a different path as the prebuilt's sources.
This change also includes a refactoring that moves genrule/filegroup.go
to android/filegroup.go so that FileGroupFactory is visible in
prebuilt_test.go.
Test: Test
https://android-review.googlesource.com/c/platform/development/+/469159
in clang-tools branch on Linux, Darwin. Test regular build in
aosp/master.
Change-Id: I3ff6215ab2e62955f039fd1086c31f1bd50ebcf6
ExpandSourcesSubDir was calling SourceFileProducer.Srcs(), and then
doing:
moduleSrcs = append(moduleSrcs[:j], moduleSrcs[j+1:]...)
This modifies the backing store of the slice, which may affect the
original data stored in the SourceFileProducer. Make all Srcs
implementations return slice that points to a copy of the backing
array.
Test: m out/target/common/obj/PACKAGING/checkpublicapi-current-timestamp
Bug: b/76179848 b/76397326
Change-Id: I2432ce196984814daafc5aa9a2746e81de74494c
While the rule may really need all of the generated header files to
exist, only one of them (per genrule task) needs to be in the dependency
list, since the rest are essentially aliases.
This brings an AOSP aosp_arm-userdebug out/soong/build.ninja file from
372MB to 156MB, with equivalent functionality. The Android-aosp_arm.mk
file is reduced from 11MB to 6.5MB.
Bug: 73745773
Test: diff out/soong/build.ninja
Test: diff out/soong/Android-aosp_arm.mk
Test: rm -rf out; m
Change-Id: If17377666292cc20957417fc4c3cd52f98971d0c
The path to the output directory may be arbitrarily long, use
__SBOX_OUT_DIR__ in the list of output files passed to sbox
to avoid expanding it multiple times in the command line.
Fixes:
ninja: fatal: posix_spawn: Argument list too long
09:40:14 ninja failed with: exit status 1
when building libchrome with a long OUT or OUT_DIR_COMMON_BASE.
Bug: 73726635
Test: m checkbuild
Change-Id: I59024b2164287c8e531711afd9273b692ce9c28a
This reverts commit 606e9de344.
Reason for revert: <try to fix the broken build yesterday>
Change-Id: I2963b9af63c7c7398159e5e9a1e448266e1c81d5
Test: unittest
Bug: 70387174
Test: Put this text into an Android.bp:
genrule {
name: "test_genrule",
tool_files: ["foo"],
out: ["foo.c"],
cmd: "for i in a b c; do echo $$i; done > $(out)",
}
cc_library {
name: "libtest_genrule",
srcs: [":test_genrule"],
}
and then run `m -j libtest_genrule`. Although the library
shouldn't compile, check that it produces a foo.c that has
"a\nb\n\c\n" and not "\n\n\n".
Change-Id: I139106479439c9b3a95f1a2ecc23e73570d7bd59
Also allow commands with no tools for the case when standard
shell utilities are used to munge an input file.
Test: m checkbuild
Change-Id: Ie061e90cafe1a0a0db004a89e9a17fb48709cb03
tool_files can be used to add dependencies on files used by tools,
so let it support globs.
Test: m checkbuild
Change-Id: I1c1cb098190e1bb1c81292f6eb0c4ed0e240c1e1
AConfig() now duplicates Config(). Replace the uses of AConfig()
with Config(). Leave AConfig() for now until code in other
projects is cleaned up.
Test: m checkbuild
Change-Id: Ic88be643049d21dba45dbd1a65588ed94bf43bdc
genrules lost the ability to depend on Go tools after
I05e945f38915d49cd3c0ab72a86576949bc7eff2 which converted
VisitDirectDeps from blueprint Modules to android Modules.
Add VisitDirectDepsBlueprint to visit all modules including
blueprint Modules, and use it in genrule. Also add a check
for disabled modules that was being handled by
VisitDirectDeps.
Test: m checkbuild
Change-Id: I65724283166c63596d071e598c08fed87ef32896
