Prepare for making the image mutator available to all modules and
moving it between the os and arch mutators by moving it into the
android package and using an interface implemented by the module
types to control it.
Bug: 142286466
Test: No unexpected changes to out/soong/build.ninja
Change-Id: I0dcc9c7b5ec80edffade340c367f6ae4da34151b
The arch variants are hardcoded in every module type. Refactor
them out into a Target.Variations() method in preparation for
splitting the arch mutator into two, which will require using
different variations.
Test: m checkbuild
Change-Id: I28ef7cd5168095ac888fe77f04e27f9ad81978c0
gensrcs modules use a single command to convert all of the sources,
which can lead to command line length limits, long critical path
times, and excessive rebuilds. Shard them into multiple rules,
defaulting to 100 commands in each.
Test: TestGenSrcs
Test: m
Fixes: 70221552
Fixes: 138438756
Fixes: 138829091
Change-Id: I8409e43011a4754e095ad1b368570a2ba8d75e50
Instead of using android.Phony, which is a "real" rule, use
blueprint.Phony, which is ninja's built-in "phony" rule. That's desired
in this case, since we only want this to be an alias to the inputs.
Test: cd system/bt; mma; mma (second run had nothing to do)
Test: treehugger
Change-Id: Ib0b630e23653c05a12cf617350f8b4c6d6d6cdff
For remote builds, we actually need to know that we depend on all
outputs from a genrule, not just the first one. But having every user
depend on every genrule output increases the ninja file size by >40%.
So to work around the increase in file size, use a ninja phony rule to
produce a single alias to all of the files. Unlike make, where phony
rules are always dirty, ninja does not attempt to stat phony rules as
long as they have inputs, so they act as an alias. (If they don't have
inputs, it will stat the file an only consider it dirty if it doesn't
exist.)
My remote build tooling can then see that it's a phony rule with inputs,
and collapse all of these inputs onto the actual rule.
This only applies to genrules that have >6 outputs, in order to keep the
graph simpler and easier to read for the common case. That's ~10% of the
genrules in AOSP, and it increases the ninja file size by ~1.3%.
Test: manual ninja file inspection
Test: treehugger
Change-Id: I1a26a02fe983f0c92ce726ada3133707223e662e
RuleBuilder does its own ninja escaping, so values that will be
passed to RuleBuilder must not be pre-escaped. Add a new
android.ExpandNinjaEscaped method that explicitly handles ninja
escaping. Some of the expansion functions return ninja values
(like "${in}") that need to stay unescaped, so add a bool return
value to the expansion function in android.ExpandNinjaEscaped.
Test: expand_test.go
Change-Id: Ib03d4db38b5e3e5bffbd87acf14f55e276a53d04
* changes:
Consolidate *MutatorContext and ModuleContext into BaseModuleContext
Add test for missing defaults modules with AllowMissingDependencies
Capture missing dependency error rules
Share buildDir for android/soong/android tests
Reimplement AddMissingDependencies
Add GenerateAndroidBuildActions to DefaultsModuleBase
genrule modules were not mutated by the apex mutator. As a result, a
module that is mutated for an apex have depended on the genrule modules
host_bionic_linker_{asm|flags} that are not mutated for the APEX.
This in turn caused an inconsistency problem because the
host_bionic_linker_* genrules use the dynamic linker that isn't mutated
for APEX as an input, while the outputs of the genrules are used by
modules mutated for APEX. The inconsistency caused b/135008828.
Fixing this issue by correctly let genrule to be mutated by the apex
mutator.
Bug: 135008828
Test: checkout master-art-host
ALLOW_MISSING_DEPENDENCIES=true DIST_DIR=out/dist
./art/tools/dist_linux_bionic.sh -j50 showcommands
com.android.runtime.host
is passing
Change-Id: I92d90400c61d3951f6202d932832d0cb63c95a7d
Add an empty GenerateAndroidBuildActiosn to DefaultsModuleBase
so that every defaults module doesn't need to provide one. This
will also allow adding an implementation in the next patch.
Test: m checkbuild
Change-Id: I13554bdb3a287c2f18e1efab74d4f08a1ba8620c
Set the location label for missing srcs and tools to avoid
nonsensical errors when parsing the command.
Test: genrule_test.go
Test: paths_test.go
Test: unbundled branch with missing framework-res module needed by robolectric genrule
Change-Id: I9c1f1cd82a80f048c0e903b8e93910b1ae34b0b1
Add a proto.plugin property to allow specifying a custom protoc
plugin to generate the code.
Fixes: 70706119
Test: m am StreamingProtoTest
Change-Id: I1ecdd346284b42bbcc8297019d98d2cd564eb94c
Export genrules to Make, which will allow it to support
dist-for-goals.
Test: m PRODUCT-sdk_phone_armv7-win_sdk sdk_repo dist
Bug: 80494441
Bug: 128878015
Change-Id: Idb8dfcec2be5bfab94d17395826ef451b0dd9b63
Move the logic from ctx.ExpandSources into android.PathsForModuleSrc
and ctx.ExpandSource into android.PathForModuleSrc, and deprecate
them. When combined with the pathDepsMutator this will let all
properties that take source paths also take filegroups or genrule
outputs, as long as they are tagged with `android:"path"`.
Test: All soong tests
Change-Id: I01625e76b5da19240e9649bf26a014eeeafcab8f
Add `android:"path"` to all properties that take paths to source
files, and remove the calls to ExtractSource(s)Deps, the
pathsDepsMutator will add the necessary SourceDepTag dependency.
Test: All soong tests
Change-Id: I488ba1a5d680aaa50b04fc38acf693e23c6d4d6d
Follow the change to blueprint to make *Escape take and return a string
and add *EscapeList that take and return slices of strings. Fix up
a few places that were unnecessarily converting a string to a slice
and back to a string.
Test: m nothing
Change-Id: I3fa87de175522205f36544ef76aa2f04aef1b936
Add an empty DepsMutator to ModuleBase so it doesn't have to be
implemented on every module that doesn't need it.
Test: all soong tests
Change-Id: I545a832a0dbf27386d3080377a75ea482cd9ce59
Add arch_variant to the genrule srcs property so that cc_genrule
modules can vary the srcs and outs by architecture.
Test: cc/genrule_test.go
Change-Id: Idb17013e137d7ca21ca4cfc3e4c1fa7d89977043
There was no way to select a single source file from a genrule that
has multiple source files. Make Soong's genrule closer to Bazel
by supporting $(location) for input and outputs. Change the
label used for tools referenced through filegroups to the name
of the module instead of the name of the file, which means it
matches the string used in the tools property.
Also support :module format in the tools property in preparation
for deprecating the tool_files property and putting both files
and modules in the tools property.
Bug: 117354232
Test: genrule_test.go
Change-Id: Id31a4ac4ff7064076a576c1d8ffeb7c19fc6b9a4
Merged-In: Id31a4ac4ff7064076a576c1d8ffeb7c19fc6b9a4
(cherry picked from commit 098d22b5f6)
- Register a singleton and implement GenerateBuildActions func in java/jdeps.go.
- Declare a interface and a struct to collect info in android/module.go.
- Implement IDEInfo for Library & Import module in java/jdeps.go.
- Implement IDEInfo for Genrule module in genrule/genrule.go.
- Implement IDEInfo for fileGroup module in android/filegroup.go.
- Test codes for jdeps.go in java/jdeps_test.go.
Bug: 111044346
Test: export SOONG_COLLECT_JAVA_DEPS=1;mmm packages/apps/Settings
out/soong/module_bp_java_deps.json will be generated
Change-Id: If61da77b4d7614c2c5da438b6af4c725ceccc5c3
Report errors inside the android.Expand lambda and don't return an
error so that processing can continue and multiple errors can be
reported. Check for errors (for example incorrect syntax of variables)
immediately after Expand returns. Avoids a misleading error message:
specified depfile=true but did not include a reference to '${depfile}' in cmd
Bug: 111219250
Test: m checkbuild
Change-Id: Id9a16c1609f5fd9345bfa1a2191261cff72fd382
SourceDepTag is going to become a set of tags of the same type
instead of a single tag, remove references to it outside the
android module.
Bug: 80144045
Test: soong tests
Change-Id: I00b2ea5040e4fc95dfbfdd79e21579853c478fcb
Bug: http://b/64121881
Bug: http://b/78188880
- Allow filegroup's properties to be extended by a LoadHook
- Support a filegroup (':module') in a prebuilt's 'Srcs' property to
export files from a different path as the prebuilt's sources.
This change also includes a refactoring that moves genrule/filegroup.go
to android/filegroup.go so that FileGroupFactory is visible in
prebuilt_test.go.
Test: Test
https://android-review.googlesource.com/c/platform/development/+/469159
in clang-tools branch on Linux, Darwin. Test regular build in
aosp/master.
Change-Id: I3ff6215ab2e62955f039fd1086c31f1bd50ebcf6
ExpandSourcesSubDir was calling SourceFileProducer.Srcs(), and then
doing:
moduleSrcs = append(moduleSrcs[:j], moduleSrcs[j+1:]...)
This modifies the backing store of the slice, which may affect the
original data stored in the SourceFileProducer. Make all Srcs
implementations return slice that points to a copy of the backing
array.
Test: m out/target/common/obj/PACKAGING/checkpublicapi-current-timestamp
Bug: b/76179848 b/76397326
Change-Id: I2432ce196984814daafc5aa9a2746e81de74494c
While the rule may really need all of the generated header files to
exist, only one of them (per genrule task) needs to be in the dependency
list, since the rest are essentially aliases.
This brings an AOSP aosp_arm-userdebug out/soong/build.ninja file from
372MB to 156MB, with equivalent functionality. The Android-aosp_arm.mk
file is reduced from 11MB to 6.5MB.
Bug: 73745773
Test: diff out/soong/build.ninja
Test: diff out/soong/Android-aosp_arm.mk
Test: rm -rf out; m
Change-Id: If17377666292cc20957417fc4c3cd52f98971d0c
The path to the output directory may be arbitrarily long, use
__SBOX_OUT_DIR__ in the list of output files passed to sbox
to avoid expanding it multiple times in the command line.
Fixes:
ninja: fatal: posix_spawn: Argument list too long
09:40:14 ninja failed with: exit status 1
when building libchrome with a long OUT or OUT_DIR_COMMON_BASE.
Bug: 73726635
Test: m checkbuild
Change-Id: I59024b2164287c8e531711afd9273b692ce9c28a
This reverts commit 606e9de344.
Reason for revert: <try to fix the broken build yesterday>
Change-Id: I2963b9af63c7c7398159e5e9a1e448266e1c81d5
Test: unittest
Bug: 70387174
Test: Put this text into an Android.bp:
genrule {
name: "test_genrule",
tool_files: ["foo"],
out: ["foo.c"],
cmd: "for i in a b c; do echo $$i; done > $(out)",
}
cc_library {
name: "libtest_genrule",
srcs: [":test_genrule"],
}
and then run `m -j libtest_genrule`. Although the library
shouldn't compile, check that it produces a foo.c that has
"a\nb\n\c\n" and not "\n\n\n".
Change-Id: I139106479439c9b3a95f1a2ecc23e73570d7bd59
Also allow commands with no tools for the case when standard
shell utilities are used to munge an input file.
Test: m checkbuild
Change-Id: Ie061e90cafe1a0a0db004a89e9a17fb48709cb03
tool_files can be used to add dependencies on files used by tools,
so let it support globs.
Test: m checkbuild
Change-Id: I1c1cb098190e1bb1c81292f6eb0c4ed0e240c1e1
AConfig() now duplicates Config(). Replace the uses of AConfig()
with Config(). Leave AConfig() for now until code in other
projects is cleaned up.
Test: m checkbuild
Change-Id: Ic88be643049d21dba45dbd1a65588ed94bf43bdc
genrules lost the ability to depend on Go tools after
I05e945f38915d49cd3c0ab72a86576949bc7eff2 which converted
VisitDirectDeps from blueprint Modules to android Modules.
Add VisitDirectDepsBlueprint to visit all modules including
blueprint Modules, and use it in genrule. Also add a check
for disabled modules that was being handled by
VisitDirectDeps.
Test: m checkbuild
Change-Id: I65724283166c63596d071e598c08fed87ef32896
in preparation for each task to delete its output dir
before running.
Bug: 38205169
Test: rm out -rf && m -j
Change-Id: Ifcd42073d4fb74f532adc0c095555b4a542feb99
rather than passing buildDir .
This will make it reasonable for sbox to delete its --output-root
Bug: 38205169
Test: m -j
Change-Id: Ic3a82a8df040117870c251c068a88545143257cf
Most notably, the sandbox depfile path should be passed into the tool.
Bug: 68336760
Test: m -j checkbuild
Change-Id: I22f944a3f57d613fda26de0ea777a915cafcd020
Also adds checks that the dependencies are android.Modules and
are not disabled.
Test: m checkbuild
Change-Id: I05e945f38915d49cd3c0ab72a86576949bc7eff2
Now that android.ModuleContext does not include blueprint.ModuleContext
we can rename android.ModuleContext.ModuleBuild to
android.ModuleContext.Build without colliding with
blueprint.ModuleContext.Build. Leave ModuleBuild as a wrapper around
Build for now to avoid having to update all the users outside
build/soong simultaneously.
Test: m checkbuild
Change-Id: I18eb8cc04faf002049a11d9aac97e9732ff5d638
Ninja doesn't support depfiles on a rule with multiple outputs.
Use a single output and put all the rest as implicit outputs.
Bug: 68057449
Test: java_test.go
Change-Id: Ia544493b1b3b51b185c865149d8f3e0eb3c57ee2
cc_genrule is the same as a normal genrule, but can depend on other cc
modules (like cc_object).
Test: mmma external/minijail
Change-Id: I8df87665c7bdc76ce89c92755c054f967a818e57
Allow sharing lists of files between make and soong by allowing
filegroups to export them as a make variable.
Test: m -j checkbuild
Change-Id: I27ae2f6d180bf01d69a628dbe59edcdba93da015
So that we don't get confused when using :<module> in srcs to depend on
a module that could also be a HostBinTool.
Test: m -j
Change-Id: Ia3b1c26826e70f84c6dc5ff78c95dd11d76901b6
Change module factories from returning a blueprint.Module and a list
of property structs to returning an android.Module, which holds the
list of property structs.
Test: build.ninja identical except for Factory: comment lines
Change-Id: Ica1d823f009db812c518f271a386fbff39c9766f
This causes Soong to put the outputs of each genrule into a temporary
location and copy the declared outputs back to the output directory.
This gets the process closer to having an actual sandbox.
Bug: 35562758
Test: make
Change-Id: I8048fbf1a3899a86fb99d71b60669b6633b07b3e
Descriptions currently look like:
[ 0% 4/29328] cc out-soong/.intermediates/external/clang/lib/Sema/libclangSema/android_arm_armv7-a-neon_denver_static_core/obj/external/clang/lib/Sema/SemaCodeComplete.o
This is not very helpful - most of the characters are used to show the
output path, which contains useful information like target architecture,
but also contains most of the path to the source files twice, and less
useful information like the exact variant name used by soong.
Make the descriptions look like:
[ 0% 3/29329] //external/clang/lib/Sema:libclangSema clang++ SemaTemplateInstantiate.cpp
This is //path/to/module:modulename tool relative/path/to/source/file
Test: builds, looks pretty
Change-Id: I3087aa7d4eb1860ef6239d77407b8b35445616d7
cmd may be empty if android.Expand fails, and continuing results
in a panic. Return immediately.
Test: m -j checkbuild
Change-Id: Ibf4d57ce048db9f7f16ce118ed9e9ecea5a6551c
Calling android.NewContext() in tests results in a context that
contains all the mutators, module types, and singletons, which
causes unexpected interactions in unit tests. Create an empty
context instead, and add in only the necessary mutators, module
types, and singletons.
Bug: 36366816
Test: soong tests
Change-Id: Ic61262c37e3436b3ad4ccaca18b737021c304be6
The genrule tool dependency handling was rejecting any dependencies
that were not tools. This caused a failure when ExtractSourceDeps
added a source file dependency on a filegroup module. Remove the
unnecessary check in the tool handling.
Test: builds
Bug: 35002681
Change-Id: Ibd8dff306be3ef55aac2e12a28ddc187be2234d9
Allow tests to specify a data property that lists files or filegroup
modules that will be packaged alongside the test. Also add a path
property to filegroup modules to allow shifting the path of the
packaged files, and add ExpandSourcesSubDir to expand the filegroup
sources while including a shifted relative path in the Paths objects.
Test: soong tests, manually adding data to a module
Change-Id: I52a48942660e12755d313ef13279313361b4fc35
filegroup is a module that contains a list of files, and can be used
to export files across package boundaries. filegroups (and genrules)
can be referenced from srcs properties of other modules using the
syntax ":module".
Test: m -j
Change-Id: I3d6fc4819c0b4225b474e0ad42f0d947f55a5961
Instead of exporting the generated sources dir as headers, let
the Android.bp file specify subdirectories as exported.
Test: m -j checkbuild
Change-Id: I18dd900d63ce7485c8fbfcc39dc77abad6f733d7
If a genrule sets deps_file: true, use a file in the gen directory
as a GCC-style deps file.
Test: m -j libLLVMObject
Change-Id: Id410165847e4eaea1853a392512e38787c431523
To allow genrules with more than one tool, rename the tool property to
tools and make it an array, replace $tool with $(location <label>),
and use $() for other variables for consistency.
Also remove the host bin directory from the genrule path, and the
$srcDir variable, using either of them would have caused dependency
issues.
Bug: 31948427
Test: compare build.ninja
Change-Id: Icf6d3bce2bea00fec1363fd65c0bdf96d09281bf
(cherry picked from commit de6bd86d24)
We were emulating this for proto files, standardize it and make the
other generators use it as well.
Test: Compare out/soong/build.ninja before/after change
Test: mmma -j system/tools/hidl
Change-Id: I1888c7b981749060a398387bbb9b481270bf6d75
Register mutators inside lambdas that are called in a defined order to
correctly order mutators before and after the arch and deps mutators.
Test: build.ninja identical
Change-Id: Iefe2a3515aee8570e76a6e76925db4cda0e9e822
Mutator registration is tightly coupled with the android package, move
all registration from the soong package to the android package.
Test: build.ninja identical
Change-Id: Ie183d0b52cc7431c9e05b231934d189208ef1efe
The underlying code to expose outputs to other modules has suppported
this already for gensrcs (which runs the command multiple times to
create multiple outputs). This allows a genrule to run a command once,
and output multiple files.
Bug: 31742855
Test: Add multi-output genrule; Inspect build.ninja
Change-Id: I72054cc39c0d0e3e44feebba72005e5af1a1709a
Instead of just passing this to ninja, use os.Expand to parse the
variable substitutions. While we're here, define a new "genDir" variable
that is the root of this module's generated directory.
Bug: 31742855
Test: Ensure invalid variables cause error
Test: Inspect commands in build.ninja
Change-Id: Iff6d0499ab57367669e73df52ab7c647358da13b
Append .Parallel() to all of the RegisterBottomUpMutator calls to tell
Blueprint it can run them in parallel.
Test: identical build.ninja, passes race detector
Change-Id: I969a0689522d4cba7c8ff51e2aa00fe2fd338a89
Soong's multi-architecture building has grown complex, with the
combination of HostOrDevice+HostType+Arch necessary to determine how to
build a variant of a module, and three separate mutators to split each
into its variations.
Combine HostOrDevice+HostType into Os, which will be Linux, Darwin,
Windows, or Android. Store Os+Arch as a single Target.
Change-Id: I92f2e2dac53617d595a35cc285d2bd348baa0fbd
Soong's multi-architecture building has grown complex, with the
combination of HostOrDevice+HostType+Arch necessary to determine how to
build a variant of a module, and three separate mutators to split each
into its variations.
Combine HostOrDevice+HostType into Os, which will be Linux, Darwin,
Windows, or Android. Store Os+Arch as a single Target.
Change-Id: Iae677eff61a851b65a7192a47f2dc17c1abb4160
This will allow modules to specify a local script file that will be
expanded to the proper path in $tool, and will automatically be added to
the dependencies.
Change-Id: Icea4046b1fe2fd7c543aa33913d1a53201b15353
Add two cc properties: `generated_sources` and `generated_headers`,
instead of just adding genrule modules to `deps`. Label these with the
dep tagging mechanism, add the generated_headers paths to the include
paths, and add dependencies to generated headers for each compile.
Add dependencies so that regular sources can include generated yacc
headers, not just other generated lex/yacc files. Static/shared specific
sources still don't have dependencies to or from regular sources though.
Switch from an implicit dependency on generated files to an orderonly
dependency, since the depfile will take care of the necessary implicit
dependencies for incremental builds.
Change-Id: I436675acb1f57329d98b81c56dcb4384201a68ea
Blueprint now requres DependencyTags to be passed to AddDependency
calls. Use the tags to avoid setting member variables on modules, which
will be lost after the mutators have been called.
Change-Id: I8c1d9ed1db85a300e14394b911a516d361ba9f75
This centralizes verification and common operations, like converting the
path to a source file to the path for a built object.
It also embeds the configuration knowledge into the path, so that we can
remove "${SrcDir}/path" from the ninja file. When SrcDir is '.', that
leads to paths like './path' instead of just 'path' like make is doing,
causing differences in compiled binaries.
Change-Id: Ib4e8910a6e867ce1b7b420d927c04f1142a7589e
This defines another mutator between HostOrDevice and Arch that will
expand host modules into a module for each host type
(Darwin/Linux/Windows) that is currently being built.
Change-Id: I4c8ac6b616c229f6bd45ad8a35902652fb6a4fff
EarlyMutators are identical to BottomUpMutators, except they run before
DynamicDependencies. DynamicDependencies can be replaced with a
BottomUpMutator. Replace both EarlyMutators and DynamicDependencies
with BottomUpMutators, which allows setting the order between all
mutators through registration order.
Change-Id: Id1305d798d3d2da592061c89d7c10a71780b71a3
Instead of putting all the blueprint registrations in soong_build,
put them all in init() functions. This puts the registration next
to the implementation.
Change-Id: Ide1a749518f5e9d1367a18ab3bb1d91da3310c76
To align with the current make build system, add exclude_srcs and
exclude_java_resource_dirs. These replace the functionality of
arch_subtract and glob exclusions that use "-file" to exclude a file.
Change-Id: I91c23d5e3c9409f2d9f7921f950153a03a68ad61
Improve the comments associated with properties to work better with
Blueprint's auto-documenting feature.
Make all properties structs into named types so that thet types can be
found using reflection and cross-referenced to the source code to
auto-extract docs.
Remove the leading <property>: text from properties, the documentation
will include the lowercased name of the property.
Add filter tags to the nested arch properties.
Change-Id: I4ef5db86358886fe61456c24eb2dbe6f7b876115
Make common.Glob take an interface that is the intersection of
blueprint.ModuleContext and blueprint.SingletonContext used by
Glob and GlobRule, allowing it to be called on either. Also
move ExpandSources and Glob into AndroidModuleContext.
Change-Id: I8d1a3160c5dd201033afdb37210e82d8065b137d
A far dependency will not inherit any of the variations from the current
module, allowing a device genrule to depend on a host tool to generate
the sources.
Change-Id: I7f622ddd31a81d605f55d5946a109ea53e864084
Add genrule, which uses a host executable (possibly built by the
build system) to generate a single source file. This differs slightly
from gensrcs, which takes a list of sources and translates them through
a host executable to individual source files.
Change-Id: I94bda62c4c53fb3f3817def190e6a7561508d297
Support -file or -path/glob in file lists that contain globs to
subtract files from the resulting glob. Also move source file
prefixing and handling into a common function.
Change-Id: Ib6d74ce22f53cae7348c4ba35b779976d90359a6
Using ctx.Config().(Config) everywhere is a mouthful, and it is
inefficient to do the type assertion. Put the Config interface into
the context, and provide an AConfig() to return the Config already
converted to the right type.
Change-Id: I301a1fd7d2a005580aabca7866a37c5d42ad8c69
Art needs a custom module type in order to perform complicated
build logic like depending on environment variables and varying
cflags based on cpu variant. Export enough of the types and
functions from cc for art_cc_library to inherit from cc_library.
While I'm touching every line, also rename the New* methods
to *Factory.
Change-Id: I7123aa47019c4ced7a1ab57c394225bc7844b5ea
