In conjunction with soong/build changes, this materialized runtime
metrics for various soong_build events.
Test: Manually verified materialized protos for bp2build, mixed builds,
and legacy build.
Change-Id: Ia92403605e3063028dbf6a1ded8449c190b9e63e
Variant names are part of the name of the generated Ninja rule.
toNinjaName needs to be applied to it as well.
Test: presubmit
Change-Id: I4fb908b6824179440b1c207240bad59dc85009fa
This was the only one in the source tree.
Side cleanup: remove some dead code that I assume comes from the time
where Blueprint files had to specify what subdirectories other Blueprint
files are in.
Test: Presubmits.
Change-Id: If84c4e85bc5516f30da97c1be29b56e50dddb3c4
These are just out/ and out/soong/ and the old names were quite
confusing.
Test: Presubmits.
Merged-In: Idd9ce3c38a259faabcc56f0cd3fdac8b289123b1
Merged-In: I334ab40c668d2a94536f3e63d5f1fa0b401388ac
Change-Id: Ib7c568c6a97701f2240c5e3f0f2ce67397819ac0
Propagate the ninja file deps from a LoadHook to the build.ninja.d
file.
Bug: 188547846
Test: next CL
Change-Id: If8176474b5094ee40d07df12f5da79a906ce7290
Use moduleInfo.String() to print dependency cycle errors so they
contain the variant.
Test: Test_parallelVisit
Change-Id: I2bddaa35c8abb57c42b4c424e861361a8813d588
parallelVisit supports mutating the dependency graph while it is being
visited by proceeding until there are no modules with their dependencies
satisfied, and then checking if there are modules that haven't been
visited yet. If so, it assumes there was a newly introduced dependency
cycle and tries to find it to return as an error.
Finding the dependency cycle could traverse outside of the cycle.
If the dependency cycle occurs near the bottom of the dependency graph,
that traversal could be both long and wide, leading to very long
runtimes.
Memoize traversed modules that were not found to be part of the
dependency cycle to prevent repeated traversals.
Fixes: 186572387
Test: introduce cycle into libc, m nothing
Test: Test_parallelVisit
Change-Id: I38d0749dbedffbe8a39e433d97fbe08486451321
There are a few cases that force all globs to be rerun at the beginning
of the build (changes to bpglob or dependencies, second build after a
clean build). The number of globs has gotten high enough that rerunning
them all can have significant overhead to start bpglob for each one.
Replace the per-glob bpglob invocations with sharded invocations using
1024 hash buckets.
Bug: 159845846
Test: glob_test.go
Test: m nothing && m nothing
Test: build/soong/bootstrap_test.sh
Change-Id: Ife1f7a03c8f6b25d1be01531425d8dc2c76d1ea0
Fix detecting cycles in parallelVisit when the first alphabetical
module is not part of the cycle, but depends on the cycle. Instead of
starting from the first alphabetical module, check every module in a
determinsitic order and return the first time a cycle is found.
Test: Test_parallelVisit
Change-Id: I03726f838ec42975251088ba75158103940115c2
Avoid reallocating module.forwardDeps and module.reverseDeps every
time through updateDependencies by resetting the slices without
reducing their capacity. Accumulate dependencies to visit directly
into module.forwardDeps. Use a loop instead of a map to check
for duplicates, average number of dependencies is measured to be
9.5, although there are a few outliers with up to 2108.
Reduces mean soong_build execution time on internal master from 87s
to 82.7s (5%).
Test: context_test.go
Change-Id: I58fcd5514e494bafa965443461851b21b7bce382
Variables, pools and rules each computed their full names every time
they were referenced, which required string concatenations. Since
every one is guaranteed to be accessed at least twice, once when the
definition is written into the ninja file and once for each reference,
precompute the full name. For local variables that can be done
during initialization, but for global variables add a pass to
PrepareBuildActions to compute the name for each live variable using
the final package names.
Test: ninja_writer_test.go
Change-Id: I2264b05e0409e36651db2fb5d463c16c698d4d5e
ninjaWriter repeatedly called io.WriteString() on its writer, which
does a type assertion every time. Replace its io.Writer with an
io.StringWriter and call WriteString on it directly.
Test: ninja_writer_test.go
Change-Id: Ie073d996a319190242bf6a00af07a13a60d078b5
* Invalidate module group cache if deps are modified
PreSingletons run after the blueprints have been parsed and can run
VisitAllModules; however, this seeds the cache of sorted modules which
may change after mutators have run.
This causes recomputation of the cache if any deps have been
modified since the last time it was run.
Change-Id: I79fc822dd630f84790f309ba4e6024588a8fe28e
In particular the formatting of dependencies with variants which lacked
braces that caused them to float together with the dependency names.
Also add some context to the ReplaceDependenciesIf panic message.
Test: m
Change-Id: Ibd03690317ca53f3ecbdd95033404d89d849b4dd
Providers are a new concept for Blueprint, based on providers in Bazel:
https://docs.bazel.build/versions/master/skylark/rules.html#providers
Providers aim to simplify the interaction between modules by replacing
type asserting to arbitrary interfaces with requesting optional data
objects from modules. This will also move Blueprint closer to supporting
incremental analysis by serializing the providers and only rerunning
the analysis phase on modules whose inputs have changed.
Change-Id: I39f5f78b372412a7dbf151ceccb3f917f6c874bf
The motivaion for this change is to allow writing code that uses the
newly added dependency module in the same mutator pass, for example to
add more dependencies. Like this:
for _, m := range ctx.AddVariationDependencies(nil, tag, deps...) {
if someModuleProperty(m); ok {
ctx.AddVariationDependencies(nil, tag, otherDep)
}
}
Note that there is no guarantee that the returned module has already
been processed by the current mutator.
The patch does not add runtime overhead on findng dependency modules,
as this has already been done previously.
Test: go test
Pass a channel to visitor functions called by parallelVisit that
allows them to pause the current visitor until a given visitor
has finished. This allows parallelVisit to work on a dependency
graph while it is being mutated.
Test: Test_parallelVisit
Change-Id: Id8b1542c22ac9914439310e31d992ae0d7318d69
Pull request #317 made the equality check in moduleMatchingVariant
more correct by comparing the variant maps instead of the names.
Using only the names effectively ignores any variation with an
empty name. Unfortuantely this broke a current user of
ReplaceDependenciesIf. Go back to the relaxed check for now.
Test: build on a branch with sdk modules
Change-Id: I11016c8df7bd91fd022d04fd3033756f54d7fa0b
AddFarVariationDependencies claims to take the first variant that
matches all the requested variations, but it did nothing of the sort.
It took the first variant that either matched or did not contain each
of the requested variations. A module with no variations was always
matched, and requesting variations that didn't apply to a module
still matched (for example, requesting an image variation for a
host module that was ignored by the image mutator).
Fix AddFarVariationDependencies by making subset actually check
for subsets.
Test: Test_findVariant
Change-Id: I10063fec342db2a1c0685a7db08e4a650d14bd4e
AddFarVariationDependencies takes the first matching variant. To
maintain sensible behavior on a module with aliases, the ordering
of aliases and module variants needs to be maintained so that
AddFarVariationDependencies can find an earlier matching alias
instead of a more specific variant.
Test: go test .
Change-Id: I78f4e96edd98159f3a62d94e240e5d652667bec4
Add tests for findVariant behavior that provides the matching
behaviors of AddVariationDependencies, AddFarVariationDependencies,
etc.
Test: Test_findVariant
Change-Id: I3494d57179c8b3d62f7d32e5a1b43c9b9672c2df
CreateAliasVariation creates a new variation for the current mutator
that is an alias to a variation that was created with CreateVarations
by the same mutator. It is similar to AliasVariation, except that
AliasVariation creates an alias from the pre-mutated variant to one
of the new variations, while CreateAliasVariation creates an alias
from a new post-mutated variation to one of the new variations.
Bug: 164216768
Test: TestCreateAliasVariation
Change-Id: I8847b8d6fc1d3248abc910b2fe2399881f9bdaee
Context.ModulePath is a duplicate of Context.BlueprintFile, and is
misleading because it returns the path to the Android.bp, and not
the path to the directory that contains the Android.bp file like
ModuleContext.ModuleDir.
Change-Id: I505136fce3c3d37595cd4b4e08de5e2691a2a0f6
Adds a CheckBlueprintSyntax(...) method to check the syntax of a
Blueprint file.
Changes processModuleDef and newModule from being method on *Context to
being standalone functions. That ensures that CheckBlueprintSyntax(...)
does not need to take a context and so there is no chance that it can
change its state.
Previously a LoadHook could not modify the name of a module because the
module was registered before the LoadHooks were run. That made it very
complicated (requiring mutators and auto generated names) to create a
module type whose name was determined by say the directory in which it
is defined.
This change moves the LoadHook execution slightly earlier so it runs
before registration of the module.
That caused one slight side problem which was that the
moduleInfo.Name() would fail when called in a LoadHook. That was
because that gets the name from group.name but was group was nil
because it is only set when the module is registered.
Modifying the moduleInfo.Name() method to get the name from the module
logicModule.Name() if group is nil fixed that. The reason for getting
the name from the group.name rather than the logicModule.Name() is that
the former tracks renames but the latter does not. However that is not
an issue in this case as there has been no opportunity for the module
to be renamed until after the LoadHook has returned.
Previously, WalkDeps() would record that a module was visited after the
first time it encountered the module irrespective of whether it recursed
into or not. This change moves the recording so it happens only after it
has been recursed into.
Added TestWalkDepsDuplicates_IgnoreFirstPath to test the change. Without
the change the test fails because it does not visit E.
Test refactoring:
* A depsMutator was added instead of relying on blueprintDepsMutator to
allow different tags to be used for different dependency types.
* Modified barModule and fooModule to support the new depsMutator and
add support for another type of dependency that is ignored by the
walking code.
* Extracted walkDependencyGraph() function to reuse common code.
Mutators were not propagating the results of ctx.AddNinjaFileDeps.
Test: examine out/soong/build.ninja.d
Fixes: 150689149
Change-Id: Ia1e69ebc9dfa94a05f4ecd9cc2a8691ee63c9dd5
There are 8935901 *ninjaString objects generated in an AOSP
aosp_blueline-userdebug build, and 7865180 of those are a literal
string with no ninja variables.
Each of those *ninjaString objects takes a minimum of 48 bytes for
2 slices, plus 8 bytes for the pointer to the ninjaString. For
the literal string case, one of those slices has a single element,
(costing another 16 bytes for the backing array), and the other
slice is empty, for a total of 72 bytes.
Replace *ninjaString with a ninjaString interface. This increases
the size of the reference from 8 bytes to 16 bytes, but using
a type alias of a string for the literal string implementation uses
only 16 bytes, saving 40 bytes per literal string or 314 MB.
Test: ninja_strings_test
Change-Id: Ic5fe16ed1f2a244fe6a8ccdf762919634d825cbe
The proptools functions took an inconsistent variety of
struct and *struct types. Some methods even took a struct
but returned a *struct. Make all the exported methods
take a *struct, with internal helpers for the ones that need
to take a struct.
Test: proptools tests
Change-Id: I60ce212606e96adcef66c531d57f69c39e1a1638
Add RegisterScopedModuleType to LoadHookContext that registers
a module type factory that will be visible for the remainder of
the file. Also add ModuleFactories that returns the globally
register module factories.
Test: all blueprint tests
Change-Id: If646a73befe3b8e45c4b0984531c6a39ddc8d066
Move LoadHooks from Blueprint and run them during ParseBlueprintsFiles.
This will allow them to add scoped module types that are visible to the
rest of the Blueprints file. Requires passing the config object to
ParseBlueprintsFiles.
Test: all blueprint tests
Change-Id: Ia2a2c9a0223d5458bfd48bd22ebed0fdbd0156c6
