Commit graph

212 commits

Author SHA1 Message Date
Colin Cross
7ff2e8d2a4 Optimize updateDependencies
Avoid reallocating module.forwardDeps and module.reverseDeps every
time through updateDependencies by resetting the slices without
reducing their capacity.  Accumulate dependencies to visit directly
into module.forwardDeps.  Use a loop instead of a map to check
for duplicates, average number of dependencies is measured to be
9.5, although there are a few outliers with up to 2108.

Reduces mean soong_build execution time on internal master from 87s
to 82.7s (5%).

Test: context_test.go
Change-Id: I58fcd5514e494bafa965443461851b21b7bce382
2021-01-22 10:22:21 -08:00
Colin Cross
92054a49d2 Memoize full names of variables, pools and rules
Variables, pools and rules each computed their full names every time
they were referenced, which required string concatenations.  Since
every one is guaranteed to be accessed at least twice, once when the
definition is written into the ninja file and once for each reference,
precompute the full name.  For local variables that can be done
during initialization, but for global variables add a pass to
PrepareBuildActions to compute the name for each live variable using
the final package names.

Test: ninja_writer_test.go
Change-Id: I2264b05e0409e36651db2fb5d463c16c698d4d5e
2021-01-21 22:02:30 -08:00
Colin Cross
0335e0900d Use io.StringWriter in ninjaWriter
ninjaWriter repeatedly called io.WriteString() on its writer, which
does a type assertion every time.  Replace its io.Writer with an
io.StringWriter and call WriteString on it directly.

Test: ninja_writer_test.go
Change-Id: Ie073d996a319190242bf6a00af07a13a60d078b5
2021-01-21 22:02:30 -08:00
Liz Kammer
9ae14f12f9
Invalidate module group cache if deps are modified (#334)
* Invalidate module group cache if deps are modified

PreSingletons run after the blueprints have been parsed and can run
VisitAllModules; however, this seeds the cache of sorted modules which
may change after mutators have run. 

This causes recomputation of the cache if any deps have been
modified since the last time it was run.

Change-Id: I79fc822dd630f84790f309ba4e6024588a8fe28e
2020-11-30 16:30:45 -07:00
Martin Stjernholm
0f1637b19b
Improve formatting of some error messages. (#327)
In particular the formatting of dependencies with variants which lacked
braces that caused them to float together with the dependency names.

Also add some context to the ReplaceDependenciesIf panic message.

Test: m
Change-Id: Ibd03690317ca53f3ecbdd95033404d89d849b4dd
2020-11-16 12:15:36 -08:00
Colin Cross
2da8492b9d Add Providers to Blueprint
Providers are a new concept for Blueprint, based on providers in Bazel:
https://docs.bazel.build/versions/master/skylark/rules.html#providers

Providers aim to simplify the interaction between modules by replacing
type asserting to arbitrary interfaces with requesting optional data
objects from modules.  This will also move Blueprint closer to supporting
incremental analysis by serializing the providers and only rerunning
the analysis phase on modules whose inputs have changed.

Change-Id: I39f5f78b372412a7dbf151ceccb3f917f6c874bf
2020-09-22 18:20:18 -07:00
colincross
66fa73dd6e
Merge pull request #316 from skvadrik/add-dep-ret-mod
Return dependency modules from dependency-adding methods.
2020-09-22 10:59:46 -07:00
Ulya Trafimovich
9577bbc922 Return dependency modules from dependency-adding methods.
The motivaion for this change is to allow writing code that uses the
newly added dependency module in the same mutator pass, for example to
add more dependencies. Like this:

  for _, m := range ctx.AddVariationDependencies(nil, tag, deps...) {
      if someModuleProperty(m); ok {
          ctx.AddVariationDependencies(nil, tag, otherDep)
      }
  }

Note that there is no guarantee that the returned module has already
been processed by the current mutator.

The patch does not add runtime overhead on findng dependency modules,
as this has already been done previously.

Test: go test
2020-09-17 11:43:15 +01:00
Colin Cross
c4773d90a2 Support pausing parallelVisit
Pass a channel to visitor functions called by parallelVisit that
allows them to pause the current visitor until a given visitor
has finished.  This allows parallelVisit to work on a dependency
graph while it is being mutated.

Test: Test_parallelVisit
Change-Id: Id8b1542c22ac9914439310e31d992ae0d7318d69
2020-09-16 12:58:39 -07:00
Colin Cross
edbdb8c2d3 Relax check in moduleMatchingVariant
Pull request #317 made the equality check in moduleMatchingVariant
more correct by comparing the variant maps instead of the names.
Using only the names effectively ignores any variation with an
empty name.  Unfortuantely this broke a current user of
ReplaceDependenciesIf.  Go back to the relaxed check for now.

Test: build on a branch with sdk modules
Change-Id: I11016c8df7bd91fd022d04fd3033756f54d7fa0b
2020-09-11 19:24:59 -07:00
Colin Cross
5dc6759951 Fix AddFarVariationDependencies subset checks
AddFarVariationDependencies claims to take the first variant that
matches all the requested variations, but it did nothing of the sort.
It took the first variant that either matched or did not contain each
of the requested variations.  A module with no variations was always
matched, and requesting variations that didn't apply to a module
still matched (for example, requesting an image variation for a
host module that was ignored by the image mutator).

Fix AddFarVariationDependencies by making subset actually check
for subsets.

Test: Test_findVariant
Change-Id: I10063fec342db2a1c0685a7db08e4a650d14bd4e
2020-09-09 18:29:15 -07:00
Colin Cross
5df74a8e38 Maintain ordering between variants and aliases
AddFarVariationDependencies takes the first matching variant.  To
maintain sensible behavior on a module with aliases, the ordering
of aliases and module variants needs to be maintained so that
AddFarVariationDependencies can find an earlier matching alias
instead of a more specific variant.

Test: go test .
Change-Id: I78f4e96edd98159f3a62d94e240e5d652667bec4
2020-09-09 18:29:15 -07:00
Colin Cross
39644c0903 Add tests for findVariant
Add tests for findVariant behavior that provides the matching
behaviors of AddVariationDependencies, AddFarVariationDependencies,
etc.

Test: Test_findVariant
Change-Id: I3494d57179c8b3d62f7d32e5a1b43c9b9672c2df
2020-09-09 18:27:32 -07:00
Colin Cross
279489c017 Add CreateAliasVariation
CreateAliasVariation creates a new variation for the current mutator
that is an alias to a variation that was created with CreateVarations
by the same mutator.  It is similar to AliasVariation, except that
AliasVariation creates an alias from the pre-mutated variant to one
of the new variations, while CreateAliasVariation creates an alias
from a new post-mutated variation to one of the new variations.

Bug: 164216768
Test: TestCreateAliasVariation
Change-Id: I8847b8d6fc1d3248abc910b2fe2399881f9bdaee
2020-08-14 11:15:58 -07:00
Colin Cross
edc41769fe Combine variant fields into variant struct
Combine variant, variations and dependencyVariations into a single
struct.

Test: blueprint tests
Change-Id: I76811bb2bdf76f1b4cffc6a7b9bc7362ecb5f7b9
2020-08-14 11:15:58 -07:00
Colin Cross
8e454c5575 Remove unused Context.ModulePath
Context.ModulePath is a duplicate of Context.BlueprintFile, and is
misleading because it returns the path to the Android.bp, and not
the path to the directory that contains the Android.bp file like
ModuleContext.ModuleDir.

Change-Id: I505136fce3c3d37595cd4b4e08de5e2691a2a0f6
2020-07-06 13:34:42 -07:00
Paul Duffin
8969cb6170 Add ReplaceDependeciesIf to allow for conditional replacement
Adds support for making the replacement of one dependency with another
conditional on the from and to module as well as the tag used.
2020-06-30 12:15:26 +01:00
Martin Stjernholm
2f212479e2 Add ctx.OtherModule(Reverse)DependencyVariantExists.
This allows guarding calls to AddVariationDependencies and
AddReverseDependency to register dependencies on optional modules.
2020-05-26 20:52:43 +01:00
Paul Duffin
2a2c58ef46 Support checking syntax of generated Blueprint files
Adds a CheckBlueprintSyntax(...) method to check the syntax of a
Blueprint file.

Changes processModuleDef and newModule from being method on *Context to
being standalone functions. That ensures that CheckBlueprintSyntax(...)
does not need to take a context and so there is no chance that it can
change its state.
2020-05-13 09:06:17 +01:00
Paul Duffin
244033b20f Run LoadHooks before registering module
Previously a LoadHook could not modify the name of a module because the
module was registered before the LoadHooks were run. That made it very
complicated (requiring mutators and auto generated names) to create a
module type whose name was determined by say the directory in which it
is defined.

This change moves the LoadHook execution slightly earlier so it runs
before registration of the module.

That caused one slight side problem which was that the
moduleInfo.Name() would fail when called in a LoadHook. That was
because that gets the name from group.name but was group was nil
because it is only set when the module is registered.

Modifying the moduleInfo.Name() method to get the name from the module
logicModule.Name() if group is nil fixed that. The reason for getting
the name from the group.name rather than the logicModule.Name() is that
the former tracks renames but the latter does not. However that is not
an issue in this case as there has been no opportunity for the module
to be renamed until after the LoadHook has returned.
2020-05-04 14:16:03 +01:00
Paul Duffin
72bab1707e WalkDeps - only record module visited when it has been recursed into
Previously, WalkDeps() would record that a module was visited after the
first time it encountered the module irrespective of whether it recursed
into or not. This change moves the recording so it happens only after it
has been recursed into.

Added TestWalkDepsDuplicates_IgnoreFirstPath to test the change. Without
the change the test fails because it does not visit E.

Test refactoring:
* A depsMutator was added instead of relying on blueprintDepsMutator to
  allow different tags to be used for different dependency types.
* Modified barModule and fooModule to support the new depsMutator and
  add support for another type of dependency that is ignored by the
  walking code.
* Extracted walkDependencyGraph() function to reuse common code.
2020-04-02 10:56:13 +01:00
Paul Duffin
b77556bdca Allow missing variants when allowMissingDependencies=true 2020-03-24 19:17:27 +00:00
Colin Cross
ab0a83f09c Fix missing dependencies from mutators
Mutators were not propagating the results of ctx.AddNinjaFileDeps.

Test: examine out/soong/build.ninja.d
Fixes: 150689149
Change-Id: Ia1e69ebc9dfa94a05f4ecd9cc2a8691ee63c9dd5
2020-03-03 14:24:24 -08:00
Colin Cross
2ce594e446 Make ninjaString an interface
There are 8935901 *ninjaString objects generated in an AOSP
aosp_blueline-userdebug build, and 7865180 of those are a literal
string with no ninja variables.
Each of those *ninjaString objects takes a minimum of 48 bytes for
2 slices, plus 8 bytes for the pointer to the ninjaString.  For
the literal string case, one of those slices has a single element,
(costing another 16 bytes for the backing array), and the other
slice is empty, for a total of 72 bytes.

Replace *ninjaString with a ninjaString interface.  This increases
the size of the reference from 8 bytes to 16 bytes, but using
a type alias of a string for the literal string implementation uses
only 16 bytes, saving 40 bytes per literal string or 314 MB.

Test: ninja_strings_test
Change-Id: Ic5fe16ed1f2a244fe6a8ccdf762919634d825cbe
2020-01-29 16:23:40 -08:00
Colin Cross
5d57b2d347 Make proptools functions consistently take *struct types
The proptools functions took an inconsistent variety of
struct and *struct types.  Some methods even took a struct
but returned a *struct.  Make all the exported methods
take a *struct, with internal helpers for the ones that need
to take a struct.

Test: proptools tests
Change-Id: I60ce212606e96adcef66c531d57f69c39e1a1638
2020-01-28 09:51:19 -08:00
Colin Cross
c5fa50e057 Allow primary builder to change working directory
Bug: 146437378
Test: pathtools/fs_test.go
Change-Id: I513ceb9b8b0b4f18223bc34ecad9846fe220709b
2020-01-08 15:54:58 -08:00
Colin Cross
f27c5e470b Move unpackProperties to proptools and export it
Test: unpack_test.go
Change-Id: I145369323bd7d5003a261c13dfb8ed31d0be51b5
2020-01-02 20:32:51 -08:00
Colin Cross
9672d86a87 Add scoped module factories
Add RegisterScopedModuleType to LoadHookContext that registers
a module type factory that will be visible for the remainder of
the file.  Also add ModuleFactories that returns the globally
register module factories.

Test: all blueprint tests
Change-Id: If646a73befe3b8e45c4b0984531c6a39ddc8d066
2020-01-02 20:32:51 -08:00
Colin Cross
da70fd0b84 Move LoadHooks from Soong to Blueprint
Move LoadHooks from Blueprint and run them during ParseBlueprintsFiles.
This will allow them to add scoped module types that are visible to the
rest of the Blueprints file.  Requires passing the config object to
ParseBlueprintsFiles.

Test: all blueprint tests
Change-Id: Ia2a2c9a0223d5458bfd48bd22ebed0fdbd0156c6
2020-01-02 20:32:51 -08:00
Colin Cross
8cde425b60 Add Context.SetFs
Allow the primary builder to set the filesystem directly.

Bug: 146437378
Test: m checkbuild
Change-Id: Ic8cc3e6a03943d5f47b041a66df6a11f4336c97b
2019-12-17 13:14:31 -08:00
Colin Cross
f7beb89df5 Add support for module variant aliases
Adding a dependency on a module with variants can be problematic
if the code adding the dependency is not aware of every mutator
that has created variants.  Add an AliasVariations to
BottomUpMutatorContext, which allows a mutator to alias the
original variant of a module to one of the new variants of the
module, which will allow future dependencies to be added using
the original list of variations.  The aliases are transient,
and only exist until the next mutator that calls CreateVariations
when visiting the module without also calling AliasVariations.

Test: TestAlises
Change-Id: Ieaa04b5a6bdcb5a1ff5114b1e03460de795d4479
2019-11-14 12:58:52 -08:00
Colin Cross
9403b5a790 Delay allocating variationMaps until populating them
Modules are created with a nil variantionMap, which avoids allocating
an empty map if the module will never be split.
Context.addVariationDependencies constructs a variationMap to
compare against the variationMap of each variant, but constructed
an empty variationMap if no variations were specified.  A nil
map is not the same as an empty map, so consistently use a nil
map and create it when populating the first entry.

Change-Id: I48b604659f9cdb23326b504a093cdfe5a3eb4f68
2019-11-14 12:58:52 -08:00
Colin Cross
d03b59d03e Use module groups more widely
Use module groups instead of passing around the list of modules
extracted from a module group.

Test: blueprint tests
Change-Id: I02a79950c6377441c49129ebeb5f12be257df668
2019-11-14 12:58:52 -08:00
Jiyong Park
df4ed95f43 Don't ignore local variations when creating reverse dep
This fixes a bug that reverse dependency can't be made for modules
having local variations. Previously, when module A having local variants
calls AddReverseDependency to module B having local variants, the match
is tested between the non-local variants of module A against all
variants of module B, which can never be successful.

This change fixes it by using all variants of module A when
findMatchingVariants is called for AddReverseDependency.

Test: m
Change-Id: Ib289188a5dd58c72bd6ba07e3c0f825f8b1c6b1b
2019-09-22 07:44:52 +09:00
Jiyong Park
1e2e56dc62 Add SetDefaultDependencyVariation
SetDefaultDependencyVariation sets the variation name that will be used
when a dangling dependency is found while a module is being split. A
dangling dependency can occur if a module is split to a variant that one
of its dependencies is not split into. When the default variation is not
set, such dangling dependency is a hard error. But with the new
function, the default variation can be set and subsequent calls to
CreateVariations and its variations on the same context uses the default
variation when necessary.

(If even the default variation does not exist for the dependent module,
it is an hard error)

Note that this is different from calling SetDependencyVariation("foo")
followed by CreateVariations("foo", "bar"). In that case, regardless of
whether a dependency of the current module has the variant 'bar' or not,
only the 'foo' variant is chosen.

With SetDefaultDependencyVariation("foo") followed by
CreateVariations("foo", "bar"), 'foo' variant is used only when the
'bar' variant of the current module depends on a module that does not
have 'bar' variant.

Bug: 138103882
Test: m
Change-Id: I4520ca87487994de024fdbacda3bef6636225f0d
2019-08-09 12:49:11 -07:00
Colin Cross
322cc01803 Report creating modules in errors on created modules
Bug: 133172285
Change-Id: I0782c722f5877508691868f96dbf7dc0bed3f9f4
2019-05-20 15:09:48 -07:00
Colin Cross
99bdb2ab4f Consolidate mutator contexts
Move most of the mutator context methods into BaseModuleContext
so they are available to both top down and bottom up mutators.
Only CreateModule is unique to TopDownMutatorContext, and the
dependency and variation adding methods are unique to the
BottomUpMutatorContext.

The dependency visiting methods are now available on
BottomUpMutatorContext, which requires delaying making newly
added dependencies visible to maintain the invariant that
the mutator has been called on the dependency before the
dependency can be visited by its parents.

Test: m checkbuild
Change-Id: Ie14afc02ac76d0b5a66b0e52de2aa9e17fd1bec0
2019-05-20 14:58:14 -07:00
Patrice Arruda
b0a40a717a Moving the ParseBlueprintsFiles go comment in context.go.
The go comment for the blueprint.Context.ParseBlueprintsFiles was
actually place above the blueprint.Context.ParseFileList.

Bug: N/A
Test: N/A

Change-Id: I065d2f6ec034c614e40de745931f4c87cdb73422
2019-03-08 14:17:43 -08:00
Colin Cross
9226d6c9f5 Allow Context to query Singletons
Add Context.Singletons and Context.SingletonName to allow primary
builders to query Singletons returned by all registered
SingletonFactories.

Change-Id: Iee85643dd47f7472e6bb5ae8004374bd6ea0419e
2019-02-26 10:15:40 -08:00
Jaewoong Jung
781f6b2896 bpdoc improvements
1. Extract module type documentation.
2. Support primary builder customization of factory function to use for
documentation for each module type.
3. Change the ModuleType list order so that they are grouped by package.

This is basically minor refactoring + readability improvement done on
top of https://github.com/google/blueprint/pull/232.

Change-Id: If7413e5ac23486b85f18d02fb3ba288a38730c32
2019-02-08 15:48:27 -08:00
Colin Cross
de7afaaf74 Write ninja file directly to the output file
Writing the ninja file to a byte buffer causes a significant amount
of time to be spent in memmove when growing the byte slice.  Write
the file directly to disk instead.

This also fixes some unhandled error warnings, which become more
likely when doing disk IO instead of byte buffer writes.

Change-Id: I5094e4c45cab4012713037f60c5a4fb00718f92e
2019-01-23 13:26:42 -08:00
Colin Cross
3a8c025648 Add pprof labels
Add pprof labels to the larger stages of blueprint, including
labels for each mutator.

Change-Id: Ie91daff51811187c1b702a775c05d0b32f7170fc
2019-01-23 13:26:42 -08:00
Jaewoong Jung
ccc3494088 Make sure all modules have a name. (#226)
Throw an error if run into a module without a name when generating
contexts.

Test: context_test.go
Change-Id: I3976d86d1f15b8ac10a7a38aa42ae277740a8f3b
2018-10-11 13:01:05 -07:00
Dan Willemsen
734f20c205 Fix issues found by go vet
Change-Id: If3e20a1f394d757554d6a7da5ed4c41fe194f55f
2018-07-21 13:10:25 -07:00
Dan Willemsen
ab223a512b Run globs during earlier bootstrap phases
Instead of sometimes re-running minibp/the primary builder during the
next phase, run bpglob earlier to check dependencies.

We've run into issues where the environment is slightly different
between bootstrapping phase and the main build phase. It's also a
problem because our primary builder (Soong) exports information used by
another tool (Kati) that runs in between the bootstrapping phases and
the main phase. When Soong would run in the main phase, it could get out
of sync, and would require the build to be run again.

To do this, add a "subninja" include a build-globs.ninja file to each
build.ninja file. The first time, this will be an empty file, but we'll
always run minibp / the primary builder anyway. When the builder runs,
in addition to writing a dependency file, write out the
build-globs.ninja file with the rules to run bpglob.

Since bpglob may need to be run very early, before it would normally be
built, build it with microfactory.

Change-Id: I89fcd849a8729e892f163d40060ab90b5d4dfa5d
2018-07-06 10:39:38 -07:00
Colin Cross
526e02f0c6 Prevent duplicate visit calls in WalkDeps
WalkDeps was following every possible path through the dependency
tree, which can be enormous.  Modify it to only call visit for
any particular (child, parent) pair once for each direct dependency
by not recursing into child if visitDown returns true but child
has already been visited.

Test: TestWalkDeps, TestWalkDepsDuplicates
Change-Id: Ieef28399bd10e744417cdeb661dfa04fbeb4ec60
2018-06-21 13:41:42 -07:00
Colin Cross
9607a9f248 Allow multiple dependencies on the same module
Allow adding multiple dependencies on the same module.  Adds a flag
to Context.walkDeps to determine whether dependencies should be
visited multiple times, and clarifies the behavior of
VisitDepsDepthFirst (continues to visit each module once, even if
there are multiple dependencies on it) and VisitDirectDeps (may
visit a module multiple times).

Test: TestWalkDepsDuplicates, TestVisit
Change-Id: I58afbe90490aca102d242d63e185386e1fe55d73
2018-06-20 14:10:18 -07:00
Colin Cross
7e72337259 Reduce maximum goroutine count
Running with the data race detector enabled hits a 8192 active
goroutine limit.  Reduce the maximum number of active goroutines
by limiting parallelVisit to 1000 goroutines.  Also rewrite
cloneModule in terms of parallelVisit.

Change-Id: Icdd63648e8e010557b624e12592156490b40adb9
2018-04-10 16:51:29 -07:00
Dan Willemsen
e6d45fe39f Support go1.10
Add stubs for the new testDeps interface functions. Also removes testing
for 1.8.

Change-Id: Ice58cca20658d905df9fb87e822d7861abf60976
2018-02-27 01:49:49 -08:00
Dan Willemsen
b6c90239d6 Append / to directories in Glob results
This makes it easy for users of Glob to detect whether the match is a
file or directory. Doing the check at this level means that the filelist
file used as a dependency will be updated if a directory is replaced
with a file of the same name, or vice versa.

Change-Id: I79ebba39327218bcdcf50b393498306119de9d6c
2018-02-23 17:21:37 -08:00
Jeff Gaston
a7e408af0a prevent file=nil panic if syntax error in Blueprints
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
      'build/blueprint:work^^^' 'build/blueprint:work'
Test: put a syntax error in a file and see that the
      reported error reports the location of the violation

Change-Id: Iaeedb91ea8e816cb8e9ee954f21cd6c6bc4afa48
2017-12-05 16:09:02 -08:00
Jeff Gaston
8fd9578a6a have openAndParse use return values rather than channels
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
      'build/blueprint:work^^^' 'build/blueprint:work'

Change-Id: I941dfa76b94178198994a280eb40bcded891eda1
2017-12-05 15:26:57 -08:00
Jeff Gaston
5800d046c1 rename parseOneAsync to openAndParse
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
      'build/blueprint:work^^^' 'build/blueprint:work'

Change-Id: I56e08716057c3746b01b7a5e43f0223ee1f940db
2017-12-05 15:26:57 -08:00
Jeff Gaston
0e90759bfe Make ninja file deterministic even with dup module names
Bug: 65683273
Test: ./build/soong/scripts/diff_build_graphs.sh \
      --products=aosp_arm '' ''

Change-Id: I5e45b2309ba4993ba2180b5f9a4785f31e28d503
2017-12-04 17:16:21 -08:00
Jeff Gaston
f23e36690e Disallow bp 'build' includes to reference other dirs
Bug: 65683273
Test: add 'build = ["sub/nope.bp"]' to an Android.bp; notice the error

Change-Id: Ic0f171f283edda074f65a76029e660dfaab2504b
2017-12-04 17:16:21 -08:00
Jeff Gaston
3c8c3346d2 Pass ModulePath to NameInterface
To allow it to validate that the filename equals Android.bp

Bug: 65683273
Test: m -j nothing # which runs tests
Change-Id: I171dddd102590df558053b615d39c75c00b6ac6e
2017-12-04 17:16:21 -08:00
Jeff Gaston
656870fbca Have handleOneFile(child) wait for handleOneFile(parent).
Bug: 65683273
Test: m -j nothing # which runs unit tests

Change-Id: I850e78ebcdbee68637f66d81fc127f19dd28508a
2017-12-04 17:16:21 -08:00
Jeff Gaston
d70bf75491 Extract module naming into an interface
in facilitate moving name resolution to Soong

Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
      --products=aosp_arm \
      'build/blueprint:work^' 'build/blueprint:work'
      # and see that the only changes were:
      # 1. adding the name_interface.go file
      # 2. changing some line numbers

Change-Id: Ifa7603cca59b3b3d592f2f146fdafe57012bd4b9
2017-11-29 12:01:09 -08:00
Jeff Gaston
9f63090a4c Support files named Android.bp in tests
Bug: 65683273
Test: m -j nothing # which runs unit tests

Change-Id: I00862cd9673719424a2b18e347c7f9fe84be2857
2017-11-20 17:03:49 -08:00
colincross
7ab9124b27
Merge pull request #183 from colincross/presingleton
Add presingletons
2017-11-07 14:50:05 -08:00
Colin Cross
5f03f11c0a Add presingletons
PreSingletons use the same interface as Singletons, but run after
parsing all the blueprint files but before running any mutators
or GenerateBuildActions.  They can be used to perform global setup,
including generating rules, that may be referenced by later
mutator or generate passes.

Test: m checkbuild
Change-Id: I4b93425a724e739f5b8843262efb8804b3bf3531
2017-11-07 13:31:33 -08:00
Jeff Gaston
3802ba9f18 No longer follow subdirs in Android.bp files
because there some symlinks named Android.bp that point to
files named Android.bp that contain <subdir> entries that
are supposed to be interpreted relative to the original
symlink. It's more convenient to just skip following subdirs.

Bug: 64363847
Test: make -j
Change-Id: Ib15f1d1c7d511191fd841b1a29ceed559dd6cdbb
2017-10-30 15:00:19 -07:00
Jeff Gaston
c3e2844dfe Support for a custom list of Blueprints files to parse
Bug: 64363847
Test: BLUEPRINT_LIST_FILE=out/.module_paths/Android.bp.list minibp

Change-Id: Id7f8cb1ab3a6684b3f8265d77bb32413957f1c93
2017-10-30 15:00:19 -07:00
Jeff Gaston
a12f22fb69 Move parseOne for readability
Bug: 64363847
Test: m -j
Change-Id: Ie3e973dadbff139def127b0bb05c57bafb79165b
2017-10-30 15:00:19 -07:00
Jeff Gaston
5f763d0511 Clarify context.go somewhat
in preparation to support parsing a predefined list of Blueprints files

Bug: 64363847
Test: m -j
Change-Id: I98cc98dc17ecaf69b95272d32cd83adbdb0e8c5b
2017-10-30 15:00:19 -07:00
colincross
b67c1d433c Merge pull request #166 from colincross/create
Add TopDownMutatorContext.CreateModule
2017-09-20 14:09:15 -07:00
Jeff Gaston
aca4220583 Clearer error in case of Android.bp being unreadable
Bug: 64600838
Test: mkdir errtest \
      && ln -s /tmp/dontexist errtest/Android.bp \
      # and add errtest to ./Android.bp \
      && m nothing \
      # and check that the error message mentions a symlink

Change-Id: I841ec12d613f61ccc3396538062bee48c8c1ca27
2017-09-01 17:29:30 -07:00
Colin Cross
874a346904 Let mutators use ctx.AddNinjaFileDeps
Propagate extra ninja file deps through mutators so that they
can use ctx.AddNinjaFileDeps.

Test: blueprint tests
Change-Id: I299a0665c3f63b020ae345889fd78b91b91b215a
2017-08-01 15:12:12 -07:00
Colin Cross
af4fd215eb Add TopDownMutatorContext.CreateModule
Allow a module to create other modules as if they were specified
in a blueprint file.  CreateModule takes the name of a module type,
calls the factory function the module type, and then inserts
properties into the property structs returned by the factory.

Bug: 64161749
Test: TestCreateModule
Change-Id: Ic1903305d6b08eae1edc7f0fb4137e85544aac0f
2017-08-01 15:12:12 -07:00
Colin Cross
5fe225f5f9 Rename newModules to newVariations
The next patch will introduce creating new modules, so rename the
current variable that contains new variations of existing modules
to avoid confusion.

Test: blueprint tests
Change-Id: Ic1d3824e54e51c282a08f8ecef8fc658cc503a65
2017-08-01 15:12:12 -07:00
Colin Cross
d2f4ac1224 Rename moduleInfo.moduleProperties to properties
Remove unnecessary name duplication.

Test: blueprint tests
Change-Id: I2c65ce6a164e47e855dd3bb1c7bee957dfd422ae
2017-08-01 15:12:12 -07:00
Colin Cross
4a02a3019a Cap concurrency when parsing blueprint files
Darwin has a default limit of 256 open files per process.  Parsing
too many blueprint files in parallel can hit the limit.  Cap the
concurrency at 200.

Test: manual testing with limit set to 32
Change-Id: Ic64d21d2c0ffd7c86bf3f02fb51216ee5684a80c
2017-05-16 10:55:29 -07:00
Dan Willemsen
978c4aa92f Improve error messages when dependency variants are missing
Instead of just saying:

error: .../Android.bp:48:1: dependency "libc++" of "libtest" missing variant "arch:android_arm_armv7-a, link:shared, vndk:"

Include a list of currently existing variants:

error: .../Android.bp:48:1: dependency "libc++" of "libtest" missing variant:
  arch:android_arm_armv7-a, link:shared, vndk:
available variants:
  arch:android_arm_armv7-a, link:shared
  arch:linux_x86, link:shared
  arch:linux_x86_64, link:shared
  arch:windows_x86, link:shared
  arch:windows_x86_64, link:shared

This still isn't the best experience for users, but it at least provides
enough information for someone more familiar with the build to
understand the problem.
2017-03-20 14:23:43 -07:00
Colin Cross
080c1336b5 Add Context.VisitDirectDeps
Context already has VisitDepsDepthFirst[If], add VisitDirectDeps[If].

Change-Id: Id550bc14275db230c19fd6ca14b67b305dd7d96b
2017-03-17 13:46:47 -07:00
Nan Zhang
346b2d0d80 Added a check in BP level for BaseDependencyTag
It is not allowed to directly use BaseDependencyTag as customized
user dependency tag passed down to BP since it might cause issues
that different type of modules will be mixed when fetched based on
Tag.
2017-03-10 16:39:27 -08:00
Colin Cross
b519a7e1b6 Add globbing to filesystem mocking
Add globbing to filesystem mocking so that more code can be tested
against the mock.  Also moves the filesystem mock to pathtools,
and renames pathtools.GlobWithExcludes to pathtools.Glob, replacing
the existing pathtools.Glob.

Test: blueprint tests
Change-Id: I722df8121bc870c4a861d7c249245c57dcb607be
2017-02-02 16:48:06 -08:00
Colin Cross
0ce142ca05 Fix data race when applying replacements
Mutator context goroutines appending directly to the global Context's
replacements list causes a data race.  Send them over a channel
instead.

The renames and replacements are local to the mutator, so move them
out of Context and into the runMutator method.

Change-Id: I797edb1e27ee29f8946c58101b40fcfb50a32eb9
2016-12-09 10:35:33 -08:00
Colin Cross
08e4954a8c Fix optional_subdirs globbing
Bypassing c.glob() and using filepath.Glob() directly for non-glob
paths does not add dependencies on directories that contain missing
files.  For optional_subdirs, this means no dependency is added to
rerun the primary builder when an Android.bp file is added to an
optional_subdirs directory.  Always use c.glob(), for the non-optional
case it will not insert any dependencies if the file exists (as tested
by glob_test.go's no-wild tests), and if the file doesn't exist the
len(matches) == 0 will error out.

Change-Id: I370479c6e89f5ff590897702e256256a4dca6952
2016-11-14 15:41:41 -08:00
Colin Cross
127d2eae8b Import globbing from Soong
Add globbing with dependency checking to blueprint.  Calling
ModuleContext.GlobWithDeps or SingletonContext.GlobWithDeps will return
a list of files that match the globs, while also adding efficient
dependencies to rerun the primary builder if a file that matches the
glob is added or removed.

Also use the globbing support for optional_subdirs=, subdirs= and build=
lines in blueprints files.  The globbing slightly changes the behavior
of subname= lines, it no longer falls back to looking for a file called
"Blueprints".  Blueprint files that need to include a subdirectory with
a different name can use build= instead of subdir= to directly include
them.  The Blueprints file is updated to reset subname="Blueprints" in
case we want to include subdirectories inside blueprint and the primary
builder has changed the subname.

Also adds a new test directory that contains a simple primary builder
tree to test regeneration for globbing, and runs the tests in travis.

Change-Id: I83ce525fd11e11579cc58ba5308d01ca8eea7bc6
2016-11-03 13:54:03 -07:00
Dan Willemsen
5c43e07937 Support implicit outputs
Added in Ninja 1.7, for outputs that will not show up in $out.
2016-10-31 17:36:49 -07:00
Colin Cross
9cfd198dd8 Add ReplaceDependencies
ReplaceDependencies allows a module to replace dependencies on a
matching variant of a target module with itself.

Change-Id: I22946dec23c38ed5e1ad23b87121d72668268c01
2016-10-12 13:33:37 -07:00
Colin Cross
c4e5b8157b Add Rename
Allow modules to be renamed by mutators.  This will allow modules to
dynamically adapt to the presence or absence of modules with the same
name.  For example, if a source module does not exist, a prebuilt module
could rename itself to the name of the source module so that
dependenices on the module name are satisified.

Change-Id: I44004604c6d9db041bb7d38fe6c1ca877bc7d6f1
2016-10-12 13:33:37 -07:00
Colin Cross
0b7e83e11a Ask primary builder for module names
Remove the assumed Name and Deps properties.  Instead, ask the Module
for the Name, and require them to use the existing replacements for
Deps.

Change-Id: I729045d86277686078b3aa0bba71c67d612ead2c
2016-10-12 13:33:37 -07:00
Colin Cross
2c62844274 Improve error reporting
Report module name and variant and property names in errors.

Change-Id: I747f840402497b2c92bf8c565d7c50af33e6ab0e
2016-10-12 11:14:18 -07:00
Colin Cross
2458712c0b Add dependency on build= blueprint files
Blueprint files included with build= statements were not adding a
dependency to cause regenerating when they were changed.

Bug: 32085516
Test: touch external/boringssl/sources.bp && mmma -j external/boringssl
Change-Id: Id4fdf3b6788ae5c1e94547dc63ec6b55424a66a0
2016-10-11 10:03:14 -07:00
colincross
e412f09f7e Merge pull request #117 from colincross/variant
Elide empty variations
2016-08-15 12:22:37 -07:00
colincross
087b721ea8 Merge pull request #115 from colincross/parallel
Parallelize TopDownMutators
2016-08-11 17:10:59 -07:00
Colin Cross
badc8817fa Elide empty variations
Don't add a "_" to the variant name if the variation name is empty.  For
example, when splitting a variant with name "foo" into variations "" and
"bar", the new variants would be named "foo" and "foo_bar" instead of
"foo_" and "foo_bar".

Change-Id: I82342d57e2a8e9f2d65a7d8d2872dcb7b3512899
2016-08-11 17:03:26 -07:00
Colin Cross
0fff742172 Fix error reporting in parallel mutators
runMutators was sending the wrong value into errsCh, hiding all errors.
parallelVisit had a data race on error, send the cancel signal over a
channel instead of writing to a bool from a goroutine.

Change-Id: I18919d71464c094db5846badbfac80f99366d70c
2016-08-11 15:41:57 -07:00
Colin Cross
3702ac7531 Parallelize TopDownMutators
Convert parallelVisitAllBottomUp into a generic parallelVisit that takes
a visitOrderer to select top down or bottom up.  Combine
runTopDownMutator and runBottomUpMutator into runMutators that takes a
mutatorDirection to select the visitOrderer for parallelVisit and which
function pointer in the mutatorInfo to run.  Optimize out the
updateDependencies to only run if dependencies have been modified to
avoid running it after TopDownMutators that cannot modify dependencies.

Change-Id: Ib00302db1108ebab2ce8e01b20aa026140d382a4
2016-08-11 14:40:53 -07:00
Colin Cross
c93490c679 Parallelize cloneModules
cloneModules duplicates every module, which can be slow.  Spawn a new
goroutine for each module to spread the factory and CopyProperties calls
over multiple cpus.

Change-Id: I2597f921dadfd321ccb705aec9a904c71b75f9ef
2016-08-10 16:31:35 -07:00
Colin Cross
49c279ab26 Parallelize BottomUpMutators
Allow BottomUpMutators to run in parallel by calling Parallel() on the
return value of RegisterBottomUpMutator.  To avoid locking, moves
updates of global state into a separate goroutine that receives updates
over channels from the mutator goroutines.

Change-Id: Ic59b612da9b406cf59ec44940f0c1dee0c051a51
2016-08-10 16:31:35 -07:00
Colin Cross
f8b5042c86 Implement EarlyMutators with BottomUpMutators
Get rid of runEarlyMutators, and implement EarlyMutators as a shim
around BottomUpMutators.  Removes mostly-duplicate code between
runEarlyMutators and runBottomUpMutator to allow parallelizing
runBottomUpMutator.

Change-Id: I66a051cb8710770b47da21d35e55aa49c84e9be8
2016-08-10 16:31:35 -07:00
Colin Cross
bafd5f5a2b Reimplement VisitDirectDeps[If] in on Context.walkdeps
VisitDirectDeps[If] need to know the parent in order to support
OtherModuleDependencyTag(), remove Context.visitDirectDeps[If] and
reimplement [Module]Context.VisitDirectDeps[If] using Context.walkdeps.

Change-Id: I2faac2d100d2a0ad2a4f1d8250997eed1a7ef6f0
Test: TestVisit
2016-08-08 17:30:38 -07:00
Colin Cross
d7b0f60802 Allow tests to mock filesystem
Tests can call Context.MockFileSystem to pass in a map of filenames to
contents.

Change-Id: Idd67c68f7cb43bc2117cc78336347db7e2f21991
2016-08-08 17:26:57 -07:00
Dan Willemsen
c98e55b0af Expose ModuleType to Singletons
In order to implement some build statistics in Soong, expose the module
type (the string used to define the module) to singletons.

Change-Id: I441d12c7782bcf338b3654cfe907b8d2a7253594
2016-07-25 15:51:50 -07:00
Colin Cross
b3d0b8dab4 Rename Pos members
Pos is going to be part of the Node interface, rename the Pos member
of structs to be more specific.

Change-Id: Ibd31119863b96d38bf8dac216e026200a54bbe18
2016-06-14 15:26:49 -07:00
Colin Cross
c32c47938f Remove blueprint/parser.Ident
It wasn't adding anything useful, and it resulted in Name.Name to get to
the identifier.  Replace Name Ident with Name string; NamePos
scanner.Position.

Change-Id: Idf9b18b31dd563a18f27c602c2d14298955af371
2016-06-14 15:26:49 -07:00
Colin Cross
e32cc80f20 Refactor blueprint parser nodes to an interface
Refactor the blueprint parser Value object, which contained a Type enum
and members to hold every possible type, into an interface (now called
Expression).  Rename the existing Expression object that represented a binary
operator Operator.

Also adds and fixes some new printer test cases with mulitline expressions.

Change-Id: Icf4a20f92c8c2a27f18df8ca515a9d7f282ff133
2016-06-08 14:48:53 -07:00
Colin Cross
f1875466fc Support dependencies between variants
Variants of a module may need to depend on other variants, for example
to reuse results between variants.  Support AddInterVariantDependency to
add an explicit dependency from one variant to another, along with a
dependency tag.  Both modules must have just been returned by a call to
CreateVariants.

Change-Id: I8f4878f94ced74dd00cfac8303b15ef70cdebf36
2016-04-12 14:52:46 -07:00