AddFarVariationDependencies claims to take the first variant that
matches all the requested variations, but it did nothing of the sort.
It took the first variant that either matched or did not contain each
of the requested variations. A module with no variations was always
matched, and requesting variations that didn't apply to a module
still matched (for example, requesting an image variation for a
host module that was ignored by the image mutator).
Fix AddFarVariationDependencies by making subset actually check
for subsets.
Test: Test_findVariant
Change-Id: I10063fec342db2a1c0685a7db08e4a650d14bd4e
AddFarVariationDependencies takes the first matching variant. To
maintain sensible behavior on a module with aliases, the ordering
of aliases and module variants needs to be maintained so that
AddFarVariationDependencies can find an earlier matching alias
instead of a more specific variant.
Test: go test .
Change-Id: I78f4e96edd98159f3a62d94e240e5d652667bec4
Add tests for findVariant behavior that provides the matching
behaviors of AddVariationDependencies, AddFarVariationDependencies,
etc.
Test: Test_findVariant
Change-Id: I3494d57179c8b3d62f7d32e5a1b43c9b9672c2df
CreateAliasVariation creates a new variation for the current mutator
that is an alias to a variation that was created with CreateVarations
by the same mutator. It is similar to AliasVariation, except that
AliasVariation creates an alias from the pre-mutated variant to one
of the new variations, while CreateAliasVariation creates an alias
from a new post-mutated variation to one of the new variations.
Bug: 164216768
Test: TestCreateAliasVariation
Change-Id: I8847b8d6fc1d3248abc910b2fe2399881f9bdaee
Context.ModulePath is a duplicate of Context.BlueprintFile, and is
misleading because it returns the path to the Android.bp, and not
the path to the directory that contains the Android.bp file like
ModuleContext.ModuleDir.
Change-Id: I505136fce3c3d37595cd4b4e08de5e2691a2a0f6
Adds a CheckBlueprintSyntax(...) method to check the syntax of a
Blueprint file.
Changes processModuleDef and newModule from being method on *Context to
being standalone functions. That ensures that CheckBlueprintSyntax(...)
does not need to take a context and so there is no chance that it can
change its state.
Previously a LoadHook could not modify the name of a module because the
module was registered before the LoadHooks were run. That made it very
complicated (requiring mutators and auto generated names) to create a
module type whose name was determined by say the directory in which it
is defined.
This change moves the LoadHook execution slightly earlier so it runs
before registration of the module.
That caused one slight side problem which was that the
moduleInfo.Name() would fail when called in a LoadHook. That was
because that gets the name from group.name but was group was nil
because it is only set when the module is registered.
Modifying the moduleInfo.Name() method to get the name from the module
logicModule.Name() if group is nil fixed that. The reason for getting
the name from the group.name rather than the logicModule.Name() is that
the former tracks renames but the latter does not. However that is not
an issue in this case as there has been no opportunity for the module
to be renamed until after the LoadHook has returned.
Previously, WalkDeps() would record that a module was visited after the
first time it encountered the module irrespective of whether it recursed
into or not. This change moves the recording so it happens only after it
has been recursed into.
Added TestWalkDepsDuplicates_IgnoreFirstPath to test the change. Without
the change the test fails because it does not visit E.
Test refactoring:
* A depsMutator was added instead of relying on blueprintDepsMutator to
allow different tags to be used for different dependency types.
* Modified barModule and fooModule to support the new depsMutator and
add support for another type of dependency that is ignored by the
walking code.
* Extracted walkDependencyGraph() function to reuse common code.
Mutators were not propagating the results of ctx.AddNinjaFileDeps.
Test: examine out/soong/build.ninja.d
Fixes: 150689149
Change-Id: Ia1e69ebc9dfa94a05f4ecd9cc2a8691ee63c9dd5
There are 8935901 *ninjaString objects generated in an AOSP
aosp_blueline-userdebug build, and 7865180 of those are a literal
string with no ninja variables.
Each of those *ninjaString objects takes a minimum of 48 bytes for
2 slices, plus 8 bytes for the pointer to the ninjaString. For
the literal string case, one of those slices has a single element,
(costing another 16 bytes for the backing array), and the other
slice is empty, for a total of 72 bytes.
Replace *ninjaString with a ninjaString interface. This increases
the size of the reference from 8 bytes to 16 bytes, but using
a type alias of a string for the literal string implementation uses
only 16 bytes, saving 40 bytes per literal string or 314 MB.
Test: ninja_strings_test
Change-Id: Ic5fe16ed1f2a244fe6a8ccdf762919634d825cbe
The proptools functions took an inconsistent variety of
struct and *struct types. Some methods even took a struct
but returned a *struct. Make all the exported methods
take a *struct, with internal helpers for the ones that need
to take a struct.
Test: proptools tests
Change-Id: I60ce212606e96adcef66c531d57f69c39e1a1638
Add RegisterScopedModuleType to LoadHookContext that registers
a module type factory that will be visible for the remainder of
the file. Also add ModuleFactories that returns the globally
register module factories.
Test: all blueprint tests
Change-Id: If646a73befe3b8e45c4b0984531c6a39ddc8d066
Move LoadHooks from Blueprint and run them during ParseBlueprintsFiles.
This will allow them to add scoped module types that are visible to the
rest of the Blueprints file. Requires passing the config object to
ParseBlueprintsFiles.
Test: all blueprint tests
Change-Id: Ia2a2c9a0223d5458bfd48bd22ebed0fdbd0156c6
Adding a dependency on a module with variants can be problematic
if the code adding the dependency is not aware of every mutator
that has created variants. Add an AliasVariations to
BottomUpMutatorContext, which allows a mutator to alias the
original variant of a module to one of the new variants of the
module, which will allow future dependencies to be added using
the original list of variations. The aliases are transient,
and only exist until the next mutator that calls CreateVariations
when visiting the module without also calling AliasVariations.
Test: TestAlises
Change-Id: Ieaa04b5a6bdcb5a1ff5114b1e03460de795d4479
Modules are created with a nil variantionMap, which avoids allocating
an empty map if the module will never be split.
Context.addVariationDependencies constructs a variationMap to
compare against the variationMap of each variant, but constructed
an empty variationMap if no variations were specified. A nil
map is not the same as an empty map, so consistently use a nil
map and create it when populating the first entry.
Change-Id: I48b604659f9cdb23326b504a093cdfe5a3eb4f68
Use module groups instead of passing around the list of modules
extracted from a module group.
Test: blueprint tests
Change-Id: I02a79950c6377441c49129ebeb5f12be257df668
This fixes a bug that reverse dependency can't be made for modules
having local variations. Previously, when module A having local variants
calls AddReverseDependency to module B having local variants, the match
is tested between the non-local variants of module A against all
variants of module B, which can never be successful.
This change fixes it by using all variants of module A when
findMatchingVariants is called for AddReverseDependency.
Test: m
Change-Id: Ib289188a5dd58c72bd6ba07e3c0f825f8b1c6b1b
SetDefaultDependencyVariation sets the variation name that will be used
when a dangling dependency is found while a module is being split. A
dangling dependency can occur if a module is split to a variant that one
of its dependencies is not split into. When the default variation is not
set, such dangling dependency is a hard error. But with the new
function, the default variation can be set and subsequent calls to
CreateVariations and its variations on the same context uses the default
variation when necessary.
(If even the default variation does not exist for the dependent module,
it is an hard error)
Note that this is different from calling SetDependencyVariation("foo")
followed by CreateVariations("foo", "bar"). In that case, regardless of
whether a dependency of the current module has the variant 'bar' or not,
only the 'foo' variant is chosen.
With SetDefaultDependencyVariation("foo") followed by
CreateVariations("foo", "bar"), 'foo' variant is used only when the
'bar' variant of the current module depends on a module that does not
have 'bar' variant.
Bug: 138103882
Test: m
Change-Id: I4520ca87487994de024fdbacda3bef6636225f0d
Move most of the mutator context methods into BaseModuleContext
so they are available to both top down and bottom up mutators.
Only CreateModule is unique to TopDownMutatorContext, and the
dependency and variation adding methods are unique to the
BottomUpMutatorContext.
The dependency visiting methods are now available on
BottomUpMutatorContext, which requires delaying making newly
added dependencies visible to maintain the invariant that
the mutator has been called on the dependency before the
dependency can be visited by its parents.
Test: m checkbuild
Change-Id: Ie14afc02ac76d0b5a66b0e52de2aa9e17fd1bec0
The go comment for the blueprint.Context.ParseBlueprintsFiles was
actually place above the blueprint.Context.ParseFileList.
Bug: N/A
Test: N/A
Change-Id: I065d2f6ec034c614e40de745931f4c87cdb73422
Add Context.Singletons and Context.SingletonName to allow primary
builders to query Singletons returned by all registered
SingletonFactories.
Change-Id: Iee85643dd47f7472e6bb5ae8004374bd6ea0419e
1. Extract module type documentation.
2. Support primary builder customization of factory function to use for
documentation for each module type.
3. Change the ModuleType list order so that they are grouped by package.
This is basically minor refactoring + readability improvement done on
top of https://github.com/google/blueprint/pull/232.
Change-Id: If7413e5ac23486b85f18d02fb3ba288a38730c32
Writing the ninja file to a byte buffer causes a significant amount
of time to be spent in memmove when growing the byte slice. Write
the file directly to disk instead.
This also fixes some unhandled error warnings, which become more
likely when doing disk IO instead of byte buffer writes.
Change-Id: I5094e4c45cab4012713037f60c5a4fb00718f92e
Throw an error if run into a module without a name when generating
contexts.
Test: context_test.go
Change-Id: I3976d86d1f15b8ac10a7a38aa42ae277740a8f3b
Instead of sometimes re-running minibp/the primary builder during the
next phase, run bpglob earlier to check dependencies.
We've run into issues where the environment is slightly different
between bootstrapping phase and the main build phase. It's also a
problem because our primary builder (Soong) exports information used by
another tool (Kati) that runs in between the bootstrapping phases and
the main phase. When Soong would run in the main phase, it could get out
of sync, and would require the build to be run again.
To do this, add a "subninja" include a build-globs.ninja file to each
build.ninja file. The first time, this will be an empty file, but we'll
always run minibp / the primary builder anyway. When the builder runs,
in addition to writing a dependency file, write out the
build-globs.ninja file with the rules to run bpglob.
Since bpglob may need to be run very early, before it would normally be
built, build it with microfactory.
Change-Id: I89fcd849a8729e892f163d40060ab90b5d4dfa5d
WalkDeps was following every possible path through the dependency
tree, which can be enormous. Modify it to only call visit for
any particular (child, parent) pair once for each direct dependency
by not recursing into child if visitDown returns true but child
has already been visited.
Test: TestWalkDeps, TestWalkDepsDuplicates
Change-Id: Ieef28399bd10e744417cdeb661dfa04fbeb4ec60
Allow adding multiple dependencies on the same module. Adds a flag
to Context.walkDeps to determine whether dependencies should be
visited multiple times, and clarifies the behavior of
VisitDepsDepthFirst (continues to visit each module once, even if
there are multiple dependencies on it) and VisitDirectDeps (may
visit a module multiple times).
Test: TestWalkDepsDuplicates, TestVisit
Change-Id: I58afbe90490aca102d242d63e185386e1fe55d73
Running with the data race detector enabled hits a 8192 active
goroutine limit. Reduce the maximum number of active goroutines
by limiting parallelVisit to 1000 goroutines. Also rewrite
cloneModule in terms of parallelVisit.
Change-Id: Icdd63648e8e010557b624e12592156490b40adb9
This makes it easy for users of Glob to detect whether the match is a
file or directory. Doing the check at this level means that the filelist
file used as a dependency will be updated if a directory is replaced
with a file of the same name, or vice versa.
Change-Id: I79ebba39327218bcdcf50b393498306119de9d6c
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
'build/blueprint:work^^^' 'build/blueprint:work'
Test: put a syntax error in a file and see that the
reported error reports the location of the violation
Change-Id: Iaeedb91ea8e816cb8e9ee954f21cd6c6bc4afa48
To allow it to validate that the filename equals Android.bp
Bug: 65683273
Test: m -j nothing # which runs tests
Change-Id: I171dddd102590df558053b615d39c75c00b6ac6e
in facilitate moving name resolution to Soong
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
--products=aosp_arm \
'build/blueprint:work^' 'build/blueprint:work'
# and see that the only changes were:
# 1. adding the name_interface.go file
# 2. changing some line numbers
Change-Id: Ifa7603cca59b3b3d592f2f146fdafe57012bd4b9
PreSingletons use the same interface as Singletons, but run after
parsing all the blueprint files but before running any mutators
or GenerateBuildActions. They can be used to perform global setup,
including generating rules, that may be referenced by later
mutator or generate passes.
Test: m checkbuild
Change-Id: I4b93425a724e739f5b8843262efb8804b3bf3531
because there some symlinks named Android.bp that point to
files named Android.bp that contain <subdir> entries that
are supposed to be interpreted relative to the original
symlink. It's more convenient to just skip following subdirs.
Bug: 64363847
Test: make -j
Change-Id: Ib15f1d1c7d511191fd841b1a29ceed559dd6cdbb
Bug: 64600838
Test: mkdir errtest \
&& ln -s /tmp/dontexist errtest/Android.bp \
# and add errtest to ./Android.bp \
&& m nothing \
# and check that the error message mentions a symlink
Change-Id: I841ec12d613f61ccc3396538062bee48c8c1ca27
Propagate extra ninja file deps through mutators so that they
can use ctx.AddNinjaFileDeps.
Test: blueprint tests
Change-Id: I299a0665c3f63b020ae345889fd78b91b91b215a
Allow a module to create other modules as if they were specified
in a blueprint file. CreateModule takes the name of a module type,
calls the factory function the module type, and then inserts
properties into the property structs returned by the factory.
Bug: 64161749
Test: TestCreateModule
Change-Id: Ic1903305d6b08eae1edc7f0fb4137e85544aac0f
The next patch will introduce creating new modules, so rename the
current variable that contains new variations of existing modules
to avoid confusion.
Test: blueprint tests
Change-Id: Ic1d3824e54e51c282a08f8ecef8fc658cc503a65
Darwin has a default limit of 256 open files per process. Parsing
too many blueprint files in parallel can hit the limit. Cap the
concurrency at 200.
Test: manual testing with limit set to 32
Change-Id: Ic64d21d2c0ffd7c86bf3f02fb51216ee5684a80c
Instead of just saying:
error: .../Android.bp:48:1: dependency "libc++" of "libtest" missing variant "arch:android_arm_armv7-a, link:shared, vndk:"
Include a list of currently existing variants:
error: .../Android.bp:48:1: dependency "libc++" of "libtest" missing variant:
arch:android_arm_armv7-a, link:shared, vndk:
available variants:
arch:android_arm_armv7-a, link:shared
arch:linux_x86, link:shared
arch:linux_x86_64, link:shared
arch:windows_x86, link:shared
arch:windows_x86_64, link:shared
This still isn't the best experience for users, but it at least provides
enough information for someone more familiar with the build to
understand the problem.
It is not allowed to directly use BaseDependencyTag as customized
user dependency tag passed down to BP since it might cause issues
that different type of modules will be mixed when fetched based on
Tag.
Add globbing to filesystem mocking so that more code can be tested
against the mock. Also moves the filesystem mock to pathtools,
and renames pathtools.GlobWithExcludes to pathtools.Glob, replacing
the existing pathtools.Glob.
Test: blueprint tests
Change-Id: I722df8121bc870c4a861d7c249245c57dcb607be
Mutator context goroutines appending directly to the global Context's
replacements list causes a data race. Send them over a channel
instead.
The renames and replacements are local to the mutator, so move them
out of Context and into the runMutator method.
Change-Id: I797edb1e27ee29f8946c58101b40fcfb50a32eb9
Bypassing c.glob() and using filepath.Glob() directly for non-glob
paths does not add dependencies on directories that contain missing
files. For optional_subdirs, this means no dependency is added to
rerun the primary builder when an Android.bp file is added to an
optional_subdirs directory. Always use c.glob(), for the non-optional
case it will not insert any dependencies if the file exists (as tested
by glob_test.go's no-wild tests), and if the file doesn't exist the
len(matches) == 0 will error out.
Change-Id: I370479c6e89f5ff590897702e256256a4dca6952
Add globbing with dependency checking to blueprint. Calling
ModuleContext.GlobWithDeps or SingletonContext.GlobWithDeps will return
a list of files that match the globs, while also adding efficient
dependencies to rerun the primary builder if a file that matches the
glob is added or removed.
Also use the globbing support for optional_subdirs=, subdirs= and build=
lines in blueprints files. The globbing slightly changes the behavior
of subname= lines, it no longer falls back to looking for a file called
"Blueprints". Blueprint files that need to include a subdirectory with
a different name can use build= instead of subdir= to directly include
them. The Blueprints file is updated to reset subname="Blueprints" in
case we want to include subdirectories inside blueprint and the primary
builder has changed the subname.
Also adds a new test directory that contains a simple primary builder
tree to test regeneration for globbing, and runs the tests in travis.
Change-Id: I83ce525fd11e11579cc58ba5308d01ca8eea7bc6
ReplaceDependencies allows a module to replace dependencies on a
matching variant of a target module with itself.
Change-Id: I22946dec23c38ed5e1ad23b87121d72668268c01
Allow modules to be renamed by mutators. This will allow modules to
dynamically adapt to the presence or absence of modules with the same
name. For example, if a source module does not exist, a prebuilt module
could rename itself to the name of the source module so that
dependenices on the module name are satisified.
Change-Id: I44004604c6d9db041bb7d38fe6c1ca877bc7d6f1
Remove the assumed Name and Deps properties. Instead, ask the Module
for the Name, and require them to use the existing replacements for
Deps.
Change-Id: I729045d86277686078b3aa0bba71c67d612ead2c
Blueprint files included with build= statements were not adding a
dependency to cause regenerating when they were changed.
Bug: 32085516
Test: touch external/boringssl/sources.bp && mmma -j external/boringssl
Change-Id: Id4fdf3b6788ae5c1e94547dc63ec6b55424a66a0
Don't add a "_" to the variant name if the variation name is empty. For
example, when splitting a variant with name "foo" into variations "" and
"bar", the new variants would be named "foo" and "foo_bar" instead of
"foo_" and "foo_bar".
Change-Id: I82342d57e2a8e9f2d65a7d8d2872dcb7b3512899
runMutators was sending the wrong value into errsCh, hiding all errors.
parallelVisit had a data race on error, send the cancel signal over a
channel instead of writing to a bool from a goroutine.
Change-Id: I18919d71464c094db5846badbfac80f99366d70c
Convert parallelVisitAllBottomUp into a generic parallelVisit that takes
a visitOrderer to select top down or bottom up. Combine
runTopDownMutator and runBottomUpMutator into runMutators that takes a
mutatorDirection to select the visitOrderer for parallelVisit and which
function pointer in the mutatorInfo to run. Optimize out the
updateDependencies to only run if dependencies have been modified to
avoid running it after TopDownMutators that cannot modify dependencies.
Change-Id: Ib00302db1108ebab2ce8e01b20aa026140d382a4
cloneModules duplicates every module, which can be slow. Spawn a new
goroutine for each module to spread the factory and CopyProperties calls
over multiple cpus.
Change-Id: I2597f921dadfd321ccb705aec9a904c71b75f9ef
Allow BottomUpMutators to run in parallel by calling Parallel() on the
return value of RegisterBottomUpMutator. To avoid locking, moves
updates of global state into a separate goroutine that receives updates
over channels from the mutator goroutines.
Change-Id: Ic59b612da9b406cf59ec44940f0c1dee0c051a51
Get rid of runEarlyMutators, and implement EarlyMutators as a shim
around BottomUpMutators. Removes mostly-duplicate code between
runEarlyMutators and runBottomUpMutator to allow parallelizing
runBottomUpMutator.
Change-Id: I66a051cb8710770b47da21d35e55aa49c84e9be8
VisitDirectDeps[If] need to know the parent in order to support
OtherModuleDependencyTag(), remove Context.visitDirectDeps[If] and
reimplement [Module]Context.VisitDirectDeps[If] using Context.walkdeps.
Change-Id: I2faac2d100d2a0ad2a4f1d8250997eed1a7ef6f0
Test: TestVisit
In order to implement some build statistics in Soong, expose the module
type (the string used to define the module) to singletons.
Change-Id: I441d12c7782bcf338b3654cfe907b8d2a7253594
Pos is going to be part of the Node interface, rename the Pos member
of structs to be more specific.
Change-Id: Ibd31119863b96d38bf8dac216e026200a54bbe18
It wasn't adding anything useful, and it resulted in Name.Name to get to
the identifier. Replace Name Ident with Name string; NamePos
scanner.Position.
Change-Id: Idf9b18b31dd563a18f27c602c2d14298955af371
Refactor the blueprint parser Value object, which contained a Type enum
and members to hold every possible type, into an interface (now called
Expression). Rename the existing Expression object that represented a binary
operator Operator.
Also adds and fixes some new printer test cases with mulitline expressions.
Change-Id: Icf4a20f92c8c2a27f18df8ca515a9d7f282ff133
Variants of a module may need to depend on other variants, for example
to reuse results between variants. Support AddInterVariantDependency to
add an explicit dependency from one variant to another, along with a
dependency tag. Both modules must have just been returned by a call to
CreateVariants.
Change-Id: I8f4878f94ced74dd00cfac8303b15ef70cdebf36
Primary builder logic is becoming complicated due to the two pass nature
of mutators that add dependencies and GenerateBuildActions that handle
the dependencies. The reason why the dependency was added is lost by
the time GenerateBuildActions is called, resulting in build logic that
has to recreate all the dependencies and try to match them up to the
modules returned by VisitDirectDeps.
Change the API of AddDependency to take a DependencyTag interface, which
is satisifed by anything that embeds BaseDependencyTag. Mutators and
GenerateBuildActions that call VisitDirectDeps can pass each Module to
ctx.OtherModuleDependencyTag to retreive the DependencyTag that was
passed when adding the dependency.
Change-Id: I0814dcd26d1670302d340b77e8dc8704ed7b60bf
A common pattern in mutators is to set a property on a module to be used
later in GenerateBuildActions. A common anti-pattern is to set a member
variable instead of a property. Setting member variables will appear to
work until a mutator calls CreateVariations after the member variable is
set. The first variant will have the member variables, but any other
variants will have zero values for all member variables.
To catch this common case early, replace all modules with clones after
running all the mutators but before GenerateBuildActions.
This catches the anti-pattern used in bootstrap, replace the buildStage
member variable with a property.
Change-Id: I6ff37580783a6227ebba2b46d577b443566a79bb
Our use case is to write out some configuration variables to give to our
legacy build system. But in order to do that, we either need to keep a
Go copy of all of the configuration, and still have all the calls to
convert them to ninja variables, or evaluate our ninja variables.
Change-Id: If9dda305ed41bc8aabe57dd750a74d8b9af1d1a4
visited map was not being updated when the dependency was being
visited, only if it was being walked. This led to cases where
duplicate traversals were possible.
Catch panics in the build logic in order to provide extra data on when
the panic occurred, for example which module had GenerateBuildActions
called on it.
In some cases the primary builder may need to handle missing
dependencies. Add Context.SetAllowMissingDependencies to cause
Blueprint to store the list of missing dependencies without immediately
emitting an error, and ModuleContext.GetMissingDependencies to return
the missing dependencies to the primary builder. If the primary builder
does not call ModuleContext.GetMissingDependencies Blueprint will emit
dependency errors.
When building a subset of a source tree, some of the subdirectories
mentioned in the subdirs variable may be missing. Add support for an
optional_subdirs variable, which won't error out if the requested
directory is missing.
Turn PackageContext into an interface so that build systems can wrap it
to add more custom helpers.
This does introduce an API change, though it should be fairly simple.
NewPackageContext used to provide an opaque *PackageContext struct, now it
provides a PackageContext interface.
Change-Id: I383c64a303d857ef5e0dec86ad77f791ba4c9639
Removing abandoned files needs to know where the .ninja_log file is
stored. Export the ninja builddir value from Context and use it to
determine the .ninja_log path in any stage.
The ninja builddir (where ninja stores its .ninja_log and .ninja_deps
files) and the bootstrap.BuildDir (where build output files are written)
are distinct, so to reduce confusion replace SetBuildDir with
SetNinjaBuildDir.
ctx.SetBuildDir is called by the bootstrap singleton, which is always
last. In order to allow the primary builder to set the directory for
.ninja_log/.ninja_deps, remove the panic on multiple calls to
ctx.SetBuildDir, and always used the value passed by the first caller.
Dependency errors were prefixed with ??? because they were associated
with the position of the "deps" property, which is often not used. Use
the position of the module instead.
beforeInModuleList panic'd when checking if a variant was before itself
in the module list. Return a real error instead of calling
beforeInModuleList, and also fix beforeInModuleList to return false for
the same module.
AddReverseDependencies would add modules the target's dependency list in
a non-deterministic order based on the order the modules were parsed.
Redefine AddReverseDependencies to collect dependencies until the end of
the mutator and then add them sorted by name.
DynamicDependencies can be implemented more flexibly by a
BottomUpMutator. If there are no DynamicDependencies, then
EarlyMutators are identical to BottomUpMutators. Deperecate both, and
reimplement DynamicDependencies inside a BottomUpMutator that is
guaranteed to be registered first.
walkDeps performs a pre-order DFS (unlike visitDepsDepthFirst which is
a post-order DFS). The visit function takes in both a parent and child
node and returns a bool indicating if the child node should be
traversed.
Now that we have multi-stage bootstrapping, we can make the primary
builder build more dynamic. Add the concept of plugins that will be
linked and loaded into bootstrap_go_binary or bootstrap_go_package
modules. It's expected that the plugin's init() functions will do
whatever registration is necessary.
Example Blueprint definition:
bootstrap_go_binary {
name: "builder",
...
}
bootstrap_go_package {
name: "plugin1",
pluginFor: ["builder"],
}
A package may specify more than one plugin if it will be inserted into
more than one go module.
Change-Id: I109835f444196b66fc4018c3fa36ba0875823184
The Go race detector found a race condition in the parser, which
highlighted a few related bugs. A variable could be defined but
not referenced in a Blueprints file, then appended to in multiple
subdirs= Blueprints files. The race detector caught the multiple
writes to assignment.Referenced from the parsers for the subdirs
Blueprints files, but multiple appends would be much more serious.
To fix this, keep local and inherited variables separate in the
Scope object and export that info to the parser. Disallow
appending to non-local variables, which was already the intended
behavior. Only update the referenced boolean for local variables.
Together, this should prevent all writes to Assignment objects
from parsers other than the one that created them.
Also improves the error handling code and some error messages.
Change-Id: Idb4f7d2e61bbe28d90b93074764e64e60d1eba8f
The workaround no longer works with the new stage selector. We may run
bootstrap.ninja.in twice before running the next stage, but we can't
encode whether to run another ninja instance in the checked in
build.ninja.in.
This can likely be solved, but now that there's an official release with
support for multiple passes, just push up the required version.
Change-Id: I76e321912e323d60e462aabec61bdfcc7118cd5e
Modules may decide not to output any build definitions in some cases.
Clean up the ninja file by not adding headers for empty sections.
One particular usecase is my upcoming multi-stage bootstrapping -
bootstrap_go_* will not output any rules in the first stage unless it's
required to build the primary builder.
Change-Id: I6a6b54da7e1702c63bfa736bcf8daf16956f9449
This change fixes a bug where directories that could contain a Blueprints file
but currently do not were not being added as a dependency. This meant that if
a Blueprints file were added to that directory it would not automatically get
picked up by the build.
Refactor parsing Blueprints and walking the Blueprints tree to
allow reuse for tasks that don't involve creating moduleInfo
structures.
Change-Id: I677857a3462999426c8306432074ea97fdcb86c8
The primary builder will now generate a rule to call itself with
--docs=.bootstrap/docs/<name>.html to produce an automatically
generated documentation file.
The documentation generation process is:
- Call each factory once to get empty property structs associated
with the module type
- Use reflection to determine the names of the type of each property
struct
- Use the bootstrap_go_package modules from reading the Blueprints files
to find the source files for each Go package used to build the primary
builder
- Use the go/parser module to find the type declaration for each
property struct
- Extract comments for the property struct and each property declaration
- Format all the comments into HTML
Change-Id: Icae9307cc10549a30bfc14d6922824099de5a9b0
AddVariationDependencies is used to add dependency modules that
have the same variations as the depending module, but with additional
variataions. It cannot be used to add a dependency that is unrelated
to the depending module, for example a dependency on a code generator
that needs to run on the host to generate a target source file.
Add AddFarVariationDependencies, which adds a dependency on a module
that contains all the passed variations, but ignores the variations
of the depending module, as well as any unspecified variations on
the dependency module.
Change-Id: Ief696ec85cf33ad5fb187227d215c1c2e894f962
Setting the "subname" variable will modify the name of Blueprints
files searched in sub directories. If the specified name is not
found it will fall back to trying "Blueprints". The subname
variable stays in scope for all subdirectories.
Change-Id: If3c8ac54735042076fb264d33719d9ea4b8934b3
build is a magic variable that acts similarly to subdirs, but specifies
files to build instead of subdirs. Using files provides a few advantages:
files can be named something more appropriate for the project than
"Blueprints", whatever you want instead of Blueprints, and multiple
Blueprints files can exist in the same directory.
A new variable is used instead of putting filenames into subdirs to avoid
unexpected behavior when a glob matches both files and directories.
subdirs= and build= entries that don't match any directories or files
are now reported as errors.
Change-Id: Id329504ace251eab4ccea1081a3c8665a4c52f5a
This fixes a bug where a split module could be reached by subsequent
mutator passes, but it is no longer in module group's list of modules.
Change-Id: I74c3d849ed2614dcdb6f54a6760122b933bb84c2
makeUniquePackageNames never assigned to pkgs, so all packages
were using their short names, even if another package had the
same short name.
Change-Id: I5420f7839efd25362d6c5d85265399ce7b29ddb2
The patch "Fix panic when dependency cycle includes the first
module visited" caused cycles to print incorrectly by initializing
the current module to be the last module in the cycle, when it
should be the first module in the cycle.
Change-Id: Iaf939283a48faa4cc6eeb9b19aed57993575a687
Running bpfmt or bpmodify on a Blueprints file that references
variables defined in a parent Blueprints file causes parse errors
on unknown variables. Modify the parser to parse in two modes,
parser.Parse and parser.ParseAndEval. The first parses without
attempting to evaluate variables and expressions, the second
performs a full evaluation.
Change-Id: Ic11948ea77c8e65379d7260591ada15db0e4f5b9
Calling CreateVariations with no variants will corrupt the
modules list with a module with nil logicModule, panic early
with a useful message instead.
Change-Id: Ic5c921efcba70c54efb5bb21a9626b2999376a69
createVariations relies on reusing the logicModule for the first
variant in order to replace the original module in the global module
map, which prevents leaving old split modules around in the global
module map. Add comments to explain the behavior.
Change-Id: Ia06d7fa4bc777e807241996d4e8bf977e641d741
spliceModules was carefully reallocating the slice if necessary, and
then throwing away the reallocated slice and writing to the too-short
original slice. Fix the bug, and add tests that would have caught it.
Change-Id: Ifc13b2025198b270c97097fd7d28cd36e460c98c
"Variant" has been used to mean two slightly concepts in Blueprint.
The first and most obvious is a variant of a module, meaning a list
of all the ways that on version of a module differs from other
versions of the same modules, for example "arch:arm, link:static".
The other is for the specific way that a version of a module differs,
"arch:arm".
Rename all of uses of "variant" as the second meaning to "variation",
and update the documentation to clarify "variant" vs. "variation".
This modifies the build logic api to convert CreateVariants to
CreateVariations.
Change-Id: I789ef209ae6ddd39ec12e0c032f9f47f86698fe6
The mutators that run after dependencies are resolved can be too late
to support build logic that needs to vary the dependencies based on
the mutated axis, for example architecture. This patch provides an
EarlyMutator interface that can be used to mutate modules before
any dependencies have been resolved.
In order for dependencies to be satisifed in a later pass, all
dependencies of a module must either have an identical variant,
must have a single variant, or must be inserted using
DynamicDependencyModuleContext.AddVariantDependency.
Change-Id: Ic6ae57e98edfd6c8c09a7788983128d3e4e992f0
Move actionDefs from moduleGroup to moduleInfo. This will result
in multiple build.ninja headers for a single Blueprint file module
if it has multiple variants, each one specifying the variant
used to generate the following rules.
Change-Id: I414cd4d3266da8c2e92118b295569627ddf48cdd
Instead of iterating through groups and then iterating through the
group's modules, iterate through the modules directly. This will
allow circular dependencies between groups as long as the individual
modules don't have circular dependencies. In order to maintain the
ordering of modules in a group, each module is considered to have
an implicit dependency on any earlier modules in the same group.
Change-Id: Ibce67167c7234db70ede0a6b5c2b43fb8e0bb05d
Replace the array of mutator name/variant name pairs with
a map of mutator name to variant name, and store the string
variant name separately.
Change-Id: I181c2fcb05724c8755b90aaf866fdce9ef720b01
Move most of the contents of moduleGroup into moduleInfo. This will
eventually reduce moduleGroup to a simple list of variants of the
same module.
Change-Id: I4289eb9953509751d7637558cd6db73373ccdf78
Forcing module names to be valid ninja names is an unnecessary
restraint on the project build logic. Allow any string as a
module name, and sanitize and uniquify the module name for use
in module-scoped variables.
Also move the module scope to be per-module instead of per-group
so that modules can use the same local variable name for each variant.
Change-Id: If44cca20712305e2c0b6d6b39daa5eace335c148
The cycle check can panic if the first call to check happens to land
on the first module in a cycle. Print the cycle instead of panicking.
Change-Id: I6fc1c66dcc37b1eb6b11b9e65343452af3c8538d
generateModuleBuildActions was continuing to call GenerateBuildActions
on all modules even after one returned an error. This could cause a
later module to panic because it was missing information that was
supposed to be generated by the module that returned an error, hiding
the earlier error message.
If a module returns an error, stop triggering any new goroutines to
call GenerateBuildActions on later modules.
