Providers are a new concept for Blueprint, based on providers in Bazel:
https://docs.bazel.build/versions/master/skylark/rules.html#providers
Providers aim to simplify the interaction between modules by replacing
type asserting to arbitrary interfaces with requesting optional data
objects from modules. This will also move Blueprint closer to supporting
incremental analysis by serializing the providers and only rerunning
the analysis phase on modules whose inputs have changed.
Change-Id: I39f5f78b372412a7dbf151ceccb3f917f6c874bf
Adding a dependency on a module with variants can be problematic
if the code adding the dependency is not aware of every mutator
that has created variants. Add an AliasVariations to
BottomUpMutatorContext, which allows a mutator to alias the
original variant of a module to one of the new variants of the
module, which will allow future dependencies to be added using
the original list of variations. The aliases are transient,
and only exist until the next mutator that calls CreateVariations
when visiting the module without also calling AliasVariations.
Test: TestAlises
Change-Id: Ieaa04b5a6bdcb5a1ff5114b1e03460de795d4479
Move some code from Soong to support creating a property struct
at runtime by filtering fields out of another property struct.
Test: TestFilterPropertyStruct
Change-Id: Ic5ae390a885195bebad6f3ecb7c752c0582a60b1
This change fixes a bug where the `blueprint:mutated` tag was ignored
when used for struct or interface properties. It also fixes a dormant
ExcludeByTag func bug.
Test: m soong_docs, properties_test.go, bpdoc_test.go
Change-Id: I926de2aa203ec552ced897174e9b78e817701a7d
Add a function that returns all of the indexes to properties in
a property struct that are tagged with `name:"value"`.
Test: proptools/tag_test.go
Change-Id: I00294934c1a0383c8b64ecaabc0e138682efb2e5
1. Extract module type documentation.
2. Support primary builder customization of factory function to use for
documentation for each module type.
3. Change the ModuleType list order so that they are grouped by package.
This is basically minor refactoring + readability improvement done on
top of https://github.com/google/blueprint/pull/232.
Change-Id: If7413e5ac23486b85f18d02fb3ba288a38730c32
Add support for specifying symlinks in mock filesystems to prepare
for glob symlink tests.
This patch leaves incorrect behavior by not walking symlinks in
mockFs.ListDirsRecursive, but it matches what osFs does.
Test: fs_test.go
Change-Id: If87a83c00f21e14696faf890b7b09e88b18e95b9
Patch and PatchList provide an API for making changes to substrings
of a Blueprint file by parsing the AST and using the token positions
to perform text replacements.
Test: modify_test.go
Change-Id: Ibb8993221982b54602ba5a05486198fab8d35a67
Performing the same glob twice, once with excludes=nil and once
with excludes=[]string{} would hit the same entry in the glob
cache (since the glob filename would be the same), but fail
the verifyGlob check because DeepEqual considers []string(nil)
and []string{} to be different. Use a manual array check
instead.
Test: glob_test.go
Change-Id: If0d4fe80163a871077b7276e1b4a3e888a4a4898
in facilitate moving name resolution to Soong
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
--products=aosp_arm \
'build/blueprint:work^' 'build/blueprint:work'
# and see that the only changes were:
# 1. adding the name_interface.go file
# 2. changing some line numbers
Change-Id: Ifa7603cca59b3b3d592f2f146fdafe57012bd4b9
An ill-fated experiment with using a format that was closer to
Bazel left Blueprint preferring the original style, but documenting
and using the new style. Update the documentation and run bpfmt -w.
Test: builds
Change-Id: I3fb70c8fa50c332da01cf5912b179c60d1638814
In addition to running with `go run` and creating a microfactory binary,
allow microfactory to be used as a package from other go tools as well.
To allow other packages to use this, it needs to be in a non-main
package, but `go run` requires a main package. So microfactory.bash runs
a sed script before running microfactory with `go run`.
This could also be solved by using a relative import, but neither
blueprint nor microfactory currently support that.
Change-Id: I084163b14720102b3fb93a3c9d44b5d0225ff2c8
This duplicates building common blueprint go packages between minibp and
the primary builder, but drastically simplifies the first stage,
removing the need to check in a generated build.ninja.in.
Change-Id: I639a9637f1ed36d4210823ef276c0f7a064a83bd
If tests have a TestMain method the os import is unused and
breaks building the test.
Test: gotestmain/testmain_test.go
Change-Id: Ia46cdd0df71b0fc7a53d08644d220ecfd779b2ff
Add globbing to filesystem mocking so that more code can be tested
against the mock. Also moves the filesystem mock to pathtools,
and renames pathtools.GlobWithExcludes to pathtools.Glob, replacing
the existing pathtools.Glob.
Test: blueprint tests
Change-Id: I722df8121bc870c4a861d7c249245c57dcb607be
Add globbing with dependency checking to blueprint. Calling
ModuleContext.GlobWithDeps or SingletonContext.GlobWithDeps will return
a list of files that match the globs, while also adding efficient
dependencies to rerun the primary builder if a file that matches the
glob is added or removed.
Also use the globbing support for optional_subdirs=, subdirs= and build=
lines in blueprints files. The globbing slightly changes the behavior
of subname= lines, it no longer falls back to looking for a file called
"Blueprints". Blueprint files that need to include a subdirectory with
a different name can use build= instead of subdir= to directly include
them. The Blueprints file is updated to reset subname="Blueprints" in
case we want to include subdirectories inside blueprint and the primary
builder has changed the subname.
Also adds a new test directory that contains a simple primary builder
tree to test regeneration for globbing, and runs the tests in travis.
Change-Id: I83ce525fd11e11579cc58ba5308d01ca8eea7bc6
Blueprint properties that end up as command line arguments need to be
both ninja and shell escaped. Provide helpers that primary builders can
use to appropriately escape them.
Change-Id: Ifd697d87edb1c6f0a910377835c391bbe8f95b42
tl;dr: Read if you don't use the wrapper or use SKIP_NINJA
Previously, we were relying on the ninja behavior of restarting the
build when the build.ninja file was updated to switch between different
bootstrap stages. But that means that every step that could produce a
build.ninja must pass in order to switch to a different stage. That
wasn't a big problem when we had a two stage build -- there was very
little that could fail in the second stage before we chose to go back to
the first stage. But when we had a three stage build, it was possible to
get into a state (usually during development) where you were in the
second stage, but the build was failing because the first stage needed
to be run. This was fixed in d79f1af742
by adding a wrapper that always started building at the first stage.
But this kept all of the complexity of using ninja restarts without any
of the benefits, so this change removes that complexity and just runs
each stage sequentially in the wrapper. So the wrapper is now required.
Since we're no longer going through choosestage, we can also skip the
template parsing for the later stages that don't need to be templated --
this can save a couple of seconds for large files.
In addition to all of the above, this also lets Soong reduce the number
of times the main ninja file is loaded. We had been running the wrapper
once (3 stages), then running ninja again after combining the
Soong-generated build.ninja with the Kati-generated build.ninja. This
change lets us removing the intermediate parsing of Soong's build.ninja,
so that we only execute ninja 3 times per build. It also lets us have
dependencies on pools or rules from Kati in the primary builder, since
we're never executing the main build.ninja without the Kati build.ninja.
The wrapper has a new option, NINJA to provide the path to ninja. This
used to be hardcoded to `ninja`, and will still default to that. But
we'll be running the first two bootstrap stages with $NINJA even if
SKIP_NINJA is set.
The wrapper passes "-w dupbuild=err" to ninja now -- this really should
always be turned on if you care about reliable builds.
Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
Refactor the blueprint parser Value object, which contained a Type enum
and members to hold every possible type, into an interface (now called
Expression). Rename the existing Expression object that represented a binary
operator Operator.
Also adds and fixes some new printer test cases with mulitline expressions.
Change-Id: Icf4a20f92c8c2a27f18df8ca515a9d7f282ff133
Move these tools from $buildDir/.bootstrap/bin to $buildDir/bin
(configurable by the primary builder).
Also delay building them until the main stage, and give them a phony
target "blueprint_tools".
When appending properties, it may be necessary to determine if two
property structs are the same "type". A simple Go type comparison is
not sufficient, as there may be interface{} values in the property
structs that contain different types. Add proptools.TypeEqual that
returns true if they have equal types and all embedded pointers to
structs and interfaces to pointers to structs have the same nilitude and
type.
Add tests for CloneProperties, CloneEmptyProperties and ZeroProperties
and fix detected bugs related to nil pointers to structs and interfaces
containing nil pointers to structs.
It is common for a mutator to append or prepend property structs
together. Add helper functions to append or prepend properties in property
structs. The append operation is defined as appending string and slices
of strings normally, OR-ing bool values, and recursing into embedded
structs, pointers to structs, and interfaces containing pointers to
structs. Appending or prepending the zero value of a property will
always be a no-op.
This removes the need to use $OLDPWD when running tests, which means
that the builddir may be an absolute or relative directory. It also
filters out the "PASS" message on successful test runs to clean up our
output.
Change-Id: I4ab937c7a87b74fe997a47cc0311e2f357f9f7e9
Now that we have multi-stage bootstrapping, we can make the primary
builder build more dynamic. Add the concept of plugins that will be
linked and loaded into bootstrap_go_binary or bootstrap_go_package
modules. It's expected that the plugin's init() functions will do
whatever registration is necessary.
Example Blueprint definition:
bootstrap_go_binary {
name: "builder",
...
}
bootstrap_go_package {
name: "plugin1",
pluginFor: ["builder"],
}
A package may specify more than one plugin if it will be inserted into
more than one go module.
Change-Id: I109835f444196b66fc4018c3fa36ba0875823184
This splits the current bootstrap stage into two stages:
A bootstrap stage, which like today, a reference is checked into the
tree. It just builds the "core" blueprint binaries -- minibp,
gotestmain, and choosestage. Just enough to build the next stage's ninja
file.
A primary builder stage. This builds the primary builder, the main ninja
file, and any other bootstrap binaries (bpfmt, etc).
The main advantage here is that the checked in file really only contains
references to blueprint -- not the primary builder. This will allow us
to make the primary builder more dynamic, by loading more module types
that may or may not exist in all trees.
It's even possible to reuse the build.ninja.in in the blueprint repo
directly now. We don't currently do that, since we still want to turn on
tests.
Change-Id: I18683891ed7348b0d7af93084e3a68a04fbd5dbc
This simplifies the bootstrap process while making it more flexible by
moving the stage selection into a go binary(choosestage). It will now be
possible to have more than two build stages.
Now each stage has a ninja template(main.ninja.in) and a timestamp
file(main.ninja.in.timestamp). The timestamp file may be updated by any
build stage that wishes to regenerate the ninja template. If the
choosestage binaries sees that the timestamp is newer than the template,
it will choose the prior stage.
The main stage no longer writes to the source tree to update the
build.ninja.in file. This was a problem for read-only source trees.
Instead, the choosestage binary first checks to see if that file is
newer than the last bootstrap.ninja.in, copies it in place, and starts
the boostrap stage.
The bootstrap stage regenerates it's own ninja template, but that
required a loop through the main stage to actually run it. The
choosestage binary now detects if the template has changed for the
current stage, and will restart the stage.
One change is that if dependencies do get messed up, instead of silently
failing, there's a higher chance that the bootstrap step will just
continue looping, doing nothing. This can happen if the main stage
has a dependency that triggers the bootstrap stage, but the bootstrap
stage doesn't see anything required to rebuild the main ninja file. A
side effect of this requirement is that changes to test code will now
rebuild the main ninja file.
Change-Id: I9965cfba79dc0dbbd3af05f5944f7653054455a2
The primary builder will now generate a rule to call itself with
--docs=.bootstrap/docs/<name>.html to produce an automatically
generated documentation file.
The documentation generation process is:
- Call each factory once to get empty property structs associated
with the module type
- Use reflection to determine the names of the type of each property
struct
- Use the bootstrap_go_package modules from reading the Blueprints files
to find the source files for each Go package used to build the primary
builder
- Use the go/parser module to find the type declaration for each
property struct
- Extract comments for the property struct and each property declaration
- Format all the comments into HTML
Change-Id: Icae9307cc10549a30bfc14d6922824099de5a9b0
Users that want to enable this option can use the '-t' option to
bootstrap.bash when passing '-r'. Builders that want to enable this can
set the RUN_TESTS environment variable in their bootstrap.bash.
The gotestmain tools is needed to write the main functions for the test
binaries, since 'go test' doesn't work well in this environment.
Change-Id: Iec5c2b5c9c3f5e3ba0ac8677fb88f5e963f9bd3f
Make integrating with go tools easier by putting the blueprint package
files in the top level directory of the git project instead of in a
subdirectory called blueprint.
Change-Id: I35c144c5fe7ddf34e478d0c47c50b2f6c92c2a03
bpmodify can be used to add or remove dependencies or other strings
from selected modules in Blueprint files.
Change-Id: I0df3762976e74bf46fd2922bbd48b46e526b7951
This change makes the bootstrapping process remove any files that were
previously created by invoking a Ninja rule (i.e. they appear in the .ninja_log
file) but are no longer a build output target.
Change-Id: I3c78e563393b97f8ca196ac85c7caa2b3866ffa6
This change replaces the automatic caller package divination with a
PackageContext object that must be explicitly passed in by callers.
Change-Id: I139be29ecf75a7cf8488b3958dee5e44363acc22
Blueprint is a build system component that reads Blueprints files defining
modules to be built, and generates a Ninja build manifest that can be used to
perform all the build actions. It does not dictate or implement much build
policy itself, but rather provides a framework to ease the process of defining
build logic in Go.
The "blueprint" and "blueprint/parser" Go packages contain the functionality
for reading Blueprint files and invoking build logic functions defined in other
Go packages.
The "blueprint/bootstrap" Go package contains just enough build logic to build
a binary that includes Blueprint and any pure-Go (i.e. no cgo) build logic
defined in external Go packages. This can be used to create a minimal Ninja
file that's capable of bootstrapping a Blueprint-based build system from
source.
The "blueprint/bootstrap/minibp" Go package contains code for a minimal binary
that includes the build logic defined in the "blueprint/bootstrap" package.
This binary can then create the Ninja file for the bootstrapping process.
Change-Id: I8d8390042372a72d225785cda738525001b009f1
