Add globbing with dependency checking to blueprint. Calling
ModuleContext.GlobWithDeps or SingletonContext.GlobWithDeps will return
a list of files that match the globs, while also adding efficient
dependencies to rerun the primary builder if a file that matches the
glob is added or removed.
Also use the globbing support for optional_subdirs=, subdirs= and build=
lines in blueprints files. The globbing slightly changes the behavior
of subname= lines, it no longer falls back to looking for a file called
"Blueprints". Blueprint files that need to include a subdirectory with
a different name can use build= instead of subdir= to directly include
them. The Blueprints file is updated to reset subname="Blueprints" in
case we want to include subdirectories inside blueprint and the primary
builder has changed the subname.
Also adds a new test directory that contains a simple primary builder
tree to test regeneration for globbing, and runs the tests in travis.
Change-Id: I83ce525fd11e11579cc58ba5308d01ca8eea7bc6
Other than cleaning up the code, the only change in behavior is to run
the docs build in parallel with the primary builder, saving about a
second on a Soong bootstrap (10s -> 9s).
Change-Id: Iaff51d6d1a37af842f294f12db11d5774d48d440
Soong (and soon, Blueprint) embeds the results of filesystem globs into
the build.ninja, and uses a helper tool to detect when the glob changes
and we need to rebuild the build.ninja file. This is more flexible than
listing the affected directories in the depfiles, since it can check to
see if a file we actually cared about was added, instead of re-running
anytime any file was added/removed (which happens on atomic file
modifications as well).
My recent bootstrap simplification broke this, since the helper rules
are in the main build.ninja, but I removed the ability to regenerate
that file from itself. So keep the current model, but add a rule into
the primary and main stages that allow themselves to re-run their
generator and write out a new build.ninja file if necessary. The actual
build rules of the generator aren't necessary, since we already built
them in the previous stage.
Change-Id: Ib51245920b2ec3ee5306c0c269361a5a8733caa8
Blueprint properties that end up as command line arguments need to be
both ninja and shell escaped. Provide helpers that primary builders can
use to appropriately escape them.
Change-Id: Ifd697d87edb1c6f0a910377835c391bbe8f95b42
tl;dr: Read if you don't use the wrapper or use SKIP_NINJA
Previously, we were relying on the ninja behavior of restarting the
build when the build.ninja file was updated to switch between different
bootstrap stages. But that means that every step that could produce a
build.ninja must pass in order to switch to a different stage. That
wasn't a big problem when we had a two stage build -- there was very
little that could fail in the second stage before we chose to go back to
the first stage. But when we had a three stage build, it was possible to
get into a state (usually during development) where you were in the
second stage, but the build was failing because the first stage needed
to be run. This was fixed in d79f1af742
by adding a wrapper that always started building at the first stage.
But this kept all of the complexity of using ninja restarts without any
of the benefits, so this change removes that complexity and just runs
each stage sequentially in the wrapper. So the wrapper is now required.
Since we're no longer going through choosestage, we can also skip the
template parsing for the later stages that don't need to be templated --
this can save a couple of seconds for large files.
In addition to all of the above, this also lets Soong reduce the number
of times the main ninja file is loaded. We had been running the wrapper
once (3 stages), then running ninja again after combining the
Soong-generated build.ninja with the Kati-generated build.ninja. This
change lets us removing the intermediate parsing of Soong's build.ninja,
so that we only execute ninja 3 times per build. It also lets us have
dependencies on pools or rules from Kati in the primary builder, since
we're never executing the main build.ninja without the Kati build.ninja.
The wrapper has a new option, NINJA to provide the path to ninja. This
used to be hardcoded to `ninja`, and will still default to that. But
we'll be running the first two bootstrap stages with $NINJA even if
SKIP_NINJA is set.
The wrapper passes "-w dupbuild=err" to ninja now -- this really should
always be turned on if you care about reliable builds.
Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
Refactor the blueprint parser Value object, which contained a Type enum
and members to hold every possible type, into an interface (now called
Expression). Rename the existing Expression object that represented a binary
operator Operator.
Also adds and fixes some new printer test cases with mulitline expressions.
Change-Id: Icf4a20f92c8c2a27f18df8ca515a9d7f282ff133
When appending properties, it may be necessary to determine if two
property structs are the same "type". A simple Go type comparison is
not sufficient, as there may be interface{} values in the property
structs that contain different types. Add proptools.TypeEqual that
returns true if they have equal types and all embedded pointers to
structs and interfaces to pointers to structs have the same nilitude and
type.
Add tests for CloneProperties, CloneEmptyProperties and ZeroProperties
and fix detected bugs related to nil pointers to structs and interfaces
containing nil pointers to structs.
It is common for a mutator to append or prepend property structs
together. Add helper functions to append or prepend properties in property
structs. The append operation is defined as appending string and slices
of strings normally, OR-ing bool values, and recursing into embedded
structs, pointers to structs, and interfaces containing pointers to
structs. Appending or prepending the zero value of a property will
always be a no-op.
This removes the need to use $OLDPWD when running tests, which means
that the builddir may be an absolute or relative directory. It also
filters out the "PASS" message on successful test runs to clean up our
output.
Change-Id: I4ab937c7a87b74fe997a47cc0311e2f357f9f7e9
It's difficult for wrapping scripts to handle -b properly. Just pass
BUILDDIR instead, which is easier to handle. This still accepts -b, so
that incremental builds work across this change.
Now that we have multi-stage bootstrapping, we can make the primary
builder build more dynamic. Add the concept of plugins that will be
linked and loaded into bootstrap_go_binary or bootstrap_go_package
modules. It's expected that the plugin's init() functions will do
whatever registration is necessary.
Example Blueprint definition:
bootstrap_go_binary {
name: "builder",
...
}
bootstrap_go_package {
name: "plugin1",
pluginFor: ["builder"],
}
A package may specify more than one plugin if it will be inserted into
more than one go module.
Change-Id: I109835f444196b66fc4018c3fa36ba0875823184
The go compiler and linker changed in v1.5 -- to 'go tool compile' and
'go tool link' instead of 6g and 6l. Move the selection logic to
bootstrap.bash, and have it use compile/link if 6g/6l are missing. This
way the build.ninja.in will continue working with either go 1.4 or 1.5.
Travis and the test suite will fail under 1.5, since the build.ninja.in
is still generated with 1.4, and the function names in the comments
differ between 1.4 and 1.5.
To provide a consistent __FILE__ behavior with cpp, we want to be able
to run with SRCDIR="." and the outputs be saved elsewhere. Other tools
within android also expect to be run from $TOP.
Change-Id: I572bce5c9086b0c3310b42065ae98cbf5a1c6399
API Change -- will require changes to any code using bootstrap.BinDir
This way we can put references to other variables in BinDir, and still
be used properly by other packages.
Change-Id: I497424cb254b3a170401ac9420fa0adbf8d11d1e
This splits the current bootstrap stage into two stages:
A bootstrap stage, which like today, a reference is checked into the
tree. It just builds the "core" blueprint binaries -- minibp,
gotestmain, and choosestage. Just enough to build the next stage's ninja
file.
A primary builder stage. This builds the primary builder, the main ninja
file, and any other bootstrap binaries (bpfmt, etc).
The main advantage here is that the checked in file really only contains
references to blueprint -- not the primary builder. This will allow us
to make the primary builder more dynamic, by loading more module types
that may or may not exist in all trees.
It's even possible to reuse the build.ninja.in in the blueprint repo
directly now. We don't currently do that, since we still want to turn on
tests.
Change-Id: I18683891ed7348b0d7af93084e3a68a04fbd5dbc
This simplifies the bootstrap process while making it more flexible by
moving the stage selection into a go binary(choosestage). It will now be
possible to have more than two build stages.
Now each stage has a ninja template(main.ninja.in) and a timestamp
file(main.ninja.in.timestamp). The timestamp file may be updated by any
build stage that wishes to regenerate the ninja template. If the
choosestage binaries sees that the timestamp is newer than the template,
it will choose the prior stage.
The main stage no longer writes to the source tree to update the
build.ninja.in file. This was a problem for read-only source trees.
Instead, the choosestage binary first checks to see if that file is
newer than the last bootstrap.ninja.in, copies it in place, and starts
the boostrap stage.
The bootstrap stage regenerates it's own ninja template, but that
required a loop through the main stage to actually run it. The
choosestage binary now detects if the template has changed for the
current stage, and will restart the stage.
One change is that if dependencies do get messed up, instead of silently
failing, there's a higher chance that the bootstrap step will just
continue looping, doing nothing. This can happen if the main stage
has a dependency that triggers the bootstrap stage, but the bootstrap
stage doesn't see anything required to rebuild the main ninja file. A
side effect of this requirement is that changes to test code will now
rebuild the main ninja file.
Change-Id: I9965cfba79dc0dbbd3af05f5944f7653054455a2
This only checks to make sure that for a given updated file, the
bootstrap stage is properly run. It doesn't actually check to make sure
that anything was rebuilt.
Change-Id: I9cb6ff1d483264da30e43d5580361d93b148f42c
The workaround no longer works with the new stage selector. We may run
bootstrap.ninja.in twice before running the next stage, but we can't
encode whether to run another ninja instance in the checked in
build.ninja.in.
This can likely be solved, but now that there's an official release with
support for multiple passes, just push up the required version.
Change-Id: I76e321912e323d60e462aabec61bdfcc7118cd5e
The primary builder will now generate a rule to call itself with
--docs=.bootstrap/docs/<name>.html to produce an automatically
generated documentation file.
The documentation generation process is:
- Call each factory once to get empty property structs associated
with the module type
- Use reflection to determine the names of the type of each property
struct
- Use the bootstrap_go_package modules from reading the Blueprints files
to find the source files for each Go package used to build the primary
builder
- Use the go/parser module to find the type declaration for each
property struct
- Extract comments for the property struct and each property declaration
- Format all the comments into HTML
Change-Id: Icae9307cc10549a30bfc14d6922824099de5a9b0
Users that want to enable this option can use the '-t' option to
bootstrap.bash when passing '-r'. Builders that want to enable this can
set the RUN_TESTS environment variable in their bootstrap.bash.
The gotestmain tools is needed to write the main functions for the test
binaries, since 'go test' doesn't work well in this environment.
Change-Id: Iec5c2b5c9c3f5e3ba0ac8677fb88f5e963f9bd3f
Make integrating with go tools easier by putting the blueprint package
files in the top level directory of the git project instead of in a
subdirectory called blueprint.
Change-Id: I35c144c5fe7ddf34e478d0c47c50b2f6c92c2a03
This change makes removeAbandonedFiles perform some of the string substitutions
that are done by the bootstrap script before considering whether a built target
still exists. It fixes a bug where the bootstrap manifest would get deleted
and subsequently regenerated, resulting in more re-bootstrap iterations than
was necessary.
Change-Id: I946c883dcc31fb77496f64d2573b395ad2cf095d
This change makes the bootstrapping process remove any files that were
previously created by invoking a Ninja rule (i.e. they appear in the .ninja_log
file) but are no longer a build output target.
Change-Id: I3c78e563393b97f8ca196ac85c7caa2b3866ffa6
This change replaces the automatic caller package divination with a
PackageContext object that must be explicitly passed in by callers.
Change-Id: I139be29ecf75a7cf8488b3958dee5e44363acc22
This change eliminates blueprint.ModuleType and replaces it with simple factory
functions. Rather than using the explicitly provided ModuleType names to
essentially identify the Go factory function in the generated Ninja files, we
now use the actual factory function name.
Change-Id: Ib7813e850322a82cc35cdc56bebff7d580a5c6ec
This change makes the bootstrap script take two optional flag-based command
line arguments rather than using positional arguments. The -i argument
replaces the old positional arguments (one of which was simply not needed), and
the -r flag now causes the script to regenerate the bootstrap Ninja file for
use when working on Blueprint itself.
Additionally this change allows all the configuration variables used in the
script to be overridden by environment variables. This makes it possible to
create a very simple bootstrap script for a customized Blueprint-based build
system by invoking this script after setting up the proper environment
variables.
Change-Id: I60f890061ae237a68719e1e2d345d1a8e87c851c
This change adds a bunch of docs describing the bootstrap process. It also
renames a couple things in the source to better correspond with the terminology
used in the docs. On top of that it makes the standalone build to copy minibp
from the .bootstrap/bin directory to the bin directory to make it easier to
find.
Change-Id: Ibae3e607b59e2caf7d764ec3153155fde208ec1e
This change causes the module and singleton content in generated Ninja files to
be sorted by name.
Change-Id: I6e1fe6d4211111c7dfa25ebe0fc721701a6d0256
This change fixes an issue where the re-bootstrapping process would overwrite a
newer bootstrap manifest with one that it generates using its older minibp. It
fixes the issue by only generating a new bootstrap manifest right after
rebuilding minibp (as part of the bootstrap process). It then uses an
additional rebootstrap iteration to replace the old bootstrap manifest with the
new one.
Change-Id: I16bad2f30f6ad7f10da07d77105e8745adec3650
This change makes the module definition location that's included in the build
manifest comments use paths relative to the top-level source directory. This
should make the bootstrap build manifest not get regenerated when using a new
build directory.
Change-Id: I350562ad10aa52688c8841c96c6325502b0faa59
Blueprint is a build system component that reads Blueprints files defining
modules to be built, and generates a Ninja build manifest that can be used to
perform all the build actions. It does not dictate or implement much build
policy itself, but rather provides a framework to ease the process of defining
build logic in Go.
The "blueprint" and "blueprint/parser" Go packages contain the functionality
for reading Blueprint files and invoking build logic functions defined in other
Go packages.
The "blueprint/bootstrap" Go package contains just enough build logic to build
a binary that includes Blueprint and any pure-Go (i.e. no cgo) build logic
defined in external Go packages. This can be used to create a minimal Ninja
file that's capable of bootstrapping a Blueprint-based build system from
source.
The "blueprint/bootstrap/minibp" Go package contains code for a minimal binary
that includes the build logic defined in the "blueprint/bootstrap" package.
This binary can then create the Ninja file for the bootstrapping process.
Change-Id: I8d8390042372a72d225785cda738525001b009f1
