Emit the rules to build blueprint tests even if runGoTests is not
set, and add a phony rule "blueprint_tests" to run them. This will
allow Soong to stop running the tests at the beginning of every build
but still run them as part of checkbuild or with a manual
`m blueprint_tests`.
Bug: 156428456
Test: m
Test: m blueprint_tests
Change-Id: If293a0757766d3046e78bf230a1825f15adc68fd
Don't write to $buildDir/.bootstrap in the ninja file generated for
the main stage.
Test: examine main stage build.ninja file
Change-Id: I9c6183c9a434ff11d44cdd4f9844eae192ebfe03
The go compile command in go 1.13 needs -p main instead of
-p $pkgPath when compiling binaries. This also works for older
go releases.
Also turn on travis for go 1.13.
Change-Id: I45ce185aa44cfa01a7933cc6e52290a85c851641
The primary builder should always be running on its own with no
other processes running in parallel, so put it in the console
pool so that it can print to stdout without buffering.
Bug: 80165685
Test: build with print in Soong
Change-Id: If34ecdb5fa18de7e47c4cd6965d551c504850176
In cases that we want to run blueprint-based builds in many
configurations to verify all of the logic works without errors, but
don't care about running the final ninja file, writing it out only
wastes time and disk space. So add a --empty-ninja-file option that
writes out an empty ninja file instead.
Our specific use case (Soong's build_test / multiproduct_kati) runs
Soong several hundred times for different configurations, and the ninja
files are around 1GB, which leads to several hundred gigabytes of disk
writes (and persistent use during incremental generation).
Change-Id: I0198dfb2f744ce22284c05d5214dac2ab5dc9700
The test package archive was being picked up through "-L" + testRoot,
but did not have a dependency on it. If the *_test.go file was
changed but not the implementation then the tests would not be
rerun.
Test: Add t.Fatal() to a *_test.go file and rebuild.
Change-Id: I32b8c7a0fa1d5b76b1242646887fe937e396e318
Linking tests needs the same dependencies that linking binaries
got in https://github.com/google/blueprint/pull/222.
Test: m checkbuild
Change-Id: I33330f3184b8c0fd2bc20b48736c20d6edeaea68
Many Android build tools are written in go, and changes to Soong or
Blueprint can cause them to rebuild. Almost everything in an
Android build is downstream of at least one of these tools, so
they all rebuild too.
Go binaries are static, so their contents will always change if any
of their dependencies change. Only update output files of go compile
and go link if the contents change, and enable restat for the rules.
Test: m soong_zip && m soong_zip
Test: m soong_zip && touch build/soong/zip/cmd/main.go && m soong_zip
Change-Id: I9267580f644b42b44d43fb0a2674dc5234f127e5
Instead of sometimes re-running minibp/the primary builder during the
next phase, run bpglob earlier to check dependencies.
We've run into issues where the environment is slightly different
between bootstrapping phase and the main build phase. It's also a
problem because our primary builder (Soong) exports information used by
another tool (Kati) that runs in between the bootstrapping phases and
the main phase. When Soong would run in the main phase, it could get out
of sync, and would require the build to be run again.
To do this, add a "subninja" include a build-globs.ninja file to each
build.ninja file. The first time, this will be an empty file, but we'll
always run minibp / the primary builder anyway. When the builder runs,
in addition to writing a dependency file, write out the
build-globs.ninja file with the rules to run bpglob.
Since bpglob may need to be run very early, before it would normally be
built, build it with microfactory.
Change-Id: I89fcd849a8729e892f163d40060ab90b5d4dfa5d
When passing "-c" to the Go compiler, any time this value changes, we'd
force all of the Go compiles to rebuild. This could trigger a
substantial portion of the tree to rebuild (anything that transitive
depends on a Go helper tool).
We're running into issues when moving output directories between
multiple GCE machines with different core counts, but are otherwise
identical. This could also hit users moving/mounting disks between
machines, though changes to other host tools can make an impact too.
On my 48-core machine, I get a ~15% benefit from going from -c 1 to -c
48, but also ~12% benefit from going from -c 1 to -c 8. So this will
still let us scale somewhat, but prevent rebuilds when transitioning
between machines that are more likely building Android.
We're already going to load the .ninja_deps file, so we may as well use
it. These are the only two files in our build that don't.
Test: dump the .ninja_deps file, see the dependencies
Move the docs to the main stage and make a phony rule to build
them. This will allow the primaray builder to choose not to
build them so that it can use its own rule instead.
Change-Id: Ib788f74717ccdcd85c2e5b6a581cc744d439341a
In case the primary builder wants to depend on a go binary by name,
expose an interface to find the real installed path of the binary.
Most of this change is changing it so that we're storing the install
path without any ninja references.
Change-Id: I873378ebdd47d8036d3cef5aab76f036846a3db1
Having separate .ninja_log files doesn't work very well after
04d886b110
since the last successful timestamp is not recorded in the ninja log.
This was triggering the primary builder to be run twice on every build
since it was always considered out of date in the primary, then main
stages.
The two bootstrap stages already were unified when minibootstrap was
simplified, but this combines the main stage as well. This means that
we'll save some time running globs that are shared between minibp and
the primary builder as well (subdirs globs).
The cleanup code needed to be refactored, since the ninja_log will have
entries from the main stage as well. So instead of looking at everything
in the ninja log, take a string prefix to limit what we may clean.
Change-Id: I8d43168643aa8a0c6c3e48c6101eaa45d174bbe5
Now that packages and binaries are only built when needed, they need to
be defined in both stages in case something building in the main stage
needs something that was defined, but didn't need to be built in the
earlier stage.
Change-Id: I67704fba1d670e82218eba5ce91f0c01896557a2
Instead of forcing things to be built alongside the primary builder,
mark all packages and binaries as optional, unless they're explicitly
marked as default.
Change-Id: Ie53c90a01b41ce886d651f9aa4c4900d1edb0157
This duplicates building common blueprint go packages between minibp and
the primary builder, but drastically simplifies the first stage,
removing the need to check in a generated build.ninja.in.
Change-Id: I639a9637f1ed36d4210823ef276c0f7a064a83bd
In order to use certain syscalls, some source must only be compiled on
one of darwin or linux. There are also cases where it's simplest to have
two implementations of a method (or struct) instead of choosing an
implementation at runtime.
The standard go tools use _<GOOS>.go for this, but the explicit structs
map better into the current blueprint format (and you can still use
_darwin.go / _linux.go to maintain compatiblity with those tools).
Test: Define _linux.go / _darwin.go files, ensure only one is compiled.
Change-Id: I58bffefbf839ba52dea3d8e7d3ec67eaa721c463
We haven't been building the tests for any Go tools that get built
during the primary stage, since we haven't been passing -t to the
primary builder. So enable them there so that we test those tools.
Change-Id: Ic7720c0679419f9e1ba61c7e151f37eb5c7bc8dc
Before this change, we would ensure that tests for a package were run
before anything used that package, even just to compile. With this
change, we'll let other packages compile, but still run the tests in
dependency order, and before the binaries are finally installed.
This shrinks our initial build time from ~9 seconds to ~5 seconds on my
machine. Most of the improvements come from building the "real" package
and the test package in parallel.
Change-Id: I935670feecadacbf09b250bcaa1659a3599edc58
Other than cleaning up the code, the only change in behavior is to run
the docs build in parallel with the primary builder, saving about a
second on a Soong bootstrap (10s -> 9s).
Change-Id: Iaff51d6d1a37af842f294f12db11d5774d48d440
Soong (and soon, Blueprint) embeds the results of filesystem globs into
the build.ninja, and uses a helper tool to detect when the glob changes
and we need to rebuild the build.ninja file. This is more flexible than
listing the affected directories in the depfiles, since it can check to
see if a file we actually cared about was added, instead of re-running
anytime any file was added/removed (which happens on atomic file
modifications as well).
My recent bootstrap simplification broke this, since the helper rules
are in the main build.ninja, but I removed the ability to regenerate
that file from itself. So keep the current model, but add a rule into
the primary and main stages that allow themselves to re-run their
generator and write out a new build.ninja file if necessary. The actual
build rules of the generator aren't necessary, since we already built
them in the previous stage.
Change-Id: Ib51245920b2ec3ee5306c0c269361a5a8733caa8
Remove the assumed Name and Deps properties. Instead, ask the Module
for the Name, and require them to use the existing replacements for
Deps.
Change-Id: I729045d86277686078b3aa0bba71c67d612ead2c
tl;dr: Read if you don't use the wrapper or use SKIP_NINJA
Previously, we were relying on the ninja behavior of restarting the
build when the build.ninja file was updated to switch between different
bootstrap stages. But that means that every step that could produce a
build.ninja must pass in order to switch to a different stage. That
wasn't a big problem when we had a two stage build -- there was very
little that could fail in the second stage before we chose to go back to
the first stage. But when we had a three stage build, it was possible to
get into a state (usually during development) where you were in the
second stage, but the build was failing because the first stage needed
to be run. This was fixed in d79f1af742
by adding a wrapper that always started building at the first stage.
But this kept all of the complexity of using ninja restarts without any
of the benefits, so this change removes that complexity and just runs
each stage sequentially in the wrapper. So the wrapper is now required.
Since we're no longer going through choosestage, we can also skip the
template parsing for the later stages that don't need to be templated --
this can save a couple of seconds for large files.
In addition to all of the above, this also lets Soong reduce the number
of times the main ninja file is loaded. We had been running the wrapper
once (3 stages), then running ninja again after combining the
Soong-generated build.ninja with the Kati-generated build.ninja. This
change lets us removing the intermediate parsing of Soong's build.ninja,
so that we only execute ninja 3 times per build. It also lets us have
dependencies on pools or rules from Kati in the primary builder, since
we're never executing the main build.ninja without the Kati build.ninja.
The wrapper has a new option, NINJA to provide the path to ninja. This
used to be hardcoded to `ninja`, and will still default to that. But
we'll be running the first two bootstrap stages with $NINJA even if
SKIP_NINJA is set.
The wrapper passes "-w dupbuild=err" to ninja now -- this really should
always be turned on if you care about reliable builds.
Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
Move these tools from $buildDir/.bootstrap/bin to $buildDir/bin
(configurable by the primary builder).
Also delay building them until the main stage, and give them a phony
target "blueprint_tools".
Primary builder logic is becoming complicated due to the two pass nature
of mutators that add dependencies and GenerateBuildActions that handle
the dependencies. The reason why the dependency was added is lost by
the time GenerateBuildActions is called, resulting in build logic that
has to recreate all the dependencies and try to match them up to the
modules returned by VisitDirectDeps.
Change the API of AddDependency to take a DependencyTag interface, which
is satisifed by anything that embeds BaseDependencyTag. Mutators and
GenerateBuildActions that call VisitDirectDeps can pass each Module to
ctx.OtherModuleDependencyTag to retreive the DependencyTag that was
passed when adding the dependency.
Change-Id: I0814dcd26d1670302d340b77e8dc8704ed7b60bf
A common pattern in mutators is to set a property on a module to be used
later in GenerateBuildActions. A common anti-pattern is to set a member
variable instead of a property. Setting member variables will appear to
work until a mutator calls CreateVariations after the member variable is
set. The first variant will have the member variables, but any other
variants will have zero values for all member variables.
To catch this common case early, replace all modules with clones after
running all the mutators but before GenerateBuildActions.
This catches the anti-pattern used in bootstrap, replace the buildStage
member variable with a property.
Change-Id: I6ff37580783a6227ebba2b46d577b443566a79bb
Removing abandoned files needs to know where the .ninja_log file is
stored. Export the ninja builddir value from Context and use it to
determine the .ninja_log path in any stage.
The ninja builddir (where ninja stores its .ninja_log and .ninja_deps
files) and the bootstrap.BuildDir (where build output files are written)
are distinct, so to reduce confusion replace SetBuildDir with
SetNinjaBuildDir.
For implicit dependencies that will be common to all users of a Rule,
add a new field 'CommandDeps' to the RuleParam. This is a list of
strings to be prepended to the implicit dependencies in each BuildParam.
This lets us have the dependencies declared next to where they are used,
instead of duplicated in areas that may be far apart.
I looked at passing this information down to ninja too, but it only
saves us a few percent of ninja file, and requires a modification to the
ninja file format.
Change-Id: Ifd910dee1506d4e32a76ed06206f853c4caec622
This removes the need to use $OLDPWD when running tests, which means
that the builddir may be an absolute or relative directory. It also
filters out the "PASS" message on successful test runs to clean up our
output.
Change-Id: I4ab937c7a87b74fe997a47cc0311e2f357f9f7e9
It's difficult for wrapping scripts to handle -b properly. Just pass
BUILDDIR instead, which is easier to handle. This still accepts -b, so
that incremental builds work across this change.
Now that we have multi-stage bootstrapping, we can make the primary
builder build more dynamic. Add the concept of plugins that will be
linked and loaded into bootstrap_go_binary or bootstrap_go_package
modules. It's expected that the plugin's init() functions will do
whatever registration is necessary.
Example Blueprint definition:
bootstrap_go_binary {
name: "builder",
...
}
bootstrap_go_package {
name: "plugin1",
pluginFor: ["builder"],
}
A package may specify more than one plugin if it will be inserted into
more than one go module.
Change-Id: I109835f444196b66fc4018c3fa36ba0875823184
With the introduction of $buildDir in bootstrapDir, the ninja cleanup
step hasn't been able to cleanup during the bootstrap stages. The main
stage was unaffected, as long as you were using "." as your buildDir.
Change-Id: I277dd7864989f9052d96cab9ce377548a1391a80
The go compiler and linker changed in v1.5 -- to 'go tool compile' and
'go tool link' instead of 6g and 6l. Move the selection logic to
bootstrap.bash, and have it use compile/link if 6g/6l are missing. This
way the build.ninja.in will continue working with either go 1.4 or 1.5.
Travis and the test suite will fail under 1.5, since the build.ninja.in
is still generated with 1.4, and the function names in the comments
differ between 1.4 and 1.5.
To provide a consistent __FILE__ behavior with cpp, we want to be able
to run with SRCDIR="." and the outputs be saved elsewhere. Other tools
within android also expect to be run from $TOP.
Change-Id: I572bce5c9086b0c3310b42065ae98cbf5a1c6399
API Change -- will require changes to any code using bootstrap.BinDir
This way we can put references to other variables in BinDir, and still
be used properly by other packages.
Change-Id: I497424cb254b3a170401ac9420fa0adbf8d11d1e
This splits the current bootstrap stage into two stages:
A bootstrap stage, which like today, a reference is checked into the
tree. It just builds the "core" blueprint binaries -- minibp,
gotestmain, and choosestage. Just enough to build the next stage's ninja
file.
A primary builder stage. This builds the primary builder, the main ninja
file, and any other bootstrap binaries (bpfmt, etc).
The main advantage here is that the checked in file really only contains
references to blueprint -- not the primary builder. This will allow us
to make the primary builder more dynamic, by loading more module types
that may or may not exist in all trees.
It's even possible to reuse the build.ninja.in in the blueprint repo
directly now. We don't currently do that, since we still want to turn on
tests.
Change-Id: I18683891ed7348b0d7af93084e3a68a04fbd5dbc
This simplifies the bootstrap process while making it more flexible by
moving the stage selection into a go binary(choosestage). It will now be
possible to have more than two build stages.
Now each stage has a ninja template(main.ninja.in) and a timestamp
file(main.ninja.in.timestamp). The timestamp file may be updated by any
build stage that wishes to regenerate the ninja template. If the
choosestage binaries sees that the timestamp is newer than the template,
it will choose the prior stage.
The main stage no longer writes to the source tree to update the
build.ninja.in file. This was a problem for read-only source trees.
Instead, the choosestage binary first checks to see if that file is
newer than the last bootstrap.ninja.in, copies it in place, and starts
the boostrap stage.
The bootstrap stage regenerates it's own ninja template, but that
required a loop through the main stage to actually run it. The
choosestage binary now detects if the template has changed for the
current stage, and will restart the stage.
One change is that if dependencies do get messed up, instead of silently
failing, there's a higher chance that the bootstrap step will just
continue looping, doing nothing. This can happen if the main stage
has a dependency that triggers the bootstrap stage, but the bootstrap
stage doesn't see anything required to rebuild the main ninja file. A
side effect of this requirement is that changes to test code will now
rebuild the main ninja file.
Change-Id: I9965cfba79dc0dbbd3af05f5944f7653054455a2
