In cases that we want to run blueprint-based builds in many
configurations to verify all of the logic works without errors, but
don't care about running the final ninja file, writing it out only
wastes time and disk space. So add a --empty-ninja-file option that
writes out an empty ninja file instead.
Our specific use case (Soong's build_test / multiproduct_kati) runs
Soong several hundred times for different configurations, and the ninja
files are around 1GB, which leads to several hundred gigabytes of disk
writes (and persistent use during incremental generation).
Change-Id: I0198dfb2f744ce22284c05d5214dac2ab5dc9700
Instead of sometimes re-running minibp/the primary builder during the
next phase, run bpglob earlier to check dependencies.
We've run into issues where the environment is slightly different
between bootstrapping phase and the main build phase. It's also a
problem because our primary builder (Soong) exports information used by
another tool (Kati) that runs in between the bootstrapping phases and
the main phase. When Soong would run in the main phase, it could get out
of sync, and would require the build to be run again.
To do this, add a "subninja" include a build-globs.ninja file to each
build.ninja file. The first time, this will be an empty file, but we'll
always run minibp / the primary builder anyway. When the builder runs,
in addition to writing a dependency file, write out the
build-globs.ninja file with the rules to run bpglob.
Since bpglob may need to be run very early, before it would normally be
built, build it with microfactory.
Change-Id: I89fcd849a8729e892f163d40060ab90b5d4dfa5d
Having separate .ninja_log files doesn't work very well after
04d886b110
since the last successful timestamp is not recorded in the ninja log.
This was triggering the primary builder to be run twice on every build
since it was always considered out of date in the primary, then main
stages.
The two bootstrap stages already were unified when minibootstrap was
simplified, but this combines the main stage as well. This means that
we'll save some time running globs that are shared between minibp and
the primary builder as well (subdirs globs).
The cleanup code needed to be refactored, since the ninja_log will have
entries from the main stage as well. So instead of looking at everything
in the ninja log, take a string prefix to limit what we may clean.
Change-Id: I8d43168643aa8a0c6c3e48c6101eaa45d174bbe5
This duplicates building common blueprint go packages between minibp and
the primary builder, but drastically simplifies the first stage,
removing the need to check in a generated build.ninja.in.
Change-Id: I639a9637f1ed36d4210823ef276c0f7a064a83bd
tl;dr: Read if you don't use the wrapper or use SKIP_NINJA
Previously, we were relying on the ninja behavior of restarting the
build when the build.ninja file was updated to switch between different
bootstrap stages. But that means that every step that could produce a
build.ninja must pass in order to switch to a different stage. That
wasn't a big problem when we had a two stage build -- there was very
little that could fail in the second stage before we chose to go back to
the first stage. But when we had a three stage build, it was possible to
get into a state (usually during development) where you were in the
second stage, but the build was failing because the first stage needed
to be run. This was fixed in d79f1af742
by adding a wrapper that always started building at the first stage.
But this kept all of the complexity of using ninja restarts without any
of the benefits, so this change removes that complexity and just runs
each stage sequentially in the wrapper. So the wrapper is now required.
Since we're no longer going through choosestage, we can also skip the
template parsing for the later stages that don't need to be templated --
this can save a couple of seconds for large files.
In addition to all of the above, this also lets Soong reduce the number
of times the main ninja file is loaded. We had been running the wrapper
once (3 stages), then running ninja again after combining the
Soong-generated build.ninja with the Kati-generated build.ninja. This
change lets us removing the intermediate parsing of Soong's build.ninja,
so that we only execute ninja 3 times per build. It also lets us have
dependencies on pools or rules from Kati in the primary builder, since
we're never executing the main build.ninja without the Kati build.ninja.
The wrapper has a new option, NINJA to provide the path to ninja. This
used to be hardcoded to `ninja`, and will still default to that. But
we'll be running the first two bootstrap stages with $NINJA even if
SKIP_NINJA is set.
The wrapper passes "-w dupbuild=err" to ninja now -- this really should
always be turned on if you care about reliable builds.
Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
This wrapper script can be used instead of ninja to ensure the build
won't get stuck building the primary builder.
An example of when this would happen:
1. Do a successful build
2. Sync/make a change in the primary builder (soong, etc) that depends
on a blueprint change.
3. The next build would notice that change, and rewind to the primary
stage to rebuild the builder. That build would fail.
4. Sync/fix the blueprint code.
5. The next build would still fail. The bootstrap stage would need to
be run in order to fix the primary stage, but we're still stuck in the
primary stage. The only way to switch stages is to successfully
complete everything required to choose the next stage.
This generally isn't a problem in the main stage, since there is no code
being built in the dependency chain leading up to stage selection.
Any existing wrappers (like soong) can execute this wrapper (optionally
skipping ninja execution) if they're worried about the above situation.
This isn't strictly required -- running ninja directly will still work
in most cases, you'll just need to re-run bootstrap.bash if you get into
a bad state.
Change-Id: I5901d7240a1daa388a786ceb1c8259502fc14058
The go compiler and linker changed in v1.5 -- to 'go tool compile' and
'go tool link' instead of 6g and 6l. Move the selection logic to
bootstrap.bash, and have it use compile/link if 6g/6l are missing. This
way the build.ninja.in will continue working with either go 1.4 or 1.5.
Travis and the test suite will fail under 1.5, since the build.ninja.in
is still generated with 1.4, and the function names in the comments
differ between 1.4 and 1.5.
To provide a consistent __FILE__ behavior with cpp, we want to be able
to run with SRCDIR="." and the outputs be saved elsewhere. Other tools
within android also expect to be run from $TOP.
Change-Id: I572bce5c9086b0c3310b42065ae98cbf5a1c6399
Users that want to enable this option can use the '-t' option to
bootstrap.bash when passing '-r'. Builders that want to enable this can
set the RUN_TESTS environment variable in their bootstrap.bash.
The gotestmain tools is needed to write the main functions for the test
binaries, since 'go test' doesn't work well in this environment.
Change-Id: Iec5c2b5c9c3f5e3ba0ac8677fb88f5e963f9bd3f
This change makes the bootstrap script take two optional flag-based command
line arguments rather than using positional arguments. The -i argument
replaces the old positional arguments (one of which was simply not needed), and
the -r flag now causes the script to regenerate the bootstrap Ninja file for
use when working on Blueprint itself.
Additionally this change allows all the configuration variables used in the
script to be overridden by environment variables. This makes it possible to
create a very simple bootstrap script for a customized Blueprint-based build
system by invoking this script after setting up the proper environment
variables.
Change-Id: I60f890061ae237a68719e1e2d345d1a8e87c851c
Blueprint is a build system component that reads Blueprints files defining
modules to be built, and generates a Ninja build manifest that can be used to
perform all the build actions. It does not dictate or implement much build
policy itself, but rather provides a framework to ease the process of defining
build logic in Go.
The "blueprint" and "blueprint/parser" Go packages contain the functionality
for reading Blueprint files and invoking build logic functions defined in other
Go packages.
The "blueprint/bootstrap" Go package contains just enough build logic to build
a binary that includes Blueprint and any pure-Go (i.e. no cgo) build logic
defined in external Go packages. This can be used to create a minimal Ninja
file that's capable of bootstrapping a Blueprint-based build system from
source.
The "blueprint/bootstrap/minibp" Go package contains code for a minimal binary
that includes the build logic defined in the "blueprint/bootstrap" package.
This binary can then create the Ninja file for the bootstrapping process.
Change-Id: I8d8390042372a72d225785cda738525001b009f1
