2015-01-23 23:15:10 +01:00
|
|
|
// Copyright 2014 Google Inc. All rights reserved.
|
|
|
|
//
|
|
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
|
|
// you may not use this file except in compliance with the License.
|
|
|
|
// You may obtain a copy of the License at
|
|
|
|
//
|
|
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
//
|
|
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
// See the License for the specific language governing permissions and
|
|
|
|
// limitations under the License.
|
|
|
|
|
2017-07-21 03:41:56 +02:00
|
|
|
// The Blueprint bootstrapping mechanism is intended to enable building a
|
|
|
|
// source tree with minimal prebuilts. The only prerequisites for performing
|
|
|
|
// such a build are:
|
2014-06-12 00:51:08 +02:00
|
|
|
//
|
|
|
|
// 1. A Ninja binary
|
|
|
|
// 2. A script interpreter (e.g. Bash or Python)
|
|
|
|
// 3. A Go toolchain
|
|
|
|
//
|
|
|
|
// The Primary Builder
|
|
|
|
//
|
|
|
|
// As part of the bootstrapping process, a binary called the "primary builder"
|
|
|
|
// is created. This primary builder is the binary that includes both the core
|
|
|
|
// Blueprint library and the build logic specific to the source tree. It is
|
|
|
|
// used to generate the Ninja file that describes how to build the entire source
|
|
|
|
// tree.
|
|
|
|
//
|
|
|
|
// The primary builder must be a pure Go (i.e. no cgo) module built with the
|
|
|
|
// module type 'bootstrap_go_binary'. It should have the 'primaryBuilder'
|
|
|
|
// module property set to true in its Blueprints file. If more than one module
|
|
|
|
// sets primaryBuilder to true the build will fail.
|
|
|
|
//
|
|
|
|
// The primary builder main function should look something like:
|
|
|
|
//
|
|
|
|
// package main
|
|
|
|
//
|
|
|
|
// import (
|
|
|
|
// "flag"
|
2015-03-21 03:39:29 +01:00
|
|
|
// "github.com/google/blueprint"
|
|
|
|
// "github.com/google/blueprint/bootstrap"
|
2014-06-12 00:51:08 +02:00
|
|
|
// "path/filepath"
|
|
|
|
//
|
|
|
|
// "my/custom/build/logic"
|
|
|
|
// )
|
|
|
|
//
|
|
|
|
// func main() {
|
|
|
|
// // The primary builder should use the global flag set because the
|
|
|
|
// // bootstrap package registers its own flags there.
|
|
|
|
// flag.Parse()
|
|
|
|
//
|
|
|
|
// // The top-level Blueprints file is passed as the first argument.
|
|
|
|
// srcDir := filepath.Dir(flag.Arg(0))
|
|
|
|
//
|
|
|
|
// // Create the build context.
|
|
|
|
// ctx := blueprint.NewContext()
|
|
|
|
//
|
|
|
|
// // Register custom module types
|
|
|
|
// ctx.RegisterModuleType("foo", logic.FooModule)
|
|
|
|
// ctx.RegisterModuleType("bar", logic.BarModule)
|
|
|
|
//
|
|
|
|
// // Register custom singletons
|
|
|
|
// ctx.RegisterSingleton("baz", logic.NewBazSingleton())
|
|
|
|
//
|
|
|
|
// // Create and initialize the custom Config object.
|
|
|
|
// config := logic.NewConfig(srcDir)
|
|
|
|
//
|
|
|
|
// // This call never returns
|
|
|
|
// bootstrap.Main(ctx, config)
|
|
|
|
// }
|
|
|
|
//
|
|
|
|
// Required Source Files
|
|
|
|
//
|
|
|
|
// There are three files that must be included in the source tree to facilitate
|
|
|
|
// the build bootstrapping:
|
|
|
|
//
|
|
|
|
// 1. The top-level Blueprints file
|
2017-07-21 03:41:56 +02:00
|
|
|
// 2. The bootstrap script
|
|
|
|
// 3. The build wrapper script
|
2014-06-12 00:51:08 +02:00
|
|
|
//
|
|
|
|
// The top-level Blueprints file describes how the entire source tree should be
|
|
|
|
// built. It must have a 'subdirs' assignment that includes both the core
|
|
|
|
// Blueprint library and the custom build logic for the source tree. It should
|
|
|
|
// also include (either directly or through a subdirs entry) describe all the
|
|
|
|
// modules to be built in the source tree.
|
|
|
|
//
|
2017-07-21 03:41:56 +02:00
|
|
|
// The bootstrap script is a small script to setup the build directory, writing
|
|
|
|
// a couple configuration files (including the path the source directory,
|
|
|
|
// information about the Go build environment, etc), then copying the build
|
|
|
|
// wrapper into the build directory.
|
2014-06-12 00:51:08 +02:00
|
|
|
//
|
|
|
|
// The Bootstrapping Process
|
|
|
|
//
|
Simplify bootstrap
tl;dr: Read if you don't use the wrapper or use SKIP_NINJA
Previously, we were relying on the ninja behavior of restarting the
build when the build.ninja file was updated to switch between different
bootstrap stages. But that means that every step that could produce a
build.ninja must pass in order to switch to a different stage. That
wasn't a big problem when we had a two stage build -- there was very
little that could fail in the second stage before we chose to go back to
the first stage. But when we had a three stage build, it was possible to
get into a state (usually during development) where you were in the
second stage, but the build was failing because the first stage needed
to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82
by adding a wrapper that always started building at the first stage.
But this kept all of the complexity of using ninja restarts without any
of the benefits, so this change removes that complexity and just runs
each stage sequentially in the wrapper. So the wrapper is now required.
Since we're no longer going through choosestage, we can also skip the
template parsing for the later stages that don't need to be templated --
this can save a couple of seconds for large files.
In addition to all of the above, this also lets Soong reduce the number
of times the main ninja file is loaded. We had been running the wrapper
once (3 stages), then running ninja again after combining the
Soong-generated build.ninja with the Kati-generated build.ninja. This
change lets us removing the intermediate parsing of Soong's build.ninja,
so that we only execute ninja 3 times per build. It also lets us have
dependencies on pools or rules from Kati in the primary builder, since
we're never executing the main build.ninja without the Kati build.ninja.
The wrapper has a new option, NINJA to provide the path to ninja. This
used to be hardcoded to `ninja`, and will still default to that. But
we'll be running the first two bootstrap stages with $NINJA even if
SKIP_NINJA is set.
The wrapper passes "-w dupbuild=err" to ninja now -- this really should
always be turned on if you care about reliable builds.
Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
|
|
|
// There are three stages to the bootstrapping process, each with a
|
|
|
|
// corresponding Ninja file. The stages are referred to as the "bootstrap",
|
|
|
|
// "primary", and "main" stages. Each stage builds the next stage's Ninja file.
|
2014-06-12 00:51:08 +02:00
|
|
|
//
|
|
|
|
// The bootstrapping process begins with the user running the bootstrap script
|
|
|
|
// to initialize a new build directory. The script is run from the build
|
2017-07-21 03:41:56 +02:00
|
|
|
// directory, and creates a ".minibootstrap/build.ninja" file that sets a few
|
|
|
|
// variables then includes blueprint's "bootstrap/build.ninja". It also writes
|
|
|
|
// out a ".blueprint.bootstrap" file that contains a few variables for later use:
|
|
|
|
//
|
|
|
|
// BLUEPRINT_BOOTSTRAP_VERSION - Used to detect when a user needs to run
|
|
|
|
// bootstrap.bash again
|
|
|
|
//
|
|
|
|
// SRCDIR - The path to the source directory
|
|
|
|
// BLUEPRINTDIR - The path to the blueprints directory (includes $SRCDIR)
|
|
|
|
// GOROOT - The path to the root directory of the Go toolchain
|
|
|
|
// NINJA_BUILDDIR - The path to store .ninja_log, .ninja_deps
|
2014-06-12 00:51:08 +02:00
|
|
|
//
|
Simplify bootstrap
tl;dr: Read if you don't use the wrapper or use SKIP_NINJA
Previously, we were relying on the ninja behavior of restarting the
build when the build.ninja file was updated to switch between different
bootstrap stages. But that means that every step that could produce a
build.ninja must pass in order to switch to a different stage. That
wasn't a big problem when we had a two stage build -- there was very
little that could fail in the second stage before we chose to go back to
the first stage. But when we had a three stage build, it was possible to
get into a state (usually during development) where you were in the
second stage, but the build was failing because the first stage needed
to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82
by adding a wrapper that always started building at the first stage.
But this kept all of the complexity of using ninja restarts without any
of the benefits, so this change removes that complexity and just runs
each stage sequentially in the wrapper. So the wrapper is now required.
Since we're no longer going through choosestage, we can also skip the
template parsing for the later stages that don't need to be templated --
this can save a couple of seconds for large files.
In addition to all of the above, this also lets Soong reduce the number
of times the main ninja file is loaded. We had been running the wrapper
once (3 stages), then running ninja again after combining the
Soong-generated build.ninja with the Kati-generated build.ninja. This
change lets us removing the intermediate parsing of Soong's build.ninja,
so that we only execute ninja 3 times per build. It also lets us have
dependencies on pools or rules from Kati in the primary builder, since
we're never executing the main build.ninja without the Kati build.ninja.
The wrapper has a new option, NINJA to provide the path to ninja. This
used to be hardcoded to `ninja`, and will still default to that. But
we'll be running the first two bootstrap stages with $NINJA even if
SKIP_NINJA is set.
The wrapper passes "-w dupbuild=err" to ninja now -- this really should
always be turned on if you care about reliable builds.
Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
|
|
|
// Once the script completes the build directory is initialized and ready to run
|
|
|
|
// a build. A wrapper script (blueprint.bash by default) has been installed in
|
|
|
|
// order to run a build. It iterates through the three stages of the build:
|
|
|
|
//
|
2017-07-21 03:41:56 +02:00
|
|
|
// - Runs microfactory.bash to build minibp
|
|
|
|
// - Runs the .minibootstrap/build.ninja to build .bootstrap/build.ninja
|
|
|
|
// - Runs .bootstrap/build.ninja to build and run the primary builder
|
|
|
|
// - Runs build.ninja to build your code
|
|
|
|
//
|
Run globs during earlier bootstrap phases
Instead of sometimes re-running minibp/the primary builder during the
next phase, run bpglob earlier to check dependencies.
We've run into issues where the environment is slightly different
between bootstrapping phase and the main build phase. It's also a
problem because our primary builder (Soong) exports information used by
another tool (Kati) that runs in between the bootstrapping phases and
the main phase. When Soong would run in the main phase, it could get out
of sync, and would require the build to be run again.
To do this, add a "subninja" include a build-globs.ninja file to each
build.ninja file. The first time, this will be an empty file, but we'll
always run minibp / the primary builder anyway. When the builder runs,
in addition to writing a dependency file, write out the
build-globs.ninja file with the rules to run bpglob.
Since bpglob may need to be run very early, before it would normally be
built, build it with microfactory.
Change-Id: I89fcd849a8729e892f163d40060ab90b5d4dfa5d
2018-07-06 06:56:59 +02:00
|
|
|
// Microfactory takes care of building an up to date version of `minibp` and
|
|
|
|
// `bpglob` under the .minibootstrap/ directory.
|
2017-07-21 03:41:56 +02:00
|
|
|
//
|
|
|
|
// During <builddir>/.minibootstrap/build.ninja, the following actions are
|
|
|
|
// taken, if necessary:
|
|
|
|
//
|
Simplify bootstrap
tl;dr: Read if you don't use the wrapper or use SKIP_NINJA
Previously, we were relying on the ninja behavior of restarting the
build when the build.ninja file was updated to switch between different
bootstrap stages. But that means that every step that could produce a
build.ninja must pass in order to switch to a different stage. That
wasn't a big problem when we had a two stage build -- there was very
little that could fail in the second stage before we chose to go back to
the first stage. But when we had a three stage build, it was possible to
get into a state (usually during development) where you were in the
second stage, but the build was failing because the first stage needed
to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82
by adding a wrapper that always started building at the first stage.
But this kept all of the complexity of using ninja restarts without any
of the benefits, so this change removes that complexity and just runs
each stage sequentially in the wrapper. So the wrapper is now required.
Since we're no longer going through choosestage, we can also skip the
template parsing for the later stages that don't need to be templated --
this can save a couple of seconds for large files.
In addition to all of the above, this also lets Soong reduce the number
of times the main ninja file is loaded. We had been running the wrapper
once (3 stages), then running ninja again after combining the
Soong-generated build.ninja with the Kati-generated build.ninja. This
change lets us removing the intermediate parsing of Soong's build.ninja,
so that we only execute ninja 3 times per build. It also lets us have
dependencies on pools or rules from Kati in the primary builder, since
we're never executing the main build.ninja without the Kati build.ninja.
The wrapper has a new option, NINJA to provide the path to ninja. This
used to be hardcoded to `ninja`, and will still default to that. But
we'll be running the first two bootstrap stages with $NINJA even if
SKIP_NINJA is set.
The wrapper passes "-w dupbuild=err" to ninja now -- this really should
always be turned on if you care about reliable builds.
Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
|
|
|
// - Run minibp to generate .bootstrap/build.ninja (Primary stage)
|
Run globs during earlier bootstrap phases
Instead of sometimes re-running minibp/the primary builder during the
next phase, run bpglob earlier to check dependencies.
We've run into issues where the environment is slightly different
between bootstrapping phase and the main build phase. It's also a
problem because our primary builder (Soong) exports information used by
another tool (Kati) that runs in between the bootstrapping phases and
the main phase. When Soong would run in the main phase, it could get out
of sync, and would require the build to be run again.
To do this, add a "subninja" include a build-globs.ninja file to each
build.ninja file. The first time, this will be an empty file, but we'll
always run minibp / the primary builder anyway. When the builder runs,
in addition to writing a dependency file, write out the
build-globs.ninja file with the rules to run bpglob.
Since bpglob may need to be run very early, before it would normally be
built, build it with microfactory.
Change-Id: I89fcd849a8729e892f163d40060ab90b5d4dfa5d
2018-07-06 06:56:59 +02:00
|
|
|
// - Includes .minibootstrap/build-globs.ninja, which defines rules to
|
|
|
|
// run bpglob during incremental builds. These outputs are listed in
|
|
|
|
// the dependency file output by minibp.
|
Simplify bootstrap
tl;dr: Read if you don't use the wrapper or use SKIP_NINJA
Previously, we were relying on the ninja behavior of restarting the
build when the build.ninja file was updated to switch between different
bootstrap stages. But that means that every step that could produce a
build.ninja must pass in order to switch to a different stage. That
wasn't a big problem when we had a two stage build -- there was very
little that could fail in the second stage before we chose to go back to
the first stage. But when we had a three stage build, it was possible to
get into a state (usually during development) where you were in the
second stage, but the build was failing because the first stage needed
to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82
by adding a wrapper that always started building at the first stage.
But this kept all of the complexity of using ninja restarts without any
of the benefits, so this change removes that complexity and just runs
each stage sequentially in the wrapper. So the wrapper is now required.
Since we're no longer going through choosestage, we can also skip the
template parsing for the later stages that don't need to be templated --
this can save a couple of seconds for large files.
In addition to all of the above, this also lets Soong reduce the number
of times the main ninja file is loaded. We had been running the wrapper
once (3 stages), then running ninja again after combining the
Soong-generated build.ninja with the Kati-generated build.ninja. This
change lets us removing the intermediate parsing of Soong's build.ninja,
so that we only execute ninja 3 times per build. It also lets us have
dependencies on pools or rules from Kati in the primary builder, since
we're never executing the main build.ninja without the Kati build.ninja.
The wrapper has a new option, NINJA to provide the path to ninja. This
used to be hardcoded to `ninja`, and will still default to that. But
we'll be running the first two bootstrap stages with $NINJA even if
SKIP_NINJA is set.
The wrapper passes "-w dupbuild=err" to ninja now -- this really should
always be turned on if you care about reliable builds.
Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
|
|
|
//
|
2017-07-21 03:41:56 +02:00
|
|
|
// During the <builddir>/.bootstrap/build.ninja, the following actions are
|
|
|
|
// taken, if necessary:
|
Simplify bootstrap
tl;dr: Read if you don't use the wrapper or use SKIP_NINJA
Previously, we were relying on the ninja behavior of restarting the
build when the build.ninja file was updated to switch between different
bootstrap stages. But that means that every step that could produce a
build.ninja must pass in order to switch to a different stage. That
wasn't a big problem when we had a two stage build -- there was very
little that could fail in the second stage before we chose to go back to
the first stage. But when we had a three stage build, it was possible to
get into a state (usually during development) where you were in the
second stage, but the build was failing because the first stage needed
to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82
by adding a wrapper that always started building at the first stage.
But this kept all of the complexity of using ninja restarts without any
of the benefits, so this change removes that complexity and just runs
each stage sequentially in the wrapper. So the wrapper is now required.
Since we're no longer going through choosestage, we can also skip the
template parsing for the later stages that don't need to be templated --
this can save a couple of seconds for large files.
In addition to all of the above, this also lets Soong reduce the number
of times the main ninja file is loaded. We had been running the wrapper
once (3 stages), then running ninja again after combining the
Soong-generated build.ninja with the Kati-generated build.ninja. This
change lets us removing the intermediate parsing of Soong's build.ninja,
so that we only execute ninja 3 times per build. It also lets us have
dependencies on pools or rules from Kati in the primary builder, since
we're never executing the main build.ninja without the Kati build.ninja.
The wrapper has a new option, NINJA to provide the path to ninja. This
used to be hardcoded to `ninja`, and will still default to that. But
we'll be running the first two bootstrap stages with $NINJA even if
SKIP_NINJA is set.
The wrapper passes "-w dupbuild=err" to ninja now -- this really should
always be turned on if you care about reliable builds.
Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
|
|
|
//
|
2017-07-21 03:41:56 +02:00
|
|
|
// - Build the primary builder, anything marked `default: true`, and
|
|
|
|
// any dependencies.
|
Simplify bootstrap
tl;dr: Read if you don't use the wrapper or use SKIP_NINJA
Previously, we were relying on the ninja behavior of restarting the
build when the build.ninja file was updated to switch between different
bootstrap stages. But that means that every step that could produce a
build.ninja must pass in order to switch to a different stage. That
wasn't a big problem when we had a two stage build -- there was very
little that could fail in the second stage before we chose to go back to
the first stage. But when we had a three stage build, it was possible to
get into a state (usually during development) where you were in the
second stage, but the build was failing because the first stage needed
to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82
by adding a wrapper that always started building at the first stage.
But this kept all of the complexity of using ninja restarts without any
of the benefits, so this change removes that complexity and just runs
each stage sequentially in the wrapper. So the wrapper is now required.
Since we're no longer going through choosestage, we can also skip the
template parsing for the later stages that don't need to be templated --
this can save a couple of seconds for large files.
In addition to all of the above, this also lets Soong reduce the number
of times the main ninja file is loaded. We had been running the wrapper
once (3 stages), then running ninja again after combining the
Soong-generated build.ninja with the Kati-generated build.ninja. This
change lets us removing the intermediate parsing of Soong's build.ninja,
so that we only execute ninja 3 times per build. It also lets us have
dependencies on pools or rules from Kati in the primary builder, since
we're never executing the main build.ninja without the Kati build.ninja.
The wrapper has a new option, NINJA to provide the path to ninja. This
used to be hardcoded to `ninja`, and will still default to that. But
we'll be running the first two bootstrap stages with $NINJA even if
SKIP_NINJA is set.
The wrapper passes "-w dupbuild=err" to ninja now -- this really should
always be turned on if you care about reliable builds.
Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
|
|
|
// - Run the primary builder to generate build.ninja
|
|
|
|
// - Run the primary builder to extract documentation
|
Run globs during earlier bootstrap phases
Instead of sometimes re-running minibp/the primary builder during the
next phase, run bpglob earlier to check dependencies.
We've run into issues where the environment is slightly different
between bootstrapping phase and the main build phase. It's also a
problem because our primary builder (Soong) exports information used by
another tool (Kati) that runs in between the bootstrapping phases and
the main phase. When Soong would run in the main phase, it could get out
of sync, and would require the build to be run again.
To do this, add a "subninja" include a build-globs.ninja file to each
build.ninja file. The first time, this will be an empty file, but we'll
always run minibp / the primary builder anyway. When the builder runs,
in addition to writing a dependency file, write out the
build-globs.ninja file with the rules to run bpglob.
Since bpglob may need to be run very early, before it would normally be
built, build it with microfactory.
Change-Id: I89fcd849a8729e892f163d40060ab90b5d4dfa5d
2018-07-06 06:56:59 +02:00
|
|
|
// - Includes .bootstrap/build-globs.ninja, which defines rules to run
|
|
|
|
// bpglob during incremental builds. These outputs are listed in the
|
|
|
|
// dependency file output by the primary builder.
|
Simplify bootstrap
tl;dr: Read if you don't use the wrapper or use SKIP_NINJA
Previously, we were relying on the ninja behavior of restarting the
build when the build.ninja file was updated to switch between different
bootstrap stages. But that means that every step that could produce a
build.ninja must pass in order to switch to a different stage. That
wasn't a big problem when we had a two stage build -- there was very
little that could fail in the second stage before we chose to go back to
the first stage. But when we had a three stage build, it was possible to
get into a state (usually during development) where you were in the
second stage, but the build was failing because the first stage needed
to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82
by adding a wrapper that always started building at the first stage.
But this kept all of the complexity of using ninja restarts without any
of the benefits, so this change removes that complexity and just runs
each stage sequentially in the wrapper. So the wrapper is now required.
Since we're no longer going through choosestage, we can also skip the
template parsing for the later stages that don't need to be templated --
this can save a couple of seconds for large files.
In addition to all of the above, this also lets Soong reduce the number
of times the main ninja file is loaded. We had been running the wrapper
once (3 stages), then running ninja again after combining the
Soong-generated build.ninja with the Kati-generated build.ninja. This
change lets us removing the intermediate parsing of Soong's build.ninja,
so that we only execute ninja 3 times per build. It also lets us have
dependencies on pools or rules from Kati in the primary builder, since
we're never executing the main build.ninja without the Kati build.ninja.
The wrapper has a new option, NINJA to provide the path to ninja. This
used to be hardcoded to `ninja`, and will still default to that. But
we'll be running the first two bootstrap stages with $NINJA even if
SKIP_NINJA is set.
The wrapper passes "-w dupbuild=err" to ninja now -- this really should
always be turned on if you care about reliable builds.
Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
|
|
|
//
|
|
|
|
// Then the main stage is at <builddir>/build.ninja, and will contain all the
|
|
|
|
// rules generated by the primary builder. In addition, the bootstrap code
|
|
|
|
// adds a phony rule "blueprint_tools" that depends on all blueprint_go_binary
|
|
|
|
// rules (bpfmt, bpmodify, etc).
|
2014-06-12 00:51:08 +02:00
|
|
|
//
|
2014-05-28 01:34:41 +02:00
|
|
|
package bootstrap
|