Simplify bootstrap
tl;dr: Read if you don't use the wrapper or use SKIP_NINJA
Previously, we were relying on the ninja behavior of restarting the
build when the build.ninja file was updated to switch between different
bootstrap stages. But that means that every step that could produce a
build.ninja must pass in order to switch to a different stage. That
wasn't a big problem when we had a two stage build -- there was very
little that could fail in the second stage before we chose to go back to
the first stage. But when we had a three stage build, it was possible to
get into a state (usually during development) where you were in the
second stage, but the build was failing because the first stage needed
to be run. This was fixed in d79f1af742
by adding a wrapper that always started building at the first stage.
But this kept all of the complexity of using ninja restarts without any
of the benefits, so this change removes that complexity and just runs
each stage sequentially in the wrapper. So the wrapper is now required.
Since we're no longer going through choosestage, we can also skip the
template parsing for the later stages that don't need to be templated --
this can save a couple of seconds for large files.
In addition to all of the above, this also lets Soong reduce the number
of times the main ninja file is loaded. We had been running the wrapper
once (3 stages), then running ninja again after combining the
Soong-generated build.ninja with the Kati-generated build.ninja. This
change lets us removing the intermediate parsing of Soong's build.ninja,
so that we only execute ninja 3 times per build. It also lets us have
dependencies on pools or rules from Kati in the primary builder, since
we're never executing the main build.ninja without the Kati build.ninja.
The wrapper has a new option, NINJA to provide the path to ninja. This
used to be hardcoded to `ninja`, and will still default to that. But
we'll be running the first two bootstrap stages with $NINJA even if
SKIP_NINJA is set.
The wrapper passes "-w dupbuild=err" to ninja now -- this really should
always be turned on if you care about reliable builds.
Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
This commit is contained in:
Dan Willemsen
parent
3fc513f94c
commit
7d0dddd84d
26 changed files with 212 additions and 771 deletions
|
|
@ -21,5 +21,5 @@ script:
|
|||
- cd stage
|
||||
- ../bootstrap.bash
|
||||
- ./blueprint.bash
|
||||
- diff -us ../build.ninja.in .bootstrap/bootstrap.ninja.in
|
||||
- diff -us ../build.ninja.in .minibootstrap/build.ninja.in
|
||||
- ../tests/test.sh
|
||||
|
|
|
|||
|
|
@ -142,11 +142,6 @@ bootstrap_core_go_binary(
|
|||
srcs = ["gotestrunner/gotestrunner.go"],
|
||||
)
|
||||
|
||||
bootstrap_core_go_binary(
|
||||
name = "choosestage",
|
||||
srcs = ["choosestage/choosestage.go"],
|
||||
)
|
||||
|
||||
bootstrap_go_binary{
|
||||
name = "loadplugins",
|
||||
srcs = ["loadplugins/loadplugins.go"],
|
||||
|
|
|
|||
|
|
@ -26,36 +26,41 @@ set -e
|
|||
# if the custom build system only wants to install their own wrapper.
|
||||
[ -z "$BUILDDIR" ] && BUILDDIR=`dirname "${BASH_SOURCE[0]}"`
|
||||
|
||||
# NINJA should be set to the path of the ninja executable. By default, this
|
||||
# is just "ninja", and will be looked up in $PATH.
|
||||
[ -z "$NINJA" ] && NINJA=ninja
|
||||
|
||||
|
||||
if [ ! -f "${BUILDDIR}/.blueprint.bootstrap" ]; then
|
||||
echo "Please run bootstrap.bash (.blueprint.bootstrap missing)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# .blueprint.bootstrap provides saved values from the bootstrap.bash script:
|
||||
#
|
||||
# BOOTSTRAP
|
||||
# BOOTSTRAP_MANIFEST
|
||||
#
|
||||
# If it doesn't exist, we probably just need to re-run bootstrap.bash, which
|
||||
# ninja will do when switching stages. So just skip to ninja.
|
||||
if [ -f "${BUILDDIR}/.blueprint.bootstrap" ]; then
|
||||
source "${BUILDDIR}/.blueprint.bootstrap"
|
||||
source "${BUILDDIR}/.blueprint.bootstrap"
|
||||
|
||||
# Pick the newer of .bootstrap/bootstrap.ninja.in or $BOOTSTRAP_MANIFEST,
|
||||
# and copy it to .bootstrap/build.ninja.in
|
||||
GEN_BOOTSTRAP_MANIFEST="${BUILDDIR}/.bootstrap/bootstrap.ninja.in"
|
||||
if [ -f "${GEN_BOOTSTRAP_MANIFEST}" ]; then
|
||||
if [ "${GEN_BOOTSTRAP_MANIFEST}" -nt "${BOOTSTRAP_MANIFEST}" ]; then
|
||||
BOOTSTRAP_MANIFEST="${GEN_BOOTSTRAP_MANIFEST}"
|
||||
GEN_BOOTSTRAP_MANIFEST="${BUILDDIR}/.minibootstrap/build.ninja.in"
|
||||
if [ -f "${GEN_BOOTSTRAP_MANIFEST}" ]; then
|
||||
if [ "${BOOTSTRAP_MANIFEST}" -nt "${GEN_BOOTSTRAP_MANIFEST}" ]; then
|
||||
"${BOOTSTRAP}" -i "${BOOTSTRAP_MANIFEST}"
|
||||
fi
|
||||
fi
|
||||
|
||||
# Copy the selected manifest to $BUILDDIR/.bootstrap/build.ninja.in
|
||||
mkdir -p "${BUILDDIR}/.bootstrap"
|
||||
cp "${BOOTSTRAP_MANIFEST}" "${BUILDDIR}/.bootstrap/build.ninja.in"
|
||||
|
||||
# Bootstrap it to $BUILDDIR/build.ninja
|
||||
"${BOOTSTRAP}" -i "${BUILDDIR}/.bootstrap/build.ninja.in"
|
||||
else
|
||||
"${BOOTSTRAP}" -i "${BOOTSTRAP_MANIFEST}"
|
||||
fi
|
||||
|
||||
# Build minibp and the primary build.ninja
|
||||
"${NINJA}" -w dupbuild=err -f "${BUILDDIR}/.minibootstrap/build.ninja" "${BUILDDIR}/.bootstrap/build.ninja"
|
||||
|
||||
# Build the primary builder and the main build.ninja
|
||||
"${NINJA}" -w dupbuild=err -f "${BUILDDIR}/.bootstrap/build.ninja" "${BUILDDIR}/build.ninja"
|
||||
|
||||
# SKIP_NINJA can be used by wrappers that wish to run ninja themselves.
|
||||
if [ -z "$SKIP_NINJA" ]; then
|
||||
ninja -C "${BUILDDIR}" "$@"
|
||||
"${NINJA}" -w dupbuild=err -f "${BUILDDIR}/build.ninja" "$@"
|
||||
else
|
||||
exit 0
|
||||
fi
|
||||
|
|
|
|||
|
|
@ -126,7 +126,7 @@ if [ $REGEN_BOOTSTRAP_MANIFEST = true ]; then
|
|||
fi
|
||||
fi
|
||||
|
||||
mkdir -p $BUILDDIR
|
||||
mkdir -p $BUILDDIR/.minibootstrap
|
||||
|
||||
sed -e "s|@@SrcDir@@|$SRCDIR|g" \
|
||||
-e "s|@@BuildDir@@|$BUILDDIR|g" \
|
||||
|
|
@ -135,7 +135,7 @@ sed -e "s|@@SrcDir@@|$SRCDIR|g" \
|
|||
-e "s|@@GoLink@@|$GOLINK|g" \
|
||||
-e "s|@@Bootstrap@@|$BOOTSTRAP|g" \
|
||||
-e "s|@@BootstrapManifest@@|$BOOTSTRAP_MANIFEST|g" \
|
||||
$IN > $BUILDDIR/build.ninja
|
||||
$IN > $BUILDDIR/.minibootstrap/build.ninja
|
||||
|
||||
echo "BOOTSTRAP=\"${BOOTSTRAP}\"" > $BUILDDIR/.blueprint.bootstrap
|
||||
echo "BOOTSTRAP_MANIFEST=\"${BOOTSTRAP_MANIFEST}\"" >> $BUILDDIR/.blueprint.bootstrap
|
||||
|
|
|
|||
|
|
@ -31,7 +31,6 @@ var (
|
|||
|
||||
goTestMainCmd = pctx.StaticVariable("goTestMainCmd", filepath.Join(bootstrapDir, "bin", "gotestmain"))
|
||||
goTestRunnerCmd = pctx.StaticVariable("goTestRunnerCmd", filepath.Join(bootstrapDir, "bin", "gotestrunner"))
|
||||
chooseStageCmd = pctx.StaticVariable("chooseStageCmd", filepath.Join(bootstrapDir, "bin", "choosestage"))
|
||||
pluginGenSrcCmd = pctx.StaticVariable("pluginGenSrcCmd", filepath.Join(bootstrapDir, "bin", "loadplugins"))
|
||||
|
||||
compile = pctx.StaticRule("compile",
|
||||
|
|
@ -90,14 +89,6 @@ var (
|
|||
Generator: true,
|
||||
})
|
||||
|
||||
chooseStage = pctx.StaticRule("chooseStage",
|
||||
blueprint.RuleParams{
|
||||
Command: "$chooseStageCmd --current $current --bootstrap $bootstrapManifest -o $out $in",
|
||||
CommandDeps: []string{"$chooseStageCmd", "$bootstrapManifest"},
|
||||
Description: "choosing next stage",
|
||||
},
|
||||
"current", "generator")
|
||||
|
||||
touch = pctx.StaticRule("touch",
|
||||
blueprint.RuleParams{
|
||||
Command: "touch $out",
|
||||
|
|
@ -318,11 +309,6 @@ func (g *goPackage) GenerateBuildActions(ctx blueprint.ModuleContext) {
|
|||
|
||||
buildGoPackage(ctx, g.pkgRoot, g.properties.PkgPath, g.archiveFile,
|
||||
g.properties.Srcs, genSrcs, deps)
|
||||
} else if g.config.stage > g.BuildStage() {
|
||||
if len(g.properties.TestSrcs) > 0 && g.config.runGoTests {
|
||||
phonyGoTarget(ctx, g.testArchiveFile, g.properties.TestSrcs, nil, nil)
|
||||
}
|
||||
phonyGoTarget(ctx, g.archiveFile, g.properties.Srcs, genSrcs, nil)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -396,11 +382,6 @@ func (g *goBinary) GenerateBuildActions(ctx blueprint.ModuleContext) {
|
|||
genSrcs = append(genSrcs, pluginSrc)
|
||||
}
|
||||
|
||||
// We only actually want to build the builder modules if we're running as
|
||||
// minibp (i.e. we're generating a bootstrap Ninja file). This is to break
|
||||
// the circular dependence that occurs when the builder requires a new Ninja
|
||||
// file to be built, but building a new ninja file requires the builder to
|
||||
// be built.
|
||||
if g.config.stage == g.BuildStage() {
|
||||
var deps []string
|
||||
|
||||
|
|
@ -440,13 +421,6 @@ func (g *goBinary) GenerateBuildActions(ctx blueprint.ModuleContext) {
|
|||
Outputs: []string{binaryFile},
|
||||
Inputs: []string{aoutFile},
|
||||
})
|
||||
} else if g.config.stage > g.BuildStage() {
|
||||
if len(g.properties.TestSrcs) > 0 && g.config.runGoTests {
|
||||
phonyGoTarget(ctx, g.testArchiveFile, g.properties.TestSrcs, nil, nil)
|
||||
}
|
||||
|
||||
intermediates := []string{aoutFile, archiveFile}
|
||||
phonyGoTarget(ctx, binaryFile, g.properties.Srcs, genSrcs, intermediates)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -582,52 +556,6 @@ func buildGoTest(ctx blueprint.ModuleContext, testRoot, testPkgArchive,
|
|||
return []string{testPassed}
|
||||
}
|
||||
|
||||
func phonyGoTarget(ctx blueprint.ModuleContext, target string, srcs []string,
|
||||
gensrcs []string, intermediates []string) {
|
||||
|
||||
var depTargets []string
|
||||
ctx.VisitDepsDepthFirstIf(isGoPackageProducer,
|
||||
func(module blueprint.Module) {
|
||||
dep := module.(goPackageProducer)
|
||||
target := dep.GoPackageTarget()
|
||||
depTargets = append(depTargets, target)
|
||||
})
|
||||
|
||||
moduleDir := ctx.ModuleDir()
|
||||
srcs = pathtools.PrefixPaths(srcs, filepath.Join("$srcDir", moduleDir))
|
||||
srcs = append(srcs, gensrcs...)
|
||||
|
||||
ctx.Build(pctx, blueprint.BuildParams{
|
||||
Rule: phony,
|
||||
Outputs: []string{target},
|
||||
Inputs: srcs,
|
||||
Implicits: depTargets,
|
||||
})
|
||||
|
||||
// If one of the source files gets deleted or renamed that will prevent the
|
||||
// re-bootstrapping happening because it depends on the missing source file.
|
||||
// To get around this we add a build statement using the built-in phony rule
|
||||
// for each source file, which will cause Ninja to treat it as dirty if its
|
||||
// missing.
|
||||
for _, src := range srcs {
|
||||
ctx.Build(pctx, blueprint.BuildParams{
|
||||
Rule: blueprint.Phony,
|
||||
Outputs: []string{src},
|
||||
})
|
||||
}
|
||||
|
||||
// If there is no rule to build the intermediate files of a bootstrap go package
|
||||
// the cleanup phase of the primary builder will delete the intermediate files,
|
||||
// forcing an unnecessary rebuild. Add phony rules for all of them.
|
||||
for _, intermediate := range intermediates {
|
||||
ctx.Build(pctx, blueprint.BuildParams{
|
||||
Rule: blueprint.Phony,
|
||||
Outputs: []string{intermediate},
|
||||
})
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
type singleton struct {
|
||||
// The bootstrap Config
|
||||
config *Config
|
||||
|
|
@ -646,10 +574,10 @@ func (s *singleton) GenerateBuildActions(ctx blueprint.SingletonContext) {
|
|||
// creating the binary that we'll use to generate the non-bootstrap
|
||||
// build.ninja file.
|
||||
var primaryBuilders []*goBinary
|
||||
// rebootstrapDeps contains modules that will be built in StageBootstrap
|
||||
var rebootstrapDeps []string
|
||||
// primaryRebootstrapDeps contains modules that will be built in StagePrimary
|
||||
var primaryRebootstrapDeps []string
|
||||
// bootstrapDeps contains modules that will be built in StageBootstrap
|
||||
var bootstrapDeps []string
|
||||
// primaryBootstrapDeps contains modules that will be built in StagePrimary
|
||||
var primaryBootstrapDeps []string
|
||||
// blueprintTools contains blueprint go binaries that will be built in StageMain
|
||||
var blueprintTools []string
|
||||
ctx.VisitAllModulesIf(isBootstrapBinaryModule,
|
||||
|
|
@ -660,9 +588,9 @@ func (s *singleton) GenerateBuildActions(ctx blueprint.SingletonContext) {
|
|||
|
||||
switch binaryModule.BuildStage() {
|
||||
case StageBootstrap:
|
||||
rebootstrapDeps = append(rebootstrapDeps, installPath)
|
||||
bootstrapDeps = append(bootstrapDeps, installPath)
|
||||
case StagePrimary:
|
||||
primaryRebootstrapDeps = append(primaryRebootstrapDeps, installPath)
|
||||
primaryBootstrapDeps = append(primaryBootstrapDeps, installPath)
|
||||
case StageMain:
|
||||
blueprintTools = append(blueprintTools, installPath)
|
||||
}
|
||||
|
|
@ -702,24 +630,19 @@ func (s *singleton) GenerateBuildActions(ctx blueprint.SingletonContext) {
|
|||
topLevelBlueprints := filepath.Join("$srcDir",
|
||||
filepath.Base(s.config.topLevelBlueprintsFile))
|
||||
|
||||
rebootstrapDeps = append(rebootstrapDeps, topLevelBlueprints)
|
||||
primaryRebootstrapDeps = append(primaryRebootstrapDeps, topLevelBlueprints)
|
||||
bootstrapDeps = append(bootstrapDeps, topLevelBlueprints)
|
||||
|
||||
mainNinjaFile := filepath.Join(bootstrapDir, "main.ninja.in")
|
||||
mainNinjaTimestampFile := mainNinjaFile + ".timestamp"
|
||||
mainNinjaTimestampDepFile := mainNinjaTimestampFile + ".d"
|
||||
primaryBuilderNinjaFile := filepath.Join(bootstrapDir, "primary.ninja.in")
|
||||
primaryBuilderNinjaTimestampFile := primaryBuilderNinjaFile + ".timestamp"
|
||||
primaryBuilderNinjaTimestampDepFile := primaryBuilderNinjaTimestampFile + ".d"
|
||||
bootstrapNinjaFile := filepath.Join(bootstrapDir, "bootstrap.ninja.in")
|
||||
mainNinjaFile := filepath.Join("$buildDir", "build.ninja")
|
||||
primaryBuilderNinjaFile := filepath.Join(bootstrapDir, "build.ninja")
|
||||
bootstrapNinjaFileTemplate := filepath.Join(miniBootstrapDir, "build.ninja.in")
|
||||
bootstrapNinjaFile := filepath.Join(miniBootstrapDir, "build.ninja")
|
||||
docsFile := filepath.Join(docsDir, primaryBuilderName+".html")
|
||||
|
||||
primaryRebootstrapDeps = append(primaryRebootstrapDeps, docsFile)
|
||||
primaryBootstrapDeps = append(primaryBootstrapDeps, docsFile)
|
||||
|
||||
// If the tests change, be sure to re-run them. These need to be
|
||||
// dependencies for the ninja file so that it's updated after these
|
||||
// run. Otherwise we'd never leave the bootstrap stage, since the
|
||||
// timestamp file would be newer than the ninja file.
|
||||
// run.
|
||||
ctx.VisitAllModulesIf(isGoTestProducer,
|
||||
func(module blueprint.Module) {
|
||||
testModule := module.(goTestProducer)
|
||||
|
|
@ -727,9 +650,9 @@ func (s *singleton) GenerateBuildActions(ctx blueprint.SingletonContext) {
|
|||
if target != "" {
|
||||
switch testModule.BuildStage() {
|
||||
case StageBootstrap:
|
||||
rebootstrapDeps = append(rebootstrapDeps, target)
|
||||
bootstrapDeps = append(bootstrapDeps, target)
|
||||
case StagePrimary:
|
||||
primaryRebootstrapDeps = append(primaryRebootstrapDeps, target)
|
||||
primaryBootstrapDeps = append(primaryBootstrapDeps, target)
|
||||
case StageMain:
|
||||
blueprintTools = append(blueprintTools, target)
|
||||
}
|
||||
|
|
@ -750,18 +673,15 @@ func (s *singleton) GenerateBuildActions(ctx blueprint.SingletonContext) {
|
|||
// it needs to be regenerated.
|
||||
primarybp := ctx.Rule(pctx, "primarybp",
|
||||
blueprint.RuleParams{
|
||||
Command: fmt.Sprintf("%s --build-primary $runTests -m $bootstrapManifest "+
|
||||
"--timestamp $timestamp --timestampdep $timestampdep "+
|
||||
Command: fmt.Sprintf("%s --build-primary $runTests "+
|
||||
"-b $buildDir -d $outfile.d -o $outfile $in", minibpFile),
|
||||
Description: "minibp $outfile",
|
||||
Depfile: "$outfile.d",
|
||||
},
|
||||
"runTests", "timestamp", "timestampdep", "outfile")
|
||||
"runTests", "outfile")
|
||||
|
||||
args := map[string]string{
|
||||
"outfile": primaryBuilderNinjaFile,
|
||||
"timestamp": primaryBuilderNinjaTimestampFile,
|
||||
"timestampdep": primaryBuilderNinjaTimestampDepFile,
|
||||
}
|
||||
|
||||
if s.config.runGoTests {
|
||||
|
|
@ -770,26 +690,22 @@ func (s *singleton) GenerateBuildActions(ctx blueprint.SingletonContext) {
|
|||
|
||||
ctx.Build(pctx, blueprint.BuildParams{
|
||||
Rule: primarybp,
|
||||
Outputs: []string{primaryBuilderNinjaFile, primaryBuilderNinjaTimestampFile},
|
||||
Outputs: []string{primaryBuilderNinjaFile},
|
||||
Inputs: []string{topLevelBlueprints},
|
||||
Implicits: rebootstrapDeps,
|
||||
Implicits: bootstrapDeps,
|
||||
Args: args,
|
||||
})
|
||||
|
||||
// Rebuild the bootstrap Ninja file using the minibp that we just built.
|
||||
// If this produces a difference, choosestage will retrigger this stage.
|
||||
minibp := ctx.Rule(pctx, "minibp",
|
||||
blueprint.RuleParams{
|
||||
Command: fmt.Sprintf("%s $runTests -m $bootstrapManifest "+
|
||||
Command: fmt.Sprintf("%s $runTests "+
|
||||
"-b $buildDir -d $out.d -o $out $in", minibpFile),
|
||||
// $bootstrapManifest is here so that when it is updated, we
|
||||
// force a rebuild of bootstrap.ninja.in. chooseStage should
|
||||
// have already copied the new version over, but kept the old
|
||||
// timestamps to force this regeneration.
|
||||
CommandDeps: []string{"$bootstrapManifest", minibpFile},
|
||||
CommandDeps: []string{minibpFile},
|
||||
Description: "minibp $out",
|
||||
Generator: true,
|
||||
Depfile: "$out.d",
|
||||
// So that we don't trigger a restart if this hasn't changed
|
||||
Restat: true,
|
||||
},
|
||||
"runTests")
|
||||
|
||||
|
|
@ -801,29 +717,15 @@ func (s *singleton) GenerateBuildActions(ctx blueprint.SingletonContext) {
|
|||
|
||||
ctx.Build(pctx, blueprint.BuildParams{
|
||||
Rule: minibp,
|
||||
Outputs: []string{bootstrapNinjaFile},
|
||||
Outputs: []string{bootstrapNinjaFileTemplate},
|
||||
Inputs: []string{topLevelBlueprints},
|
||||
Args: args,
|
||||
})
|
||||
|
||||
// When the current build.ninja file is a bootstrapper, we always want
|
||||
// to have it replace itself with a non-bootstrapper build.ninja. To
|
||||
// accomplish that we depend on a file that should never exist and
|
||||
// "build" it using Ninja's built-in phony rule.
|
||||
notAFile := filepath.Join(bootstrapDir, "notAFile")
|
||||
ctx.Build(pctx, blueprint.BuildParams{
|
||||
Rule: blueprint.Phony,
|
||||
Outputs: []string{notAFile},
|
||||
})
|
||||
|
||||
ctx.Build(pctx, blueprint.BuildParams{
|
||||
Rule: chooseStage,
|
||||
Outputs: []string{filepath.Join(bootstrapDir, "build.ninja.in")},
|
||||
Inputs: []string{bootstrapNinjaFile, primaryBuilderNinjaFile},
|
||||
Implicits: []string{notAFile},
|
||||
Args: map[string]string{
|
||||
"current": bootstrapNinjaFile,
|
||||
},
|
||||
Rule: bootstrap,
|
||||
Outputs: []string{bootstrapNinjaFile},
|
||||
Inputs: []string{bootstrapNinjaFileTemplate},
|
||||
})
|
||||
|
||||
case StagePrimary:
|
||||
|
|
@ -836,28 +738,23 @@ func (s *singleton) GenerateBuildActions(ctx blueprint.SingletonContext) {
|
|||
|
||||
// We generate the depfile here that includes the dependencies for all
|
||||
// the Blueprints files that contribute to generating the big build
|
||||
// manifest (build.ninja file). This depfile will be used by the non-
|
||||
// bootstrap build manifest to determine whether it should touch the
|
||||
// timestamp file to trigger a re-bootstrap.
|
||||
// manifest (build.ninja file).
|
||||
bigbp := ctx.Rule(pctx, "bigbp",
|
||||
blueprint.RuleParams{
|
||||
Command: fmt.Sprintf("%s %s -m $bootstrapManifest "+
|
||||
"--timestamp $timestamp --timestampdep $timestampdep "+
|
||||
Command: fmt.Sprintf("%s %s "+
|
||||
"-b $buildDir -d $outfile.d -o $outfile $in", primaryBuilderFile,
|
||||
primaryBuilderExtraFlags),
|
||||
Description: fmt.Sprintf("%s $outfile", primaryBuilderName),
|
||||
Depfile: "$outfile.d",
|
||||
},
|
||||
"timestamp", "timestampdep", "outfile")
|
||||
"outfile")
|
||||
|
||||
ctx.Build(pctx, blueprint.BuildParams{
|
||||
Rule: bigbp,
|
||||
Outputs: []string{mainNinjaFile, mainNinjaTimestampFile},
|
||||
Outputs: []string{mainNinjaFile},
|
||||
Inputs: []string{topLevelBlueprints},
|
||||
Implicits: primaryRebootstrapDeps,
|
||||
Implicits: primaryBootstrapDeps,
|
||||
Args: map[string]string{
|
||||
"timestamp": mainNinjaTimestampFile,
|
||||
"timestampdep": mainNinjaTimestampDepFile,
|
||||
"outfile": mainNinjaFile,
|
||||
},
|
||||
})
|
||||
|
|
@ -880,98 +777,9 @@ func (s *singleton) GenerateBuildActions(ctx blueprint.SingletonContext) {
|
|||
Outputs: []string{docsFile},
|
||||
})
|
||||
|
||||
// Detect whether we need to rebuild the primary stage by going back to
|
||||
// the bootstrapper. If this is newer than the primaryBuilderNinjaFile,
|
||||
// then chooseStage will trigger a rebuild of primaryBuilderNinjaFile by
|
||||
// returning to the bootstrap stage.
|
||||
ctx.Build(pctx, blueprint.BuildParams{
|
||||
Rule: touch,
|
||||
Outputs: []string{primaryBuilderNinjaTimestampFile},
|
||||
Implicits: rebootstrapDeps,
|
||||
Args: map[string]string{
|
||||
"depfile": primaryBuilderNinjaTimestampDepFile,
|
||||
"generator": "true",
|
||||
},
|
||||
})
|
||||
|
||||
// When the current build.ninja file is a bootstrapper, we always want
|
||||
// to have it replace itself with a non-bootstrapper build.ninja. To
|
||||
// accomplish that we depend on a file that should never exist and
|
||||
// "build" it using Ninja's built-in phony rule.
|
||||
notAFile := filepath.Join(bootstrapDir, "notAFile")
|
||||
ctx.Build(pctx, blueprint.BuildParams{
|
||||
Rule: blueprint.Phony,
|
||||
Outputs: []string{notAFile},
|
||||
})
|
||||
|
||||
ctx.Build(pctx, blueprint.BuildParams{
|
||||
Rule: chooseStage,
|
||||
Outputs: []string{filepath.Join(bootstrapDir, "build.ninja.in")},
|
||||
Inputs: []string{bootstrapNinjaFile, primaryBuilderNinjaFile, mainNinjaFile},
|
||||
Implicits: []string{notAFile, primaryBuilderNinjaTimestampFile},
|
||||
Args: map[string]string{
|
||||
"current": primaryBuilderNinjaFile,
|
||||
},
|
||||
})
|
||||
|
||||
// Create this phony rule so that upgrades don't delete these during
|
||||
// cleanup
|
||||
ctx.Build(pctx, blueprint.BuildParams{
|
||||
Rule: blueprint.Phony,
|
||||
Outputs: []string{bootstrapNinjaFile},
|
||||
})
|
||||
|
||||
case StageMain:
|
||||
ctx.SetNinjaBuildDir(pctx, "${buildDir}")
|
||||
|
||||
// We're generating a non-bootstrapper Ninja file, so we need to set it
|
||||
// up to re-bootstrap if necessary. We do this by making build.ninja.in
|
||||
// depend on the various Ninja files, the source build.ninja.in, and
|
||||
// on the timestamp files.
|
||||
//
|
||||
// The timestamp files themselves are set up with the same dependencies
|
||||
// as their Ninja files, including their own depfile. If any of the
|
||||
// dependencies need to be updated, we'll touch the timestamp file,
|
||||
// which will tell choosestage to switch to the stage that rebuilds
|
||||
// that Ninja file.
|
||||
ctx.Build(pctx, blueprint.BuildParams{
|
||||
Rule: touch,
|
||||
Outputs: []string{primaryBuilderNinjaTimestampFile},
|
||||
Implicits: rebootstrapDeps,
|
||||
Args: map[string]string{
|
||||
"depfile": primaryBuilderNinjaTimestampDepFile,
|
||||
"generator": "true",
|
||||
},
|
||||
})
|
||||
|
||||
ctx.Build(pctx, blueprint.BuildParams{
|
||||
Rule: touch,
|
||||
Outputs: []string{mainNinjaTimestampFile},
|
||||
Implicits: primaryRebootstrapDeps,
|
||||
Args: map[string]string{
|
||||
"depfile": mainNinjaTimestampDepFile,
|
||||
"generator": "true",
|
||||
},
|
||||
})
|
||||
|
||||
ctx.Build(pctx, blueprint.BuildParams{
|
||||
Rule: chooseStage,
|
||||
Outputs: []string{filepath.Join(bootstrapDir, "build.ninja.in")},
|
||||
Inputs: []string{bootstrapNinjaFile, primaryBuilderNinjaFile, mainNinjaFile},
|
||||
Implicits: []string{primaryBuilderNinjaTimestampFile, mainNinjaTimestampFile},
|
||||
Args: map[string]string{
|
||||
"current": mainNinjaFile,
|
||||
"generator": "true",
|
||||
},
|
||||
})
|
||||
|
||||
// Create this phony rule so that upgrades don't delete these during
|
||||
// cleanup
|
||||
ctx.Build(pctx, blueprint.BuildParams{
|
||||
Rule: blueprint.Phony,
|
||||
Outputs: []string{mainNinjaFile, docsFile, "$bootstrapManifest"},
|
||||
})
|
||||
|
||||
if primaryBuilderName == "minibp" {
|
||||
// This is a standalone Blueprint build, so we copy the minibp
|
||||
// binary to the "bin" directory to make it easier to find.
|
||||
|
|
@ -989,12 +797,6 @@ func (s *singleton) GenerateBuildActions(ctx blueprint.SingletonContext) {
|
|||
Inputs: blueprintTools,
|
||||
})
|
||||
}
|
||||
|
||||
ctx.Build(pctx, blueprint.BuildParams{
|
||||
Rule: bootstrap,
|
||||
Outputs: []string{"$buildDir/build.ninja"},
|
||||
Inputs: []string{filepath.Join(bootstrapDir, "build.ninja.in")},
|
||||
})
|
||||
}
|
||||
|
||||
// packageRoot returns the module-specific package root directory path. This
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ const logFileName = ".ninja_log"
|
|||
// removeAbandonedFiles removes any files that appear in the Ninja log that are
|
||||
// not currently build targets.
|
||||
func removeAbandonedFiles(ctx *blueprint.Context, config *Config,
|
||||
srcDir, manifestFile string) error {
|
||||
srcDir string) error {
|
||||
|
||||
ninjaBuildDir, err := ctx.NinjaBuildDir()
|
||||
if err != nil {
|
||||
|
|
@ -45,8 +45,7 @@ func removeAbandonedFiles(ctx *blueprint.Context, config *Config,
|
|||
|
||||
replacer := strings.NewReplacer(
|
||||
"@@SrcDir@@", srcDir,
|
||||
"@@BuildDir@@", BuildDir,
|
||||
"@@BootstrapManifest@@", manifestFile)
|
||||
"@@BuildDir@@", BuildDir)
|
||||
ninjaBuildDir = replacer.Replace(ninjaBuildDir)
|
||||
targets := make(map[string]bool)
|
||||
for target := range targetRules {
|
||||
|
|
|
|||
|
|
@ -32,24 +32,19 @@ import (
|
|||
var (
|
||||
outFile string
|
||||
depFile string
|
||||
timestampFile string
|
||||
timestampDepFile string
|
||||
manifestFile string
|
||||
docFile string
|
||||
cpuprofile string
|
||||
traceFile string
|
||||
runGoTests bool
|
||||
|
||||
BuildDir string
|
||||
SrcDir string
|
||||
)
|
||||
|
||||
func init() {
|
||||
flag.StringVar(&outFile, "o", "build.ninja.in", "the Ninja file to output")
|
||||
flag.StringVar(&BuildDir, "b", ".", "the build output directory")
|
||||
flag.StringVar(&depFile, "d", "", "the dependency file to output")
|
||||
flag.StringVar(×tampFile, "timestamp", "", "file to write before the output file")
|
||||
flag.StringVar(×tampDepFile, "timestampdep", "", "the dependency file for the timestamp file")
|
||||
flag.StringVar(&manifestFile, "m", "", "the bootstrap manifest file")
|
||||
flag.StringVar(&docFile, "docs", "", "build documentation file to output")
|
||||
flag.StringVar(&cpuprofile, "cpuprofile", "", "write cpu profile to file")
|
||||
flag.StringVar(&traceFile, "trace", "", "write trace to file")
|
||||
|
|
@ -87,6 +82,8 @@ func Main(ctx *blueprint.Context, config interface{}, extraNinjaFileDeps ...stri
|
|||
fatalf("no Blueprints file specified")
|
||||
}
|
||||
|
||||
SrcDir = filepath.Dir(flag.Arg(0))
|
||||
|
||||
stage := StageMain
|
||||
if c, ok := config.(ConfigInterface); ok {
|
||||
if c.GeneratingBootstrapper() {
|
||||
|
|
@ -145,20 +142,6 @@ func Main(ctx *blueprint.Context, config interface{}, extraNinjaFileDeps ...stri
|
|||
}
|
||||
|
||||
const outFilePermissions = 0666
|
||||
if timestampFile != "" {
|
||||
err := ioutil.WriteFile(timestampFile, []byte{}, outFilePermissions)
|
||||
if err != nil {
|
||||
fatalf("error writing %s: %s", timestampFile, err)
|
||||
}
|
||||
|
||||
if timestampDepFile != "" {
|
||||
err := deptools.WriteDepFile(timestampDepFile, timestampFile, deps)
|
||||
if err != nil {
|
||||
fatalf("error writing depfile: %s", err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
err = ioutil.WriteFile(outFile, buf.Bytes(), outFilePermissions)
|
||||
if err != nil {
|
||||
fatalf("error writing %s: %s", outFile, err)
|
||||
|
|
@ -169,15 +152,10 @@ func Main(ctx *blueprint.Context, config interface{}, extraNinjaFileDeps ...stri
|
|||
if err != nil {
|
||||
fatalf("error writing depfile: %s", err)
|
||||
}
|
||||
err = deptools.WriteDepFile(depFile+".timestamp", outFile+".timestamp", deps)
|
||||
if err != nil {
|
||||
fatalf("error writing depfile: %s", err)
|
||||
}
|
||||
}
|
||||
|
||||
if c, ok := config.(ConfigRemoveAbandonedFiles); !ok || c.RemoveAbandonedFiles() {
|
||||
srcDir := filepath.Dir(bootstrapConfig.topLevelBlueprintsFile)
|
||||
err := removeAbandonedFiles(ctx, bootstrapConfig, srcDir, manifestFile)
|
||||
err := removeAbandonedFiles(ctx, bootstrapConfig, SrcDir)
|
||||
if err != nil {
|
||||
fatalf("error removing abandoned files: %s", err)
|
||||
}
|
||||
|
|
|
|||
|
|
@ -14,27 +14,54 @@
|
|||
|
||||
package bootstrap
|
||||
|
||||
import (
|
||||
"runtime"
|
||||
|
||||
"github.com/google/blueprint"
|
||||
)
|
||||
|
||||
func bootstrapVariable(name, template string, value func() string) blueprint.Variable {
|
||||
return pctx.VariableFunc(name, func(config interface{}) (string, error) {
|
||||
if c, ok := config.(ConfigInterface); ok && c.GeneratingBootstrapper() {
|
||||
return template, nil
|
||||
}
|
||||
return value(), nil
|
||||
})
|
||||
}
|
||||
|
||||
var (
|
||||
// These variables are the only configuration needed by the boostrap
|
||||
// modules. They are always set to the variable name enclosed in "@@" so
|
||||
// that their values can be easily replaced in the generated Ninja file.
|
||||
srcDir = pctx.StaticVariable("srcDir", "@@SrcDir@@")
|
||||
buildDir = pctx.StaticVariable("buildDir", "@@BuildDir@@")
|
||||
goRoot = pctx.StaticVariable("goRoot", "@@GoRoot@@")
|
||||
compileCmd = pctx.StaticVariable("compileCmd", "@@GoCompile@@")
|
||||
linkCmd = pctx.StaticVariable("linkCmd", "@@GoLink@@")
|
||||
bootstrapCmd = pctx.StaticVariable("bootstrapCmd", "@@Bootstrap@@")
|
||||
bootstrapManifest = pctx.StaticVariable("bootstrapManifest",
|
||||
"@@BootstrapManifest@@")
|
||||
// modules. For the first bootstrap stage, they are set to the
|
||||
// variable name enclosed in "@@" so that their values can be easily
|
||||
// replaced in the generated Ninja file.
|
||||
srcDir = bootstrapVariable("srcDir", "@@SrcDir@@", func() string {
|
||||
return SrcDir
|
||||
})
|
||||
buildDir = bootstrapVariable("buildDir", "@@BuildDir@@", func() string {
|
||||
return BuildDir
|
||||
})
|
||||
goRoot = bootstrapVariable("goRoot", "@@GoRoot@@", func() string {
|
||||
return runtime.GOROOT()
|
||||
})
|
||||
compileCmd = bootstrapVariable("compileCmd", "@@GoCompile@@", func() string {
|
||||
return "$goRoot/pkg/tool/" + runtime.GOOS + "_" + runtime.GOARCH + "/compile"
|
||||
})
|
||||
linkCmd = bootstrapVariable("linkCmd", "@@GoLink@@", func() string {
|
||||
return "$goRoot/pkg/tool/" + runtime.GOOS + "_" + runtime.GOARCH + "/link"
|
||||
})
|
||||
bootstrapCmd = bootstrapVariable("bootstrapCmd", "@@Bootstrap@@", func() string {
|
||||
panic("bootstrapCmd is only available for minibootstrap")
|
||||
})
|
||||
)
|
||||
|
||||
type ConfigInterface interface {
|
||||
// GeneratingBootstrapper should return true if this build invocation is
|
||||
// creating a build.ninja.in file to be used in a build bootstrapping
|
||||
// sequence.
|
||||
// creating a .minibootstrap/build.ninja file to be used in a build
|
||||
// bootstrapping sequence.
|
||||
GeneratingBootstrapper() bool
|
||||
// GeneratingPrimaryBuilder should return true if this build invocation is
|
||||
// creating a build.ninja.in file to be used to build the primary builder
|
||||
// creating a .bootstrap/build.ninja file to be used to build the
|
||||
// primary builder
|
||||
GeneratingPrimaryBuilder() bool
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -103,67 +103,76 @@
|
|||
//
|
||||
// The Bootstrapping Process
|
||||
//
|
||||
// A bootstrap-enabled build directory has two states, each with a corresponding
|
||||
// Ninja file. The states are referred to as the "bootstrap" state and the
|
||||
// "main" state. Changing the directory to a particular state means replacing
|
||||
// the build.ninja file with one that will perform the build actions for the
|
||||
// state.
|
||||
// There are three stages to the bootstrapping process, each with a
|
||||
// corresponding Ninja file. The stages are referred to as the "bootstrap",
|
||||
// "primary", and "main" stages. Each stage builds the next stage's Ninja file.
|
||||
//
|
||||
// The bootstrapping process begins with the user running the bootstrap script
|
||||
// to initialize a new build directory. The script is run from the build
|
||||
// directory, and when run with no arguments it copies the source bootstrap
|
||||
// Ninja file into the build directory as "build.ninja". It also performs a set
|
||||
// of string substitutions on the file to configure it for the user's build
|
||||
// environment. Specifically, the following strings are substituted in the file:
|
||||
// Ninja file into the build directory as ".minibootstrap/build.ninja". It
|
||||
// also performs a set of string substitutions on the file to configure it for
|
||||
// the user's build environment. Specifically, the following strings are
|
||||
// substituted in the file:
|
||||
//
|
||||
// @@SrcDir@@ - The path to the root source directory (either
|
||||
// absolute or relative to the build dir)
|
||||
// @@BuildDir@@ - The path to the build directory
|
||||
// @@GoRoot@@ - The path to the root directory of the Go toolchain
|
||||
// @@GoCompile@@ - The path to the Go compiler (6g or compile)
|
||||
// @@GoLink@@ - The path to the Go linker (6l or link)
|
||||
// @@Bootstrap@@ - The path to the bootstrap script
|
||||
// @@BootstrapManifest@@ - The path to the source bootstrap Ninja file
|
||||
//
|
||||
// Once the script completes the build directory is initialized in the bootstrap
|
||||
// build state. In this state, running Ninja may perform the following build
|
||||
// actions. Each one but the last can be skipped if its output is determined to
|
||||
// be up-to-date.
|
||||
// Once the script completes the build directory is initialized and ready to run
|
||||
// a build. A wrapper script (blueprint.bash by default) has been installed in
|
||||
// order to run a build. It iterates through the three stages of the build:
|
||||
//
|
||||
// - Build the minibp binary
|
||||
// - Run minibp to generate .bootstrap/bootstrap.ninja.in
|
||||
// - Build the primary builder binary
|
||||
// - Run the primary builder to generate .bootstrap/main.ninja.in
|
||||
// - Run the bootstrap script to "copy" .bootstrap/main.ninja.in to build.ninja
|
||||
// - Checks to see if the source bootstrap Ninja file is newer than the
|
||||
// one that is in the build directory, if so, update the build dir copy.
|
||||
// - Run the Bootstrap stage
|
||||
// - Run the Primary stage
|
||||
// - Run the Main stage
|
||||
//
|
||||
// The last of these build actions results in transitioning the build directory
|
||||
// to the main build state.
|
||||
// Previously, we were keeping track of the "state" of the build directory and
|
||||
// only going back to previous stages when something had changed. But that
|
||||
// added complexity, and failed when there was a build error in the Primary
|
||||
// stage that would have been fixed if the Bootstrap stage was re-run (we
|
||||
// would only evaluate which stages needed to be run at the end of the stage).
|
||||
// So now we always run through each stage, and the first two stages will do
|
||||
// nothing when nothing has changed.
|
||||
//
|
||||
// The main state (potentially) performs the following actions:
|
||||
// - Copy .bootstrap/bootstrap.ninja.in to the source bootstrap Ninja location
|
||||
// - Run the bootstrap script to "copy" the source bootstrap Ninja file to
|
||||
// build.ninja
|
||||
// - Build all the non-bootstrap modules defined in Blueprints files
|
||||
// During the Bootstrap stage, <builddir>/.minibootstrap/build.ninja, the
|
||||
// following actions are taken, if necessary:
|
||||
//
|
||||
// - Build all bootstrap_core_go_binary rules, and dependencies --
|
||||
// minibp and some test helpers.
|
||||
// - Run minibp to generate .bootstrap/build.ninja (Primary stage)
|
||||
// - Run minibp to generate .minibootstrap/build.ninja.in
|
||||
// - Restart if .minibootstrap/build.ninja.in has changed
|
||||
//
|
||||
// During the Primary stage, <builddir>/.bootstrap/build.ninja, the following
|
||||
// actions are taken, if necessary:
|
||||
//
|
||||
// - Build any bootstrap_go_binary rules and dependencies -- usually the
|
||||
// primary builder and any build or runtime dependencies.
|
||||
// - Run the primary builder to generate build.ninja
|
||||
// - Run the primary builder to extract documentation
|
||||
//
|
||||
// Then the main stage is at <builddir>/build.ninja, and will contain all the
|
||||
// rules generated by the primary builder. In addition, the bootstrap code
|
||||
// adds a phony rule "blueprint_tools" that depends on all blueprint_go_binary
|
||||
// rules (bpfmt, bpmodify, etc).
|
||||
//
|
||||
// Updating the Bootstrap Ninja File Template
|
||||
//
|
||||
// The main purpose of the bootstrap state is to generate the Ninja file for the
|
||||
// main state. The one additional thing it does is generate a new bootstrap
|
||||
// Ninja file template at .bootstrap/bootstrap.ninja.in. When generating this
|
||||
// The main purpose of the bootstrap stage is to generate the Ninja file for the
|
||||
// primary stage. The one additional thing it does is generate a new bootstrap
|
||||
// Ninja file template at .minibootstrap/build.ninja.in. When generating this
|
||||
// file, minibp will compare the new bootstrap Ninja file contents with the
|
||||
// original (in the source tree). If the contents match, the new file will be
|
||||
// created with a timestamp that matches that of the original, indicating that
|
||||
// the original file in the source tree is up-to-date.
|
||||
//
|
||||
// This is done so that in the main state if the bootstrap Ninja file template
|
||||
// in the source tree is out of date it can be automatically updated. Note,
|
||||
// however, that we can't have the main state generate the new bootstrap Ninja
|
||||
// file template contents itself, because it may be using an older minibp.
|
||||
// Recall that minibp is only built during the bootstrap state (to break a
|
||||
// circular dependence), so if a new bootstrap Ninja file template were
|
||||
// generated then it could replace a new file (from an updated source tree) with
|
||||
// one generated using an old minibp.
|
||||
// original (in the source tree).
|
||||
//
|
||||
// This scheme ensures that updates to the source tree are always incorporated
|
||||
// into the build process and that changes that require a new bootstrap Ninja
|
||||
// file template automatically update the template in the source tree.
|
||||
// into the build process.
|
||||
//
|
||||
package bootstrap
|
||||
|
|
|
|||
|
|
@ -15,10 +15,6 @@ g.bootstrap.BinDir = ${g.bootstrap.buildDir}/.bootstrap/bin
|
|||
|
||||
g.bootstrap.bootstrapCmd = @@Bootstrap@@
|
||||
|
||||
g.bootstrap.bootstrapManifest = @@BootstrapManifest@@
|
||||
|
||||
g.bootstrap.chooseStageCmd = ${g.bootstrap.buildDir}/.bootstrap/bin/choosestage
|
||||
|
||||
g.bootstrap.compileCmd = @@GoCompile@@
|
||||
|
||||
g.bootstrap.goRoot = @@GoRoot@@
|
||||
|
|
@ -34,10 +30,6 @@ rule g.bootstrap.bootstrap
|
|||
description = bootstrap ${in}
|
||||
generator = true
|
||||
|
||||
rule g.bootstrap.chooseStage
|
||||
command = ${g.bootstrap.chooseStageCmd} --current ${current} --bootstrap ${g.bootstrap.bootstrapManifest} -o ${out} ${in}
|
||||
description = choosing next stage
|
||||
|
||||
rule g.bootstrap.compile
|
||||
command = GOROOT='${g.bootstrap.goRoot}' ${g.bootstrap.compileCmd} -o ${out} -p ${pkgPath} -complete ${incFlags} -pack ${in}
|
||||
description = compile ${out}
|
||||
|
|
@ -189,29 +181,6 @@ build $
|
|||
default $
|
||||
${g.bootstrap.buildDir}/.bootstrap/blueprint-proptools/pkg/github.com/google/blueprint/proptools.a
|
||||
|
||||
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
|
||||
# Module: choosestage
|
||||
# Variant:
|
||||
# Type: bootstrap_core_go_binary
|
||||
# Factory: github.com/google/blueprint/bootstrap.newGoBinaryModuleFactory.func1
|
||||
# Defined: Blueprints:145:1
|
||||
|
||||
build ${g.bootstrap.buildDir}/.bootstrap/choosestage/obj/choosestage.a: $
|
||||
g.bootstrap.compile ${g.bootstrap.srcDir}/choosestage/choosestage.go | $
|
||||
${g.bootstrap.compileCmd}
|
||||
pkgPath = choosestage
|
||||
default ${g.bootstrap.buildDir}/.bootstrap/choosestage/obj/choosestage.a
|
||||
|
||||
build ${g.bootstrap.buildDir}/.bootstrap/choosestage/obj/a.out: $
|
||||
g.bootstrap.link $
|
||||
${g.bootstrap.buildDir}/.bootstrap/choosestage/obj/choosestage.a | $
|
||||
${g.bootstrap.linkCmd}
|
||||
default ${g.bootstrap.buildDir}/.bootstrap/choosestage/obj/a.out
|
||||
|
||||
build ${g.bootstrap.BinDir}/choosestage: g.bootstrap.cp $
|
||||
${g.bootstrap.buildDir}/.bootstrap/choosestage/obj/a.out
|
||||
default ${g.bootstrap.BinDir}/choosestage
|
||||
|
||||
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
|
||||
# Module: gotestmain
|
||||
# Variant:
|
||||
|
|
@ -294,47 +263,31 @@ default ${g.bootstrap.BinDir}/minibp
|
|||
# Factory: github.com/google/blueprint/bootstrap.newSingletonFactory.func1
|
||||
|
||||
rule s.bootstrap.primarybp
|
||||
command = ${g.bootstrap.BinDir}/minibp --build-primary ${runTests} -m ${g.bootstrap.bootstrapManifest} --timestamp ${timestamp} --timestampdep ${timestampdep} -b ${g.bootstrap.buildDir} -d ${outfile}.d -o ${outfile} ${in}
|
||||
command = ${g.bootstrap.BinDir}/minibp --build-primary ${runTests} -b ${g.bootstrap.buildDir} -d ${outfile}.d -o ${outfile} ${in}
|
||||
depfile = ${outfile}.d
|
||||
description = minibp ${outfile}
|
||||
|
||||
rule s.bootstrap.minibp
|
||||
command = ${g.bootstrap.BinDir}/minibp ${runTests} -m ${g.bootstrap.bootstrapManifest} -b ${g.bootstrap.buildDir} -d ${out}.d -o ${out} ${in}
|
||||
command = ${g.bootstrap.BinDir}/minibp ${runTests} -b ${g.bootstrap.buildDir} -d ${out}.d -o ${out} ${in}
|
||||
depfile = ${out}.d
|
||||
description = minibp ${out}
|
||||
generator = true
|
||||
restat = true
|
||||
|
||||
build ${g.bootstrap.buildDir}/.bootstrap/primary.ninja.in $
|
||||
${g.bootstrap.buildDir}/.bootstrap/primary.ninja.in.timestamp: $
|
||||
s.bootstrap.primarybp ${g.bootstrap.srcDir}/Blueprints | $
|
||||
${g.bootstrap.BinDir}/choosestage ${g.bootstrap.BinDir}/gotestmain $
|
||||
build ${g.bootstrap.buildDir}/.bootstrap/build.ninja: s.bootstrap.primarybp $
|
||||
${g.bootstrap.srcDir}/Blueprints | ${g.bootstrap.BinDir}/gotestmain $
|
||||
${g.bootstrap.BinDir}/gotestrunner ${g.bootstrap.BinDir}/minibp $
|
||||
${g.bootstrap.srcDir}/Blueprints
|
||||
outfile = ${g.bootstrap.buildDir}/.bootstrap/primary.ninja.in
|
||||
timestamp = ${g.bootstrap.buildDir}/.bootstrap/primary.ninja.in.timestamp
|
||||
timestampdep = ${g.bootstrap.buildDir}/.bootstrap/primary.ninja.in.timestamp.d
|
||||
default ${g.bootstrap.buildDir}/.bootstrap/primary.ninja.in $
|
||||
${g.bootstrap.buildDir}/.bootstrap/primary.ninja.in.timestamp
|
||||
outfile = ${g.bootstrap.buildDir}/.bootstrap/build.ninja
|
||||
default ${g.bootstrap.buildDir}/.bootstrap/build.ninja
|
||||
|
||||
build ${g.bootstrap.buildDir}/.bootstrap/bootstrap.ninja.in: $
|
||||
build ${g.bootstrap.buildDir}/.minibootstrap/build.ninja.in: $
|
||||
s.bootstrap.minibp ${g.bootstrap.srcDir}/Blueprints | $
|
||||
${g.bootstrap.bootstrapManifest} ${g.bootstrap.BinDir}/minibp
|
||||
default ${g.bootstrap.buildDir}/.bootstrap/bootstrap.ninja.in
|
||||
${g.bootstrap.BinDir}/minibp
|
||||
default ${g.bootstrap.buildDir}/.minibootstrap/build.ninja.in
|
||||
|
||||
build ${g.bootstrap.buildDir}/.bootstrap/notAFile: phony
|
||||
default ${g.bootstrap.buildDir}/.bootstrap/notAFile
|
||||
|
||||
build ${g.bootstrap.buildDir}/.bootstrap/build.ninja.in: $
|
||||
g.bootstrap.chooseStage $
|
||||
${g.bootstrap.buildDir}/.bootstrap/bootstrap.ninja.in $
|
||||
${g.bootstrap.buildDir}/.bootstrap/primary.ninja.in | $
|
||||
${g.bootstrap.chooseStageCmd} ${g.bootstrap.bootstrapManifest} $
|
||||
${g.bootstrap.buildDir}/.bootstrap/notAFile
|
||||
current = ${g.bootstrap.buildDir}/.bootstrap/bootstrap.ninja.in
|
||||
default ${g.bootstrap.buildDir}/.bootstrap/build.ninja.in
|
||||
|
||||
build ${g.bootstrap.buildDir}/build.ninja: g.bootstrap.bootstrap $
|
||||
${g.bootstrap.buildDir}/.bootstrap/build.ninja.in | $
|
||||
build ${g.bootstrap.buildDir}/.minibootstrap/build.ninja: $
|
||||
g.bootstrap.bootstrap $
|
||||
${g.bootstrap.buildDir}/.minibootstrap/build.ninja.in | $
|
||||
${g.bootstrap.bootstrapCmd}
|
||||
default ${g.bootstrap.buildDir}/build.ninja
|
||||
default ${g.bootstrap.buildDir}/.minibootstrap/build.ninja
|
||||
|
||||
|
|
|
|||
|
|
@ -1,194 +0,0 @@
|
|||
// Copyright 2015 Google Inc. All rights reserved.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
// Choose which ninja file (stage) to run next
|
||||
//
|
||||
// In the common case, this program takes a list of ninja files, compares their
|
||||
// mtimes against their $file.timestamp mtimes, and picks the last up to date
|
||||
// ninja file to output. That stage is expected to rebuild the next file in the
|
||||
// list and call this program again. If none of the ninja files are considered
|
||||
// dirty, the last stage is output.
|
||||
//
|
||||
// One exception is if the current stage's ninja file was rewritten, it will be
|
||||
// run again.
|
||||
//
|
||||
// Another exception is if the source bootstrap file has been updated more
|
||||
// recently than the first stage, the source file will be copied to the first
|
||||
// stage, and output. This would be expected with a new source drop via git.
|
||||
// The timestamp of the first file is not updated so that it can be regenerated
|
||||
// with any local changes.
|
||||
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"flag"
|
||||
"fmt"
|
||||
"io/ioutil"
|
||||
"os"
|
||||
"path/filepath"
|
||||
)
|
||||
|
||||
var (
|
||||
outputFile string
|
||||
currentFile string
|
||||
bootstrapFile string
|
||||
verbose bool
|
||||
)
|
||||
|
||||
func init() {
|
||||
flag.StringVar(&outputFile, "o", "", "Output file")
|
||||
flag.StringVar(¤tFile, "current", "", "Current stage's file")
|
||||
flag.StringVar(&bootstrapFile, "bootstrap", "", "Bootstrap file checked into source")
|
||||
flag.BoolVar(&verbose, "v", false, "Verbose mode")
|
||||
}
|
||||
|
||||
func compareFiles(a, b string) (bool, error) {
|
||||
aData, err := ioutil.ReadFile(a)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
|
||||
bData, err := ioutil.ReadFile(b)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
|
||||
return bytes.Equal(aData, bData), nil
|
||||
}
|
||||
|
||||
// If the source bootstrap reference file is newer, then we may have gotten
|
||||
// other source updates too. So we need to restart everything with the file
|
||||
// that was checked in instead of the bootstrap that we last built.
|
||||
func copyBootstrapIfNecessary(bootstrapFile, filename string) (bool, error) {
|
||||
if bootstrapFile == "" {
|
||||
return false, nil
|
||||
}
|
||||
|
||||
bootstrapStat, err := os.Stat(bootstrapFile)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
|
||||
fileStat, err := os.Stat(filename)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
|
||||
time := fileStat.ModTime()
|
||||
if !bootstrapStat.ModTime().After(time) {
|
||||
return false, nil
|
||||
}
|
||||
|
||||
fmt.Printf("Newer source version of %s. Copying to %s\n", filepath.Base(bootstrapFile), filepath.Base(filename))
|
||||
if verbose {
|
||||
fmt.Printf("Source: %s\nBuilt: %s\n", bootstrapStat.ModTime(), time)
|
||||
}
|
||||
|
||||
data, err := ioutil.ReadFile(bootstrapFile)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
|
||||
err = ioutil.WriteFile(filename, data, 0666)
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
|
||||
// Restore timestamp to force regeneration of the bootstrap.ninja.in
|
||||
err = os.Chtimes(filename, time, time)
|
||||
return true, err
|
||||
}
|
||||
|
||||
func main() {
|
||||
flag.Parse()
|
||||
|
||||
if flag.NArg() == 0 {
|
||||
fmt.Fprintf(os.Stderr, "Must specify at least one ninja file\n")
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
if outputFile == "" {
|
||||
fmt.Fprintf(os.Stderr, "Must specify an output file\n")
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
gotoFile := flag.Arg(0)
|
||||
if copied, err := copyBootstrapIfNecessary(bootstrapFile, flag.Arg(0)); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Failed to copy bootstrap ninja file: %s\n", err)
|
||||
os.Exit(1)
|
||||
} else if !copied {
|
||||
for _, fileName := range flag.Args() {
|
||||
timestampName := fileName + ".timestamp"
|
||||
|
||||
// If we're currently running this stage, and the build.ninja.in
|
||||
// file differs from the current stage file, then it has been rebuilt.
|
||||
// Restart the stage.
|
||||
if filepath.Clean(currentFile) == filepath.Clean(fileName) {
|
||||
if _, err := os.Stat(outputFile); !os.IsNotExist(err) {
|
||||
if ok, err := compareFiles(fileName, outputFile); err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Failure when comparing files: %s\n", err)
|
||||
os.Exit(1)
|
||||
} else if !ok {
|
||||
fmt.Printf("Stage %s has changed, restarting\n", filepath.Base(fileName))
|
||||
gotoFile = fileName
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fileStat, err := os.Stat(fileName)
|
||||
if err != nil {
|
||||
// Regenerate this stage on error
|
||||
break
|
||||
}
|
||||
|
||||
timestampStat, err := os.Stat(timestampName)
|
||||
if err != nil {
|
||||
// This file may not exist. There's no point for
|
||||
// the first stage to have one, as it should be
|
||||
// a subset of the second stage dependencies,
|
||||
// and both will return to the first stage.
|
||||
continue
|
||||
}
|
||||
|
||||
if verbose {
|
||||
fmt.Printf("For %s:\n file: %s\n time: %s\n", fileName, fileStat.ModTime(), timestampStat.ModTime())
|
||||
}
|
||||
|
||||
// If the timestamp file has a later modification time, that
|
||||
// means that this stage needs to be regenerated. Break, so
|
||||
// that we run the last found stage.
|
||||
if timestampStat.ModTime().After(fileStat.ModTime()) {
|
||||
break
|
||||
}
|
||||
|
||||
gotoFile = fileName
|
||||
}
|
||||
}
|
||||
|
||||
fmt.Printf("Choosing %s for next stage\n", filepath.Base(gotoFile))
|
||||
|
||||
data, err := ioutil.ReadFile(gotoFile)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Can't read file: %s", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
err = ioutil.WriteFile(outputFile, data, 0666)
|
||||
if err != nil {
|
||||
fmt.Fprintf(os.Stderr, "Can't write file: %s", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
|
@ -3,5 +3,6 @@
|
|||
export BOOTSTRAP="${BASH_SOURCE[0]}"
|
||||
export SRCDIR=".."
|
||||
export BOOTSTRAP_MANIFEST="src.build.ninja.in"
|
||||
export WRAPPER="../blueprint.bash"
|
||||
|
||||
../bootstrap.bash "$@"
|
||||
|
|
|
|||
|
|
@ -1,3 +0,0 @@
|
|||
Choosing bootstrap.ninja.in for next stage
|
||||
Choosing primary.ninja.in for next stage
|
||||
Choosing main.ninja.in for next stage
|
||||
|
|
@ -1,4 +0,0 @@
|
|||
Newer source version of build.ninja.in. Copying to bootstrap.ninja.in
|
||||
Choosing bootstrap.ninja.in for next stage
|
||||
Choosing primary.ninja.in for next stage
|
||||
Choosing main.ninja.in for next stage
|
||||
|
|
@ -1,2 +0,0 @@
|
|||
Choosing primary.ninja.in for next stage
|
||||
Choosing main.ninja.in for next stage
|
||||
|
|
@ -1,3 +0,0 @@
|
|||
Choosing bootstrap.ninja.in for next stage
|
||||
Choosing primary.ninja.in for next stage
|
||||
Choosing main.ninja.in for next stage
|
||||
|
|
@ -1,6 +0,0 @@
|
|||
Newer source version of src.build.ninja.in. Copying to bootstrap.ninja.in
|
||||
Choosing bootstrap.ninja.in for next stage
|
||||
Stage bootstrap.ninja.in has changed, restarting
|
||||
Choosing bootstrap.ninja.in for next stage
|
||||
Choosing primary.ninja.in for next stage
|
||||
Choosing main.ninja.in for next stage
|
||||
|
|
@ -1,2 +0,0 @@
|
|||
Choosing primary.ninja.in for next stage
|
||||
Choosing main.ninja.in for next stage
|
||||
|
|
@ -1,4 +0,0 @@
|
|||
Stage bootstrap.ninja.in has changed, restarting
|
||||
Choosing bootstrap.ninja.in for next stage
|
||||
Choosing primary.ninja.in for next stage
|
||||
Choosing main.ninja.in for next stage
|
||||
|
|
@ -1,4 +0,0 @@
|
|||
Stage bootstrap.ninja.in has changed, restarting
|
||||
Choosing bootstrap.ninja.in for next stage
|
||||
Choosing primary.ninja.in for next stage
|
||||
Choosing main.ninja.in for next stage
|
||||
|
|
@ -1,2 +0,0 @@
|
|||
Choosing primary.ninja.in for next stage
|
||||
Choosing main.ninja.in for next stage
|
||||
|
|
@ -1,4 +0,0 @@
|
|||
Stage bootstrap.ninja.in has changed, restarting
|
||||
Choosing bootstrap.ninja.in for next stage
|
||||
Choosing primary.ninja.in for next stage
|
||||
Choosing main.ninja.in for next stage
|
||||
|
|
@ -1,2 +0,0 @@
|
|||
Choosing primary.ninja.in for next stage
|
||||
Choosing main.ninja.in for next stage
|
||||
|
|
@ -1,2 +0,0 @@
|
|||
Choosing primary.ninja.in for next stage
|
||||
Choosing main.ninja.in for next stage
|
||||
164
tests/test.sh
164
tests/test.sh
|
|
@ -1,4 +1,4 @@
|
|||
#!/bin/bash
|
||||
#!/bin/bash -ex
|
||||
|
||||
# Go to srcdir
|
||||
cd $(dirname ${BASH_SOURCE[0]})/..
|
||||
|
|
@ -6,136 +6,40 @@ cd $(dirname ${BASH_SOURCE[0]})/..
|
|||
rm -rf out.test
|
||||
mkdir out.test
|
||||
cd out.test
|
||||
../bootstrap.bash
|
||||
cp ../build.ninja.in src.build.ninja.in
|
||||
../tests/bootstrap.bash
|
||||
|
||||
# Run ninja, filter the output, and compare against expectations
|
||||
# $1: Name of test
|
||||
function testcase()
|
||||
{
|
||||
echo -n "Running $1..."
|
||||
if ! ninja -v -d explain >log_$1 2>&1; then
|
||||
echo " Failed."
|
||||
echo "Test $1 Failed:" >>failed
|
||||
tail log_$1 >>failed
|
||||
return
|
||||
fi
|
||||
grep -E "^(Choosing|Newer|Stage)" log_$1 >test_$1
|
||||
if ! cmp -s test_$1 ../tests/expected_$1; then
|
||||
echo " Failed."
|
||||
echo "Test $1 Failed:" >>failed
|
||||
diff -u ../tests/expected_$1 test_$1 >>failed
|
||||
else
|
||||
echo " Passed."
|
||||
fi
|
||||
}
|
||||
./blueprint.bash
|
||||
|
||||
# Run wrapper, filter the output, and compare against expectations
|
||||
# $1: Name of test
|
||||
function testcase_wrapper()
|
||||
{
|
||||
echo -n "Running wrapper_$1..."
|
||||
if ! ./blueprint.bash -v -d explain >log_wrapper_$1 2>&1; then
|
||||
echo " Failed."
|
||||
echo "Test wrapper_$1 Failed:" >>failed
|
||||
tail log_wrapper_$1 >>failed
|
||||
return
|
||||
fi
|
||||
grep -E "^(Choosing|Newer|Stage)" log_wrapper_$1 >test_wrapper_$1
|
||||
if ! cmp -s test_wrapper_$1 ../tests/expected_wrapper_$1; then
|
||||
echo " Failed."
|
||||
echo "Test wrapper_$1 Failed:" >>failed
|
||||
diff -u ../tests/expected_wrapper_$1 test_wrapper_$1 >>failed
|
||||
else
|
||||
echo " Passed."
|
||||
fi
|
||||
}
|
||||
|
||||
|
||||
testcase start
|
||||
|
||||
# The 2 second sleeps are needed until ninja understands sub-second timestamps
|
||||
# https://github.com/martine/ninja/issues/371
|
||||
|
||||
# This test affects all bootstrap stages
|
||||
sleep 2
|
||||
touch ../Blueprints
|
||||
testcase all
|
||||
|
||||
# This test affects only the primary bootstrap stage
|
||||
sleep 2
|
||||
touch ../bpmodify/bpmodify.go
|
||||
testcase primary
|
||||
|
||||
# This test affects nothing, nothing should be done
|
||||
sleep 2
|
||||
testcase none
|
||||
|
||||
# This test will cause the source build.ninja.in to be copied into the first
|
||||
# stage.
|
||||
sleep 2
|
||||
touch ../build.ninja.in
|
||||
testcase manifest
|
||||
|
||||
# From now on, we're going to be modifying the build.ninja.in, so let's make our
|
||||
# own copy
|
||||
sleep 2
|
||||
../tests/bootstrap.bash -r
|
||||
|
||||
sleep 2
|
||||
testcase start2
|
||||
|
||||
# This is similar to the last test, but incorporates a change into the source
|
||||
# build.ninja.in, so that we'll restart into the new version created by the
|
||||
# build.
|
||||
sleep 2
|
||||
echo "# test" >>src.build.ninja.in
|
||||
testcase regen
|
||||
|
||||
# Add tests to our build by using '-t'
|
||||
sleep 2
|
||||
../tests/bootstrap.bash -r -t
|
||||
|
||||
sleep 2
|
||||
testcase start_add_tests
|
||||
|
||||
# Make sure that updating a test file causes us to go back to the bootstrap
|
||||
# stage
|
||||
sleep 2
|
||||
touch ../parser/parser_test.go
|
||||
testcase rebuild_test
|
||||
|
||||
# Restart testing using the wrapper instead of going straight to ninja. This
|
||||
# will force each test to start in the correct bootstrap stage, so there are
|
||||
# less cases to test.
|
||||
cd ..
|
||||
rm -rf out.test
|
||||
mkdir -p out.test
|
||||
cd out.test
|
||||
../bootstrap.bash
|
||||
|
||||
testcase_wrapper start
|
||||
|
||||
# This test affects all bootstrap stages
|
||||
sleep 2
|
||||
touch ../Blueprints
|
||||
testcase_wrapper all
|
||||
|
||||
# From now on, we're going to be modifying the build.ninja.in, so let's make our
|
||||
# own copy
|
||||
sleep 2
|
||||
../tests/bootstrap.bash -r
|
||||
|
||||
sleep 2
|
||||
testcase_wrapper start2
|
||||
|
||||
# This is similar to the last test, but incorporates a change into the source
|
||||
# build.ninja.in, so that we'll restart into the new version created by the
|
||||
# build.
|
||||
sleep 2
|
||||
echo "# test" >>src.build.ninja.in
|
||||
testcase_wrapper regen
|
||||
|
||||
if [ -f failed ]; then
|
||||
cat failed
|
||||
if [[ -d .bootstrap/blueprint/test ]]; then
|
||||
echo "Tests should not be enabled here" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
sleep 2
|
||||
sed -i 's/${runTests}/-t/' src.build.ninja.in
|
||||
./blueprint.bash
|
||||
|
||||
if [[ ! -d .bootstrap/blueprint/test ]]; then
|
||||
echo "Tests should be enabled here" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if cmp -s src.build.ninja.in .minibootstrap/build.ninja.in; then
|
||||
echo "src.build.ninja.in and .minibootstrap/build.ninja.in should be different" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
sleep 2
|
||||
cp ../build.ninja.in src.build.ninja.in
|
||||
./blueprint.bash
|
||||
|
||||
if [[ -d .bootstrap/blueprint/test ]]; then
|
||||
echo "Tests should not be enabled here (2)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! cmp -s src.build.ninja.in .minibootstrap/build.ninja.in; then
|
||||
echo "src.build.ninja.in and .minibootstrap/build.ninja.in should be the same" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
|
|
|||
Loading…
Reference in a new issue