platform_build_blueprint/bootstrap/command.go

202 lines
4.9 KiB
Go
Raw Normal View History

// Copyright 2014 Google Inc. All rights reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package bootstrap
import (
"bytes"
"flag"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"runtime"
"runtime/debug"
"runtime/pprof"
"runtime/trace"
"github.com/google/blueprint"
"github.com/google/blueprint/deptools"
)
var (
Simplify bootstrap tl;dr: Read if you don't use the wrapper or use SKIP_NINJA Previously, we were relying on the ninja behavior of restarting the build when the build.ninja file was updated to switch between different bootstrap stages. But that means that every step that could produce a build.ninja must pass in order to switch to a different stage. That wasn't a big problem when we had a two stage build -- there was very little that could fail in the second stage before we chose to go back to the first stage. But when we had a three stage build, it was possible to get into a state (usually during development) where you were in the second stage, but the build was failing because the first stage needed to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82 by adding a wrapper that always started building at the first stage. But this kept all of the complexity of using ninja restarts without any of the benefits, so this change removes that complexity and just runs each stage sequentially in the wrapper. So the wrapper is now required. Since we're no longer going through choosestage, we can also skip the template parsing for the later stages that don't need to be templated -- this can save a couple of seconds for large files. In addition to all of the above, this also lets Soong reduce the number of times the main ninja file is loaded. We had been running the wrapper once (3 stages), then running ninja again after combining the Soong-generated build.ninja with the Kati-generated build.ninja. This change lets us removing the intermediate parsing of Soong's build.ninja, so that we only execute ninja 3 times per build. It also lets us have dependencies on pools or rules from Kati in the primary builder, since we're never executing the main build.ninja without the Kati build.ninja. The wrapper has a new option, NINJA to provide the path to ninja. This used to be hardcoded to `ninja`, and will still default to that. But we'll be running the first two bootstrap stages with $NINJA even if SKIP_NINJA is set. The wrapper passes "-w dupbuild=err" to ninja now -- this really should always be turned on if you care about reliable builds. Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
outFile string
depFile string
docFile string
cpuprofile string
memprofile string
Simplify bootstrap tl;dr: Read if you don't use the wrapper or use SKIP_NINJA Previously, we were relying on the ninja behavior of restarting the build when the build.ninja file was updated to switch between different bootstrap stages. But that means that every step that could produce a build.ninja must pass in order to switch to a different stage. That wasn't a big problem when we had a two stage build -- there was very little that could fail in the second stage before we chose to go back to the first stage. But when we had a three stage build, it was possible to get into a state (usually during development) where you were in the second stage, but the build was failing because the first stage needed to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82 by adding a wrapper that always started building at the first stage. But this kept all of the complexity of using ninja restarts without any of the benefits, so this change removes that complexity and just runs each stage sequentially in the wrapper. So the wrapper is now required. Since we're no longer going through choosestage, we can also skip the template parsing for the later stages that don't need to be templated -- this can save a couple of seconds for large files. In addition to all of the above, this also lets Soong reduce the number of times the main ninja file is loaded. We had been running the wrapper once (3 stages), then running ninja again after combining the Soong-generated build.ninja with the Kati-generated build.ninja. This change lets us removing the intermediate parsing of Soong's build.ninja, so that we only execute ninja 3 times per build. It also lets us have dependencies on pools or rules from Kati in the primary builder, since we're never executing the main build.ninja without the Kati build.ninja. The wrapper has a new option, NINJA to provide the path to ninja. This used to be hardcoded to `ninja`, and will still default to that. But we'll be running the first two bootstrap stages with $NINJA even if SKIP_NINJA is set. The wrapper passes "-w dupbuild=err" to ninja now -- this really should always be turned on if you care about reliable builds. Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
traceFile string
runGoTests bool
noGC bool
BuildDir string
Simplify bootstrap tl;dr: Read if you don't use the wrapper or use SKIP_NINJA Previously, we were relying on the ninja behavior of restarting the build when the build.ninja file was updated to switch between different bootstrap stages. But that means that every step that could produce a build.ninja must pass in order to switch to a different stage. That wasn't a big problem when we had a two stage build -- there was very little that could fail in the second stage before we chose to go back to the first stage. But when we had a three stage build, it was possible to get into a state (usually during development) where you were in the second stage, but the build was failing because the first stage needed to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82 by adding a wrapper that always started building at the first stage. But this kept all of the complexity of using ninja restarts without any of the benefits, so this change removes that complexity and just runs each stage sequentially in the wrapper. So the wrapper is now required. Since we're no longer going through choosestage, we can also skip the template parsing for the later stages that don't need to be templated -- this can save a couple of seconds for large files. In addition to all of the above, this also lets Soong reduce the number of times the main ninja file is loaded. We had been running the wrapper once (3 stages), then running ninja again after combining the Soong-generated build.ninja with the Kati-generated build.ninja. This change lets us removing the intermediate parsing of Soong's build.ninja, so that we only execute ninja 3 times per build. It also lets us have dependencies on pools or rules from Kati in the primary builder, since we're never executing the main build.ninja without the Kati build.ninja. The wrapper has a new option, NINJA to provide the path to ninja. This used to be hardcoded to `ninja`, and will still default to that. But we'll be running the first two bootstrap stages with $NINJA even if SKIP_NINJA is set. The wrapper passes "-w dupbuild=err" to ninja now -- this really should always be turned on if you care about reliable builds. Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
SrcDir string
)
func init() {
flag.StringVar(&outFile, "o", "build.ninja", "the Ninja file to output")
flag.StringVar(&BuildDir, "b", ".", "the build output directory")
flag.StringVar(&depFile, "d", "", "the dependency file to output")
flag.StringVar(&docFile, "docs", "", "build documentation file to output")
flag.StringVar(&cpuprofile, "cpuprofile", "", "write cpu profile to file")
flag.StringVar(&traceFile, "trace", "", "write trace to file")
flag.StringVar(&memprofile, "memprofile", "", "write memory profile to file")
flag.BoolVar(&noGC, "nogc", false, "turn off GC for debugging")
flag.BoolVar(&runGoTests, "t", false, "build and run go tests during bootstrap")
}
func Main(ctx *blueprint.Context, config interface{}, extraNinjaFileDeps ...string) {
if !flag.Parsed() {
flag.Parse()
}
runtime.GOMAXPROCS(runtime.NumCPU())
if noGC {
debug.SetGCPercent(-1)
}
if cpuprofile != "" {
f, err := os.Create(cpuprofile)
if err != nil {
fatalf("error opening cpuprofile: %s", err)
}
pprof.StartCPUProfile(f)
defer f.Close()
defer pprof.StopCPUProfile()
}
if traceFile != "" {
f, err := os.Create(traceFile)
if err != nil {
fatalf("error opening trace: %s", err)
}
trace.Start(f)
defer f.Close()
defer trace.Stop()
}
if flag.NArg() != 1 {
fatalf("no Blueprints file specified")
}
Simplify bootstrap tl;dr: Read if you don't use the wrapper or use SKIP_NINJA Previously, we were relying on the ninja behavior of restarting the build when the build.ninja file was updated to switch between different bootstrap stages. But that means that every step that could produce a build.ninja must pass in order to switch to a different stage. That wasn't a big problem when we had a two stage build -- there was very little that could fail in the second stage before we chose to go back to the first stage. But when we had a three stage build, it was possible to get into a state (usually during development) where you were in the second stage, but the build was failing because the first stage needed to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82 by adding a wrapper that always started building at the first stage. But this kept all of the complexity of using ninja restarts without any of the benefits, so this change removes that complexity and just runs each stage sequentially in the wrapper. So the wrapper is now required. Since we're no longer going through choosestage, we can also skip the template parsing for the later stages that don't need to be templated -- this can save a couple of seconds for large files. In addition to all of the above, this also lets Soong reduce the number of times the main ninja file is loaded. We had been running the wrapper once (3 stages), then running ninja again after combining the Soong-generated build.ninja with the Kati-generated build.ninja. This change lets us removing the intermediate parsing of Soong's build.ninja, so that we only execute ninja 3 times per build. It also lets us have dependencies on pools or rules from Kati in the primary builder, since we're never executing the main build.ninja without the Kati build.ninja. The wrapper has a new option, NINJA to provide the path to ninja. This used to be hardcoded to `ninja`, and will still default to that. But we'll be running the first two bootstrap stages with $NINJA even if SKIP_NINJA is set. The wrapper passes "-w dupbuild=err" to ninja now -- this really should always be turned on if you care about reliable builds. Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
SrcDir = filepath.Dir(flag.Arg(0))
Enhance bootstrap stage selection This simplifies the bootstrap process while making it more flexible by moving the stage selection into a go binary(choosestage). It will now be possible to have more than two build stages. Now each stage has a ninja template(main.ninja.in) and a timestamp file(main.ninja.in.timestamp). The timestamp file may be updated by any build stage that wishes to regenerate the ninja template. If the choosestage binaries sees that the timestamp is newer than the template, it will choose the prior stage. The main stage no longer writes to the source tree to update the build.ninja.in file. This was a problem for read-only source trees. Instead, the choosestage binary first checks to see if that file is newer than the last bootstrap.ninja.in, copies it in place, and starts the boostrap stage. The bootstrap stage regenerates it's own ninja template, but that required a loop through the main stage to actually run it. The choosestage binary now detects if the template has changed for the current stage, and will restart the stage. One change is that if dependencies do get messed up, instead of silently failing, there's a higher chance that the bootstrap step will just continue looping, doing nothing. This can happen if the main stage has a dependency that triggers the bootstrap stage, but the bootstrap stage doesn't see anything required to rebuild the main ninja file. A side effect of this requirement is that changes to test code will now rebuild the main ninja file. Change-Id: I9965cfba79dc0dbbd3af05f5944f7653054455a2
2015-07-23 02:05:59 +02:00
stage := StageMain
if c, ok := config.(ConfigInterface); ok {
if c.GeneratingPrimaryBuilder() {
stage = StagePrimary
}
}
bootstrapConfig := &Config{
Enhance bootstrap stage selection This simplifies the bootstrap process while making it more flexible by moving the stage selection into a go binary(choosestage). It will now be possible to have more than two build stages. Now each stage has a ninja template(main.ninja.in) and a timestamp file(main.ninja.in.timestamp). The timestamp file may be updated by any build stage that wishes to regenerate the ninja template. If the choosestage binaries sees that the timestamp is newer than the template, it will choose the prior stage. The main stage no longer writes to the source tree to update the build.ninja.in file. This was a problem for read-only source trees. Instead, the choosestage binary first checks to see if that file is newer than the last bootstrap.ninja.in, copies it in place, and starts the boostrap stage. The bootstrap stage regenerates it's own ninja template, but that required a loop through the main stage to actually run it. The choosestage binary now detects if the template has changed for the current stage, and will restart the stage. One change is that if dependencies do get messed up, instead of silently failing, there's a higher chance that the bootstrap step will just continue looping, doing nothing. This can happen if the main stage has a dependency that triggers the bootstrap stage, but the bootstrap stage doesn't see anything required to rebuild the main ninja file. A side effect of this requirement is that changes to test code will now rebuild the main ninja file. Change-Id: I9965cfba79dc0dbbd3af05f5944f7653054455a2
2015-07-23 02:05:59 +02:00
stage: stage,
topLevelBlueprintsFile: flag.Arg(0),
runGoTests: runGoTests,
}
ctx.RegisterBottomUpMutator("bootstrap_plugin_deps", pluginDeps)
ctx.RegisterModuleType("bootstrap_go_package", newGoPackageModuleFactory(bootstrapConfig))
ctx.RegisterModuleType("bootstrap_go_binary", newGoBinaryModuleFactory(bootstrapConfig, false))
ctx.RegisterModuleType("blueprint_go_binary", newGoBinaryModuleFactory(bootstrapConfig, true))
ctx.RegisterSingletonType("bootstrap", newSingletonFactory(bootstrapConfig))
ctx.RegisterSingletonType("glob", globSingletonFactory(ctx))
deps, errs := ctx.ParseBlueprintsFiles(bootstrapConfig.topLevelBlueprintsFile)
if len(errs) > 0 {
fatalErrors(errs)
}
// Add extra ninja file dependencies
deps = append(deps, extraNinjaFileDeps...)
errs = ctx.ResolveDependencies(config)
if len(errs) > 0 {
fatalErrors(errs)
}
if docFile != "" {
err := writeDocs(ctx, filepath.Dir(bootstrapConfig.topLevelBlueprintsFile), docFile)
if err != nil {
fatalErrors([]error{err})
}
return
}
extraDeps, errs := ctx.PrepareBuildActions(config)
if len(errs) > 0 {
fatalErrors(errs)
}
deps = append(deps, extraDeps...)
buf := bytes.NewBuffer(nil)
err := ctx.WriteBuildFile(buf)
if err != nil {
fatalf("error generating Ninja file contents: %s", err)
}
const outFilePermissions = 0666
Enhance bootstrap stage selection This simplifies the bootstrap process while making it more flexible by moving the stage selection into a go binary(choosestage). It will now be possible to have more than two build stages. Now each stage has a ninja template(main.ninja.in) and a timestamp file(main.ninja.in.timestamp). The timestamp file may be updated by any build stage that wishes to regenerate the ninja template. If the choosestage binaries sees that the timestamp is newer than the template, it will choose the prior stage. The main stage no longer writes to the source tree to update the build.ninja.in file. This was a problem for read-only source trees. Instead, the choosestage binary first checks to see if that file is newer than the last bootstrap.ninja.in, copies it in place, and starts the boostrap stage. The bootstrap stage regenerates it's own ninja template, but that required a loop through the main stage to actually run it. The choosestage binary now detects if the template has changed for the current stage, and will restart the stage. One change is that if dependencies do get messed up, instead of silently failing, there's a higher chance that the bootstrap step will just continue looping, doing nothing. This can happen if the main stage has a dependency that triggers the bootstrap stage, but the bootstrap stage doesn't see anything required to rebuild the main ninja file. A side effect of this requirement is that changes to test code will now rebuild the main ninja file. Change-Id: I9965cfba79dc0dbbd3af05f5944f7653054455a2
2015-07-23 02:05:59 +02:00
err = ioutil.WriteFile(outFile, buf.Bytes(), outFilePermissions)
if err != nil {
fatalf("error writing %s: %s", outFile, err)
}
if depFile != "" {
err := deptools.WriteDepFile(depFile, outFile, deps)
if err != nil {
fatalf("error writing depfile: %s", err)
}
}
if c, ok := config.(ConfigRemoveAbandonedFiles); !ok || c.RemoveAbandonedFiles() {
Simplify bootstrap tl;dr: Read if you don't use the wrapper or use SKIP_NINJA Previously, we were relying on the ninja behavior of restarting the build when the build.ninja file was updated to switch between different bootstrap stages. But that means that every step that could produce a build.ninja must pass in order to switch to a different stage. That wasn't a big problem when we had a two stage build -- there was very little that could fail in the second stage before we chose to go back to the first stage. But when we had a three stage build, it was possible to get into a state (usually during development) where you were in the second stage, but the build was failing because the first stage needed to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82 by adding a wrapper that always started building at the first stage. But this kept all of the complexity of using ninja restarts without any of the benefits, so this change removes that complexity and just runs each stage sequentially in the wrapper. So the wrapper is now required. Since we're no longer going through choosestage, we can also skip the template parsing for the later stages that don't need to be templated -- this can save a couple of seconds for large files. In addition to all of the above, this also lets Soong reduce the number of times the main ninja file is loaded. We had been running the wrapper once (3 stages), then running ninja again after combining the Soong-generated build.ninja with the Kati-generated build.ninja. This change lets us removing the intermediate parsing of Soong's build.ninja, so that we only execute ninja 3 times per build. It also lets us have dependencies on pools or rules from Kati in the primary builder, since we're never executing the main build.ninja without the Kati build.ninja. The wrapper has a new option, NINJA to provide the path to ninja. This used to be hardcoded to `ninja`, and will still default to that. But we'll be running the first two bootstrap stages with $NINJA even if SKIP_NINJA is set. The wrapper passes "-w dupbuild=err" to ninja now -- this really should always be turned on if you care about reliable builds. Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
err := removeAbandonedFiles(ctx, bootstrapConfig, SrcDir)
if err != nil {
fatalf("error removing abandoned files: %s", err)
}
}
if memprofile != "" {
f, err := os.Create(memprofile)
if err != nil {
fatalf("error opening memprofile: %s", err)
}
defer f.Close()
pprof.WriteHeapProfile(f)
}
}
func fatalf(format string, args ...interface{}) {
fmt.Printf(format, args...)
fmt.Print("\n")
os.Exit(1)
}
func fatalErrors(errs []error) {
red := "\x1b[31m"
unred := "\x1b[0m"
for _, err := range errs {
switch err := err.(type) {
case *blueprint.BlueprintError,
*blueprint.ModuleError,
*blueprint.PropertyError:
fmt.Printf("%serror:%s %s\n", red, unred, err.Error())
default:
fmt.Printf("%sinternal error:%s %s\n", red, unred, err)
}
}
os.Exit(1)
}