platform_build_blueprint/bootstrap/config.go

117 lines
3.3 KiB
Go
Raw Normal View History

// Copyright 2014 Google Inc. All rights reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package bootstrap
Simplify bootstrap tl;dr: Read if you don't use the wrapper or use SKIP_NINJA Previously, we were relying on the ninja behavior of restarting the build when the build.ninja file was updated to switch between different bootstrap stages. But that means that every step that could produce a build.ninja must pass in order to switch to a different stage. That wasn't a big problem when we had a two stage build -- there was very little that could fail in the second stage before we chose to go back to the first stage. But when we had a three stage build, it was possible to get into a state (usually during development) where you were in the second stage, but the build was failing because the first stage needed to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82 by adding a wrapper that always started building at the first stage. But this kept all of the complexity of using ninja restarts without any of the benefits, so this change removes that complexity and just runs each stage sequentially in the wrapper. So the wrapper is now required. Since we're no longer going through choosestage, we can also skip the template parsing for the later stages that don't need to be templated -- this can save a couple of seconds for large files. In addition to all of the above, this also lets Soong reduce the number of times the main ninja file is loaded. We had been running the wrapper once (3 stages), then running ninja again after combining the Soong-generated build.ninja with the Kati-generated build.ninja. This change lets us removing the intermediate parsing of Soong's build.ninja, so that we only execute ninja 3 times per build. It also lets us have dependencies on pools or rules from Kati in the primary builder, since we're never executing the main build.ninja without the Kati build.ninja. The wrapper has a new option, NINJA to provide the path to ninja. This used to be hardcoded to `ninja`, and will still default to that. But we'll be running the first two bootstrap stages with $NINJA even if SKIP_NINJA is set. The wrapper passes "-w dupbuild=err" to ninja now -- this really should always be turned on if you care about reliable builds. Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
import (
"fmt"
"os"
"path/filepath"
Simplify bootstrap tl;dr: Read if you don't use the wrapper or use SKIP_NINJA Previously, we were relying on the ninja behavior of restarting the build when the build.ninja file was updated to switch between different bootstrap stages. But that means that every step that could produce a build.ninja must pass in order to switch to a different stage. That wasn't a big problem when we had a two stage build -- there was very little that could fail in the second stage before we chose to go back to the first stage. But when we had a three stage build, it was possible to get into a state (usually during development) where you were in the second stage, but the build was failing because the first stage needed to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82 by adding a wrapper that always started building at the first stage. But this kept all of the complexity of using ninja restarts without any of the benefits, so this change removes that complexity and just runs each stage sequentially in the wrapper. So the wrapper is now required. Since we're no longer going through choosestage, we can also skip the template parsing for the later stages that don't need to be templated -- this can save a couple of seconds for large files. In addition to all of the above, this also lets Soong reduce the number of times the main ninja file is loaded. We had been running the wrapper once (3 stages), then running ninja again after combining the Soong-generated build.ninja with the Kati-generated build.ninja. This change lets us removing the intermediate parsing of Soong's build.ninja, so that we only execute ninja 3 times per build. It also lets us have dependencies on pools or rules from Kati in the primary builder, since we're never executing the main build.ninja without the Kati build.ninja. The wrapper has a new option, NINJA to provide the path to ninja. This used to be hardcoded to `ninja`, and will still default to that. But we'll be running the first two bootstrap stages with $NINJA even if SKIP_NINJA is set. The wrapper passes "-w dupbuild=err" to ninja now -- this really should always be turned on if you care about reliable builds. Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
"runtime"
"strings"
Simplify bootstrap tl;dr: Read if you don't use the wrapper or use SKIP_NINJA Previously, we were relying on the ninja behavior of restarting the build when the build.ninja file was updated to switch between different bootstrap stages. But that means that every step that could produce a build.ninja must pass in order to switch to a different stage. That wasn't a big problem when we had a two stage build -- there was very little that could fail in the second stage before we chose to go back to the first stage. But when we had a three stage build, it was possible to get into a state (usually during development) where you were in the second stage, but the build was failing because the first stage needed to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82 by adding a wrapper that always started building at the first stage. But this kept all of the complexity of using ninja restarts without any of the benefits, so this change removes that complexity and just runs each stage sequentially in the wrapper. So the wrapper is now required. Since we're no longer going through choosestage, we can also skip the template parsing for the later stages that don't need to be templated -- this can save a couple of seconds for large files. In addition to all of the above, this also lets Soong reduce the number of times the main ninja file is loaded. We had been running the wrapper once (3 stages), then running ninja again after combining the Soong-generated build.ninja with the Kati-generated build.ninja. This change lets us removing the intermediate parsing of Soong's build.ninja, so that we only execute ninja 3 times per build. It also lets us have dependencies on pools or rules from Kati in the primary builder, since we're never executing the main build.ninja without the Kati build.ninja. The wrapper has a new option, NINJA to provide the path to ninja. This used to be hardcoded to `ninja`, and will still default to that. But we'll be running the first two bootstrap stages with $NINJA even if SKIP_NINJA is set. The wrapper passes "-w dupbuild=err" to ninja now -- this really should always be turned on if you care about reliable builds. Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
"github.com/google/blueprint"
)
func bootstrapVariable(name string, value func(BootstrapConfig) string) blueprint.Variable {
return pctx.VariableFunc(name, func(ctx blueprint.VariableFuncContext, config interface{}) (string, error) {
c, ok := config.(BootstrapConfig)
if !ok {
panic(fmt.Sprintf("Bootstrap rules were passed a configuration that does not include theirs, config=%q",
config))
}
return value(c), nil
Simplify bootstrap tl;dr: Read if you don't use the wrapper or use SKIP_NINJA Previously, we were relying on the ninja behavior of restarting the build when the build.ninja file was updated to switch between different bootstrap stages. But that means that every step that could produce a build.ninja must pass in order to switch to a different stage. That wasn't a big problem when we had a two stage build -- there was very little that could fail in the second stage before we chose to go back to the first stage. But when we had a three stage build, it was possible to get into a state (usually during development) where you were in the second stage, but the build was failing because the first stage needed to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82 by adding a wrapper that always started building at the first stage. But this kept all of the complexity of using ninja restarts without any of the benefits, so this change removes that complexity and just runs each stage sequentially in the wrapper. So the wrapper is now required. Since we're no longer going through choosestage, we can also skip the template parsing for the later stages that don't need to be templated -- this can save a couple of seconds for large files. In addition to all of the above, this also lets Soong reduce the number of times the main ninja file is loaded. We had been running the wrapper once (3 stages), then running ninja again after combining the Soong-generated build.ninja with the Kati-generated build.ninja. This change lets us removing the intermediate parsing of Soong's build.ninja, so that we only execute ninja 3 times per build. It also lets us have dependencies on pools or rules from Kati in the primary builder, since we're never executing the main build.ninja without the Kati build.ninja. The wrapper has a new option, NINJA to provide the path to ninja. This used to be hardcoded to `ninja`, and will still default to that. But we'll be running the first two bootstrap stages with $NINJA even if SKIP_NINJA is set. The wrapper passes "-w dupbuild=err" to ninja now -- this really should always be turned on if you care about reliable builds. Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
})
}
var (
// These variables are the only configuration needed by the bootstrap
// modules.
srcDirVariable = bootstrapVariable("srcDir", func(c BootstrapConfig) string {
return "."
Simplify bootstrap tl;dr: Read if you don't use the wrapper or use SKIP_NINJA Previously, we were relying on the ninja behavior of restarting the build when the build.ninja file was updated to switch between different bootstrap stages. But that means that every step that could produce a build.ninja must pass in order to switch to a different stage. That wasn't a big problem when we had a two stage build -- there was very little that could fail in the second stage before we chose to go back to the first stage. But when we had a three stage build, it was possible to get into a state (usually during development) where you were in the second stage, but the build was failing because the first stage needed to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82 by adding a wrapper that always started building at the first stage. But this kept all of the complexity of using ninja restarts without any of the benefits, so this change removes that complexity and just runs each stage sequentially in the wrapper. So the wrapper is now required. Since we're no longer going through choosestage, we can also skip the template parsing for the later stages that don't need to be templated -- this can save a couple of seconds for large files. In addition to all of the above, this also lets Soong reduce the number of times the main ninja file is loaded. We had been running the wrapper once (3 stages), then running ninja again after combining the Soong-generated build.ninja with the Kati-generated build.ninja. This change lets us removing the intermediate parsing of Soong's build.ninja, so that we only execute ninja 3 times per build. It also lets us have dependencies on pools or rules from Kati in the primary builder, since we're never executing the main build.ninja without the Kati build.ninja. The wrapper has a new option, NINJA to provide the path to ninja. This used to be hardcoded to `ninja`, and will still default to that. But we'll be running the first two bootstrap stages with $NINJA even if SKIP_NINJA is set. The wrapper passes "-w dupbuild=err" to ninja now -- this really should always be turned on if you care about reliable builds. Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
})
soongOutDirVariable = bootstrapVariable("soongOutDir", func(c BootstrapConfig) string {
return c.SoongOutDir()
Simplify bootstrap tl;dr: Read if you don't use the wrapper or use SKIP_NINJA Previously, we were relying on the ninja behavior of restarting the build when the build.ninja file was updated to switch between different bootstrap stages. But that means that every step that could produce a build.ninja must pass in order to switch to a different stage. That wasn't a big problem when we had a two stage build -- there was very little that could fail in the second stage before we chose to go back to the first stage. But when we had a three stage build, it was possible to get into a state (usually during development) where you were in the second stage, but the build was failing because the first stage needed to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82 by adding a wrapper that always started building at the first stage. But this kept all of the complexity of using ninja restarts without any of the benefits, so this change removes that complexity and just runs each stage sequentially in the wrapper. So the wrapper is now required. Since we're no longer going through choosestage, we can also skip the template parsing for the later stages that don't need to be templated -- this can save a couple of seconds for large files. In addition to all of the above, this also lets Soong reduce the number of times the main ninja file is loaded. We had been running the wrapper once (3 stages), then running ninja again after combining the Soong-generated build.ninja with the Kati-generated build.ninja. This change lets us removing the intermediate parsing of Soong's build.ninja, so that we only execute ninja 3 times per build. It also lets us have dependencies on pools or rules from Kati in the primary builder, since we're never executing the main build.ninja without the Kati build.ninja. The wrapper has a new option, NINJA to provide the path to ninja. This used to be hardcoded to `ninja`, and will still default to that. But we'll be running the first two bootstrap stages with $NINJA even if SKIP_NINJA is set. The wrapper passes "-w dupbuild=err" to ninja now -- this really should always be turned on if you care about reliable builds. Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
})
outDirVariable = bootstrapVariable("outDir", func(c BootstrapConfig) string {
return c.OutDir()
})
goRootVariable = bootstrapVariable("goRoot", func(c BootstrapConfig) string {
goroot := runtime.GOROOT()
// Prefer to omit absolute paths from the ninja file
if cwd, err := os.Getwd(); err == nil {
if relpath, err := filepath.Rel(cwd, goroot); err == nil {
if !strings.HasPrefix(relpath, "../") {
goroot = relpath
}
}
}
return goroot
Simplify bootstrap tl;dr: Read if you don't use the wrapper or use SKIP_NINJA Previously, we were relying on the ninja behavior of restarting the build when the build.ninja file was updated to switch between different bootstrap stages. But that means that every step that could produce a build.ninja must pass in order to switch to a different stage. That wasn't a big problem when we had a two stage build -- there was very little that could fail in the second stage before we chose to go back to the first stage. But when we had a three stage build, it was possible to get into a state (usually during development) where you were in the second stage, but the build was failing because the first stage needed to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82 by adding a wrapper that always started building at the first stage. But this kept all of the complexity of using ninja restarts without any of the benefits, so this change removes that complexity and just runs each stage sequentially in the wrapper. So the wrapper is now required. Since we're no longer going through choosestage, we can also skip the template parsing for the later stages that don't need to be templated -- this can save a couple of seconds for large files. In addition to all of the above, this also lets Soong reduce the number of times the main ninja file is loaded. We had been running the wrapper once (3 stages), then running ninja again after combining the Soong-generated build.ninja with the Kati-generated build.ninja. This change lets us removing the intermediate parsing of Soong's build.ninja, so that we only execute ninja 3 times per build. It also lets us have dependencies on pools or rules from Kati in the primary builder, since we're never executing the main build.ninja without the Kati build.ninja. The wrapper has a new option, NINJA to provide the path to ninja. This used to be hardcoded to `ninja`, and will still default to that. But we'll be running the first two bootstrap stages with $NINJA even if SKIP_NINJA is set. The wrapper passes "-w dupbuild=err" to ninja now -- this really should always be turned on if you care about reliable builds. Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
})
compileCmdVariable = bootstrapVariable("compileCmd", func(c BootstrapConfig) string {
Simplify bootstrap tl;dr: Read if you don't use the wrapper or use SKIP_NINJA Previously, we were relying on the ninja behavior of restarting the build when the build.ninja file was updated to switch between different bootstrap stages. But that means that every step that could produce a build.ninja must pass in order to switch to a different stage. That wasn't a big problem when we had a two stage build -- there was very little that could fail in the second stage before we chose to go back to the first stage. But when we had a three stage build, it was possible to get into a state (usually during development) where you were in the second stage, but the build was failing because the first stage needed to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82 by adding a wrapper that always started building at the first stage. But this kept all of the complexity of using ninja restarts without any of the benefits, so this change removes that complexity and just runs each stage sequentially in the wrapper. So the wrapper is now required. Since we're no longer going through choosestage, we can also skip the template parsing for the later stages that don't need to be templated -- this can save a couple of seconds for large files. In addition to all of the above, this also lets Soong reduce the number of times the main ninja file is loaded. We had been running the wrapper once (3 stages), then running ninja again after combining the Soong-generated build.ninja with the Kati-generated build.ninja. This change lets us removing the intermediate parsing of Soong's build.ninja, so that we only execute ninja 3 times per build. It also lets us have dependencies on pools or rules from Kati in the primary builder, since we're never executing the main build.ninja without the Kati build.ninja. The wrapper has a new option, NINJA to provide the path to ninja. This used to be hardcoded to `ninja`, and will still default to that. But we'll be running the first two bootstrap stages with $NINJA even if SKIP_NINJA is set. The wrapper passes "-w dupbuild=err" to ninja now -- this really should always be turned on if you care about reliable builds. Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
return "$goRoot/pkg/tool/" + runtime.GOOS + "_" + runtime.GOARCH + "/compile"
})
linkCmdVariable = bootstrapVariable("linkCmd", func(c BootstrapConfig) string {
Simplify bootstrap tl;dr: Read if you don't use the wrapper or use SKIP_NINJA Previously, we were relying on the ninja behavior of restarting the build when the build.ninja file was updated to switch between different bootstrap stages. But that means that every step that could produce a build.ninja must pass in order to switch to a different stage. That wasn't a big problem when we had a two stage build -- there was very little that could fail in the second stage before we chose to go back to the first stage. But when we had a three stage build, it was possible to get into a state (usually during development) where you were in the second stage, but the build was failing because the first stage needed to be run. This was fixed in d79f1af7423e0ef7a13573efdae5100a57fabc82 by adding a wrapper that always started building at the first stage. But this kept all of the complexity of using ninja restarts without any of the benefits, so this change removes that complexity and just runs each stage sequentially in the wrapper. So the wrapper is now required. Since we're no longer going through choosestage, we can also skip the template parsing for the later stages that don't need to be templated -- this can save a couple of seconds for large files. In addition to all of the above, this also lets Soong reduce the number of times the main ninja file is loaded. We had been running the wrapper once (3 stages), then running ninja again after combining the Soong-generated build.ninja with the Kati-generated build.ninja. This change lets us removing the intermediate parsing of Soong's build.ninja, so that we only execute ninja 3 times per build. It also lets us have dependencies on pools or rules from Kati in the primary builder, since we're never executing the main build.ninja without the Kati build.ninja. The wrapper has a new option, NINJA to provide the path to ninja. This used to be hardcoded to `ninja`, and will still default to that. But we'll be running the first two bootstrap stages with $NINJA even if SKIP_NINJA is set. The wrapper passes "-w dupbuild=err" to ninja now -- this really should always be turned on if you care about reliable builds. Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-13 21:42:11 +02:00
return "$goRoot/pkg/tool/" + runtime.GOOS + "_" + runtime.GOARCH + "/link"
})
debugFlagsVariable = bootstrapVariable("debugFlags", func(c BootstrapConfig) string {
if c.DebugCompilation() {
// -N: disable optimizations, -l: disable inlining
return "-N -l"
} else {
return ""
}
})
)
type BootstrapConfig interface {
// The directory where tools run during the build are located.
HostToolDir() string
// The directory where files emitted during bootstrapping are located.
// Usually OutDir() + "/soong".
SoongOutDir() string
// The output directory for the build.
OutDir() string
// Whether to compile Go code in such a way that it can be debugged
DebugCompilation() bool
// Whether to run tests for Go code
RunGoTests() bool
Subninjas() []string
PrimaryBuilderInvocations() []PrimaryBuilderInvocation
}
type StopBefore int
const (
DoEverything StopBefore = iota
StopBeforePrepareBuildActions
StopBeforeWriteNinja
)
type PrimaryBuilderInvocation struct {
Inputs []string
Implicits []string
OrderOnlyInputs []string
Outputs []string
Args []string
Console bool
Description string
Env map[string]string
}