platform_build_soong/ui/build/kati.go

367 lines
12 KiB
Go
Raw Normal View History

Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
// Copyright 2017 Google Inc. All rights reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package build
import (
"crypto/md5"
"fmt"
"io/ioutil"
"os"
"os/user"
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
"path/filepath"
"strings"
Add a unified status reporting UI This adds a new status package that merges the running of "actions" (ninja calls them edges) of multiple tools into one view of the current state, and gives that to a number of different outputs. For inputs: Kati's output parser has been rewritten (and moved) to map onto the StartAction/FinishAction API. A byproduct of this is that the build servers should be able to extract errors from Kati better, since they look like the errors that Ninja used to write. Ninja is no longer directly connected to the terminal, but its output is read via the protobuf frontend API, so it's just another tool whose output becomes merged together. multiproduct_kati loses its custom status routines, and uses the common one instead. For outputs: The primary output is the ui/terminal.Status type, which along with ui/terminal.Writer now controls everything about the terminal output. Today, this doesn't really change any behaviors, but having all terminal output going through here allows a more complicated (multi-line / full window) status display in the future. The tracer acts as an output of the status package, tracing all the action start / finish events. This replaces reading the .ninja_log file, so it now properly handles multiple output files from a single action. A new rotated log file (out/error.log, or out/dist/logs/error.log) just contains a description of all of the errors that happened during the current build. Another new compressed and rotated log file (out/verbose.log.gz, or out/dist/logs/verbose.log.gz) contains the full verbose (showcommands) log of every execution run by the build. Since this is now written on every build, the showcommands argument is now ignored -- if you want to get the commands run, look at the log file after the build. Test: m Test: <built-in tests> Test: NINJA_ARGS="-t list" m Test: check the build.trace.gz Test: check the new log files Change-Id: If1d8994890d43ef68f65aa10ddd8e6e06dc7013a
2018-05-18 01:37:09 +02:00
"android/soong/ui/metrics"
Add a unified status reporting UI This adds a new status package that merges the running of "actions" (ninja calls them edges) of multiple tools into one view of the current state, and gives that to a number of different outputs. For inputs: Kati's output parser has been rewritten (and moved) to map onto the StartAction/FinishAction API. A byproduct of this is that the build servers should be able to extract errors from Kati better, since they look like the errors that Ninja used to write. Ninja is no longer directly connected to the terminal, but its output is read via the protobuf frontend API, so it's just another tool whose output becomes merged together. multiproduct_kati loses its custom status routines, and uses the common one instead. For outputs: The primary output is the ui/terminal.Status type, which along with ui/terminal.Writer now controls everything about the terminal output. Today, this doesn't really change any behaviors, but having all terminal output going through here allows a more complicated (multi-line / full window) status display in the future. The tracer acts as an output of the status package, tracing all the action start / finish events. This replaces reading the .ninja_log file, so it now properly handles multiple output files from a single action. A new rotated log file (out/error.log, or out/dist/logs/error.log) just contains a description of all of the errors that happened during the current build. Another new compressed and rotated log file (out/verbose.log.gz, or out/dist/logs/verbose.log.gz) contains the full verbose (showcommands) log of every execution run by the build. Since this is now written on every build, the showcommands argument is now ignored -- if you want to get the commands run, look at the log file after the build. Test: m Test: <built-in tests> Test: NINJA_ARGS="-t list" m Test: check the build.trace.gz Test: check the new log files Change-Id: If1d8994890d43ef68f65aa10ddd8e6e06dc7013a
2018-05-18 01:37:09 +02:00
"android/soong/ui/status"
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
)
var spaceSlashReplacer = strings.NewReplacer("/", "_", " ", "_")
const katiBuildSuffix = ""
const katiCleanspecSuffix = "-cleanspec"
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
const katiPackageSuffix = "-package"
// genKatiSuffix creates a filename suffix for kati-generated files so that we
// can cache them based on their inputs. Such files include the generated Ninja
// files and env.sh environment variable setup files.
//
// The filename suffix should encode all common changes to Kati inputs.
// Currently that includes the TARGET_PRODUCT and kati-processed command line
// arguments.
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
func genKatiSuffix(ctx Context, config Config) {
// Construct the base suffix.
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
katiSuffix := "-" + config.TargetProduct()
// Append kati arguments to the suffix.
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
if args := config.KatiArgs(); len(args) > 0 {
katiSuffix += "-" + spaceSlashReplacer.Replace(strings.Join(args, "_"))
}
// If the suffix is too long, replace it with a md5 hash and write a
// file that contains the original suffix.
if len(katiSuffix) > 64 {
shortSuffix := "-" + fmt.Sprintf("%x", md5.Sum([]byte(katiSuffix)))
config.SetKatiSuffix(shortSuffix)
ctx.Verbosef("Kati ninja suffix too long: %q", katiSuffix)
ctx.Verbosef("Replacing with: %q", shortSuffix)
if err := ioutil.WriteFile(strings.TrimSuffix(config.KatiBuildNinjaFile(), "ninja")+"suf", []byte(katiSuffix), 0777); err != nil {
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
ctx.Println("Error writing suffix file:", err)
}
} else {
config.SetKatiSuffix(katiSuffix)
}
}
// Base function to construct and run the Kati command line with additional
// arguments, and a custom function closure to mutate the environment Kati runs
// in.
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
func runKati(ctx Context, config Config, extraSuffix string, args []string, envFunc func(*Environment)) {
executable := config.PrebuiltBuildTool("ckati")
// cKati arguments.
args = append([]string{
// Instead of executing commands directly, generate a Ninja file.
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
"--ninja",
// Generate Ninja files in the output directory.
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
"--ninja_dir=" + config.OutDir(),
// Filename suffix of the generated Ninja file.
"--ninja_suffix=" + config.KatiSuffix() + extraSuffix,
// Remove common parts at the beginning of a Ninja file, like build_dir,
// local_pool and _kati_always_build_. Allows Kati to be run multiple
// times, with generated Ninja files combined in a single invocation
// using 'include'.
"--no_ninja_prelude",
// Support declaring phony outputs in AOSP Ninja.
"--use_ninja_phony_output",
// Support declaring symlink outputs in AOSP Ninja.
"--use_ninja_symlink_outputs",
// Regenerate the Ninja file if environment inputs have changed. e.g.
// CLI flags, .mk file timestamps, env vars, $(wildcard ..) and some
// $(shell ..) results.
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
"--regen",
// Skip '-include' directives starting with the specified path. Used to
// ignore generated .mk files.
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
"--ignore_optional_include=" + filepath.Join(config.OutDir(), "%.P"),
// Detect the use of $(shell echo ...).
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
"--detect_android_echo",
// Colorful ANSI-based warning and error messages.
"--color_warnings",
// Generate all targets, not just the top level requested ones.
"--gen_all_targets",
// Use the built-in emulator of GNU find for better file finding
// performance. Used with $(shell find ...).
"--use_find_emulator",
// Fail when the find emulator encounters problems.
"--werror_find_emulator",
// Do not provide any built-in rules.
"--no_builtin_rules",
// Fail when suffix rules are used.
"--werror_suffix_rules",
// Fail when a real target depends on a phony target.
"--werror_real_to_phony",
// Makes real_to_phony checks assume that any top-level or leaf
// dependencies that does *not* have a '/' in it is a phony target.
"--top_level_phony",
// Fail when a phony target contains slashes.
"--werror_phony_looks_real",
// Fail when writing to a read-only directory.
"--werror_writable",
// Print Kati's internal statistics, such as the number of variables,
// implicit/explicit/suffix rules, and so on.
"--kati_stats",
}, args...)
// Generate a minimal Ninja file.
//
// Used for build_test and multiproduct_kati, which runs Kati several
// hundred times for different configurations to test file generation logic.
// These can result in generating Ninja files reaching ~1GB or more,
// resulting in ~hundreds of GBs of writes.
//
// Since we don't care about executing the Ninja files in these test cases,
// generating the Ninja file content wastes time, so skip writing any
// information out with --empty_ninja_file.
//
// From https://github.com/google/kati/commit/87b8da7af2c8bea28b1d8ab17679453d859f96e5
if config.EmptyNinjaFile() {
args = append(args, "--empty_ninja_file")
}
// Apply 'local_pool' to to all rules that don't specify a pool.
if config.UseRemoteBuild() {
args = append(args, "--default_pool=local_pool")
}
cmd := Command(ctx, config, "ckati", executable, args...)
// Set up the nsjail sandbox.
cmd.Sandbox = katiSandbox
// Set up stdout and stderr.
pipe, err := cmd.StdoutPipe()
if err != nil {
ctx.Fatalln("Error getting output pipe for ckati:", err)
}
cmd.Stderr = cmd.Stdout
// Apply the caller's function closure to mutate the environment variables.
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
envFunc(cmd.Environment)
// Pass on various build environment metadata to Kati.
if _, ok := cmd.Environment.Get("BUILD_USERNAME"); !ok {
username := "unknown"
if u, err := user.Current(); err == nil {
username = u.Username
} else {
ctx.Println("Failed to get current user:", err)
}
cmd.Environment.Set("BUILD_USERNAME", username)
}
if _, ok := cmd.Environment.Get("BUILD_HOSTNAME"); !ok {
hostname, err := os.Hostname()
if err != nil {
ctx.Println("Failed to read hostname:", err)
hostname = "unknown"
}
cmd.Environment.Set("BUILD_HOSTNAME", hostname)
}
cmd.StartOrFatal()
// Set up the ToolStatus command line reader for Kati for a consistent UI
// for the user.
status.KatiReader(ctx.Status.StartTool(), pipe)
cmd.WaitOrFatal()
}
func runKatiBuild(ctx Context, config Config) {
ctx.BeginTrace(metrics.RunKati, "kati build")
defer ctx.EndTrace()
args := []string{
// Mark the output directory as writable.
"--writable", config.OutDir() + "/",
// Fail when encountering implicit rules. e.g.
// %.foo: %.bar
// cp $< $@
"--werror_implicit_rules",
// Entry point for the Kati Ninja file generation.
"-f", "build/make/core/main.mk",
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
}
if !config.BuildBrokenDupRules() {
// Fail when redefining / duplicating a target.
args = append(args, "--werror_overriding_commands")
}
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
args = append(args, config.KatiArgs()...)
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
args = append(args,
// Location of the Make vars .mk file generated by Soong.
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
"SOONG_MAKEVARS_MK="+config.SoongMakeVarsMk(),
// Location of the Android.mk file generated by Soong. This
// file contains Soong modules represented as Kati modules,
// allowing Kati modules to depend on Soong modules.
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
"SOONG_ANDROID_MK="+config.SoongAndroidMk(),
// Directory containing outputs for the target device.
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
"TARGET_DEVICE_DIR="+config.TargetDeviceDir(),
// Directory containing .mk files for packaging purposes, such as
// the dist.mk file, containing dist-for-goals data.
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
"KATI_PACKAGE_MK_DIR="+config.KatiPackageMkDir())
runKati(ctx, config, katiBuildSuffix, args, func(env *Environment) {})
// compress and dist the main build ninja file.
distGzipFile(ctx, config, config.KatiBuildNinjaFile())
// Cleanup steps.
cleanCopyHeaders(ctx, config)
cleanOldInstalledFiles(ctx, config)
}
// Clean out obsolete header files on the disk that were *not copied* during the
// build with BUILD_COPY_HEADERS and LOCAL_COPY_HEADERS.
//
// These should be increasingly uncommon, as it's a deprecated feature and there
// isn't an equivalent feature in Soong.
func cleanCopyHeaders(ctx Context, config Config) {
ctx.BeginTrace("clean", "clean copy headers")
defer ctx.EndTrace()
// Read and parse the list of copied headers from a file in the product
// output directory.
data, err := ioutil.ReadFile(filepath.Join(config.ProductOut(), ".copied_headers_list"))
if err != nil {
if os.IsNotExist(err) {
return
}
ctx.Fatalf("Failed to read copied headers list: %v", err)
}
headers := strings.Fields(string(data))
if len(headers) < 1 {
ctx.Fatal("Failed to parse copied headers list: %q", string(data))
}
headerDir := headers[0]
headers = headers[1:]
// Walk the tree and remove any headers that are not in the list of copied
// headers in the current build.
filepath.Walk(headerDir,
func(path string, info os.FileInfo, err error) error {
if err != nil {
return nil
}
if info.IsDir() {
return nil
}
if !inList(path, headers) {
ctx.Printf("Removing obsolete header %q", path)
if err := os.Remove(path); err != nil {
ctx.Fatalf("Failed to remove obsolete header %q: %v", path, err)
}
}
return nil
})
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
}
// Clean out any previously installed files from the disk that are not installed
// in the current build.
func cleanOldInstalledFiles(ctx Context, config Config) {
ctx.BeginTrace("clean", "clean old installed files")
defer ctx.EndTrace()
// We shouldn't be removing files from one side of the two-step asan builds
var suffix string
if v, ok := config.Environment().Get("SANITIZE_TARGET"); ok {
if sanitize := strings.Fields(v); inList("address", sanitize) {
suffix = "_asan"
}
}
cleanOldFiles(ctx, config.ProductOut(), ".installable_files"+suffix)
cleanOldFiles(ctx, config.HostOut(), ".installable_test_files")
}
// Generate the Ninja file containing the packaging command lines for the dist
// dir.
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
func runKatiPackage(ctx Context, config Config) {
ctx.BeginTrace(metrics.RunKati, "kati package")
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
defer ctx.EndTrace()
args := []string{
// Mark the dist dir as writable.
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
"--writable", config.DistDir() + "/",
// Fail when encountering implicit rules. e.g.
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
"--werror_implicit_rules",
// Fail when redefining / duplicating a target.
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
"--werror_overriding_commands",
// Entry point.
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
"-f", "build/make/packaging/main.mk",
// Directory containing .mk files for packaging purposes, such as
// the dist.mk file, containing dist-for-goals data.
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
"KATI_PACKAGE_MK_DIR=" + config.KatiPackageMkDir(),
}
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
// Run Kati against a restricted set of environment variables.
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
runKati(ctx, config, katiPackageSuffix, args, func(env *Environment) {
env.Allow([]string{
// Some generic basics
"LANG",
"LC_MESSAGES",
"PATH",
"PWD",
"TMPDIR",
// Tool configs
"ASAN_SYMBOLIZER_PATH",
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
"JAVA_HOME",
"PYTHONDONTWRITEBYTECODE",
// Build configuration
"ANDROID_BUILD_SHELL",
"DIST_DIR",
"OUT_DIR",
}...)
if config.Dist() {
env.Set("DIST", "true")
env.Set("DIST_DIR", config.DistDir())
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
}
})
// Compress and dist the packaging Ninja file.
distGzipFile(ctx, config, config.KatiPackageNinjaFile())
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
}
// Run Kati on the cleanspec files to clean the build.
func runKatiCleanSpec(ctx Context, config Config) {
ctx.BeginTrace(metrics.RunKati, "kati cleanspec")
defer ctx.EndTrace()
runKati(ctx, config, katiCleanspecSuffix, []string{
// Fail when encountering implicit rules. e.g.
"--werror_implicit_rules",
// Fail when redefining / duplicating a target.
"--werror_overriding_commands",
// Entry point.
"-f", "build/make/core/cleanbuild.mk",
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
"SOONG_MAKEVARS_MK=" + config.SoongMakeVarsMk(),
"TARGET_DEVICE_DIR=" + config.TargetDeviceDir(),
}, func(env *Environment) {})
}