platform_build_soong/ui/build/environment.go

188 lines
4.9 KiB
Go
Raw Normal View History

Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
// Copyright 2017 Google Inc. All rights reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package build
import (
"bufio"
"fmt"
"io"
"os"
"strconv"
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
"strings"
)
// Environment adds a number of useful manipulation functions to the list of
// strings returned by os.Environ() and used in exec.Cmd.Env.
type Environment []string
// OsEnvironment wraps the current environment returned by os.Environ()
func OsEnvironment() *Environment {
env := Environment(os.Environ())
return &env
}
// Get returns the value associated with the key, and whether it exists.
// It's equivalent to the os.LookupEnv function, but with this copy of the
// Environment.
func (e *Environment) Get(key string) (string, bool) {
for _, env := range *e {
if k, v, ok := decodeKeyValue(env); ok && k == key {
return v, true
}
}
return "", false
}
// Get returns the int value associated with the key, and whether it exists
// and is a valid int.
func (e *Environment) GetInt(key string) (int, bool) {
if v, ok := e.Get(key); ok {
if i, err := strconv.Atoi(v); err == nil {
return i, true
}
}
return 0, false
}
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
// Set sets the value associated with the key, overwriting the current value
// if it exists.
func (e *Environment) Set(key, value string) {
e.Unset(key)
*e = append(*e, key+"="+value)
}
// Unset removes the specified keys from the Environment.
func (e *Environment) Unset(keys ...string) {
out := (*e)[:0]
for _, env := range *e {
if key, _, ok := decodeKeyValue(env); ok && inList(key, keys) {
continue
}
out = append(out, env)
}
*e = out
}
// UnsetWithPrefix removes all keys that start with prefix.
func (e *Environment) UnsetWithPrefix(prefix string) {
out := (*e)[:0]
for _, env := range *e {
if key, _, ok := decodeKeyValue(env); ok && strings.HasPrefix(key, prefix) {
continue
}
out = append(out, env)
}
*e = out
}
Add a Kati-based packaging step The idea is that we'd move the installation and packaging tasks over to it, using data from Soong & the Kati reading Android.mk files. This would allow us to make more fundamental changes about how we package things without having to adjust makefiles throughout the tree. Possible use cases: * Moving some information from Soong's Android.mk output to a file read by the packaging step may allow us to read the Android.mk files less often, speeding up builds. * Refactoring our current two-stage ASAN builds to run the Kati build step twice, writing into different object directories, then have a single packaging step that reads both outputs. Soong already has the capability of writing out a single ninja file with all the asan combinations. * Running two build steps, one building the system-related modules using a "generic" device configuration, and one building the vendor modules using a specific device configuration. This could enforce a GSI/mainline system vs vendor split in a single build invocation. * If all installation is through this tool, it will be much easier to track what should no longer be installed on an incremental build, reducing the need for installclean. * Changing PRODUCT_PACKAGES should be a much faster operation, which means we could keep track of local additions to the images. Then `mma` would be more persistent, instead of installing something once, then never updating it again. Eventually we plan on switching from Kati to something Go-based, but this is a more incremental approach while we clean up everything else. Currently, this just moves the dist-for-goal handling over to the packaging step, so that we don't need to read Android.mk files when DIST_DIR changes, or we switch between dist vs not. Bug: 116968624 Bug: 117463001 Test: m nothing Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
// Allow removes all keys that are not present in the input list
func (e *Environment) Allow(keys ...string) {
out := (*e)[:0]
for _, env := range *e {
if key, _, ok := decodeKeyValue(env); ok && inList(key, keys) {
out = append(out, env)
}
}
*e = out
}
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
// Environ returns the []string required for exec.Cmd.Env
func (e *Environment) Environ() []string {
return []string(*e)
}
// Copy returns a copy of the Environment so that independent changes may be made.
func (e *Environment) Copy() *Environment {
ret := Environment(make([]string, len(*e)))
for i, v := range *e {
ret[i] = v
}
return &ret
}
// IsTrue returns whether an environment variable is set to a positive value (1,y,yes,on,true)
func (e *Environment) IsEnvTrue(key string) bool {
if value, ok := e.Get(key); ok {
return value == "1" || value == "y" || value == "yes" || value == "on" || value == "true"
}
return false
}
// IsFalse returns whether an environment variable is set to a negative value (0,n,no,off,false)
func (e *Environment) IsFalse(key string) bool {
if value, ok := e.Get(key); ok {
return value == "0" || value == "n" || value == "no" || value == "off" || value == "false"
}
return false
}
// AppendFromKati reads a shell script written by Kati that exports or unsets
// environment variables, and applies those to the local Environment.
func (e *Environment) AppendFromKati(filename string) error {
file, err := os.Open(filename)
if err != nil {
return err
}
defer file.Close()
return e.appendFromKati(file)
}
func (e *Environment) appendFromKati(reader io.Reader) error {
scanner := bufio.NewScanner(reader)
for scanner.Scan() {
text := strings.TrimSpace(scanner.Text())
if len(text) == 0 || text[0] == '#' {
continue
}
cmd := strings.SplitN(text, " ", 2)
if len(cmd) != 2 {
return fmt.Errorf("Unknown kati environment line: %q", text)
}
if cmd[0] == "unset" {
str, ok := singleUnquote(cmd[1])
if !ok {
return fmt.Errorf("Failed to unquote kati line: %q", text)
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
}
e.Unset(str)
} else if cmd[0] == "export" {
key, value, ok := decodeKeyValue(cmd[1])
if !ok {
return fmt.Errorf("Failed to parse export: %v", cmd)
}
key, ok = singleUnquote(key)
if !ok {
return fmt.Errorf("Failed to unquote kati line: %q", text)
}
value, ok = singleUnquote(value)
if !ok {
return fmt.Errorf("Failed to unquote kati line: %q", text)
}
e.Set(key, value)
} else {
return fmt.Errorf("Unknown kati environment command: %q", text)
}
}
if err := scanner.Err(); err != nil {
return err
}
return nil
}