platform_build_soong/ui/build/util.go

173 lines
4.3 KiB
Go
Raw Normal View History

Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
// Copyright 2017 Google Inc. All rights reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package build
import (
"compress/gzip"
"fmt"
"io"
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
"os"
"path/filepath"
"strings"
)
func absPath(ctx Context, p string) string {
ret, err := filepath.Abs(p)
if err != nil {
ctx.Fatalf("Failed to get absolute path: %v", err)
}
return ret
}
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
// indexList finds the index of a string in a []string
func indexList(s string, list []string) int {
for i, l := range list {
if l == s {
return i
}
}
return -1
}
// inList determines whether a string is in a []string
func inList(s string, list []string) bool {
return indexList(s, list) != -1
}
// removeFromlist removes all occurrences of the string in list.
func removeFromList(s string, list []string) []string {
filteredList := make([]string, 0, len(list))
for _, ls := range list {
if s != ls {
filteredList = append(filteredList, ls)
}
}
return filteredList
}
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
// ensureDirectoriesExist is a shortcut to os.MkdirAll, sending errors to the ctx logger.
func ensureDirectoriesExist(ctx Context, dirs ...string) {
for _, dir := range dirs {
err := os.MkdirAll(dir, 0777)
if err != nil {
ctx.Fatalf("Error creating %s: %q\n", dir, err)
}
}
}
// ensureEmptyDirectoriesExist ensures that the given directories exist and are empty
func ensureEmptyDirectoriesExist(ctx Context, dirs ...string) {
// remove all the directories
for _, dir := range dirs {
seenErr := map[string]bool{}
for {
err := os.RemoveAll(dir)
if err == nil {
break
}
if pathErr, ok := err.(*os.PathError); !ok ||
dir == pathErr.Path || seenErr[pathErr.Path] {
ctx.Fatalf("Error removing %s: %q\n", dir, err)
} else {
seenErr[pathErr.Path] = true
err = os.Chmod(filepath.Dir(pathErr.Path), 0700)
if err != nil {
ctx.Fatal(err)
}
}
}
}
// recreate all the directories
ensureDirectoriesExist(ctx, dirs...)
}
Add a Go replacement for our top-level Make wrapper Right now this mostly just copies what Make is doing in build/core/ninja.mk and build/core/soong.mk. The only major feature it adds is a rotating log file with some verbose logging. There is one major functional difference -- you cannot override random Make variables during the Make phase anymore. The environment variable is set, and if Make uses ?= or the equivalent, it can still use those variables. We already made this change for Kati, which also loads all of the same code and actually does the build, so it has been half-removed for a while. The only "UI" this implements is what I'll call "Make Emulation" mode -- it's expected that current command lines will continue working, and we'll explore alternate user interfaces later. We're still using Make as a wrapper, but all it does is call into this single Go program, it won't even load the product configuration. Once this is default, we can start moving individual users over to using this directly (still in Make emulation mode), skipping the Make wrapper. Ideas for the future: * Generating trace files showing time spent in Make/Kati/Soong/Ninja (also importing ninja traces into the same stream). I had this working in a previous version of this patch, but removed it to keep the size down and focus on the current features. * More intelligent SIGALRM handling, once we fully remove the Make wrapper (which hides the SIGALRM) * Reading the experimental binary output stream from Ninja, so that we can always save the verbose log even if we're not printing it out to the console Test: USE_SOONG_UI=true m -j blueprint_tools Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
// ensureEmptyFileExists ensures that the containing directory exists, and the
// specified file exists. If it doesn't exist, it will write an empty file.
func ensureEmptyFileExists(ctx Context, file string) {
ensureDirectoriesExist(ctx, filepath.Dir(file))
if _, err := os.Stat(file); os.IsNotExist(err) {
f, err := os.Create(file)
if err != nil {
ctx.Fatalf("Error creating %s: %q\n", file, err)
}
f.Close()
} else if err != nil {
ctx.Fatalf("Error checking %s: %q\n", file, err)
}
}
// singleUnquote is similar to strconv.Unquote, but can handle multi-character strings inside single quotes.
func singleUnquote(str string) (string, bool) {
if len(str) < 2 || str[0] != '\'' || str[len(str)-1] != '\'' {
return "", false
}
return str[1 : len(str)-1], true
}
// decodeKeyValue decodes a key=value string
func decodeKeyValue(str string) (string, string, bool) {
idx := strings.IndexRune(str, '=')
if idx == -1 {
return "", "", false
}
return str[:idx], str[idx+1:], true
}
// copyFile copies a file from src to dst. filepath.Dir(dst) must exist.
func copyFile(src, dst string) (int64, error) {
source, err := os.Open(src)
if err != nil {
return 0, err
}
defer source.Close()
destination, err := os.Create(dst)
if err != nil {
return 0, err
}
defer destination.Close()
return io.Copy(destination, source)
}
// gzipFileToDir writes a compressed copy of src to destDir with the suffix ".gz".
func gzipFileToDir(src, destDir string) error {
in, err := os.Open(src)
if err != nil {
return fmt.Errorf("failed to open %s: %s", src, err.Error())
}
defer in.Close()
dest := filepath.Join(destDir, filepath.Base(src)+".gz")
out, err := os.OpenFile(dest, os.O_CREATE|os.O_WRONLY, 0666)
if err != nil {
return fmt.Errorf("failed to open %s: %s", dest, err.Error())
}
defer out.Close()
gz := gzip.NewWriter(out)
defer gz.Close()
_, err = io.Copy(gz, in)
if err != nil {
return fmt.Errorf("failed to gzip %s: %s", dest, err.Error())
}
return nil
}