Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
// Copyright 2017 Google Inc. All rights reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package build
import (
2022-01-27 05:32:22 +01:00
"context"
2022-01-26 04:11:01 +01:00
"encoding/json"
2020-08-12 07:26:23 +02:00
"fmt"
2022-01-26 04:11:01 +01:00
"io/ioutil"
2022-05-27 13:48:37 +02:00
"math/rand"
2017-01-20 23:10:01 +01:00
"os"
2022-01-27 05:32:22 +01:00
"os/exec"
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
"path/filepath"
"runtime"
"strconv"
"strings"
2022-05-27 13:48:37 +02:00
"syscall"
2018-02-14 22:27:26 +01:00
"time"
2017-03-30 02:29:06 +02:00
"android/soong/shared"
2020-09-21 19:39:24 +02:00
2021-05-24 23:24:12 +02:00
"google.golang.org/protobuf/proto"
2020-08-11 22:41:11 +02:00
smpb "android/soong/ui/metrics/metrics_proto"
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
)
2022-01-26 04:11:01 +01:00
const (
2022-03-03 18:01:40 +01:00
envConfigDir = "vendor/google/tools/soong_config"
jsonSuffix = "json"
2022-01-27 05:32:22 +01:00
2022-03-03 18:01:40 +01:00
configFetcher = "vendor/google/tools/soong/expconfigfetcher"
2022-01-27 05:32:22 +01:00
envConfigFetchTimeout = 10 * time . Second
2022-01-26 04:11:01 +01:00
)
2022-05-27 13:48:37 +02:00
var (
2023-03-03 20:47:17 +01:00
rbeRandPrefix int
googleProdCredsExistCache bool
2022-05-27 13:48:37 +02:00
)
func init ( ) {
rand . Seed ( time . Now ( ) . UnixNano ( ) )
rbeRandPrefix = rand . Intn ( 1000 )
}
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
type Config struct { * configImpl }
type configImpl struct {
2021-09-02 17:23:06 +02:00
// Some targets that are implemented in soong_build
// (bp2build, json-module-graph) are not here and have their own bits below.
2019-11-27 01:19:04 +01:00
arguments [ ] string
goma bool
environ * Environment
distDir string
buildDateTime string
2022-12-07 22:57:38 +01:00
logsPrefix string
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
// From the arguments
2022-12-02 23:22:40 +01:00
parallel int
keepGoing int
verbose bool
checkbuild bool
dist bool
jsonModuleGraph bool
apiBp2build bool // Generate BUILD files for Soong modules that contribute APIs
bp2build bool
queryview bool
reportMkMetrics bool // Collect and report mk2bp migration progress metrics.
soongDocs bool
2023-03-09 19:19:35 +01:00
multitreeBuild bool // This is a multitree build.
2022-12-02 23:22:40 +01:00
skipConfig bool
skipKati bool
skipKatiNinja bool
skipSoong bool
skipNinja bool
skipSoongTests bool
searchApiDir bool // Scan the Android.bp files generated in out/api_surfaces
skipMetricsUpload bool
2022-12-07 22:57:38 +01:00
buildStartedTime int64 // For metrics-upload-only - manually specify a build-started time
2023-01-17 21:40:22 +01:00
buildFromTextStub bool
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
// From the product config
2018-05-02 09:06:28 +02:00
katiArgs [ ] string
ninjaArgs [ ] string
katiSuffix string
targetDevice string
targetDeviceDir string
2021-05-25 21:14:02 +02:00
sandboxConfig * SandboxConfig
2018-04-05 07:25:56 +02:00
2019-12-27 18:35:42 +01:00
// Autodetected
totalRAM uint64
2020-01-03 04:10:38 +01:00
brokenDupRules bool
brokenUsesNetwork bool
brokenNinjaEnvVars [ ] string
2018-05-26 01:30:04 +02:00
pathReplaced bool
2020-12-10 12:32:38 +01:00
2022-10-18 22:10:16 +02:00
bazelProdMode bool
bazelDevMode bool
bazelStagingMode bool
2020-12-10 12:32:38 +01:00
2021-06-01 20:43:55 +02:00
// Set by multiproduct_kati
emptyNinjaFile bool
2021-07-27 23:29:06 +02:00
metricsUploader string
2022-11-29 01:47:59 +01:00
bazelForceEnabledModules string
2022-11-08 19:42:16 +01:00
2023-02-21 17:50:29 +01:00
includeTags [ ] string
sourceRootDirs [ ] string
2023-03-14 16:10:45 +01:00
// Data source to write ninja weight list
ninjaWeightListSource NinjaWeightListSource
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
}
2023-03-14 16:10:45 +01:00
type NinjaWeightListSource uint
const (
// ninja doesn't use weight list.
NOT_USED NinjaWeightListSource = iota
// ninja uses weight list based on previous builds by ninja log
NINJA_LOG
// ninja thinks every task has the same weight.
EVENLY_DISTRIBUTED
2023-03-18 16:12:39 +01:00
// ninja uses an external custom weight list
EXTERNAL_FILE
2023-03-18 16:12:39 +01:00
// ninja uses a prioritized module list from Soong
HINT_FROM_SOONG
2023-03-14 16:10:45 +01:00
)
2017-01-20 23:10:01 +01:00
const srcDirFileCheck = "build/soong/root.bp"
2019-07-08 20:06:46 +02:00
var buildFiles = [ ] string { "Android.mk" , "Android.bp" }
2019-04-23 02:12:02 +02:00
type BuildAction uint
const (
// Builds all of the modules and their dependencies of a specified directory, relative to the root
// directory of the source tree.
BUILD_MODULES_IN_A_DIRECTORY BuildAction = iota
// Builds all of the modules and their dependencies of a list of specified directories. All specified
// directories are relative to the root directory of the source tree.
BUILD_MODULES_IN_DIRECTORIES
2019-06-21 01:35:12 +02:00
// Build a list of specified modules. If none was specified, simply build the whole source tree.
BUILD_MODULES
2019-04-23 02:12:02 +02:00
)
// checkTopDir validates that the current directory is at the root directory of the source tree.
func checkTopDir ( ctx Context ) {
if _ , err := os . Stat ( srcDirFileCheck ) ; err != nil {
if os . IsNotExist ( err ) {
ctx . Fatalf ( "Current working directory must be the source tree. %q not found." , srcDirFileCheck )
}
ctx . Fatalln ( "Error verifying tree state:" , err )
}
}
2023-01-16 17:33:05 +01:00
// fetchEnvConfig optionally fetches a configuration file that can then subsequently be
// loaded into Soong environment to control certain aspects of build behavior (e.g., enabling RBE).
// If a configuration file already exists on disk, the fetch is run in the background
// so as to NOT block the rest of the build execution.
2022-01-27 05:32:22 +01:00
func fetchEnvConfig ( ctx Context , config * configImpl , envConfigName string ) error {
2022-04-08 15:42:04 +02:00
configName := envConfigName + "." + jsonSuffix
2022-07-07 04:20:51 +02:00
expConfigFetcher := & smpb . ExpConfigFetcher { Filename : & configName }
2022-04-08 15:42:04 +02:00
defer func ( ) {
ctx . Metrics . ExpConfigFetcher ( expConfigFetcher )
} ( )
2022-07-07 04:20:51 +02:00
if ! config . GoogleProdCredsExist ( ) {
status := smpb . ExpConfigFetcher_MISSING_GCERT
expConfigFetcher . Status = & status
return nil
}
2022-04-08 15:42:04 +02:00
2022-01-27 05:32:22 +01:00
s , err := os . Stat ( configFetcher )
if err != nil {
if os . IsNotExist ( err ) {
return nil
}
return err
}
if s . Mode ( ) & 0111 == 0 {
2022-04-08 15:42:04 +02:00
status := smpb . ExpConfigFetcher_ERROR
expConfigFetcher . Status = & status
2022-01-27 05:32:22 +01:00
return fmt . Errorf ( "configuration fetcher binary %v is not executable: %v" , configFetcher , s . Mode ( ) )
}
2023-01-16 17:33:05 +01:00
configExists := false
outConfigFilePath := filepath . Join ( config . OutDir ( ) , configName )
if _ , err := os . Stat ( outConfigFilePath ) ; err == nil {
configExists = true
}
2022-01-27 05:32:22 +01:00
tCtx , cancel := context . WithTimeout ( ctx , envConfigFetchTimeout )
2022-04-08 15:42:04 +02:00
fetchStart := time . Now ( )
cmd := exec . CommandContext ( tCtx , configFetcher , "-output_config_dir" , config . OutDir ( ) ,
"-output_config_name" , configName )
2022-01-27 05:32:22 +01:00
if err := cmd . Start ( ) ; err != nil {
2022-04-08 15:42:04 +02:00
status := smpb . ExpConfigFetcher_ERROR
expConfigFetcher . Status = & status
2022-01-27 05:32:22 +01:00
return err
}
2023-01-16 17:33:05 +01:00
fetchCfg := func ( ) error {
if err := cmd . Wait ( ) ; err != nil {
status := smpb . ExpConfigFetcher_ERROR
expConfigFetcher . Status = & status
return err
}
fetchEnd := time . Now ( )
expConfigFetcher . Micros = proto . Uint64 ( uint64 ( fetchEnd . Sub ( fetchStart ) . Microseconds ( ) ) )
expConfigFetcher . Filename = proto . String ( outConfigFilePath )
if _ , err := os . Stat ( outConfigFilePath ) ; err != nil {
status := smpb . ExpConfigFetcher_NO_CONFIG
expConfigFetcher . Status = & status
return err
}
2022-04-08 15:42:04 +02:00
status := smpb . ExpConfigFetcher_CONFIG
expConfigFetcher . Status = & status
2023-01-16 17:33:05 +01:00
return nil
}
// If a config file does not exist, wait for the config file to be fetched. Otherwise
// fetch the config file in the background and return immediately.
if ! configExists {
defer cancel ( )
return fetchCfg ( )
2022-04-08 15:42:04 +02:00
}
2023-01-16 17:33:05 +01:00
go func ( ) {
defer cancel ( )
if err := fetchCfg ( ) ; err != nil {
ctx . Verbosef ( "Failed to fetch config file %v: %v\n" , configName , err )
}
} ( )
2022-01-27 05:32:22 +01:00
return nil
}
2023-01-09 20:48:01 +01:00
func loadEnvConfig ( ctx Context , config * configImpl , bc string ) error {
2022-01-26 04:11:01 +01:00
if bc == "" {
return nil
}
2022-01-27 05:32:22 +01:00
2022-01-26 04:11:01 +01:00
configDirs := [ ] string {
config . OutDir ( ) ,
2022-01-27 05:32:22 +01:00
os . Getenv ( "ANDROID_BUILD_ENVIRONMENT_CONFIG_DIR" ) ,
2022-01-26 04:11:01 +01:00
envConfigDir ,
}
for _ , dir := range configDirs {
2022-01-27 05:32:22 +01:00
cfgFile := filepath . Join ( os . Getenv ( "TOP" ) , dir , fmt . Sprintf ( "%s.%s" , bc , jsonSuffix ) )
envVarsJSON , err := ioutil . ReadFile ( cfgFile )
if err != nil {
continue
2022-01-26 04:11:01 +01:00
}
2022-01-27 05:32:22 +01:00
ctx . Verbosef ( "Loading config file %v\n" , cfgFile )
var envVars map [ string ] map [ string ] string
if err := json . Unmarshal ( envVarsJSON , & envVars ) ; err != nil {
fmt . Fprintf ( os . Stderr , "Env vars config file %s did not parse correctly: %s" , cfgFile , err . Error ( ) )
2022-01-26 04:11:01 +01:00
continue
}
2022-01-27 05:32:22 +01:00
for k , v := range envVars [ "env" ] {
if os . Getenv ( k ) != "" {
continue
}
config . environ . Set ( k , v )
}
ctx . Verbosef ( "Finished loading config file %v\n" , cfgFile )
break
2022-01-26 04:11:01 +01:00
}
2022-01-27 05:32:22 +01:00
2022-01-26 04:11:01 +01:00
return nil
}
2022-11-01 01:08:45 +01:00
func defaultBazelProdMode ( cfg * configImpl ) bool {
2022-11-29 01:47:59 +01:00
// Environment flag to disable Bazel for users which experience
2022-11-01 01:08:45 +01:00
// broken bazel-handled builds, or significant performance regressions.
if cfg . IsBazelMixedBuildForceDisabled ( ) {
return false
}
// Darwin-host builds are currently untested with Bazel.
if runtime . GOOS == "darwin" {
return false
}
2022-11-01 19:25:45 +01:00
return true
2022-11-01 01:08:45 +01:00
}
2022-12-07 22:57:38 +01:00
func UploadOnlyConfig ( ctx Context , _ ... string ) Config {
ret := & configImpl {
environ : OsEnvironment ( ) ,
sandboxConfig : & SandboxConfig { } ,
}
2023-01-09 20:48:01 +01:00
srcDir := absPath ( ctx , "." )
bc := os . Getenv ( "ANDROID_BUILD_ENVIRONMENT_CONFIG" )
if err := loadEnvConfig ( ctx , ret , bc ) ; err != nil {
ctx . Fatalln ( "Failed to parse env config files: %v" , err )
}
ret . metricsUploader = GetMetricsUploader ( srcDir , ret . environ )
2022-12-07 22:57:38 +01:00
return Config { ret }
}
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
func NewConfig ( ctx Context , args ... string ) Config {
ret := & configImpl {
2023-04-13 17:17:47 +02:00
environ : OsEnvironment ( ) ,
sandboxConfig : & SandboxConfig { } ,
ninjaWeightListSource : HINT_FROM_SOONG ,
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
}
2020-07-28 20:07:27 +02:00
// Default matching ninja
2017-07-11 07:13:00 +02:00
ret . parallel = runtime . NumCPU ( ) + 2
ret . keepGoing = 1
2019-12-27 18:35:42 +01:00
ret . totalRAM = detectTotalRAM ( ctx )
2017-07-11 07:13:00 +02:00
ret . parseArgs ( ctx , args )
2023-03-18 16:12:39 +01:00
if ret . ninjaWeightListSource == HINT_FROM_SOONG {
ret . environ . Set ( "SOONG_GENERATES_NINJA_HINT" , "true" )
}
2017-03-03 00:49:10 +01:00
// Make sure OUT_DIR is set appropriately
2017-05-12 22:50:19 +02:00
if outDir , ok := ret . environ . Get ( "OUT_DIR" ) ; ok {
ret . environ . Set ( "OUT_DIR" , filepath . Clean ( outDir ) )
} else {
2017-03-03 00:49:10 +01:00
outDir := "out"
if baseDir , ok := ret . environ . Get ( "OUT_DIR_COMMON_BASE" ) ; ok {
if wd , err := os . Getwd ( ) ; err != nil {
ctx . Fatalln ( "Failed to get working directory:" , err )
} else {
outDir = filepath . Join ( baseDir , filepath . Base ( wd ) )
}
}
ret . environ . Set ( "OUT_DIR" , outDir )
}
2022-01-26 04:11:01 +01:00
// loadEnvConfig needs to know what the OUT_DIR is, so it should
// be called after we determine the appropriate out directory.
2023-01-09 20:48:01 +01:00
bc := os . Getenv ( "ANDROID_BUILD_ENVIRONMENT_CONFIG" )
if bc != "" {
if err := fetchEnvConfig ( ctx , ret , bc ) ; err != nil {
ctx . Verbosef ( "Failed to fetch config file: %v\n" , err )
2023-01-16 17:33:05 +01:00
}
if err := loadEnvConfig ( ctx , ret , bc ) ; err != nil {
2023-01-09 20:48:01 +01:00
ctx . Fatalln ( "Failed to parse env config files: %v" , err )
}
2022-01-26 04:11:01 +01:00
}
2018-10-21 06:33:41 +02:00
if distDir , ok := ret . environ . Get ( "DIST_DIR" ) ; ok {
ret . distDir = filepath . Clean ( distDir )
} else {
ret . distDir = filepath . Join ( ret . OutDir ( ) , "dist" )
}
2018-10-17 02:49:25 +02:00
2021-06-25 03:39:04 +02:00
if srcDirIsWritable , ok := ret . environ . Get ( "BUILD_BROKEN_SRC_DIR_IS_WRITABLE" ) ; ok {
ret . sandboxConfig . SetSrcDirIsRO ( srcDirIsWritable == "false" )
}
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
ret . environ . Unset (
// We're already using it
"USE_SOONG_UI" ,
// We should never use GOROOT/GOPATH from the shell environment
"GOROOT" ,
"GOPATH" ,
// These should only come from Soong, not the environment.
"CLANG" ,
"CLANG_CXX" ,
"CCC_CC" ,
"CCC_CXX" ,
// Used by the goma compiler wrapper, but should only be set by
// gomacc
"GOMACC_PATH" ,
2017-03-03 00:49:10 +01:00
// We handle this above
"OUT_DIR_COMMON_BASE" ,
2017-04-18 22:56:57 +02:00
2018-10-21 06:33:41 +02:00
// This is handled above too, and set for individual commands later
"DIST_DIR" ,
2017-04-18 22:56:57 +02:00
// Variables that have caused problems in the past
2019-11-18 20:13:53 +01:00
"BASH_ENV" ,
2018-05-01 19:07:50 +02:00
"CDPATH" ,
2017-04-18 22:56:57 +02:00
"DISPLAY" ,
"GREP_OPTIONS" ,
2023-02-17 18:54:31 +01:00
"JAVAC" ,
2018-05-01 19:07:50 +02:00
"NDK_ROOT" ,
2018-08-16 00:35:38 +02:00
"POSIXLY_CORRECT" ,
2017-07-11 23:30:00 +02:00
// Drop make flags
"MAKEFLAGS" ,
"MAKELEVEL" ,
"MFLAGS" ,
2017-10-30 21:42:06 +01:00
// Set in envsetup.sh, reset in makefiles
"ANDROID_JAVA_TOOLCHAIN" ,
2018-07-11 23:49:31 +02:00
// Set by envsetup.sh, but shouldn't be used inside the build because envsetup.sh is optional
"ANDROID_BUILD_TOP" ,
"ANDROID_HOST_OUT" ,
"ANDROID_PRODUCT_OUT" ,
"ANDROID_HOST_OUT_TESTCASES" ,
"ANDROID_TARGET_OUT_TESTCASES" ,
"ANDROID_TOOLCHAIN" ,
"ANDROID_TOOLCHAIN_2ND_ARCH" ,
"ANDROID_DEV_SCRIPTS" ,
"ANDROID_EMULATOR_PREBUILTS" ,
"ANDROID_PRE_BUILD_PATHS" ,
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
)
2020-10-19 07:45:46 +02:00
if ret . UseGoma ( ) || ret . ForceUseGoma ( ) {
ctx . Println ( "Goma for Android has been deprecated and replaced with RBE. See go/rbe_for_android for instructions on how to use RBE." )
ctx . Fatalln ( "USE_GOMA / FORCE_USE_GOMA flag is no longer supported." )
2020-09-21 19:39:24 +02:00
}
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
// Tell python not to spam the source tree with .pyc files.
ret . environ . Set ( "PYTHONDONTWRITEBYTECODE" , "1" )
2020-07-16 18:18:37 +02:00
tmpDir := absPath ( ctx , ret . TempDir ( ) )
ret . environ . Set ( "TMPDIR" , tmpDir )
2018-03-09 04:42:00 +01:00
2019-08-21 23:56:13 +02:00
// Always set ASAN_SYMBOLIZER_PATH so that ASAN-based tools can symbolize any crashes
symbolizerPath := filepath . Join ( "prebuilts/clang/host" , ret . HostPrebuiltTag ( ) ,
"llvm-binutils-stable/llvm-symbolizer" )
ret . environ . Set ( "ASAN_SYMBOLIZER_PATH" , absPath ( ctx , symbolizerPath ) )
2017-01-20 23:10:01 +01:00
// Precondition: the current directory is the top of the source tree
2019-04-23 02:12:02 +02:00
checkTopDir ( ctx )
2017-01-20 23:10:01 +01:00
2021-07-27 23:29:06 +02:00
srcDir := absPath ( ctx , "." )
if strings . ContainsRune ( srcDir , ' ' ) {
2019-09-24 00:52:40 +02:00
ctx . Println ( "You are building in a directory whose absolute path contains a space character:" )
ctx . Println ( )
ctx . Printf ( "%q\n" , srcDir )
ctx . Println ( )
ctx . Fatalln ( "Directory names containing spaces are not supported" )
2017-05-13 01:38:17 +02:00
}
2021-07-27 23:29:06 +02:00
ret . metricsUploader = GetMetricsUploader ( srcDir , ret . environ )
2017-05-13 01:38:17 +02:00
if outDir := ret . OutDir ( ) ; strings . ContainsRune ( outDir , ' ' ) {
2019-09-24 00:52:40 +02:00
ctx . Println ( "The absolute path of your output directory ($OUT_DIR) contains a space character:" )
ctx . Println ( )
ctx . Printf ( "%q\n" , outDir )
ctx . Println ( )
ctx . Fatalln ( "Directory names containing spaces are not supported" )
2017-05-13 01:38:17 +02:00
}
2020-12-10 12:32:38 +01:00
if distDir := ret . RealDistDir ( ) ; strings . ContainsRune ( distDir , ' ' ) {
2019-09-24 00:52:40 +02:00
ctx . Println ( "The absolute path of your dist directory ($DIST_DIR) contains a space character:" )
ctx . Println ( )
ctx . Printf ( "%q\n" , distDir )
ctx . Println ( )
ctx . Fatalln ( "Directory names containing spaces are not supported" )
2017-05-13 01:38:17 +02:00
}
2017-10-30 21:42:06 +01:00
// Configure Java-related variables, including adding it to $PATH
2017-12-20 23:40:39 +01:00
java8Home := filepath . Join ( "prebuilts/jdk/jdk8" , ret . HostPrebuiltTag ( ) )
java9Home := filepath . Join ( "prebuilts/jdk/jdk9" , ret . HostPrebuiltTag ( ) )
2019-10-09 18:10:08 +02:00
java11Home := filepath . Join ( "prebuilts/jdk/jdk11" , ret . HostPrebuiltTag ( ) )
2022-03-04 22:37:19 +01:00
java17Home := filepath . Join ( "prebuilts/jdk/jdk17" , ret . HostPrebuiltTag ( ) )
2017-10-30 21:42:06 +01:00
javaHome := func ( ) string {
if override , ok := ret . environ . Get ( "OVERRIDE_ANDROID_JAVA_HOME" ) ; ok {
return override
}
2019-11-07 19:58:42 +01:00
if toolchain11 , ok := ret . environ . Get ( "EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN" ) ; ok && toolchain11 != "true" {
ctx . Fatalln ( "The environment variable EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN is no longer supported. An OpenJDK 11 toolchain is now the global default." )
2019-10-09 18:10:08 +02:00
}
2022-10-05 10:20:12 +02:00
if toolchain17 , ok := ret . environ . Get ( "EXPERIMENTAL_USE_OPENJDK17_TOOLCHAIN" ) ; ok && toolchain17 != "true" {
ctx . Fatalln ( "The environment variable EXPERIMENTAL_USE_OPENJDK17_TOOLCHAIN is no longer supported. An OpenJDK 17 toolchain is now the global default." )
}
return java17Home
2017-10-30 21:42:06 +01:00
} ( )
absJavaHome := absPath ( ctx , javaHome )
2018-01-08 23:58:46 +01:00
ret . configureLocale ( ctx )
2017-10-30 21:42:06 +01:00
newPath := [ ] string { filepath . Join ( absJavaHome , "bin" ) }
if path , ok := ret . environ . Get ( "PATH" ) ; ok && path != "" {
newPath = append ( newPath , path )
}
2019-10-09 18:10:08 +02:00
2017-10-30 21:42:06 +01:00
ret . environ . Unset ( "OVERRIDE_ANDROID_JAVA_HOME" )
ret . environ . Set ( "JAVA_HOME" , absJavaHome )
ret . environ . Set ( "ANDROID_JAVA_HOME" , javaHome )
2017-12-20 23:40:39 +01:00
ret . environ . Set ( "ANDROID_JAVA8_HOME" , java8Home )
ret . environ . Set ( "ANDROID_JAVA9_HOME" , java9Home )
2019-10-09 18:10:08 +02:00
ret . environ . Set ( "ANDROID_JAVA11_HOME" , java11Home )
2017-10-30 21:42:06 +01:00
ret . environ . Set ( "PATH" , strings . Join ( newPath , string ( filepath . ListSeparator ) ) )
2023-03-09 19:19:35 +01:00
if ret . MultitreeBuild ( ) {
ret . environ . Set ( "MULTITREE_BUILD" , "true" )
}
2018-02-14 22:27:26 +01:00
outDir := ret . OutDir ( )
buildDateTimeFile := filepath . Join ( outDir , "build_date.txt" )
if buildDateTime , ok := ret . environ . Get ( "BUILD_DATETIME" ) ; ok && buildDateTime != "" {
2019-11-27 01:19:04 +01:00
ret . buildDateTime = buildDateTime
2018-02-14 22:27:26 +01:00
} else {
2019-11-27 01:19:04 +01:00
ret . buildDateTime = strconv . FormatInt ( time . Now ( ) . Unix ( ) , 10 )
2018-02-14 22:27:26 +01:00
}
2019-11-27 01:19:04 +01:00
2018-02-14 22:27:26 +01:00
ret . environ . Set ( "BUILD_DATETIME_FILE" , buildDateTimeFile )
2020-07-16 18:18:37 +02:00
if ret . UseRBE ( ) {
2020-08-12 07:26:23 +02:00
for k , v := range getRBEVars ( ctx , Config { ret } ) {
2020-07-16 18:18:37 +02:00
ret . environ . Set ( k , v )
}
}
2023-01-17 21:40:22 +01:00
if ret . BuildFromTextStub ( ) {
// TODO(b/271443071): support hidden api check for from-text stub build
ret . environ . Set ( "UNSAFE_DISABLE_HIDDENAPI_FLAGS" , "true" )
}
2020-12-08 20:42:08 +01:00
bpd := ret . BazelMetricsDir ( )
2020-11-13 17:41:26 +01:00
if err := os . RemoveAll ( bpd ) ; err != nil {
ctx . Fatalf ( "Unable to remove bazel profile directory %q: %v" , bpd , err )
}
2020-12-10 12:32:38 +01:00
2020-08-11 22:41:11 +02:00
c := Config { ret }
storeConfigMetrics ( ctx , c )
return c
2017-07-11 07:13:00 +02:00
}
2019-04-23 02:12:02 +02:00
// NewBuildActionConfig returns a build configuration based on the build action. The arguments are
// processed based on the build action and extracts any arguments that belongs to the build action.
2019-07-30 08:39:30 +02:00
func NewBuildActionConfig ( action BuildAction , dir string , ctx Context , args ... string ) Config {
return NewConfig ( ctx , getConfigArgs ( action , dir , ctx , args ) ... )
2019-04-23 02:12:02 +02:00
}
2020-08-11 22:41:11 +02:00
// storeConfigMetrics selects a set of configuration information and store in
// the metrics system for further analysis.
func storeConfigMetrics ( ctx Context , config Config ) {
if ctx . Metrics == nil {
return
}
2021-07-14 21:29:57 +02:00
ctx . Metrics . BuildConfig ( buildConfig ( config ) )
2020-10-14 01:58:41 +02:00
s := & smpb . SystemResourceInfo {
TotalPhysicalMemory : proto . Uint64 ( config . TotalRAM ( ) ) ,
AvailableCpus : proto . Int32 ( int32 ( runtime . NumCPU ( ) ) ) ,
}
ctx . Metrics . SystemResourceInfo ( s )
2020-08-11 22:41:11 +02:00
}
2023-03-16 19:52:13 +01:00
func getNinjaWeightListSourceInMetric ( s NinjaWeightListSource ) * smpb . BuildConfig_NinjaWeightListSource {
2023-03-14 16:10:45 +01:00
switch s {
case NINJA_LOG :
2023-03-16 19:52:13 +01:00
return smpb . BuildConfig_NINJA_LOG . Enum ( )
2023-03-14 16:10:45 +01:00
case EVENLY_DISTRIBUTED :
2023-03-16 19:52:13 +01:00
return smpb . BuildConfig_EVENLY_DISTRIBUTED . Enum ( )
2023-03-18 16:12:39 +01:00
case EXTERNAL_FILE :
return smpb . BuildConfig_EXTERNAL_FILE . Enum ( )
2023-03-18 16:12:39 +01:00
case HINT_FROM_SOONG :
return smpb . BuildConfig_HINT_FROM_SOONG . Enum ( )
2023-03-14 16:10:45 +01:00
default :
2023-03-16 19:52:13 +01:00
return smpb . BuildConfig_NOT_USED . Enum ( )
2023-03-14 16:10:45 +01:00
}
}
2021-07-14 21:29:57 +02:00
func buildConfig ( config Config ) * smpb . BuildConfig {
2021-10-04 22:21:41 +02:00
c := & smpb . BuildConfig {
2022-10-26 18:56:41 +02:00
ForceUseGoma : proto . Bool ( config . ForceUseGoma ( ) ) ,
UseGoma : proto . Bool ( config . UseGoma ( ) ) ,
UseRbe : proto . Bool ( config . UseRBE ( ) ) ,
BazelMixedBuild : proto . Bool ( config . BazelBuildEnabled ( ) ) ,
ForceDisableBazelMixedBuild : proto . Bool ( config . IsBazelMixedBuildForceDisabled ( ) ) ,
2023-03-14 16:10:45 +01:00
NinjaWeightListSource : getNinjaWeightListSourceInMetric ( config . NinjaWeightListSource ( ) ) ,
2021-07-14 21:29:57 +02:00
}
2021-10-04 22:21:41 +02:00
c . Targets = append ( c . Targets , config . arguments ... )
return c
2021-07-14 21:29:57 +02:00
}
2019-04-23 02:12:02 +02:00
// getConfigArgs processes the command arguments based on the build action and creates a set of new
// arguments to be accepted by Config.
2019-07-30 08:39:30 +02:00
func getConfigArgs ( action BuildAction , dir string , ctx Context , args [ ] string ) [ ] string {
2019-04-23 02:12:02 +02:00
// The next block of code verifies that the current directory is the root directory of the source
// tree. It then finds the relative path of dir based on the root directory of the source tree
// and verify that dir is inside of the source tree.
checkTopDir ( ctx )
topDir , err := os . Getwd ( )
if err != nil {
ctx . Fatalf ( "Error retrieving top directory: %v" , err )
}
2019-07-03 19:47:34 +02:00
dir , err = filepath . EvalSymlinks ( dir )
if err != nil {
ctx . Fatalf ( "Unable to evaluate symlink of %s: %v" , dir , err )
}
2019-04-23 02:12:02 +02:00
dir , err = filepath . Abs ( dir )
if err != nil {
ctx . Fatalf ( "Unable to find absolute path %s: %v" , dir , err )
}
relDir , err := filepath . Rel ( topDir , dir )
if err != nil {
ctx . Fatalf ( "Unable to find relative path %s of %s: %v" , relDir , topDir , err )
}
// If there are ".." in the path, it's not in the source tree.
if strings . Contains ( relDir , ".." ) {
ctx . Fatalf ( "Directory %s is not under the source tree %s" , dir , topDir )
}
configArgs := args [ : ]
// If the arguments contains GET-INSTALL-PATH, change the target name prefix from MODULES-IN- to
// GET-INSTALL-PATH-IN- to extract the installation path instead of building the modules.
targetNamePrefix := "MODULES-IN-"
if inList ( "GET-INSTALL-PATH" , configArgs ) {
targetNamePrefix = "GET-INSTALL-PATH-IN-"
configArgs = removeFromList ( "GET-INSTALL-PATH" , configArgs )
}
var targets [ ] string
switch action {
2019-06-21 01:35:12 +02:00
case BUILD_MODULES :
// No additional processing is required when building a list of specific modules or all modules.
2019-04-23 02:12:02 +02:00
case BUILD_MODULES_IN_A_DIRECTORY :
// If dir is the root source tree, all the modules are built of the source tree are built so
// no need to find the build file.
if topDir == dir {
break
}
2019-07-09 02:03:33 +02:00
2019-04-23 02:12:02 +02:00
buildFile := findBuildFile ( ctx , relDir )
if buildFile == "" {
2019-07-09 02:03:33 +02:00
ctx . Fatalf ( "Build file not found for %s directory" , relDir )
2019-04-23 02:12:02 +02:00
}
targets = [ ] string { convertToTarget ( filepath . Dir ( buildFile ) , targetNamePrefix ) }
case BUILD_MODULES_IN_DIRECTORIES :
newConfigArgs , dirs := splitArgs ( configArgs )
configArgs = newConfigArgs
2019-07-30 08:39:30 +02:00
targets = getTargetsFromDirs ( ctx , relDir , dirs , targetNamePrefix )
2019-04-23 02:12:02 +02:00
}
// Tidy only override all other specified targets.
tidyOnly := os . Getenv ( "WITH_TIDY_ONLY" )
if tidyOnly == "true" || tidyOnly == "1" {
configArgs = append ( configArgs , "tidy_only" )
} else {
configArgs = append ( configArgs , targets ... )
}
return configArgs
}
// convertToTarget replaces "/" to "-" in dir and pre-append the targetNamePrefix to the target name.
func convertToTarget ( dir string , targetNamePrefix string ) string {
return targetNamePrefix + strings . ReplaceAll ( dir , "/" , "-" )
}
2019-07-08 20:06:46 +02:00
// hasBuildFile returns true if dir contains an Android build file.
func hasBuildFile ( ctx Context , dir string ) bool {
for _ , buildFile := range buildFiles {
_ , err := os . Stat ( filepath . Join ( dir , buildFile ) )
if err == nil {
return true
}
if ! os . IsNotExist ( err ) {
ctx . Fatalf ( "Error retrieving the build file stats: %v" , err )
}
}
return false
}
2019-07-09 02:03:33 +02:00
// findBuildFile finds a build file (makefile or blueprint file) by looking if there is a build file
// in the current and any sub directory of dir. If a build file is not found, traverse the path
// up by one directory and repeat again until either a build file is found or reached to the root
// source tree. The returned filename of build file is "Android.mk". If one was not found, a blank
// string is returned.
2019-04-23 02:12:02 +02:00
func findBuildFile ( ctx Context , dir string ) string {
2019-07-09 02:03:33 +02:00
// If the string is empty or ".", assume it is top directory of the source tree.
if dir == "" || dir == "." {
2019-04-23 02:12:02 +02:00
return ""
}
2019-07-09 02:03:33 +02:00
found := false
for buildDir := dir ; buildDir != "." ; buildDir = filepath . Dir ( buildDir ) {
err := filepath . Walk ( buildDir , func ( path string , info os . FileInfo , err error ) error {
if err != nil {
return err
}
if found {
return filepath . SkipDir
}
if info . IsDir ( ) {
return nil
}
for _ , buildFile := range buildFiles {
if info . Name ( ) == buildFile {
found = true
return filepath . SkipDir
}
}
return nil
} )
if err != nil {
ctx . Fatalf ( "Error finding Android build file: %v" , err )
}
if found {
return filepath . Join ( buildDir , "Android.mk" )
2019-04-23 02:12:02 +02:00
}
}
return ""
}
// splitArgs iterates over the arguments list and splits into two lists: arguments and directories.
func splitArgs ( args [ ] string ) ( newArgs [ ] string , dirs [ ] string ) {
specialArgs := map [ string ] bool {
"showcommands" : true ,
"snod" : true ,
"dist" : true ,
"checkbuild" : true ,
}
newArgs = [ ] string { }
dirs = [ ] string { }
for _ , arg := range args {
// It's a dash argument if it starts with "-" or it's a key=value pair, it's not a directory.
if strings . IndexRune ( arg , '-' ) == 0 || strings . IndexRune ( arg , '=' ) != - 1 {
newArgs = append ( newArgs , arg )
continue
}
if _ , ok := specialArgs [ arg ] ; ok {
newArgs = append ( newArgs , arg )
continue
}
dirs = append ( dirs , arg )
}
return newArgs , dirs
}
// getTargetsFromDirs iterates over the dirs list and creates a list of targets to build. If a
// directory from the dirs list does not exist, a fatal error is raised. relDir is related to the
// source root tree where the build action command was invoked. Each directory is validated if the
// build file can be found and follows the format "dir1:target1,target2,...". Target is optional.
2019-07-30 08:39:30 +02:00
func getTargetsFromDirs ( ctx Context , relDir string , dirs [ ] string , targetNamePrefix string ) ( targets [ ] string ) {
2019-04-23 02:12:02 +02:00
for _ , dir := range dirs {
// The directory may have specified specific modules to build. ":" is the separator to separate
// the directory and the list of modules.
s := strings . Split ( dir , ":" )
l := len ( s )
if l > 2 { // more than one ":" was specified.
ctx . Fatalf ( "%s not in proper directory:target1,target2,... format (\":\" was specified more than once)" , dir )
}
dir = filepath . Join ( relDir , s [ 0 ] )
if _ , err := os . Stat ( dir ) ; err != nil {
ctx . Fatalf ( "couldn't find directory %s" , dir )
}
// Verify that if there are any targets specified after ":". Each target is separated by ",".
var newTargets [ ] string
if l == 2 && s [ 1 ] != "" {
newTargets = strings . Split ( s [ 1 ] , "," )
if inList ( "" , newTargets ) {
ctx . Fatalf ( "%s not in proper directory:target1,target2,... format" , dir )
}
}
2019-07-08 20:06:46 +02:00
// If there are specified targets to build in dir, an android build file must exist for the one
// shot build. For the non-targets case, find the appropriate build file and build all the
// modules in dir (or the closest one in the dir path).
2019-04-23 02:12:02 +02:00
if len ( newTargets ) > 0 {
2019-07-08 20:06:46 +02:00
if ! hasBuildFile ( ctx , dir ) {
2019-04-23 02:12:02 +02:00
ctx . Fatalf ( "Couldn't locate a build file from %s directory" , dir )
}
} else {
2019-07-08 20:06:46 +02:00
buildFile := findBuildFile ( ctx , dir )
if buildFile == "" {
ctx . Fatalf ( "Build file not found for %s directory" , dir )
}
newTargets = [ ] string { convertToTarget ( filepath . Dir ( buildFile ) , targetNamePrefix ) }
2019-04-23 02:12:02 +02:00
}
targets = append ( targets , newTargets ... )
}
2019-07-30 08:39:30 +02:00
return targets
2019-04-23 02:12:02 +02:00
}
2017-07-11 07:13:00 +02:00
func ( c * configImpl ) parseArgs ( ctx Context , args [ ] string ) {
for i := 0 ; i < len ( args ) ; i ++ {
arg := strings . TrimSpace ( args [ i ] )
2021-06-04 11:09:01 +02:00
if arg == "showcommands" {
2017-07-11 07:13:00 +02:00
c . verbose = true
2021-08-10 15:01:13 +02:00
} else if arg == "--empty-ninja-file" {
c . emptyNinjaFile = true
2021-03-16 08:55:23 +01:00
} else if arg == "--skip-ninja" {
c . skipNinja = true
2017-08-05 00:06:27 +02:00
} else if arg == "--skip-make" {
2021-06-18 20:26:19 +02:00
// TODO(ccross): deprecate this, it has confusing behaviors. It doesn't run kati,
// but it does run a Kati ninja file if the .kati_enabled marker file was created
// by a previous build.
2020-11-27 13:35:20 +01:00
c . skipConfig = true
c . skipKati = true
} else if arg == "--skip-kati" {
2021-06-04 11:08:08 +02:00
// TODO: remove --skip-kati once module builds have been migrated to --song-only
2020-11-27 13:35:20 +01:00
c . skipKati = true
2021-06-04 11:08:08 +02:00
} else if arg == "--soong-only" {
c . skipKati = true
c . skipKatiNinja = true
2021-08-10 15:01:13 +02:00
} else if arg == "--config-only" {
c . skipKati = true
c . skipKatiNinja = true
c . skipSoong = true
2021-06-18 20:26:19 +02:00
} else if arg == "--skip-config" {
c . skipConfig = true
2020-10-29 22:08:31 +01:00
} else if arg == "--skip-soong-tests" {
c . skipSoongTests = true
2022-12-02 23:22:40 +01:00
} else if arg == "--skip-metrics-upload" {
c . skipMetricsUpload = true
2022-03-03 18:01:40 +01:00
} else if arg == "--mk-metrics" {
c . reportMkMetrics = true
2023-03-09 19:19:35 +01:00
} else if arg == "--multitree-build" {
c . multitreeBuild = true
2022-08-19 04:04:11 +02:00
} else if arg == "--bazel-mode" {
c . bazelProdMode = true
} else if arg == "--bazel-mode-dev" {
c . bazelDevMode = true
2022-10-18 22:10:16 +02:00
} else if arg == "--bazel-mode-staging" {
c . bazelStagingMode = true
2022-11-03 18:02:10 +01:00
} else if arg == "--search-api-dir" {
c . searchApiDir = true
2023-03-14 16:10:45 +01:00
} else if strings . HasPrefix ( arg , "--ninja_weight_source=" ) {
source := strings . TrimPrefix ( arg , "--ninja_weight_source=" )
if source == "ninja_log" {
c . ninjaWeightListSource = NINJA_LOG
} else if source == "evenly_distributed" {
c . ninjaWeightListSource = EVENLY_DISTRIBUTED
} else if source == "not_used" {
c . ninjaWeightListSource = NOT_USED
2023-03-18 16:12:39 +01:00
} else if source == "soong" {
c . ninjaWeightListSource = HINT_FROM_SOONG
2023-03-18 16:12:39 +01:00
} else if strings . HasPrefix ( source , "file," ) {
c . ninjaWeightListSource = EXTERNAL_FILE
filePath := strings . TrimPrefix ( source , "file," )
err := validateNinjaWeightList ( filePath )
if err != nil {
ctx . Fatalf ( "Malformed weight list from %s: %s" , filePath , err )
}
_ , err = copyFile ( filePath , filepath . Join ( c . OutDir ( ) , ".ninja_weight_list" ) )
if err != nil {
ctx . Fatalf ( "Error to copy ninja weight list from %s: %s" , filePath , err )
}
2023-03-14 16:10:45 +01:00
} else {
ctx . Fatalf ( "unknown option for ninja_weight_source: %s" , source )
}
2023-01-17 21:40:22 +01:00
} else if arg == "--build-from-text-stub" {
c . buildFromTextStub = true
2022-12-02 05:34:43 +01:00
} else if strings . HasPrefix ( arg , "--build-command=" ) {
buildCmd := strings . TrimPrefix ( arg , "--build-command=" )
// remove quotations
buildCmd = strings . TrimPrefix ( buildCmd , "\"" )
buildCmd = strings . TrimSuffix ( buildCmd , "\"" )
ctx . Metrics . SetBuildCommand ( [ ] string { buildCmd } )
2022-11-29 01:47:59 +01:00
} else if strings . HasPrefix ( arg , "--bazel-force-enabled-modules=" ) {
c . bazelForceEnabledModules = strings . TrimPrefix ( arg , "--bazel-force-enabled-modules=" )
2022-12-07 22:57:38 +01:00
} else if strings . HasPrefix ( arg , "--build-started-time-unix-millis=" ) {
buildTimeStr := strings . TrimPrefix ( arg , "--build-started-time-unix-millis=" )
val , err := strconv . ParseInt ( buildTimeStr , 10 , 64 )
if err == nil {
c . buildStartedTime = val
} else {
ctx . Fatalf ( "Error parsing build-time-started-unix-millis" , err )
}
2017-10-18 05:35:34 +02:00
} else if len ( arg ) > 0 && arg [ 0 ] == '-' {
2017-07-11 07:13:00 +02:00
parseArgNum := func ( def int ) int {
if len ( arg ) > 2 {
p , err := strconv . ParseUint ( arg [ 2 : ] , 10 , 31 )
if err != nil {
ctx . Fatalf ( "Failed to parse %q: %v" , arg , err )
}
return int ( p )
} else if i + 1 < len ( args ) {
p , err := strconv . ParseUint ( args [ i + 1 ] , 10 , 31 )
if err == nil {
i ++
return int ( p )
}
}
return def
}
2017-10-18 05:35:34 +02:00
if len ( arg ) > 1 && arg [ 1 ] == 'j' {
2017-07-11 07:13:00 +02:00
c . parallel = parseArgNum ( c . parallel )
2017-10-18 05:35:34 +02:00
} else if len ( arg ) > 1 && arg [ 1 ] == 'k' {
2017-07-11 07:13:00 +02:00
c . keepGoing = parseArgNum ( 0 )
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
} else {
ctx . Fatalln ( "Unknown option:" , arg )
}
2017-07-11 23:17:50 +02:00
} else if k , v , ok := decodeKeyValue ( arg ) ; ok && len ( k ) > 0 {
2018-09-10 21:41:10 +02:00
if k == "OUT_DIR" {
ctx . Fatalln ( "OUT_DIR may only be set in the environment, not as a command line option." )
}
2017-07-11 23:17:50 +02:00
c . environ . Set ( k , v )
2018-10-21 06:33:41 +02:00
} else if arg == "dist" {
c . dist = true
2021-09-02 17:23:06 +02:00
} else if arg == "json-module-graph" {
c . jsonModuleGraph = true
} else if arg == "bp2build" {
c . bp2build = true
2022-09-28 22:43:08 +02:00
} else if arg == "api_bp2build" {
c . apiBp2build = true
2021-09-06 17:08:02 +02:00
} else if arg == "queryview" {
c . queryview = true
2021-09-06 18:31:46 +02:00
} else if arg == "soong_docs" {
c . soongDocs = true
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
} else {
2018-10-21 06:33:41 +02:00
if arg == "checkbuild" {
2017-11-17 02:55:00 +01:00
c . checkbuild = true
2017-08-05 00:06:27 +02:00
}
2017-07-11 07:13:00 +02:00
c . arguments = append ( c . arguments , arg )
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
}
}
2022-11-01 01:08:45 +01:00
if ( ! c . bazelProdMode ) && ( ! c . bazelDevMode ) && ( ! c . bazelStagingMode ) {
c . bazelProdMode = defaultBazelProdMode ( c )
}
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
}
2023-03-18 16:12:39 +01:00
func validateNinjaWeightList ( weightListFilePath string ) ( err error ) {
data , err := os . ReadFile ( weightListFilePath )
if err != nil {
return
}
lines := strings . Split ( strings . TrimSpace ( string ( data ) ) , "\n" )
for _ , line := range lines {
fields := strings . Split ( line , "," )
if len ( fields ) != 2 {
return fmt . Errorf ( "wrong format, each line should have two fields, but '%s'" , line )
}
_ , err = strconv . Atoi ( fields [ 1 ] )
if err != nil {
return
}
}
return
}
2018-01-08 23:58:46 +01:00
func ( c * configImpl ) configureLocale ( ctx Context ) {
cmd := Command ( ctx , Config { c } , "locale" , "locale" , "-a" )
output , err := cmd . Output ( )
var locales [ ] string
if err == nil {
locales = strings . Split ( string ( output ) , "\n" )
} else {
// If we're unable to list the locales, let's assume en_US.UTF-8
locales = [ ] string { "en_US.UTF-8" }
ctx . Verbosef ( "Failed to list locales (%q), falling back to %q" , err , locales )
}
// gettext uses LANGUAGE, which is passed directly through
// For LANG and LC_*, only preserve the evaluated version of
// LC_MESSAGES
2020-12-21 18:11:10 +01:00
userLang := ""
2018-01-08 23:58:46 +01:00
if lc_all , ok := c . environ . Get ( "LC_ALL" ) ; ok {
2020-12-21 18:11:10 +01:00
userLang = lc_all
2018-01-08 23:58:46 +01:00
} else if lc_messages , ok := c . environ . Get ( "LC_MESSAGES" ) ; ok {
2020-12-21 18:11:10 +01:00
userLang = lc_messages
2018-01-08 23:58:46 +01:00
} else if lang , ok := c . environ . Get ( "LANG" ) ; ok {
2020-12-21 18:11:10 +01:00
userLang = lang
2018-01-08 23:58:46 +01:00
}
c . environ . UnsetWithPrefix ( "LC_" )
2020-12-21 18:11:10 +01:00
if userLang != "" {
c . environ . Set ( "LC_MESSAGES" , userLang )
2018-01-08 23:58:46 +01:00
}
// The for LANG, use C.UTF-8 if it exists (Debian currently, proposed
// for others)
if inList ( "C.UTF-8" , locales ) {
c . environ . Set ( "LANG" , "C.UTF-8" )
2018-08-08 02:21:36 +02:00
} else if inList ( "C.utf8" , locales ) {
// These normalize to the same thing
c . environ . Set ( "LANG" , "C.UTF-8" )
2018-01-08 23:58:46 +01:00
} else if inList ( "en_US.UTF-8" , locales ) {
c . environ . Set ( "LANG" , "en_US.UTF-8" )
} else if inList ( "en_US.utf8" , locales ) {
// These normalize to the same thing
c . environ . Set ( "LANG" , "en_US.UTF-8" )
} else {
ctx . Fatalln ( "System doesn't support either C.UTF-8 or en_US.UTF-8" )
}
}
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
func ( c * configImpl ) Environment ( ) * Environment {
return c . environ
}
func ( c * configImpl ) Arguments ( ) [ ] string {
return c . arguments
}
2021-09-02 17:23:06 +02:00
func ( c * configImpl ) SoongBuildInvocationNeeded ( ) bool {
if len ( c . Arguments ( ) ) > 0 {
// Explicit targets requested that are not special targets like b2pbuild
// or the JSON module graph
return true
}
2022-09-28 22:43:08 +02:00
if ! c . JsonModuleGraph ( ) && ! c . Bp2Build ( ) && ! c . Queryview ( ) && ! c . SoongDocs ( ) && ! c . ApiBp2build ( ) {
2021-09-02 17:23:06 +02:00
// Command line was empty, the default Ninja target is built
return true
}
2021-12-15 21:03:19 +01:00
// bp2build + dist may be used to dist bp2build logs but does not require SoongBuildInvocation
if c . Dist ( ) && ! c . Bp2Build ( ) {
return true
}
2021-09-02 17:23:06 +02:00
// build.ninja doesn't need to be generated
return false
}
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
func ( c * configImpl ) OutDir ( ) string {
if outDir , ok := c . environ . Get ( "OUT_DIR" ) ; ok {
2019-07-09 02:26:47 +02:00
return outDir
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
}
return "out"
}
2017-02-05 02:30:44 +01:00
func ( c * configImpl ) DistDir ( ) string {
2022-08-30 19:15:04 +02:00
return c . distDir
2020-12-10 12:32:38 +01:00
}
func ( c * configImpl ) RealDistDir ( ) string {
2018-10-21 06:33:41 +02:00
return c . distDir
2017-02-05 02:30:44 +01:00
}
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
func ( c * configImpl ) NinjaArgs ( ) [ ] string {
2020-11-27 13:35:20 +01:00
if c . skipKati {
2017-08-05 00:06:27 +02:00
return c . arguments
}
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
return c . ninjaArgs
}
2020-11-02 08:56:20 +01:00
func ( c * configImpl ) BazelOutDir ( ) string {
return filepath . Join ( c . OutDir ( ) , "bazel" )
}
2022-11-11 20:21:03 +01:00
func ( c * configImpl ) bazelOutputBase ( ) string {
return filepath . Join ( c . BazelOutDir ( ) , "output" )
}
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
func ( c * configImpl ) SoongOutDir ( ) string {
return filepath . Join ( c . OutDir ( ) , "soong" )
}
2022-11-03 18:02:10 +01:00
func ( c * configImpl ) ApiSurfacesOutDir ( ) string {
return filepath . Join ( c . OutDir ( ) , "api_surfaces" )
}
2021-09-01 08:57:48 +02:00
func ( c * configImpl ) PrebuiltOS ( ) string {
switch runtime . GOOS {
case "linux" :
return "linux-x86"
case "darwin" :
return "darwin-x86"
default :
panic ( "Unknown GOOS" )
}
}
2021-11-02 14:42:04 +01:00
2021-09-01 08:57:48 +02:00
func ( c * configImpl ) HostToolDir ( ) string {
2021-10-26 00:40:32 +02:00
if c . SkipKatiNinja ( ) {
return filepath . Join ( c . SoongOutDir ( ) , "host" , c . PrebuiltOS ( ) , "bin" )
} else {
return filepath . Join ( c . OutDir ( ) , "host" , c . PrebuiltOS ( ) , "bin" )
}
2021-09-01 08:57:48 +02:00
}
2021-09-07 09:10:33 +02:00
func ( c * configImpl ) NamedGlobFile ( name string ) string {
2021-11-02 14:42:04 +01:00
return shared . JoinPath ( c . SoongOutDir ( ) , "globs-" + name + ".ninja" )
2021-09-07 09:10:33 +02:00
}
2021-09-08 15:31:14 +02:00
func ( c * configImpl ) UsedEnvFile ( tag string ) string {
return shared . JoinPath ( c . SoongOutDir ( ) , usedEnvFile + "." + tag )
}
2022-10-26 09:26:50 +02:00
func ( c * configImpl ) Bp2BuildFilesMarkerFile ( ) string {
return shared . JoinPath ( c . SoongOutDir ( ) , "bp2build_files_marker" )
}
func ( c * configImpl ) Bp2BuildWorkspaceMarkerFile ( ) string {
2021-11-02 14:42:04 +01:00
return shared . JoinPath ( c . SoongOutDir ( ) , "bp2build_workspace_marker" )
2021-08-12 14:03:55 +02:00
}
2021-09-06 18:31:46 +02:00
func ( c * configImpl ) SoongDocsHtml ( ) string {
return shared . JoinPath ( c . SoongOutDir ( ) , "docs/soong_build.html" )
}
2021-09-06 17:08:02 +02:00
func ( c * configImpl ) QueryviewMarkerFile ( ) string {
return shared . JoinPath ( c . SoongOutDir ( ) , "queryview.marker" )
}
2022-09-28 22:43:08 +02:00
func ( c * configImpl ) ApiBp2buildMarkerFile ( ) string {
return shared . JoinPath ( c . SoongOutDir ( ) , "api_bp2build.marker" )
}
2021-08-25 14:14:13 +02:00
func ( c * configImpl ) ModuleGraphFile ( ) string {
return shared . JoinPath ( c . SoongOutDir ( ) , "module-graph.json" )
}
2022-01-25 06:50:25 +01:00
func ( c * configImpl ) ModuleActionsFile ( ) string {
return shared . JoinPath ( c . SoongOutDir ( ) , "module-actions.json" )
}
2017-03-30 02:29:06 +02:00
func ( c * configImpl ) TempDir ( ) string {
return shared . TempDirForOutDir ( c . SoongOutDir ( ) )
}
2017-08-04 21:30:12 +02:00
func ( c * configImpl ) FileListDir ( ) string {
return filepath . Join ( c . OutDir ( ) , ".module_paths" )
}
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
func ( c * configImpl ) KatiSuffix ( ) string {
if c . katiSuffix != "" {
return c . katiSuffix
}
panic ( "SetKatiSuffix has not been called" )
}
2017-11-17 02:55:00 +01:00
// Checkbuild returns true if "checkbuild" was one of the build goals, which means that the
// user is interested in additional checks at the expense of build time.
func ( c * configImpl ) Checkbuild ( ) bool {
return c . checkbuild
}
2017-02-05 02:30:44 +01:00
func ( c * configImpl ) Dist ( ) bool {
return c . dist
}
2021-09-02 17:23:06 +02:00
func ( c * configImpl ) JsonModuleGraph ( ) bool {
return c . jsonModuleGraph
}
func ( c * configImpl ) Bp2Build ( ) bool {
return c . bp2build
}
2022-09-28 22:43:08 +02:00
func ( c * configImpl ) ApiBp2build ( ) bool {
return c . apiBp2build
}
2021-09-06 17:08:02 +02:00
func ( c * configImpl ) Queryview ( ) bool {
return c . queryview
}
2021-09-06 18:31:46 +02:00
func ( c * configImpl ) SoongDocs ( ) bool {
return c . soongDocs
}
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
func ( c * configImpl ) IsVerbose ( ) bool {
return c . verbose
}
2023-03-09 19:19:35 +01:00
func ( c * configImpl ) MultitreeBuild ( ) bool {
return c . multitreeBuild
}
2023-03-14 16:10:45 +01:00
func ( c * configImpl ) NinjaWeightListSource ( ) NinjaWeightListSource {
return c . ninjaWeightListSource
}
2020-11-27 13:35:20 +01:00
func ( c * configImpl ) SkipKati ( ) bool {
return c . skipKati
}
2021-06-04 11:08:08 +02:00
func ( c * configImpl ) SkipKatiNinja ( ) bool {
return c . skipKatiNinja
}
2021-08-10 15:01:13 +02:00
func ( c * configImpl ) SkipSoong ( ) bool {
return c . skipSoong
}
2021-03-16 08:55:23 +01:00
func ( c * configImpl ) SkipNinja ( ) bool {
return c . skipNinja
}
2021-06-04 11:09:01 +02:00
func ( c * configImpl ) SetSkipNinja ( v bool ) {
c . skipNinja = v
}
2020-11-27 13:35:20 +01:00
func ( c * configImpl ) SkipConfig ( ) bool {
return c . skipConfig
2017-08-05 00:06:27 +02:00
}
2023-01-17 21:40:22 +01:00
func ( c * configImpl ) BuildFromTextStub ( ) bool {
return c . buildFromTextStub
}
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
func ( c * configImpl ) TargetProduct ( ) string {
if v , ok := c . environ . Get ( "TARGET_PRODUCT" ) ; ok {
return v
}
panic ( "TARGET_PRODUCT is not defined" )
}
2017-05-13 04:28:13 +02:00
func ( c * configImpl ) TargetDevice ( ) string {
return c . targetDevice
}
func ( c * configImpl ) SetTargetDevice ( device string ) {
c . targetDevice = device
}
func ( c * configImpl ) TargetBuildVariant ( ) string {
if v , ok := c . environ . Get ( "TARGET_BUILD_VARIANT" ) ; ok {
return v
}
panic ( "TARGET_BUILD_VARIANT is not defined" )
}
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
func ( c * configImpl ) KatiArgs ( ) [ ] string {
return c . katiArgs
}
func ( c * configImpl ) Parallel ( ) int {
return c . parallel
}
2023-02-21 17:50:29 +01:00
func ( c * configImpl ) GetSourceRootDirs ( ) [ ] string {
return c . sourceRootDirs
}
func ( c * configImpl ) SetSourceRootDirs ( i [ ] string ) {
c . sourceRootDirs = i
}
2022-11-08 19:42:16 +01:00
func ( c * configImpl ) GetIncludeTags ( ) [ ] string {
return c . includeTags
}
func ( c * configImpl ) SetIncludeTags ( i [ ] string ) {
c . includeTags = i
}
2022-12-07 22:57:38 +01:00
func ( c * configImpl ) GetLogsPrefix ( ) string {
return c . logsPrefix
}
func ( c * configImpl ) SetLogsPrefix ( prefix string ) {
c . logsPrefix = prefix
}
2019-11-15 22:18:43 +01:00
func ( c * configImpl ) HighmemParallel ( ) int {
if i , ok := c . environ . GetInt ( "NINJA_HIGHMEM_NUM_JOBS" ) ; ok {
return i
}
const minMemPerHighmemProcess = 8 * 1024 * 1024 * 1024
parallel := c . Parallel ( )
if c . UseRemoteBuild ( ) {
// Ninja doesn't support nested pools, and when remote builds are enabled the total ninja parallelism
// is set very high (i.e. 500). Using a large value here would cause the total number of running jobs
// to be the sum of the sizes of the local and highmem pools, which will cause extra CPU contention.
// Return 1/16th of the size of the local pool, rounding up.
return ( parallel + 15 ) / 16
} else if c . totalRAM == 0 {
// Couldn't detect the total RAM, don't restrict highmem processes.
return parallel
2020-05-27 08:02:29 +02:00
} else if c . totalRAM <= 16 * 1024 * 1024 * 1024 {
// Less than 16GB of ram, restrict to 1 highmem processes
return 1
2019-11-15 22:18:43 +01:00
} else if c . totalRAM <= 32 * 1024 * 1024 * 1024 {
// Less than 32GB of ram, restrict to 2 highmem processes
return 2
} else if p := int ( c . totalRAM / minMemPerHighmemProcess ) ; p < parallel {
// If less than 8GB total RAM per process, reduce the number of highmem processes
return p
}
// No restriction on highmem processes
return parallel
}
2019-12-27 18:35:42 +01:00
func ( c * configImpl ) TotalRAM ( ) uint64 {
return c . totalRAM
}
2020-09-21 19:39:24 +02:00
// ForceUseGoma determines whether we should override Goma deprecation
// and use Goma for the current build or not.
func ( c * configImpl ) ForceUseGoma ( ) bool {
if v , ok := c . environ . Get ( "FORCE_USE_GOMA" ) ; ok {
v = strings . TrimSpace ( v )
if v != "" && v != "false" {
return true
}
}
return false
}
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
func ( c * configImpl ) UseGoma ( ) bool {
if v , ok := c . environ . Get ( "USE_GOMA" ) ; ok {
v = strings . TrimSpace ( v )
if v != "" && v != "false" {
return true
}
}
return false
}
2019-01-10 02:14:16 +01:00
func ( c * configImpl ) StartGoma ( ) bool {
if ! c . UseGoma ( ) {
return false
}
if v , ok := c . environ . Get ( "NOSTART_GOMA" ) ; ok {
v = strings . TrimSpace ( v )
if v != "" && v != "false" {
return false
}
}
return true
}
2019-07-17 14:30:04 +02:00
func ( c * configImpl ) UseRBE ( ) bool {
2022-01-26 04:11:01 +01:00
if v , ok := c . Environment ( ) . Get ( "USE_RBE" ) ; ok {
2019-07-17 14:30:04 +02:00
v = strings . TrimSpace ( v )
2020-11-11 22:01:25 +01:00
if v != "" && v != "false" {
return true
}
}
return false
}
2022-08-19 04:04:11 +02:00
func ( c * configImpl ) BazelBuildEnabled ( ) bool {
2022-10-18 22:10:16 +02:00
return c . bazelProdMode || c . bazelDevMode || c . bazelStagingMode
2021-04-20 21:32:07 +02:00
}
2019-07-17 14:30:04 +02:00
func ( c * configImpl ) StartRBE ( ) bool {
if ! c . UseRBE ( ) {
return false
}
if v , ok := c . environ . Get ( "NOSTART_RBE" ) ; ok {
v = strings . TrimSpace ( v )
if v != "" && v != "false" {
return false
}
}
return true
}
2022-05-27 13:48:37 +02:00
func ( c * configImpl ) rbeProxyLogsDir ( ) string {
for _ , f := range [ ] string { "RBE_proxy_log_dir" , "FLAG_output_dir" } {
2020-07-07 14:48:26 +02:00
if v , ok := c . environ . Get ( f ) ; ok {
return v
}
}
2022-05-27 13:48:37 +02:00
buildTmpDir := shared . TempDirForOutDir ( c . SoongOutDir ( ) )
return filepath . Join ( buildTmpDir , "rbe" )
2020-08-12 07:26:23 +02:00
}
2022-05-27 13:48:37 +02:00
func ( c * configImpl ) shouldCleanupRBELogsDir ( ) bool {
// Perform a log directory cleanup only when the log directory
// is auto created by the build rather than user-specified.
for _ , f := range [ ] string { "RBE_proxy_log_dir" , "FLAG_output_dir" } {
if _ , ok := c . environ . Get ( f ) ; ok {
return false
2020-08-12 07:26:23 +02:00
}
}
2022-05-27 13:48:37 +02:00
return true
2020-08-12 07:26:23 +02:00
}
func ( c * configImpl ) rbeExecRoot ( ) string {
for _ , f := range [ ] string { "RBE_exec_root" , "FLAG_exec_root" } {
if v , ok := c . environ . Get ( f ) ; ok {
return v
}
}
wd , err := os . Getwd ( )
if err != nil {
return ""
}
return wd
}
func ( c * configImpl ) rbeDir ( ) string {
if v , ok := c . environ . Get ( "RBE_DIR" ) ; ok {
return v
}
return "prebuilts/remoteexecution-client/live/"
}
func ( c * configImpl ) rbeReproxy ( ) string {
for _ , f := range [ ] string { "RBE_re_proxy" , "FLAG_re_proxy" } {
if v , ok := c . environ . Get ( f ) ; ok {
return v
}
}
return filepath . Join ( c . rbeDir ( ) , "reproxy" )
}
func ( c * configImpl ) rbeAuth ( ) ( string , string ) {
2022-03-18 06:39:56 +01:00
credFlags := [ ] string {
"use_application_default_credentials" ,
"use_gce_credentials" ,
"credential_file" ,
"use_google_prod_creds" ,
}
2020-08-12 07:26:23 +02:00
for _ , cf := range credFlags {
for _ , f := range [ ] string { "RBE_" + cf , "FLAG_" + cf } {
if v , ok := c . environ . Get ( f ) ; ok {
v = strings . TrimSpace ( v )
if v != "" && v != "false" && v != "0" {
return "RBE_" + cf , v
}
}
}
}
return "RBE_use_application_default_credentials" , "true"
2020-07-07 14:48:26 +02:00
}
2022-05-27 13:48:37 +02:00
func ( c * configImpl ) rbeSockAddr ( dir string ) ( string , error ) {
maxNameLen := len ( syscall . RawSockaddrUnix { } . Path )
base := fmt . Sprintf ( "reproxy_%v.sock" , rbeRandPrefix )
name := filepath . Join ( dir , base )
if len ( name ) < maxNameLen {
return name , nil
}
name = filepath . Join ( "/tmp" , base )
if len ( name ) < maxNameLen {
return name , nil
}
return "" , fmt . Errorf ( "cannot generate a proxy socket address shorter than the limit of %v" , maxNameLen )
}
2022-04-27 20:52:56 +02:00
// IsGooglerEnvironment returns true if the current build is running
// on a Google developer machine and false otherwise.
func ( c * configImpl ) IsGooglerEnvironment ( ) bool {
cf := "ANDROID_BUILD_ENVIRONMENT_CONFIG"
if v , ok := c . environ . Get ( cf ) ; ok {
return v == "googler"
}
return false
}
// GoogleProdCredsExist determine whether credentials exist on the
// Googler machine to use remote execution.
func ( c * configImpl ) GoogleProdCredsExist ( ) bool {
2023-03-03 20:47:17 +01:00
if googleProdCredsExistCache {
return googleProdCredsExistCache
}
2022-04-27 20:52:56 +02:00
if _ , err := exec . Command ( "/usr/bin/prodcertstatus" , "--simple_output" , "--nocheck_loas" ) . Output ( ) ; err != nil {
return false
}
2023-03-03 20:47:17 +01:00
googleProdCredsExistCache = true
2022-04-27 20:52:56 +02:00
return true
}
// UseRemoteBuild indicates whether to use a remote build acceleration system
// to speed up the build.
2019-11-11 23:57:42 +01:00
func ( c * configImpl ) UseRemoteBuild ( ) bool {
return c . UseGoma ( ) || c . UseRBE ( )
}
2022-04-27 20:52:56 +02:00
// StubbyExists checks whether the stubby binary exists on the machine running
// the build.
func ( c * configImpl ) StubbyExists ( ) bool {
if _ , err := exec . LookPath ( "stubby" ) ; err != nil {
return false
}
return true
}
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
// RemoteParallel controls how many remote jobs (i.e., commands which contain
2017-03-30 02:29:06 +02:00
// gomacc) are run in parallel. Note the parallelism of all other jobs is
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
// still limited by Parallel()
func ( c * configImpl ) RemoteParallel ( ) int {
2019-11-15 22:18:43 +01:00
if ! c . UseRemoteBuild ( ) {
return 0
}
if i , ok := c . environ . GetInt ( "NINJA_REMOTE_NUM_JOBS" ) ; ok {
return i
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
}
return 500
}
func ( c * configImpl ) SetKatiArgs ( args [ ] string ) {
c . katiArgs = args
}
func ( c * configImpl ) SetNinjaArgs ( args [ ] string ) {
c . ninjaArgs = args
}
func ( c * configImpl ) SetKatiSuffix ( suffix string ) {
c . katiSuffix = suffix
}
2017-08-05 00:06:27 +02:00
func ( c * configImpl ) LastKatiSuffixFile ( ) string {
return filepath . Join ( c . OutDir ( ) , "last_kati_suffix" )
}
func ( c * configImpl ) HasKatiSuffix ( ) bool {
return c . katiSuffix != ""
}
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
func ( c * configImpl ) KatiEnvFile ( ) string {
return filepath . Join ( c . OutDir ( ) , "env" + c . KatiSuffix ( ) + ".sh" )
}
2018-09-26 23:58:30 +02:00
func ( c * configImpl ) KatiBuildNinjaFile ( ) string {
return filepath . Join ( c . OutDir ( ) , "build" + c . KatiSuffix ( ) + katiBuildSuffix + ".ninja" )
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
}
Add a Kati-based packaging step
The idea is that we'd move the installation and packaging tasks over to
it, using data from Soong & the Kati reading Android.mk files.
This would allow us to make more fundamental changes about how we
package things without having to adjust makefiles throughout the tree.
Possible use cases:
* Moving some information from Soong's Android.mk output to a file read
by the packaging step may allow us to read the Android.mk files less
often, speeding up builds.
* Refactoring our current two-stage ASAN builds to run the Kati build
step twice, writing into different object directories, then have a
single packaging step that reads both outputs. Soong already has the
capability of writing out a single ninja file with all the asan
combinations.
* Running two build steps, one building the system-related modules
using a "generic" device configuration, and one building the vendor
modules using a specific device configuration. This could enforce a
GSI/mainline system vs vendor split in a single build invocation.
* If all installation is through this tool, it will be much easier to
track what should no longer be installed on an incremental build,
reducing the need for installclean.
* Changing PRODUCT_PACKAGES should be a much faster operation, which
means we could keep track of local additions to the images. Then
`mma` would be more persistent, instead of installing something once,
then never updating it again.
Eventually we plan on switching from Kati to something Go-based, but
this is a more incremental approach while we clean up everything else.
Currently, this just moves the dist-for-goal handling over to the
packaging step, so that we don't need to read Android.mk files when
DIST_DIR changes, or we switch between dist vs not.
Bug: 116968624
Bug: 117463001
Test: m nothing
Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
func ( c * configImpl ) KatiPackageNinjaFile ( ) string {
return filepath . Join ( c . OutDir ( ) , "build" + c . KatiSuffix ( ) + katiPackageSuffix + ".ninja" )
}
2022-08-16 02:57:30 +02:00
func ( c * configImpl ) SoongVarsFile ( ) string {
return filepath . Join ( c . SoongOutDir ( ) , "soong.variables" )
}
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
func ( c * configImpl ) SoongNinjaFile ( ) string {
return filepath . Join ( c . SoongOutDir ( ) , "build.ninja" )
}
func ( c * configImpl ) CombinedNinjaFile ( ) string {
2017-08-05 00:06:27 +02:00
if c . katiSuffix == "" {
return filepath . Join ( c . OutDir ( ) , "combined.ninja" )
}
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
return filepath . Join ( c . OutDir ( ) , "combined" + c . KatiSuffix ( ) + ".ninja" )
}
func ( c * configImpl ) SoongAndroidMk ( ) string {
return filepath . Join ( c . SoongOutDir ( ) , "Android-" + c . TargetProduct ( ) + ".mk" )
}
func ( c * configImpl ) SoongMakeVarsMk ( ) string {
return filepath . Join ( c . SoongOutDir ( ) , "make_vars-" + c . TargetProduct ( ) + ".mk" )
}
2017-05-19 00:29:04 +02:00
func ( c * configImpl ) ProductOut ( ) string {
2017-09-08 23:35:43 +02:00
return filepath . Join ( c . OutDir ( ) , "target" , "product" , c . TargetDevice ( ) )
2017-05-19 00:29:04 +02:00
}
2017-05-13 04:28:13 +02:00
func ( c * configImpl ) DevicePreviousProductConfig ( ) string {
2017-05-19 00:29:04 +02:00
return filepath . Join ( c . ProductOut ( ) , "previous_build_config.mk" )
}
Add a Kati-based packaging step
The idea is that we'd move the installation and packaging tasks over to
it, using data from Soong & the Kati reading Android.mk files.
This would allow us to make more fundamental changes about how we
package things without having to adjust makefiles throughout the tree.
Possible use cases:
* Moving some information from Soong's Android.mk output to a file read
by the packaging step may allow us to read the Android.mk files less
often, speeding up builds.
* Refactoring our current two-stage ASAN builds to run the Kati build
step twice, writing into different object directories, then have a
single packaging step that reads both outputs. Soong already has the
capability of writing out a single ninja file with all the asan
combinations.
* Running two build steps, one building the system-related modules
using a "generic" device configuration, and one building the vendor
modules using a specific device configuration. This could enforce a
GSI/mainline system vs vendor split in a single build invocation.
* If all installation is through this tool, it will be much easier to
track what should no longer be installed on an incremental build,
reducing the need for installclean.
* Changing PRODUCT_PACKAGES should be a much faster operation, which
means we could keep track of local additions to the images. Then
`mma` would be more persistent, instead of installing something once,
then never updating it again.
Eventually we plan on switching from Kati to something Go-based, but
this is a more incremental approach while we clean up everything else.
Currently, this just moves the dist-for-goal handling over to the
packaging step, so that we don't need to read Android.mk files when
DIST_DIR changes, or we switch between dist vs not.
Bug: 116968624
Bug: 117463001
Test: m nothing
Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-09-27 00:00:42 +02:00
func ( c * configImpl ) KatiPackageMkDir ( ) string {
return filepath . Join ( c . ProductOut ( ) , "obj" , "CONFIG" , "kati_packaging" )
}
2017-05-19 00:29:04 +02:00
func ( c * configImpl ) hostOutRoot ( ) string {
2017-09-08 23:35:43 +02:00
return filepath . Join ( c . OutDir ( ) , "host" )
2017-05-19 00:29:04 +02:00
}
func ( c * configImpl ) HostOut ( ) string {
return filepath . Join ( c . hostOutRoot ( ) , c . HostPrebuiltTag ( ) )
}
// This probably needs to be multi-valued, so not exporting it for now
func ( c * configImpl ) hostCrossOut ( ) string {
if runtime . GOOS == "linux" {
return filepath . Join ( c . hostOutRoot ( ) , "windows-x86" )
} else {
return ""
}
2017-05-13 04:28:13 +02:00
}
Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.
There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.
The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.
We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.
Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
(also importing ninja traces into the same stream). I had this working
in a previous version of this patch, but removed it to keep the size
down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
can always save the verbose log even if we're not printing it out to
the console
Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2016-08-22 00:17:17 +02:00
func ( c * configImpl ) HostPrebuiltTag ( ) string {
if runtime . GOOS == "linux" {
return "linux-x86"
} else if runtime . GOOS == "darwin" {
return "darwin-x86"
} else {
panic ( "Unsupported OS" )
}
}
2017-04-27 23:28:00 +02:00
2017-10-13 05:20:41 +02:00
func ( c * configImpl ) PrebuiltBuildTool ( name string ) string {
2017-04-27 23:28:00 +02:00
if v , ok := c . environ . Get ( "SANITIZE_HOST" ) ; ok {
if sanitize := strings . Fields ( v ) ; inList ( "address" , sanitize ) {
2017-10-13 05:20:41 +02:00
asan := filepath . Join ( "prebuilts/build-tools" , c . HostPrebuiltTag ( ) , "asan/bin" , name )
if _ , err := os . Stat ( asan ) ; err == nil {
return asan
}
2017-04-27 23:28:00 +02:00
}
}
return filepath . Join ( "prebuilts/build-tools" , c . HostPrebuiltTag ( ) , "bin" , name )
}
2018-04-05 07:25:56 +02:00
func ( c * configImpl ) SetBuildBrokenDupRules ( val bool ) {
c . brokenDupRules = val
}
func ( c * configImpl ) BuildBrokenDupRules ( ) bool {
return c . brokenDupRules
}
2018-05-02 09:06:28 +02:00
2019-04-09 19:22:43 +02:00
func ( c * configImpl ) SetBuildBrokenUsesNetwork ( val bool ) {
c . brokenUsesNetwork = val
}
func ( c * configImpl ) BuildBrokenUsesNetwork ( ) bool {
return c . brokenUsesNetwork
}
2020-01-03 04:10:38 +01:00
func ( c * configImpl ) SetBuildBrokenNinjaUsesEnvVars ( val [ ] string ) {
c . brokenNinjaEnvVars = val
}
func ( c * configImpl ) BuildBrokenNinjaUsesEnvVars ( ) [ ] string {
return c . brokenNinjaEnvVars
}
2018-05-02 09:06:28 +02:00
func ( c * configImpl ) SetTargetDeviceDir ( dir string ) {
c . targetDeviceDir = dir
}
func ( c * configImpl ) TargetDeviceDir ( ) string {
return c . targetDeviceDir
}
2018-06-16 06:54:47 +02:00
2020-06-01 19:29:30 +02:00
func ( c * configImpl ) BuildDateTime ( ) string {
return c . buildDateTime
}
func ( c * configImpl ) MetricsUploaderApp ( ) string {
2021-07-27 23:29:06 +02:00
return c . metricsUploader
2020-06-01 19:29:30 +02:00
}
2020-12-08 20:42:08 +01:00
2021-11-10 15:55:20 +01:00
// LogsDir returns the absolute path to the logs directory where build log and
// metrics files are located. By default, the logs directory is the out
2020-12-08 20:42:08 +01:00
// directory. If the argument dist is specified, the logs directory
// is <dist_dir>/logs.
func ( c * configImpl ) LogsDir ( ) string {
2021-11-10 15:55:20 +01:00
dir := c . OutDir ( )
2020-12-08 20:42:08 +01:00
if c . Dist ( ) {
2020-12-10 12:32:38 +01:00
// Always write logs to the real dist dir, even if Bazel is using a rigged dist dir for other files
2021-11-10 15:55:20 +01:00
dir = filepath . Join ( c . RealDistDir ( ) , "logs" )
2020-12-08 20:42:08 +01:00
}
2021-11-10 15:55:20 +01:00
absDir , err := filepath . Abs ( dir )
if err != nil {
fmt . Fprintf ( os . Stderr , "\nError making log dir '%s' absolute: %s\n" , dir , err . Error ( ) )
os . Exit ( 1 )
}
return absDir
2020-12-08 20:42:08 +01:00
}
// BazelMetricsDir returns the <logs dir>/bazel_metrics directory
// where the bazel profiles are located.
func ( c * configImpl ) BazelMetricsDir ( ) string {
return filepath . Join ( c . LogsDir ( ) , "bazel_metrics" )
}
2021-06-01 20:43:55 +02:00
2022-03-03 18:01:40 +01:00
// MkFileMetrics returns the file path for make-related metrics.
func ( c * configImpl ) MkMetrics ( ) string {
return filepath . Join ( c . LogsDir ( ) , "mk_metrics.pb" )
}
2021-06-01 20:43:55 +02:00
func ( c * configImpl ) SetEmptyNinjaFile ( v bool ) {
c . emptyNinjaFile = v
}
func ( c * configImpl ) EmptyNinjaFile ( ) bool {
return c . emptyNinjaFile
}
2021-07-27 23:29:06 +02:00
2022-10-26 18:56:41 +02:00
func ( c * configImpl ) IsBazelMixedBuildForceDisabled ( ) bool {
return c . Environment ( ) . IsEnvTrue ( "BUILD_BROKEN_DISABLE_BAZEL" )
}
2023-02-23 23:28:06 +01:00
func ( c * configImpl ) IsPersistentBazelEnabled ( ) bool {
return c . Environment ( ) . IsEnvTrue ( "USE_PERSISTENT_BAZEL" )
}
2022-11-29 01:47:59 +01:00
func ( c * configImpl ) BazelModulesForceEnabledByFlag ( ) string {
return c . bazelForceEnabledModules
}
2022-12-02 23:22:40 +01:00
func ( c * configImpl ) SkipMetricsUpload ( ) bool {
return c . skipMetricsUpload
}
2022-12-07 22:57:38 +01:00
// Returns a Time object if one was passed via a command-line flag.
// Otherwise returns the passed default.
func ( c * configImpl ) BuildStartedTimeOrDefault ( defaultTime time . Time ) time . Time {
if c . buildStartedTime == 0 {
return defaultTime
}
return time . UnixMilli ( c . buildStartedTime )
}
2021-07-27 23:29:06 +02:00
func GetMetricsUploader ( topDir string , env * Environment ) string {
if p , ok := env . Get ( "METRICS_UPLOADER" ) ; ok {
metricsUploader := filepath . Join ( topDir , p )
if _ , err := os . Stat ( metricsUploader ) ; err == nil {
return metricsUploader
}
}
return ""
}