This is an initial CL that collects the environment variables
USE_GOMA and USE_RBE. This is a needed metric to determine how
many users are currently using GOMA so we can help them migrate
to RBE.
Bug: b/163598127
Test: m nothing and checked soong_metrics
Change-Id: I39536eabad01aafbb3bd7182515d72230b36f658
This hasn't worked for a couple years, and continues to bitrot. Just
remove it.
Adds a bpfix rule so that we can eventually remove the
product_variables.pdk definition, which is now always a no-op.
Test: treehugger
Change-Id: I830b54d419b59f6db1d4617b45e61a78234f57a7
Merged-In: I830b54d419b59f6db1d4617b45e61a78234f57a7
"exec format error" showed up on Mac when running the rbe_test.go.
The generated bash scripts did not have a newline at the end of
the file.
Bug: b/162600461
Fixes: b/162600461
Test: "m blueprint_tools" on Android Mac builder.
Change-Id: I425fa5f428733b6569c3fcdaa312023f9761d020
Disable the error test cases to allow developers to build
AOSP on Mac.
Bug: b/162600461
Test: m nothing
Change-Id: Ic8c6fe04743a8e4fe197a1bb71259036fb749edc
Disable the error test cases to allow developers to build
AOSP on Mac.
Bug: b/162600461
Test: m nothing
Change-Id: Iac4373dcb10710d8a2325eeb3ee20a6eeca6143f
Pass -n to blueprint to make it use validation dependencies for tests,
which will let them run in parallel with soong_build.
Bug: 156428456
Test: m nothing
Change-Id: I706c14330543e2a7728e5dc385976900c67cb2f5
Currently, the build timestamp saved to the build_date_timestamp
field in the MetricsBase metrics protobuf usually comes from a text
file named build_date.txt. It may not be accurate for metrics purposes.
Instead, use the build start time from Soong main UI for a better
timestamp.
Bug: b/140638454
Test: Ran and checked the build_date_timestamp by running the printproto
command.
Change-Id: I4d893fbbf9830ba21911e56ae13bc5272be47ae6
From aosp/1329396, the RBE metrics protobuf file is part of the
metrics uploading process. The RBE metrics protobuf file is
generated by running the bootstrap shutdown command. A new function
named DumpRBEMetrics was written in order to generate the RBE metrics
protobuf file before sending to the uploading process.
Bug: b/140638454
Test: * Unit test cases
* Ran RBE build on my local host and verified the
metrics protobuf file is created.
* Ran non-RBE build after RBE build and verified that
the previous metrics protobuf file was deleted.
Change-Id: I4b8068905cb67c4b8c2d94793917b98974fed707
Help debugging finder issues by disting module_paths/files.db for
later analysis.
Bug: 157656545
Test: treehugger
Change-Id: Idfa4cffe7ef724b399a2667efd487f9c3c76bb6b
Specifically, make Android.bp.list and soong.variables explicit inputs
While this is not a comprehensive list of all inputs
of this action (as the action depends on all blueprint
files in the source tree), this is closer to the truth.
This is a rollforward CL, which was originally rolled
back, as path validation failed for when OUT_DIR was
an absolute path. Validation has now been relaxed.
Test: Manually verified ninja output, checkbuild approved validation
for aosp-crosshatch, and manually ran frameworks/rs/build_rs.py with
patch fix
Change-Id: I4eb0d517f57336dd54eaa4bd31f46df9e93e6da2
Debugging issues on the build servers can be difficult because the
intermediate files are not visible. Gzip ninja file and Makefiles
generated by Soong and the ninja files generated by Kati to the dist
directory, and also copy all of the finder output files.
Bug: 157656545
Test: m dist nothing
Change-Id: I48d75305e551ccae81c7a55721981cf58acd838b
While this is not a comprehensive list of all inputs of this action (as the action
depends on all blueprint files in the source tree), this closer to the truth.
Test: Manually verified ninja output, checkbuild approved validation for aosp-crosshatch
Change-Id: I77a0f612afb025232bee7d3ac31257b808991829
The metrics uploader was currently running on foreground where it
would copy the metrics files in a separate directory and then forked
into the background for the upload process. As a result, running the
lunch command would take a second longer to run since each metrics
uploader run had an average of half a second.
Bug: 140638454
Test: * Wrote and updated unit test cases.
* Set ANDROID_ENABLE_METRICS_UPLOAD to point to the latest
metrics_uploader bash script. Executed the "lunch 1" command
and measured the running time. Executed "m nothing" command
and checked that the metrics were uploaded.
* Ran "lunch 1" and "m nothing" with
ANDROID_ENABLE_METRICS_UPLOAD=""
* Removed oauth from metrics_uploader and ran "m nothing" and
"lunch 1". The oauth Message appeared only to "m nothing"
Change-Id: I13c61e666c8f44613dee291a704cef6a27335188
External applications such as metrics uploader depends on the TMPDIR.
DumpMakeVars is changing the TMPDIR environment variable value. Once
the work is done, TMPDIR directory is deleted. Additional flow of
the tool or external applications can panic since TMPDIR directory
does not exist. Since this is an isolated case, pass the TMPDIR
directory to the command of dumping the makefile variables.
Bug: b/140638454
Test: * Ran lunch command with ANDROID_ENABLE_METRICS_UPLOAD set
and metrics_uploader did not crash.
* m nothing
Change-Id: Ib78a3f8cfb5da48f65bd9bce1511e9abdf3d9cca
Soong now supports the ability to upload metrics to another location
by setting the ANDROID_ENABLE_METRICS_UPLOAD to an uploader that
accepts the upload.proto proto buffer message. When the environment
variable is set, a set of build metrics files (soong_metrics,
rbe_metrics.pb and build_error) is uploaded.
Bug: 140638454
Test: * Wrote unit test cases
* Setup the uploader, built a succcessful and failed aosp_arm-eng
target and monitor the uploading of the metrics.
Change-Id: I76a65739c557dc90345e098ca03119a950ece2d2
Drop down to a single high-mem task when we've got <=16GB, as the system
probably isn't completely free RAM, ninja will be using a couple gigs,
along with whatever "normal" actions will be running concurrently. So
it's unlikely that we can handle two 6-8GB actions along with everything
else.
Also print warnings when we detect <=16GB total RAM, or when we're
running more parallel jobs than than we have GB RAM. These both notify
the user and suggest lowering the `-j` value if they run into problems.
Test: fake totalRAM to [0.5,8,16]GB, checking warning
Test: fake totalRAM to 17GB, `m -j4 nothing` has no warning
Test: `m -j187 nothing` on a 188GB system
Test: `m -j188 nothing` on a 188GB system
Change-Id: Ieb008e9f462d5f40fb65781d94cf116b1caf8446
The stats output will now use the new "DEBUG" message type, which we can
always redirect to verbose.log.gz.
Test: m aprotoc (look in verbose.log.gz)
Change-Id: Ie1b58f12c008ff7d29f11ff7a9807488dba8a504
misc_info.txt's build rule does not contain inputs to tell it to
refresh on an incremental build, so incremental builds (even after
installclean) were reusing this file.
With this change, misc_info.txt is removed by installclean so that an
incremental build generates the file fresh.
Test: # misc_info.txt exists; m installclean; # misc_info.txt gone
Bug: 155930200
Change-Id: Ia319925dad26f98049a988ee1ceb1e20db621353
This was a temporary migration tool, and has been broken ever since the
ninja environment started being restricted. Remove it, since it has
lasted two releases.
Test: TEMPORARY_DISABLE_PATH_RESTRICTIONS=true m
Change-Id: I2e3709525598030f5c24d12fa966e531dfed33ae
A user was trying to use OUT_DIR=/tmp/...-out DIST_DIR=/tmp/...-dist
Test: OUT_DIR=/tmp/test-out DIST_DIR=/tmp/test-dist m nothing
Change-Id: I811cc8e1bda8d766406603df47309a8637cef48c
By the time we do argument parsing, we've already written to $OUT_DIR or
the default, so don't allow overriding the output directory on the
command line.
Test: m OUT_DIR=newdir
Change-Id: I6f8c07cbef3a71d393b766dc5a01ae9c3df8631c
Includes the exit code, real time, user time, system time, and maxrss.
Test: m nothing; check out/soong.log
Change-Id: I93dea9fc1c7fb892150e16e11cae09f2372f4429
This replaces the _kati_always_build_ hack with a ninja feature so that
ninja can actually understand what's happening. This means that we can
turn on some more options and checks around expected output filenames:
* Remove the output file(s) before the command executes
* Error if the output file(s) don't exist after the execution
* Error if the output is a directory
They're turned on immediately during the soong bootstrap runs, as those
run a limited number of rules. The main ninja execution does not remove
the output files, and prints warnings instead of errors for the others.
I'll turn them on as we understand how often those warnings are seen.
Test: m (check for new warnings)
Test: treehugger (check for new warnings)
Change-Id: I7f5c1a361dd1498eb54a2c07a918f3b0aa086e4c
This CL adds a remoteexec package that allows adding a configurable RBE
prefix to the template.
Test: built aosp crosshatch userdebug with and without RBE_CXX_LINKS.
Change-Id: Ica920c3d7f79f2996210b9cbd448126451c1707c
The dist dir is created late in the build process. If a
soong sandbox attempts to mount it before it's created
the sandbox will create an empty file in its place.
Test: lunch aosp_cf_x86_phone-userdebug && make -j dist
Change-Id: Ie6513bf702de6e6322c78753d406d70ea3ccc04d
It's safer to disallow writing to the root filesystem of the sandbox.
Still allow write access to the source, tmp, out and dist directories.
Test: build aosp_cf_x86_64_phone-userdebug
Change-Id: Idf0d3a420669fbf6c3aba24e058f09a314ec6d0e
This makes sure that we don't keep old vendor_boot artifacts around
after a clean.
Signed-off-by: Will McVicker <willmcvicker@google.com>
Fixes: 151957122
Bug: 151949371
Test: touch $OUT/vendor-ramdisk/test-file && make installclean
Change-Id: I357929583f234212a998dceaa31506fce42e0255
If this couldn't read the current user, it would print a generic
message, then proceed to dereference the invalid user struct :(
Provide somewhat reasonable defaults, and better error messages instead.
Test: m nothing
Test: run docker with misconfigured user
Change-Id: I21af77c7d8d1d2d27cb04546667eb1094c62a7a1
We have known problems (docker; distros w/o user namespaces) which mean that we won't be turning this into a fatal error anytime soon. Also remove the bug report link, it's not useful to continue getting the same reports.
Change-Id: I271871d68150417ac938f074d3730cad4518e327
Test: treehugger
Collect the number of modules and variants and some basic statistics
on memory usage inside soong_build by writing out a proto that is
read back in by soong_ui.
Test: examine soong.log
Change-Id: I6926876377a4f6229cf41fdbf166ae03c885ea55
Trying to override the default directory for ccache by using the
CCACHE_DIR environment variable fails unless it is in the allowed
list.
Bug: 149670916
Test: manual
Change-Id: I8e7eea7a5c25d7ea5f0956fafc70d62522f3c4fc
Apparently PackageManager gets confused when the apk disappears, but the
directory still exists.
Test: lunch aosp_arm-eng; m BasicDreams
<remove BasicDreams from PRODUCT_PACKAGES>
m nothing
ls out/target/product/generic/system/app
Change-Id: I486c0ddadde55fad226049d3fb0f09af88117da0
These variables control whether we accept cached-results / not and
whether we should update the cache with locally executed results or not.
I need accpet-cached whitelisted to purge the the RBE cache for the
invalid cache entry we have set for the failure we say yesterday.
Bug: b/148387048
Change-Id: I7344fc083f82e0b7bc11084376a267d19cf30bb8
This CL adds RBE support to javac, r8, and d8 rules which is only
enabled if respective environment variables are set.
Test: an aosp_crosshatch build with and without the new variables.
Change-Id: Ic82f3627944f6a5ee7b9f3228170c2709b1bfcb8
Create a highmem pool based on the total RAM and the number of CPUs,
with an override via the NINJA_HIGHMEM_NUM_JOBS variable. Put
metalava into the highmem pool.
Ninja does not support nested pools, and when goma or RBE is enabled
the maximum ninja parallelism is set very high with local jobs in a
local pool. When both the local pool and highmem pool are enabled,
the total number of local jobs will be as high as the sum of the sizes
of the two pools. Keep the highmem pool limited to 1/16th of the
local pool when remote builds are enabled to try to minimize the
effect while still limiting highmem jobs.
Fixes: 142644983
Test: m nothing, examine pools
Test: m USE_GOMA=true nothing, examine pools
Change-Id: Id79f11f44948992960ac34ecf831dacbe21bd332
Determining the total build time from the existing metrics is
tricky because some of the metrics are overlapping. For example,
the "soong" metric includes "blueprint bootstrap",
"environment check", "minibp", "bpglob", "minibootstrap", and
"bootstrap". Create a new singleton "total" metric that
covers the interesting parts of the build.
Test: m nothing
Change-Id: Iedcf5c9cf0d27252b694d35e6ff66ca1fcf734ac
This relands I12a0f907753fefd1997ab8b4ea2ac331234093cf along with
a fix to blueprint for absolute paths.
Store the current working directory and then change to the root
directory so that all file accesses must go through helpers in
the android package that properly track dependencies.
Change-Id: I24ac485677aa102eec1a2521d16820da6ee1ae77
Fixes: 146437378
Test: m checkbuild
Test: m OUT_DIR=/tmp/out nothing
Store the current working directory and then change to the root
directory so that all file accesses must go through helpers in
the android package that properly track dependencies.
Fixes: 146437378
Test: m checkbuild
Change-Id: I12a0f907753fefd1997ab8b4ea2ac331234093cf
This is used by the art apex test.
Bug: 147197813
Test: m EMMA_INSTRUMENT=true EMMA_INSTRUMENT_FRAMEWORK=true art-check-debug-apex-gen
Change-Id: Id185cd35f16131f2c9a8afeba2e5b87834b0e620
We're passing in ASAN_OPTIONS=detect_leaks=0 on the command line, and
the build started failing after we limited the environment.
Test: forrest of aosp-master/aosp_x86_64-eng
Change-Id: I42c91897c7677e1a249412e5a8bc7bb1edb2f881
Make now exports (partial) lists of what should be installed. Those
lists are incomplete, but if something is removed from the list, we can
remove it from the filesystem so that incremental builds work better.
Test: built-in tests (m blueprint_tools)
Test: remove a module from PRODUCT_PACKAGES, see the print and file removed
Test: change the name of a cts test, see the old one removed from cts
Change-Id: I6bc14c6a5af63383ea265e97ed8b875fa51af0ed
See the change that added ALLOW_NINJA_ENV for more information.
Test: m nothing; check out/soong.log for small list
Test: ALLOW_NINJA_ENV=true m nothing; check out/soong.log
Change-Id: I7761c6a07a7f8b0acee107e9c27c7739dd4b63ab
Ninja does not track changes in environment variables, so we get
potentially incorrect builds when environment variables change during
incremental builds if some action was using one of them.
Add a variable to limit exposure of these variables to ninja, and thus,
all actions run by ninja. Kati and Soong can still read environment
variables, they explicitly track which ones they read so that we can
re-run them appropriately.
This list is just the beginning, there's no good way to detect which
environment variables are currently being used and to pass them through.
So this initial change won't have a behavioral change, and we'll flip
the switch and see what fails or who complains, flipping it off and on
and adding to the list until we can make this always happen.
Also adds a board-specific `BUILD_BROKEN_NINJA_USES_ENV_VARS := ...`
list so that we can temporarily allow board-specific variables until
they're fixed.
Test: check out/soong.log
Test: ALLOW_NINJA_ENV=false m nothing; check out/soong.log
Test: set BUILD_BROKEN_NINJA_USES_ENV_VARS := OLDPWD
ALLOW_NINJA_ENV=false m nothing; check out/soong.log
Change-Id: I08e4834ce12100a577ef7d6a9a21b9e9d345cb93
Colin originally wrote this for the highmem pool:
https://android-review.googlesource.com/c/platform/build/soong/+/1168271
But while that's a larger change, this is useful to just be in the logs,
and for use in multiproduct_kati to better limit the number of
concurrent jobs.
Test: check soong.log on linux and mac
Change-Id: I0518d303a220d775f8d78dba9f660b2954e68e3e
(Although I was the one who added the gzcat reference, I don't believe
we actually use it. And neither my Debian host nor toybox has gzcat --
as opposed to zcat -- anyway. So just quietly drop that.)
Test: treehugger
Change-Id: I95d94dab1daf72faedfa170770e8ebfaa37b70c6
When LOCAL_COPY_HEADER lines are removed, we just removed the copy rule,
we didn't actually remove the old header, so C/C++ files could still
include the old headers during incremental builds.
While we do consider LOCAL_COPY_HEADERS as near-obsolet, and it'll
disappear as we move everything over to Soong, this could produce some
unfortunate incremental build results while people are attempting to
remove them.
Fix this by ensuring that only the files currently listed in
LOCAL_COPY_HEADERS exist in TARGET_OUT_HEADERS after we run Kati.
Test: Remove a LOCAL_COPY_HEADERS entry; see the header removed
Change-Id: I817305703a6996d50490d552623d7df019b608c9
Delay writing the BUILD_DATETIME_FILE until after the out directory
has been created.
Test: cuj_tests
Change-Id: Ice6f34d003f93c26b5d2d0b64f92b11efe16c2d4
This runs an arbitrary shell script for all non-interactive bash sessions, and we've got lots of those.
It was causing problems with Fedora 30, which triggers the `manpath` executable, which isn't available in our $PATH:
https://groups.google.com/d/msg/android-building/WqEyZQuxYtY/zDXl9irOCAAJ
Change-Id: If2f63bbbcc3accb8d239f93fa06823d301d69488
Pass --default_pool=local_pool to kati when RBE or goma is enabled
to put most rules into the local_pool. Specific rules that support
RBE or goma will set .KATI_NINJA_POOL := none to remove themselves
from the local_pool. Passing --default_pool will also disable the
hack in kati that sets the pool based on the presence of the string
"/gomacc" in the command line.
Fixes: 143938974
Test: inspect pools in build-${TARGET-PRODUCT}.ninja for m USE_RBE=true
Test: inspect pools in build-${TARGET-PRODUCT}.ninja for m USE_GOMA=true
Change-Id: I839b2488383fcd63fffd613e25b0b9abcb72b567
Previously, this setting environment variable to false reverted to an
OpenJDK 9 toolchain. This change removes that option.
Test: m core-all-system-modules
Test: zcat out/verbose.log.gz | grep 'prebuilts/jdk/jdk[0-9]*'
Test: EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN=false m core-all-system-modules (fails)
Bug: 131683177
Change-Id: I0d679648e1236925ce3ef7e1652379127e846b00
Prior to this change, the default was to use and OpenJDK 9 toolchain,
with the EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN=true environment
variable available to opt into OpenJDK 11. After this change, the
default is to use OpenJDK 11, with
EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN=false available to opt out.
This change affects:
- The version of javac used.
- The version of javadoc used.
- The version of other tools used during the build process (e.g. jar,
jmod, and jlink).
- For Java bytecode executed on the host, the version of the java
executable used, and so the versions of the JVM and the OpenJDK
libraries in the system module (or bootclasspath).
This change does not affect:
- The Java language level, i.e. the version of Java expected in
source code (the -source option to javac) or the version of Java
bytecode produced (the -target option to javac).
- Anything to do with code execution on the device.
Bug: 131683177
Test: make java droid docs cts tests
Test: zcat out/verbose.log.gz | grep 'prebuilts/jdk/jdk[0-9]*'
Test: make RunBluetoothRoboTests RunCarSettingsLibRoboTests
Test: cts-tradefed help
Test: atest CtsHostTzDataTests
Test: atest CtsLibcoreTestCases
Change-Id: I09dc22f1af4d1f2d7d3b85c08cb0ed9a1105aaca
Merged-In: Iaae89ef25c92ec099575c026fc50e41da4e143e8
A new sysprop neverallow rules are mandatory only for devices launching
with R or later. For devices already launched, neverallow rules can be
relaxed with adding following line to BoardConfig.mk:
BUILD_BROKEN_TREBLE_SYSPROP_NEVERALLOW := true
Bug: 131162102
Test: Set PRODUCT_SHIPPING_API_LEVEL := 30 and try building with
changing some system_public_prop to system_internal_prop
Test: m cts sepolicy_tests
Change-Id: I5e1640f7b43fd47863bc7bd163c2f43b01fa3326
Merged-In: I5e1640f7b43fd47863bc7bd163c2f43b01fa3326
(cherry picked from commit e36f52754f)
Should unbreak the mac build while I work on toybox xargs some more.
Test: suck it and see (since there's no mac presubmit)
Change-Id: I48954f801fc980a1174f93e474d73a24ad12ce66
With this change, setting the environment variable
EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN=true switches from the OpenJDK 9
toolchain to the OpenJDK 11 one (prebuilts/jdk/jdk11).
Since the version of the java.base module has to match the version of
the jlink tool which consumes it, the --module-version argument to the
jmod create call is also switched.
Test: make
Test: EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN make
Bug: 131683177
Change-Id: I606d22538165f309fe5537bd67a26390b1dd5771
My previous change makes TMPDIR a temporary directory that's removed
before we exit. So instead of starting ckati to get this value to a
directory that no longer exists, return the semi-stable value that we
use during a normal build.
Fixes: 142277335
Test: get_build_var TMPDIR
Change-Id: I37d1219b1fda09f131dee17ef7b91e475d3e39ad
Instead of just using the host $PATH during dumpvars, use our path
configuration from the regular build. But instead of creating a ton of
symlinks to the interposer, just use a small directory of symlinks. This
only takes ~3ms (vs ~300ms), at the expense of error logging. Since we
do just about the same product configuration at the start of the build,
we can just rely on the logging there.
This fixes warnings printed by the Mac build, since we using the host
`date` instead of the toybox version:
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
TMPDIR was being shared with the main build, but we're allowed to run in
parallel. If a build is started while we're running, our TMPDIR may be
cleared. Instead, create our own temporary directory, which doesn't need
to be a stable value (as we never do caching here).
Fixes: 141893400
Test: (on mac) source build/envsetup.sh; lunch
Change-Id: I0ff536e71dc5649cae26daa0087eb80861704021
log.Fatalln calls os.Exit, which skips deferred functions and
leaves the console with the cursor disabled. Use ctx.Fatalln
instead, which uses a panic to exit and calls the deferred
function to restore the cursor.
Test: DIST_DIR=/tmp/\test\ foo m dist
Change-Id: Ie92297075e37c171d5ba48848a0ddb19652b051c
This commit adds BUILD_BROKEN_PREBUILT_ELF_FILES to
`ui/build/dumpvars.go` so that its value can be printed to
`out/soong.log`.
Bug: 141176116
Test: lunch aosp_crosshatch-userdebug && make && \
grep BUILD_BROKEN_PREBUILT_ELF_FILES out/soong.log
Change-Id: I99600a5b540be2bb93647ee8175b9f9afd1a6eb3
Before this change, host APEX (`.zipapex`) packages would still be
there after `m installclean`.
Test: Run `m installclean` and check that host APEX packages are removed.
Change-Id: I446286ce700f12666be5aa90fbe3bbc648abd280
When removing dependencies from APEXes, the lib files were still there
even after installclean.
Bug: 139016109
Test: run installclean, see apex directories go away :D
Change-Id: I4dad8e3aa0042cdd5bbb76bd6fdd01b025b98495
Otherwise tools built with ASAN can't symbolize their dumps, as
llvm-symbolizer is not allowed to be used from $PATH.
Fixes: 139825736
Test: cherry-pick https://android-review.googlesource.com/c/platform/system/tools/aidl/+/1106532
mmma ASAN_OPTIONS="" SANITIZE_HOST=address system/tools/aidl
Change-Id: Ib1eeec15ce6694d7bde54c7201b3280e38df3a83
The code still needs to exist, but for buildspec.mk files, not the
PRODUCT-* arguments.
Test: none
Change-Id: I60c6044ab2d1627af7e6e2e8831b77d9db12aa55
This is done so that prototype tooling can write logs inside the DIST_DIR.
Test: I wrote out all the environment variables being passed to
goma_ctl.py and the output is https://paste.googleplex.com/6663469954760704. It contains both DIST_DIR and other variables.
Change-Id: Ia5c70e58aa1b87db9777c59cf3e5df1c9e20f08c
Test: Built aosp_arm-user with and without USE_RBE. USE_RBE uses
a proxy script in place of rewrapper.
Change-Id: I5bf008a940513872d70b5b215bd6209f759826ae
If OUT_DIR has been specified through the command line arguments,
it is sanitized first and NewConfig sets the OUT_DIR in the build
environment. When calling OutDir from Config instance, the out
was being sanitized again which is unnecessary.
Bug: b/118730755
Test: Ran unit test cases, "lunch" followed by "m" command.
Change-Id: Ieb4ffa1026c802c76c7369e8cdb3923a4c70f4b3
Previously, any of the build commands (mma, mmma, etc...) would traverse
up the directory path until an Android build file has been found. That
directory path would eventually become the MODULES-IN-<dir name> build
target. The directory where the m* command was executed may contain a
build file in a sub directory. For that case, the build target is the
MODULES-IN-<dir name> where dir is the directory of the executed m*
command since the target has been constructed by the core makefile.
Bug: b/118730755
Test: Wrote and ran unit test cases. Ran "lunch", "m". Went to a
directory in AOSP that had build files in sub directories and
executed mma. Went to a directory that had only build files in
parent directories and executed mma.
Change-Id: I9458cbc4bf0fc25bfb0848e96e92d261442d5cb0
For mmma and mmm, the findBuildFile function in config.go is invoked every time
for specified directories and directories with targets. For directories with
targets, an Android build file must exist in the directory where mmma and mmm
was invoked. There is no need to invoke findBuildFile function as a
simple check of the build file exists in the specified directory.
This is also refactoring the code for b/118730755
Bug: b/118730755
Test: Executed unit test cases through Intellij and executed mmma
command: "mmma external/protobuf:aprotoc external/bzip2".
Change-Id: I5428e3a3c36f77ff740617564b7853705521f29f
Allow running Soong in a headless delve debugger by passing
SOONG_DELVE=<listen addr> in the environment.
Bug: 80165685
Test: SOONG_DELVE=:1234 m nothing
Change-Id: Icfc893c8a8354a9bbc99112d9c83259cb41906d1
When running ninja stream stdout to the status writer. This improves
the ninja -d explain behavior, and will also allow Soong to be put
into the ninja console pool letting it print timely output to the
console.
Bug: 80165685
Test: NINJA_ARGS="-d explain" m
Change-Id: I40f03fc01d837ad91d400eae378007a7ce807705
The build directory passed in to soong_ui can contain a
symbolic link. soong_ui was not evaluating the build
directory to retrieve the true path, hence failing to
execute the internal soong.
Fixes: b/135995632
Test: Unit test case and executed the scenario in the bug
description.
Change-Id: I5779c6aa3f3183810437dbe2b2d4e40acbafb205
There was one case that did not work:
1) Create dir in root source tree
2) Run "m libc"
The build would fail as it was using the mma logic. A separate
one named BUILD_MODULES was added to allow building specific
modules in any directories using the "m" command.
Bug: b/130049705
Test: unit test cases, ran "m libc" in a tmp directory inside
of the source tree.
Change-Id: I8d23e685a673a311001ee8edd89bd73b662392dd
Everything should be using the prebuilt now.
Bug: 117561006
Test: treehugger
Test: check bass logs for m4 uses
Change-Id: Ic69d715d72d64bb0082e2f2c247113155d8b3de0
The metrics.proto needs to be imported to Google3 in order to unmarshal
the build metrics data from bigstore. Cleaned up the enum names to use
the Camel naming convention and renamed the build_metrics to
soong_metrics to be more specific to soong and to allow quering
soong_metrics files from bigstore.
Bug: b/135280521
Test: lunch andf m -j. Checked if soong_metrics was generated correctly
by using printproto command.
Change-Id: I998c8d05db592e94a653d6ca32250b80df3c9b21
Add the following build actions {BUILD_MODULES_IN_A_DIRECTORY,
BUILD_MODULES_IN_DIRECTORIES} in soong_ui config so the bash code version of
build commands (m, mm, mma, mmm, mmma) in build/make/envsetup.sh can be deprecated.
This is to allow up to date bug fixes on the build commands.
Bug: b/130049705
Test: Unit test cases
Change-Id: I772db1d4e9c1da5273374d1994eb5e8f17cd52f2
terminal.Writer is now just a wrapper around stdio.Stdout() without
any useful functionality. Replace it with stdio.Stdout() as an
io.Writer.
Test: ui/terminal/status_test.go
Change-Id: I5bc5476afdca950b505642f0135a3af9d37fbe24
All users have been removed.
Test: treehugger
Change-Id: I35e55f3ce79c8e68585be120b5dec810563c7a83
Merged-In: I35e55f3ce79c8e68585be120b5dec810563c7a83
- Add SOONG_LOCK_TIMEOUT environmental variable for users to set a
duration (e.g. 1h), which defaults to the current 10 seconds.
- Instead of logging once per second waiting to acquire the lock,
only log once before the first wait
Fixes: 131854539
Test: make; open another tab and make
Change-Id: I0a02a1b071964f74473a5f55034f916526e1b309
Allow install-clean as an alias for installclean and data-clean as
an alias for dataclean.
Test: m install-clean
Change-Id: I9c97d60572a524ad68caecd26d52bfb987468075
This reverts commit e114b3f0a9.
Build broke as follows:
out/soong/host/linux-x86/bin/sbox --sandbox-path out/soong/.temp --output-root out/soong/.intermediates/system/update_engine/ue_unittest_disk_imgs/gen -c 'tar -jxf system/update_engine/sample_images/sample_images.tar.bz2 -C __SBOX_OUT_DIR__/gen disk_ext2_1k.img disk_ext2_4k.img disk_ext2_4k_empty.img disk_ext2_unittest.img' __SBOX_OUT_DIR__/gen/disk_ext2_1k.img __SBOX_OUT_DIR__/gen/disk_ext2_4k.img __SBOX_OUT_DIR__/gen/disk_ext2_4k_empty.img __SBOX_OUT_DIR__/gen/disk_ext2_unittest.img
"bzip2" is not allowed to be used. See https://android.googlesource.com/platform/build/+/master/Changes.md#PATH_Tools for more information.
tar: Child returned status 1
tar: Error is not recoverable: exiting now
sbox command (tar -jxf system/update_engine/sample_images/sample_images.tar.bz2 -C out/soong/.temp/sbox849645882/gen disk_ext2_1k.img disk_ext2_4k.img disk_ext2_4k_empty.img disk_ext2_unittest.img) failed with err "exit status 2"
Change-Id: I3fe7100886ea3b748f5a6b5fc6d2811391bb72a8
There are no remaining users.
Also mark BUILD_BROKEN_ANDROIDMK_EXPORTS and BUILD_BROKEN_ENG_DEBUG_TAGS
as deprecated in scripts/build_broken_logs.go
Test: treehugger
Change-Id: If7892bef1b9001f12a99565f886b395cf1985e70
So that the output of build_test can be parsed to understand which
devices need changes.
Test: read out/soong.log
Change-Id: I40b25527ef72e3f646c0cfe62114b452fa3f27f6
Some people apparently still talk to the network during their build.
Allow this temporarily with a BUILD_BROKEN_USES_NETWORK check.
Bug: 129992021
Test: attempt to talk to the network during the build with and without
this flag.
Change-Id: Ifb967c656aa24c4599e7232d0f1b5a303b5bac52
Wait for the ninja proto processing goroutine to notice the fifo
has closed and exit before continuing.
Test: m nothing
Change-Id: I8cf5f3b8bf6a91496c6d2bbbd3e811eb7f0c9d21
Dump variable in log to track remaining targets which don't set it.
Bug: 122954981
Test: $ grep VNDK out/soong.log
2019/03/07 09:22:04.235604 build/soong/ui/build/dumpvars.go:109: BOARD_VNDK_VERSION current
Change-Id: I34146c752a4bf0d799374bb93cc0d75b0f1b048c
There's more than just toybox in this group now, so let's rename our
variables to something closer to the desired behavior, rather than the
first user.
Test: treehugger
Change-Id: I76d4407792061c8110b194cfe73f1ddc84dbc22f
For ease of investigating the issue with error reports, let me make
the error message more verbose.
Test: make goma_ctl.py raise Exception and run.
Test: I confirmed verbose error message shown.
Change-Id: I04f9a5ca4808746994c41ddce8a5b22a1975fcc2
Signed-off-by: Yoshisato Yanagisawa <yyanagisawa@google.com>
Bug: 124218940
If ANDROID_QUIET_BUILD environment variable is set to '1', do not show 20
lines of the environment variables. In addition, for the failing step show
only the step name and output, omitting the command proper (the verbose log
as well as error.log still contains it).
If build succeeds, the output of the build itself consists of a single
line:
```
> m androidmk
[100% NN/NN] <last command>
```
When it fails, the output does not contain sometimes very long command
line:
```
> m androidmk
[ 97% NN/MM] test androidmk
FAILED: <step>
--- FAIL: TestEndToEnd (0.01s)
androidmk_test.go:1025: failed testcase 'prebuilt_etc_TARGET_OUT_ETC'
input:
include $(CLEAR_VARS)
LOCAL_MODULE := etc.test1
LOCAL_MODULE_CLASS := ETC
LOCAL_MODULE_PATH := $(TARGET_OUT_ETC)/foo/bar
include $(BUILD_PREBUILT)
expected:
prebuilt_etc {
name: "etc.test1",
filename: "foo/bar",
}
got:
prebuilt_etc {
name: "etc.test1",
filename: "foo/bar",
}
FAIL
17:50:53 ninja failed with: exit status 1
```
[The related change in build/make/envsetup.sh suppresses timing display
when the same variable is set.]
Change-Id: I4d3c72457de031ff58a324c2fe98f4c1d10f8239
Test: treehugger
And add a helper script that can help parse the output from
build_test.bash
Test: check for BUILD_BROKEN_ENG_DEBUG_TAGS in soong.log
Test: go run ../build/soong/scripts/build_broken_logs.go *
Change-Id: Idd0fc8b59770dcdbe44eeba262558708a9497f96
We don't want these to be warnings, may as well be errors instead of
turning off the warnings.
Bug: 123583617
Test: treehugger
Change-Id: I6ca518c9647e712426952cb88bdb044d933b23f2
As more phony rules move either to the packaging kati run, or soong, the
existing real-to-phony check failed to notice that they were phony
rules.
So I added --top_level_phony to Make that assumes that all leaf nodes
(source files, or nodes generated by another ninja file generator) that
do not have a '/' in them are phony targets.
The existing phony-looks-real check in Kati should prevent these from
being real generated files, and our source tree shouldn't have any
source files used at the root level.
Test: build_test on downstream branches
Change-Id: I77cff84b536c6cd6402307a9aa5a9c273e72c71f
We've got an internal testcase that uses minijail0 inside an Android.mk
rule. That was failing since we turned on the linux sandbox, as /proc
was mounted read-only, which prevented setting up the uid/gid mappings
for a child namespace.
Fixes: 122985455
Test: treehugger & forrest of breaking build
Change-Id: Ia77a91a7f4eeeb8a24e84075d8272287f5087587
This starts a new network namespace without any connections to the
outside.
Bug: 122270019
Test: USE_GOMA=true m libc
Test: treehugger
Test: add rule to use /usr/bin/wget, fails after this change
Change-Id: Iba262025ce0e4e3bef5c34c817cc678d6c61403b
This really only initializes the sandbox, it does not attempt to change
the view of the filesystem, nor does it turn off networking.
Bug: 122270019
Test: m
Test: trigger nsjail check failure; lunch; m; cat out/soong.log
Test: USE_GOMA=true m libc
Change-Id: Ib291072dcee8247c7a15f5b6831295ead6e4fc22
As suggested in
https://android-review.googlesource.com/c/platform/build/soong/+/839293
I am moving some features in goma.mk to goma.go in Soong UI.
With this CL, let me implement ulimit check to Soong UI.
Test: export USE_GOMA=true
Test: launch aosp_arm-eng
Test: make
Test: the command succeeds if "ulimit -n" and "ulimit -u" are large enough.
Test: Otherwise, it shows error. I confirmed both cases.
Change-Id: I5d7d5ed71f620302a0d635770d1a51a2baab51fd
Signed-off-by: Yoshisato Yanagisawa <yyanagisawa@google.com>
As suggested in b/118390303#comment18, let me run
"goma_ctl.py ensure_start" in soong UI.
Bug: 118390303
Test: stop compiler_proxy, and execute following command:
Test: USE_GOMA=true lunch aosp_arm-eng
Test: and confirms compiler_proxy is running.
Change-Id: I859daae6ae2399c5b6bce2fafd874dfdb7c6aae0
Signed-off-by: Yoshisato Yanagisawa <yyanagisawa@google.com>
As part of a future change to sandbox the build on Linux, the real
username will be switching to "nobody", and the hostname will be
switching to "android-build".
The USER environment variable will reflect the sandboxed value, so for
the build properties that want the external USER, they'll need to use
BUILD_USERNAME.
Similarly, BUILD_HOSTNAME will reflect the real value, while the
`hostname` tool will return "android-build"
Bug: 122270019
Test: check build.prop
Change-Id: I99604b9488732a63690b256dc4dd7894d369a32c
So that ninja produces an error instead of just a warning when a dep
file is not produced.
Bug: 121058584
Test: check build logs for "depfile is missing" warning
Test: treehugger
Change-Id: I1cbaba866eaf293495c3c0b2b174190bcb2b0f9a
Test: Dumped the text formated based metrics file to out dir,
and checked the file.
Bug: b/63815990
Change-Id: Iff476f72a0be74eb53b6b26ef468d11c0f24a404
This way we don't appear hung at:
No need to regenerate ninja file
Change-Id: I8dbdaa2c1b1c5a6a73187d0e6061f363b62e10c9
Fixes: 122251150
Test: m nothing
