The dist dir is created late in the build process. If a
soong sandbox attempts to mount it before it's created
the sandbox will create an empty file in its place.
Test: lunch aosp_cf_x86_phone-userdebug && make -j dist
Change-Id: Ie6513bf702de6e6322c78753d406d70ea3ccc04d
It's safer to disallow writing to the root filesystem of the sandbox.
Still allow write access to the source, tmp, out and dist directories.
Test: build aosp_cf_x86_64_phone-userdebug
Change-Id: Idf0d3a420669fbf6c3aba24e058f09a314ec6d0e
buld_error protobuf file is created the moment a build action failed.
This is for external system to detect immediately what caused the
build failure while monitoring the build_progress.pb file.
Bug: b/150401146
Test: * m nothing
* Built successfully aosp_arm-eng and checked build_error file
was not created.
* Modified a cpp file to cause a build failed. Ran m and build
failed. Checked immediately during the build that the
build_error file was created.
* m clean; m and during build, checked if build_error was
removed.
Change-Id: I778616ae8cf242b49ad263b79c93321959112caa
For successful builds, a blank build_error proto file is created.
This slows down the data processing part where it reads the build_error
file and there are no errors to be processed.
Bug: 142277430
Test: Did a successful build and checked if the build_error file
was not available. Changed a source file to make the build
failed and checked if the build_error file was available.
Change-Id: I300dffe32fb7f8bf984fa20ae368bdd02bf12992
Soong UI generates a proto file named build_progress.pb in $(OUT_DIR)
output directory that contains build action numbers (how many are executing,
finished and total) during the course of a build. This is for external
systems that invokes the Platform Build Systems and would like to know
the completion status.
Bug: b/150401146
Test: Wrote a bash script that continuously read the
build_progress.pb file and computed the build completed percentage
while building the aosp_arm-eng target. Compared the percentage between
the Soong output console and the one reported by the bash script.
Change-Id: I7c7347bc8e41958093892d8e2731c4f4169937dd
This makes sure that we don't keep old vendor_boot artifacts around
after a clean.
Signed-off-by: Will McVicker <willmcvicker@google.com>
Fixes: 151957122
Bug: 151949371
Test: touch $OUT/vendor-ramdisk/test-file && make installclean
Change-Id: I357929583f234212a998dceaa31506fce42e0255
Soong UI will generate a new proto file in the out directory for
other systems to read the build completion status during a course
of a build. The proto file contains build action counters where
total_actions: represents the total build actions of a build (will
increase and may decrease during a course of the build)
finished_actions: represents the number of executed build actions.
finished_actions never decreases and finished_actions <= total_actions.
current_actions: represents the number of build actions being executed
and current_actions + finished_actions <= total_actions.
Bug: 150401146
Test: executed regen.sh script.
Change-Id: I04f4c07855723d0684b141d88cb3529ab5d9fccd
If this couldn't read the current user, it would print a generic
message, then proceed to dereference the invalid user struct :(
Provide somewhat reasonable defaults, and better error messages instead.
Test: m nothing
Test: run docker with misconfigured user
Change-Id: I21af77c7d8d1d2d27cb04546667eb1094c62a7a1
We have known problems (docker; distros w/o user namespaces) which mean that we won't be turning this into a fatal error anytime soon. Also remove the bug report link, it's not useful to continue getting the same reports.
Change-Id: I271871d68150417ac938f074d3730cad4518e327
Test: treehugger
Collect the number of modules and variants and some basic statistics
on memory usage inside soong_build by writing out a proto that is
read back in by soong_ui.
Test: examine soong.log
Change-Id: I6926876377a4f6229cf41fdbf166ae03c885ea55
Trying to override the default directory for ccache by using the
CCACHE_DIR environment variable fails unless it is in the allowed
list.
Bug: 149670916
Test: manual
Change-Id: I8e7eea7a5c25d7ea5f0956fafc70d62522f3c4fc
Apparently PackageManager gets confused when the apk disappears, but the
directory still exists.
Test: lunch aosp_arm-eng; m BasicDreams
<remove BasicDreams from PRODUCT_PACKAGES>
m nothing
ls out/target/product/generic/system/app
Change-Id: I486c0ddadde55fad226049d3fb0f09af88117da0
These variables control whether we accept cached-results / not and
whether we should update the cache with locally executed results or not.
I need accpet-cached whitelisted to purge the the RBE cache for the
invalid cache entry we have set for the failure we say yesterday.
Bug: b/148387048
Change-Id: I7344fc083f82e0b7bc11084376a267d19cf30bb8
This CL adds RBE support to javac, r8, and d8 rules which is only
enabled if respective environment variables are set.
Test: an aosp_crosshatch build with and without the new variables.
Change-Id: Ic82f3627944f6a5ee7b9f3228170c2709b1bfcb8
Create a highmem pool based on the total RAM and the number of CPUs,
with an override via the NINJA_HIGHMEM_NUM_JOBS variable. Put
metalava into the highmem pool.
Ninja does not support nested pools, and when goma or RBE is enabled
the maximum ninja parallelism is set very high with local jobs in a
local pool. When both the local pool and highmem pool are enabled,
the total number of local jobs will be as high as the sum of the sizes
of the two pools. Keep the highmem pool limited to 1/16th of the
local pool when remote builds are enabled to try to minimize the
effect while still limiting highmem jobs.
Fixes: 142644983
Test: m nothing, examine pools
Test: m USE_GOMA=true nothing, examine pools
Change-Id: Id79f11f44948992960ac34ecf831dacbe21bd332
Determining the total build time from the existing metrics is
tricky because some of the metrics are overlapping. For example,
the "soong" metric includes "blueprint bootstrap",
"environment check", "minibp", "bpglob", "minibootstrap", and
"bootstrap". Create a new singleton "total" metric that
covers the interesting parts of the build.
Test: m nothing
Change-Id: Iedcf5c9cf0d27252b694d35e6ff66ca1fcf734ac
This relands I12a0f907753fefd1997ab8b4ea2ac331234093cf along with
a fix to blueprint for absolute paths.
Store the current working directory and then change to the root
directory so that all file accesses must go through helpers in
the android package that properly track dependencies.
Change-Id: I24ac485677aa102eec1a2521d16820da6ee1ae77
Fixes: 146437378
Test: m checkbuild
Test: m OUT_DIR=/tmp/out nothing
Store the current working directory and then change to the root
directory so that all file accesses must go through helpers in
the android package that properly track dependencies.
Fixes: 146437378
Test: m checkbuild
Change-Id: I12a0f907753fefd1997ab8b4ea2ac331234093cf
This is used by the art apex test.
Bug: 147197813
Test: m EMMA_INSTRUMENT=true EMMA_INSTRUMENT_FRAMEWORK=true art-check-debug-apex-gen
Change-Id: Id185cd35f16131f2c9a8afeba2e5b87834b0e620
We're passing in ASAN_OPTIONS=detect_leaks=0 on the command line, and
the build started failing after we limited the environment.
Test: forrest of aosp-master/aosp_x86_64-eng
Change-Id: I42c91897c7677e1a249412e5a8bc7bb1edb2f881
Make now exports (partial) lists of what should be installed. Those
lists are incomplete, but if something is removed from the list, we can
remove it from the filesystem so that incremental builds work better.
Test: built-in tests (m blueprint_tools)
Test: remove a module from PRODUCT_PACKAGES, see the print and file removed
Test: change the name of a cts test, see the old one removed from cts
Change-Id: I6bc14c6a5af63383ea265e97ed8b875fa51af0ed
See the change that added ALLOW_NINJA_ENV for more information.
Test: m nothing; check out/soong.log for small list
Test: ALLOW_NINJA_ENV=true m nothing; check out/soong.log
Change-Id: I7761c6a07a7f8b0acee107e9c27c7739dd4b63ab
Ninja does not track changes in environment variables, so we get
potentially incorrect builds when environment variables change during
incremental builds if some action was using one of them.
Add a variable to limit exposure of these variables to ninja, and thus,
all actions run by ninja. Kati and Soong can still read environment
variables, they explicitly track which ones they read so that we can
re-run them appropriately.
This list is just the beginning, there's no good way to detect which
environment variables are currently being used and to pass them through.
So this initial change won't have a behavioral change, and we'll flip
the switch and see what fails or who complains, flipping it off and on
and adding to the list until we can make this always happen.
Also adds a board-specific `BUILD_BROKEN_NINJA_USES_ENV_VARS := ...`
list so that we can temporarily allow board-specific variables until
they're fixed.
Test: check out/soong.log
Test: ALLOW_NINJA_ENV=false m nothing; check out/soong.log
Test: set BUILD_BROKEN_NINJA_USES_ENV_VARS := OLDPWD
ALLOW_NINJA_ENV=false m nothing; check out/soong.log
Change-Id: I08e4834ce12100a577ef7d6a9a21b9e9d345cb93
Colin originally wrote this for the highmem pool:
https://android-review.googlesource.com/c/platform/build/soong/+/1168271
But while that's a larger change, this is useful to just be in the logs,
and for use in multiproduct_kati to better limit the number of
concurrent jobs.
Test: check soong.log on linux and mac
Change-Id: I0518d303a220d775f8d78dba9f660b2954e68e3e
(Although I was the one who added the gzcat reference, I don't believe
we actually use it. And neither my Debian host nor toybox has gzcat --
as opposed to zcat -- anyway. So just quietly drop that.)
Test: treehugger
Change-Id: I95d94dab1daf72faedfa170770e8ebfaa37b70c6
When LOCAL_COPY_HEADER lines are removed, we just removed the copy rule,
we didn't actually remove the old header, so C/C++ files could still
include the old headers during incremental builds.
While we do consider LOCAL_COPY_HEADERS as near-obsolet, and it'll
disappear as we move everything over to Soong, this could produce some
unfortunate incremental build results while people are attempting to
remove them.
Fix this by ensuring that only the files currently listed in
LOCAL_COPY_HEADERS exist in TARGET_OUT_HEADERS after we run Kati.
Test: Remove a LOCAL_COPY_HEADERS entry; see the header removed
Change-Id: I817305703a6996d50490d552623d7df019b608c9
Add a cuj_tests binary that runs through a sequence of critical
user journey builds and collects metrics.
Bug: 145688720
Test: cuj_tests
Change-Id: I6bb1a3fc4a458fd8cea7b7c924f565d0cf3e6df3
Use Printf instead of Println for formatted strings, and pass
the filename instead of the proto.
Test: cuj_tests
Change-Id: Id261b5a34304a0caa61faa1f3bbc388aacdd25a6
Delay writing the BUILD_DATETIME_FILE until after the out directory
has been created.
Test: cuj_tests
Change-Id: Ice6f34d003f93c26b5d2d0b64f92b11efe16c2d4
This runs an arbitrary shell script for all non-interactive bash sessions, and we've got lots of those.
It was causing problems with Fedora 30, which triggers the `manpath` executable, which isn't available in our $PATH:
https://groups.google.com/d/msg/android-building/WqEyZQuxYtY/zDXl9irOCAAJ
Change-Id: If2f63bbbcc3accb8d239f93fa06823d301d69488
Pass --default_pool=local_pool to kati when RBE or goma is enabled
to put most rules into the local_pool. Specific rules that support
RBE or goma will set .KATI_NINJA_POOL := none to remove themselves
from the local_pool. Passing --default_pool will also disable the
hack in kati that sets the pool based on the presence of the string
"/gomacc" in the command line.
Fixes: 143938974
Test: inspect pools in build-${TARGET-PRODUCT}.ninja for m USE_RBE=true
Test: inspect pools in build-${TARGET-PRODUCT}.ninja for m USE_GOMA=true
Change-Id: I839b2488383fcd63fffd613e25b0b9abcb72b567
Previously, this setting environment variable to false reverted to an
OpenJDK 9 toolchain. This change removes that option.
Test: m core-all-system-modules
Test: zcat out/verbose.log.gz | grep 'prebuilts/jdk/jdk[0-9]*'
Test: EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN=false m core-all-system-modules (fails)
Bug: 131683177
Change-Id: I0d679648e1236925ce3ef7e1652379127e846b00
Prior to this change, the default was to use and OpenJDK 9 toolchain,
with the EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN=true environment
variable available to opt into OpenJDK 11. After this change, the
default is to use OpenJDK 11, with
EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN=false available to opt out.
This change affects:
- The version of javac used.
- The version of javadoc used.
- The version of other tools used during the build process (e.g. jar,
jmod, and jlink).
- For Java bytecode executed on the host, the version of the java
executable used, and so the versions of the JVM and the OpenJDK
libraries in the system module (or bootclasspath).
This change does not affect:
- The Java language level, i.e. the version of Java expected in
source code (the -source option to javac) or the version of Java
bytecode produced (the -target option to javac).
- Anything to do with code execution on the device.
Bug: 131683177
Test: make java droid docs cts tests
Test: zcat out/verbose.log.gz | grep 'prebuilts/jdk/jdk[0-9]*'
Test: make RunBluetoothRoboTests RunCarSettingsLibRoboTests
Test: cts-tradefed help
Test: atest CtsHostTzDataTests
Test: atest CtsLibcoreTestCases
Change-Id: I09dc22f1af4d1f2d7d3b85c08cb0ed9a1105aaca
Merged-In: Iaae89ef25c92ec099575c026fc50e41da4e143e8
The smartStatusOutput.done channel is read from a goroutine that
locks smartStatusOutput.lock. If the goroutine that writes to
smartStatusOutput.done is holding smartStatusOutput.lock it can
lead to an AB-BA deadlock. Call stopActionTableTick from outside
the lock.
Fixes: 143558785
Test: none
Change-Id: I93a10ef9ff16c3953a1c5ccb102b024158358fe4
A new sysprop neverallow rules are mandatory only for devices launching
with R or later. For devices already launched, neverallow rules can be
relaxed with adding following line to BoardConfig.mk:
BUILD_BROKEN_TREBLE_SYSPROP_NEVERALLOW := true
Bug: 131162102
Test: Set PRODUCT_SHIPPING_API_LEVEL := 30 and try building with
changing some system_public_prop to system_internal_prop
Test: m cts sepolicy_tests
Change-Id: I5e1640f7b43fd47863bc7bd163c2f43b01fa3326
Merged-In: I5e1640f7b43fd47863bc7bd163c2f43b01fa3326
(cherry picked from commit e36f52754f)
Should unbreak the mac build while I work on toybox xargs some more.
Test: suck it and see (since there's no mac presubmit)
Change-Id: I48954f801fc980a1174f93e474d73a24ad12ce66
With this change, setting the environment variable
EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN=true switches from the OpenJDK 9
toolchain to the OpenJDK 11 one (prebuilts/jdk/jdk11).
Since the version of the java.base module has to match the version of
the jlink tool which consumes it, the --module-version argument to the
jmod create call is also switched.
Test: make
Test: EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN make
Bug: 131683177
Change-Id: I606d22538165f309fe5537bd67a26390b1dd5771
My previous change makes TMPDIR a temporary directory that's removed
before we exit. So instead of starting ckati to get this value to a
directory that no longer exists, return the semi-stable value that we
use during a normal build.
Fixes: 142277335
Test: get_build_var TMPDIR
Change-Id: I37d1219b1fda09f131dee17ef7b91e475d3e39ad
Instead of just using the host $PATH during dumpvars, use our path
configuration from the regular build. But instead of creating a ton of
symlinks to the interposer, just use a small directory of symlinks. This
only takes ~3ms (vs ~300ms), at the expense of error logging. Since we
do just about the same product configuration at the start of the build,
we can just rely on the logging there.
This fixes warnings printed by the Mac build, since we using the host
`date` instead of the toybox version:
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
TMPDIR was being shared with the main build, but we're allowed to run in
parallel. If a build is started while we're running, our TMPDIR may be
cleared. Instead, create our own temporary directory, which doesn't need
to be a stable value (as we never do caching here).
Fixes: 141893400
Test: (on mac) source build/envsetup.sh; lunch
Change-Id: I0ff536e71dc5649cae26daa0087eb80861704021
log.Fatalln calls os.Exit, which skips deferred functions and
leaves the console with the cursor disabled. Use ctx.Fatalln
instead, which uses a panic to exit and calls the deferred
function to restore the cursor.
Test: DIST_DIR=/tmp/\test\ foo m dist
Change-Id: Ie92297075e37c171d5ba48848a0ddb19652b051c
Force soong_ui in dumpvars mode to use a dumb terminal so that it
doesn't print an empty status table. Also move the logs to files
that are prefixed with "dumpvars-" so that they don't rotate the
real build logs.
Fixes: 139307849
Test: lunch
Change-Id: If92421af07731fedc9ca42670f309e95922a2224
Multiple terminals have had issues with writing newlines into the
non-scrolling region, just set the cursor to the beginning of the
next line instead.
Test: m nothing in JediTerm
Change-Id: I2e434f4cc263ca13b82889a79d6a8bb48d084cb3
Don't pass 0 to setCursor or setScrollingMargins. Passing 0 hasn't
caused any known issues, but 1 is correct.
Test: m nothing
Change-Id: I8735032eb3f1d46df4c41c4ea9ef95b5fd842824
This commit adds BUILD_BROKEN_PREBUILT_ELF_FILES to
`ui/build/dumpvars.go` so that its value can be printed to
`out/soong.log`.
Bug: 141176116
Test: lunch aosp_crosshatch-userdebug && make && \
grep BUILD_BROKEN_PREBUILT_ELF_FILES out/soong.log
Change-Id: I99600a5b540be2bb93647ee8175b9f9afd1a6eb3
Before this change, host APEX (`.zipapex`) packages would still be
there after `m installclean`.
Test: Run `m installclean` and check that host APEX packages are removed.
Change-Id: I446286ce700f12666be5aa90fbe3bbc648abd280
When removing dependencies from APEXes, the lib files were still there
even after installclean.
Bug: 139016109
Test: run installclean, see apex directories go away :D
Change-Id: I4dad8e3aa0042cdd5bbb76bd6fdd01b025b98495
Otherwise tools built with ASAN can't symbolize their dumps, as
llvm-symbolizer is not allowed to be used from $PATH.
Fixes: 139825736
Test: cherry-pick https://android-review.googlesource.com/c/platform/system/tools/aidl/+/1106532
mmma ASAN_OPTIONS="" SANITIZE_HOST=address system/tools/aidl
Change-Id: Ib1eeec15ce6694d7bde54c7201b3280e38df3a83
For smart terminals, default to using the status table, using 1/4 of the
terminal height (with a min of 1 and a max of 10).
This behavior can still be overriden to a specific height with
SOONG_UI_TABLE_HEIGHT, and turned off by setting SOONG_UI_TABLE_HEIGHT
to 0.
Test: m <adjust terminal height, see it shrink when it gets too short>
Test: SOONG_UI_TABLE_HEIGHT=20 m
Test: SOONG_UI_TABLE_HEIGHT=0 m
Change-Id: I224348a29a6e07f168c92cf5514a94bd27e32618
The code still needs to exist, but for buildspec.mk files, not the
PRODUCT-* arguments.
Test: none
Change-Id: I60c6044ab2d1627af7e6e2e8831b77d9db12aa55
This is done so that prototype tooling can write logs inside the DIST_DIR.
Test: I wrote out all the environment variables being passed to
goma_ctl.py and the output is https://paste.googleplex.com/6663469954760704. It contains both DIST_DIR and other variables.
Change-Id: Ia5c70e58aa1b87db9777c59cf3e5df1c9e20f08c
