Commit graph

393 commits

Author SHA1 Message Date
Dan Willemsen
5cacfe1e24 Allow EMMA_INSTRUMENT_FRAMEWORK during ninja
This is used by the art apex test.

Bug: 147197813
Test: m EMMA_INSTRUMENT=true EMMA_INSTRUMENT_FRAMEWORK=true art-check-debug-apex-gen
Change-Id: Id185cd35f16131f2c9a8afeba2e5b87834b0e620
2020-01-06 12:25:40 -08:00
Dan Willemsen
7da042912b Likely fix SANITIZE_HOST=address builds
We're passing in ASAN_OPTIONS=detect_leaks=0 on the command line, and
the build started failing after we limited the environment.

Test: forrest of aosp-master/aosp_x86_64-eng
Change-Id: I42c91897c7677e1a249412e5a8bc7bb1edb2f881
2020-01-04 13:58:54 -08:00
Dan Willemsen
2de4152e61 Merge "Limit environment during ninja" 2020-01-04 17:51:03 +00:00
Colin Cross
68534a29d6 Don't print critical path parallelism ratio if elapsed time is 0.
Test: m nothing, examine out/soong.log
Change-Id: Ib23902c703d306337b2a5150d6d63b1e3d0f95b3
2020-01-04 07:26:28 +00:00
Treehugger Robot
6e3b471d3d Merge "Remove files that should no longer be installed" 2020-01-04 05:16:24 +00:00
Dan Willemsen
1e775d7619 Remove files that should no longer be installed
Make now exports (partial) lists of what should be installed. Those
lists are incomplete, but if something is removed from the list, we can
remove it from the filesystem so that incremental builds work better.

Test: built-in tests (m blueprint_tools)
Test: remove a module from PRODUCT_PACKAGES, see the print and file removed
Test: change the name of a cts test, see the old one removed from cts
Change-Id: I6bc14c6a5af63383ea265e97ed8b875fa51af0ed
2020-01-03 16:46:15 -08:00
Dan Willemsen
260db53da3 Limit environment during ninja
See the change that added ALLOW_NINJA_ENV for more information.

Test: m nothing; check out/soong.log for small list
Test: ALLOW_NINJA_ENV=true m nothing; check out/soong.log
Change-Id: I7761c6a07a7f8b0acee107e9c27c7739dd4b63ab
2020-01-03 15:36:33 -08:00
Dan Willemsen
e333635983 Add option to limit environment variables given to ninja
Ninja does not track changes in environment variables, so we get
potentially incorrect builds when environment variables change during
incremental builds if some action was using one of them.

Add a variable to limit exposure of these variables to ninja, and thus,
all actions run by ninja. Kati and Soong can still read environment
variables, they explicitly track which ones they read so that we can
re-run them appropriately.

This list is just the beginning, there's no good way to detect which
environment variables are currently being used and to pass them through.
So this initial change won't have a behavioral change, and we'll flip
the switch and see what fails or who complains, flipping it off and on
and adding to the list until we can make this always happen.

Also adds a board-specific `BUILD_BROKEN_NINJA_USES_ENV_VARS := ...`
list so that we can temporarily allow board-specific variables until
they're fixed.

Test: check out/soong.log
Test: ALLOW_NINJA_ENV=false m nothing; check out/soong.log
Test: set BUILD_BROKEN_NINJA_USES_ENV_VARS := OLDPWD
      ALLOW_NINJA_ENV=false m nothing; check out/soong.log
Change-Id: I08e4834ce12100a577ef7d6a9a21b9e9d345cb93
2020-01-03 15:36:29 -08:00
Dan Willemsen
2bb82d0011 Add TotalRAM detection
Colin originally wrote this for the highmem pool:

https://android-review.googlesource.com/c/platform/build/soong/+/1168271

But while that's a larger change, this is useful to just be in the logs,
and for use in multiproduct_kati to better limit the number of
concurrent jobs.

Test: check soong.log on linux and mac
Change-Id: I0518d303a220d775f8d78dba9f660b2954e68e3e
2019-12-27 09:43:49 -08:00
Elliott Hughes
936a18d8da Merge "Switch to toybox realpath." 2019-12-20 00:27:10 +00:00
Elliott Hughes
c5aedc4c49 Switch to toybox realpath.
Change-Id: I9f1d3df08d860e5bfa131a531e274c75b72148e7
2019-12-17 22:03:45 -08:00
Treehugger Robot
0f41daf682 Merge "Switch the mac back to toybox xargs." 2019-12-15 16:35:05 +00:00
Dan Willemsen
7842826f49 Remove obsolete copy headers
When LOCAL_COPY_HEADER lines are removed, we just removed the copy rule,
we didn't actually remove the old header, so C/C++ files could still
include the old headers during incremental builds.

While we do consider LOCAL_COPY_HEADERS as near-obsolet, and it'll
disappear as we move everything over to Soong, this could produce some
unfortunate incremental build results while people are attempting to
remove them.

Fix this by ensuring that only the files currently listed in
LOCAL_COPY_HEADERS exist in TARGET_OUT_HEADERS after we run Kati.

Test: Remove a LOCAL_COPY_HEADERS entry; see the header removed
Change-Id: I817305703a6996d50490d552623d7df019b608c9
2019-12-13 19:38:23 -08:00
Elliott Hughes
d329254328 Switch the mac back to toybox xargs.
Test: make sdk
Change-Id: I29a6b0404e248553ecfdfdeb263aa9cd0d2fdb3e
2019-12-12 16:52:06 -08:00
Elliott Hughes
e2d33d52e1 Switch to our hermetic bc.
Test: treehugger
Change-Id: I12142a7f54190edd21e719bbd8f5df831ada6161
2019-12-10 10:12:18 -08:00
Colin Cross
d0be2109cf Add CUJ tests
Add a cuj_tests binary that runs through a sequence of critical
user journey builds and collects metrics.

Bug: 145688720
Test: cuj_tests
Change-Id: I6bb1a3fc4a458fd8cea7b7c924f565d0cf3e6df3
2019-12-05 11:11:37 -08:00
Treehugger Robot
dc0e8fd131 Merge changes Id261b5a3,Ice6f34d0
* changes:
  Fix errorProtoLog error messages
  Write BUILD_DATETIME_FILE after SetupOutDir
2019-12-05 09:10:13 +00:00
Colin Cross
ff27ce4cc0 Fix errorProtoLog error messages
Use Printf instead of Println for formatted strings, and pass
the filename instead of the proto.

Test: cuj_tests
Change-Id: Id261b5a34304a0caa61faa1f3bbc388aacdd25a6
2019-11-26 16:21:58 -08:00
Colin Cross
28f527c3da Write BUILD_DATETIME_FILE after SetupOutDir
Delay writing the BUILD_DATETIME_FILE until after the out directory
has been created.

Test: cuj_tests
Change-Id: Ice6f34d003f93c26b5d2d0b64f92b11efe16c2d4
2019-11-26 16:21:58 -08:00
Dan Shi
53f1a190ff Clean vts-core in host/out to ensure old package to be removed.
Bug: None
Test: m installclean
Change-Id: I5aa451d87dc377e62dcf1b88baa58539fa661feb
2019-11-26 10:02:53 -08:00
Dan Willemsen
1c504d9c7e Remove BASH_ENV from the build environment
This runs an arbitrary shell script for all non-interactive bash sessions, and we've got lots of those.

It was causing problems with Fedora 30, which triggers the `manpath` executable, which isn't available in our $PATH:
https://groups.google.com/d/msg/android-building/WqEyZQuxYtY/zDXl9irOCAAJ

Change-Id: If2f63bbbcc3accb8d239f93fa06823d301d69488
2019-11-18 19:13:56 +00:00
Elliott Hughes
7e4452511c Merge "Switch to our own zipinfo(1)." 2019-11-16 00:31:45 +00:00
Elliott Hughes
3efacee1fd Switch to our own zipinfo(1).
Test: treehugger
Change-Id: If71e87e45a52959be9a316810fe79250771eb9d4
2019-11-14 20:09:24 -08:00
Colin Cross
9016b91290 Set default pool when RBE or goma is enabled
Pass --default_pool=local_pool to kati when RBE or goma is enabled
to put most rules into the local_pool.  Specific rules that support
RBE or goma will set .KATI_NINJA_POOL := none to remove themselves
from the local_pool.  Passing --default_pool will also disable the
hack in kati that sets the pool based on the presence of the string
"/gomacc" in the command line.

Fixes: 143938974
Test: inspect pools in build-${TARGET-PRODUCT}.ninja for m USE_RBE=true
Test: inspect pools in build-${TARGET-PRODUCT}.ninja for m USE_GOMA=true
Change-Id: I839b2488383fcd63fffd613e25b0b9abcb72b567
2019-11-11 16:08:42 -08:00
Pete Gillin
a7a3d64a36 Remove the EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN option.
Previously, this setting environment variable to false reverted to an
OpenJDK 9 toolchain. This change removes that option.

Test: m core-all-system-modules
Test: zcat out/verbose.log.gz | grep 'prebuilts/jdk/jdk[0-9]*'
Test: EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN=false m core-all-system-modules (fails)
Bug: 131683177
Change-Id: I0d679648e1236925ce3ef7e1652379127e846b00
2019-11-11 11:03:58 +00:00
Pete Gillin
abbcddaaf4 Switch to an OpenJDK 11 toolchain by default.
Prior to this change, the default was to use and OpenJDK 9 toolchain,
with the EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN=true environment
variable available to opt into OpenJDK 11. After this change, the
default is to use OpenJDK 11, with
EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN=false available to opt out.

This change affects:
 - The version of javac used.
 - The version of javadoc used.
 - The version of other tools used during the build process (e.g. jar,
   jmod, and jlink).
 - For Java bytecode executed on the host, the version of the java
   executable used, and so the versions of the JVM and the OpenJDK
   libraries in the system module (or bootclasspath).

This change does not affect:
 - The Java language level, i.e. the version of Java expected in
   source code (the -source option to javac) or the version of Java
   bytecode produced (the -target option to javac).
 - Anything to do with code execution on the device.

Bug: 131683177
Test: make java droid docs cts tests
Test: zcat out/verbose.log.gz | grep 'prebuilts/jdk/jdk[0-9]*'
Test: make RunBluetoothRoboTests RunCarSettingsLibRoboTests
Test: cts-tradefed help
Test: atest CtsHostTzDataTests
Test: atest CtsLibcoreTestCases
Change-Id: I09dc22f1af4d1f2d7d3b85c08cb0ed9a1105aaca
Merged-In: Iaae89ef25c92ec099575c026fc50e41da4e143e8
2019-11-05 11:33:58 +00:00
Colin Cross
8cc19911b1 smart_status: move channel write outside lock to avoid deadlock
The smartStatusOutput.done channel is read from a goroutine that
locks smartStatusOutput.lock.  If the goroutine that writes to
smartStatusOutput.done is holding smartStatusOutput.lock it can
lead to an AB-BA deadlock.  Call stopActionTableTick from outside
the lock.

Fixes: 143558785
Test: none
Change-Id: I93a10ef9ff16c3953a1c5ccb102b024158358fe4
2019-10-29 15:27:32 -07:00
Elliott Hughes
e23c63de21 Merge "Switch to toybox find for darwin and linux." 2019-10-23 14:52:28 +00:00
Inseob Kim
822fdcab1f Add BUILD_BROKEN_TREBLE_SYSPROP_NEVERALLOW
A new sysprop neverallow rules are mandatory only for devices launching
with R or later. For devices already launched, neverallow rules can be
relaxed with adding following line to BoardConfig.mk:

BUILD_BROKEN_TREBLE_SYSPROP_NEVERALLOW := true

Bug: 131162102
Test: Set PRODUCT_SHIPPING_API_LEVEL := 30 and try building with
changing some system_public_prop to system_internal_prop
Test: m cts sepolicy_tests

Change-Id: I5e1640f7b43fd47863bc7bd163c2f43b01fa3326
Merged-In: I5e1640f7b43fd47863bc7bd163c2f43b01fa3326
(cherry picked from commit e36f52754f)
2019-10-15 09:22:42 +09:00
Elliott Hughes
e04b73f100 Merge "Go back to the native mac xargs for now." 2019-10-10 21:18:59 +00:00
Elliott Hughes
27cc0a8f32 Go back to the native mac xargs for now.
Should unbreak the mac build while I work on toybox xargs some more.

Test: suck it and see (since there's no mac presubmit)
Change-Id: I48954f801fc980a1174f93e474d73a24ad12ce66
2019-10-09 18:12:33 -07:00
Treehugger Robot
9026090263 Merge "Add $PRODUCT_OUT/test_harness_ramdisk to installclean" 2019-10-09 23:49:46 +00:00
Pete Gillin
1f52e93db2 Add experimental option to use OpenJDK 11 toolchain.
With this change, setting the environment variable
EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN=true switches from the OpenJDK 9
toolchain to the OpenJDK 11 one (prebuilts/jdk/jdk11).

Since the version of the java.base module has to match the version of
the jlink tool which consumes it, the --module-version argument to the
jmod create call is also switched.

Test: make
Test: EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN make
Bug: 131683177
Change-Id: I606d22538165f309fe5537bd67a26390b1dd5771
2019-10-09 18:38:14 +01:00
Bowgo Tsai
5145c2c11f Add $PRODUCT_OUT/test_harness_ramdisk to installclean
Also removes $PRODUCT_OUT/debug_ramdisk.

Bug: 140036184
Test: `make installclean` removes those directories
Change-Id: I442753817bdda4293b98ac1a7fd0173820dc2a2f
2019-10-09 01:59:51 +00:00
Dan Willemsen
b6699a1414 Restore previous behavior of get_build_var TMPDIR
My previous change makes TMPDIR a temporary directory that's removed
before we exit. So instead of starting ckati to get this value to a
directory that no longer exists, return the semi-stable value that we
use during a normal build.

Fixes: 142277335
Test: get_build_var TMPDIR
Change-Id: I37d1219b1fda09f131dee17ef7b91e475d3e39ad
2019-10-07 15:29:23 -07:00
Dan Willemsen
4e2456bc09 Fix dumpvars $PATH / $TMPDIR
Instead of just using the host $PATH during dumpvars, use our path
configuration from the regular build. But instead of creating a ton of
symlinks to the interposer, just use a small directory of symlinks. This
only takes ~3ms (vs ~300ms), at the expense of error logging. Since we
do just about the same product configuration at the start of the build,
we can just rely on the logging there.

This fixes warnings printed by the Mac build, since we using the host
`date` instead of the toybox version:

usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
            [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
            [-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]

TMPDIR was being shared with the main build, but we're allowed to run in
parallel. If a build is started while we're running, our TMPDIR may be
cleared. Instead, create our own temporary directory, which doesn't need
to be a stable value (as we never do caching here).

Fixes: 141893400
Test: (on mac) source build/envsetup.sh; lunch
Change-Id: I0ff536e71dc5649cae26daa0087eb80861704021
2019-10-04 12:53:51 -07:00
Logan Chien
65ab468452 Merge "Add BUILD_BROKEN_PREBUILT_ELF_FILES to dumpvars.go" 2019-10-03 15:51:09 +00:00
Colin Cross
1f6faeb50f Use ctx.Fatalln instead of log.Fatalln for absolute path errors
log.Fatalln calls os.Exit, which skips deferred functions and
leaves the console with the cursor disabled.  Use ctx.Fatalln
instead, which uses a panic to exit and calls the deferred
function to restore the cursor.

Test: DIST_DIR=/tmp/\test\ foo m dist
Change-Id: Ie92297075e37c171d5ba48848a0ddb19652b051c
2019-09-23 15:55:54 -07:00
Colin Cross
c0b9f6b9d2 Force dumpvars into dumb terminal mode and move log files
Force soong_ui in dumpvars mode to use a dumb terminal so that it
doesn't print an empty status table.  Also move the logs to files
that are prefixed with "dumpvars-" so that they don't rotate the
real build logs.

Fixes: 139307849
Test: lunch
Change-Id: If92421af07731fedc9ca42670f309e95922a2224
2019-09-23 14:24:57 -07:00
Colin Cross
f0b987ecba status table: don't write newlines in non-scrolling terminals
Multiple terminals have had issues with writing newlines into the
non-scrolling region, just set the cursor to the beginning of the
next line instead.

Test: m nothing in JediTerm
Change-Id: I2e434f4cc263ca13b82889a79d6a8bb48d084cb3
2019-09-20 15:03:31 -07:00
Colin Cross
bf8f57e2f6 status table: ANSI code rows and columns are 1-based
Don't pass 0 to setCursor or setScrollingMargins.  Passing 0 hasn't
caused any known issues, but 1 is correct.

Test: m nothing
Change-Id: I8735032eb3f1d46df4c41c4ea9ef95b5fd842824
2019-09-20 15:03:11 -07:00
Logan Chien
f9cf9acd7a Add BUILD_BROKEN_PREBUILT_ELF_FILES to dumpvars.go
This commit adds BUILD_BROKEN_PREBUILT_ELF_FILES to
`ui/build/dumpvars.go` so that its value can be printed to
`out/soong.log`.

Bug: 141176116
Test: lunch aosp_crosshatch-userdebug && make && \
      grep BUILD_BROKEN_PREBUILT_ELF_FILES out/soong.log

Change-Id: I99600a5b540be2bb93647ee8175b9f9afd1a6eb3
2019-09-20 11:40:49 -07:00
Roland Levillain
9c27df7d4d Merge "Add $ANDROID_HOST_OUT/apex to the file set removed by target installclean." 2019-09-18 10:47:06 +00:00
Elliott Hughes
26bce340e3 Track switch to toybox date for macOS.
Bug: http://b/139450866
Test: builds
Change-Id: I5ab0d20d761bcd03f3bee18b8343a2f8aa5e2275
2019-09-12 15:05:13 -07:00
Roland Levillain
e5f9ee56b5 Add $ANDROID_HOST_OUT/apex to the file set removed by target installclean.
Before this change, host APEX (`.zipapex`) packages would still be
there after `m installclean`.

Test: Run `m installclean` and check that host APEX packages are removed.
Change-Id: I446286ce700f12666be5aa90fbe3bbc648abd280
2019-09-11 14:50:08 +01:00
Elliott Hughes
34b49d1402 Use toybox md5sum instead of md5 on macOS.
Test: builds locally
Change-Id: I04aa52de96e0525a44331cb0c057a8e18a95891d
2019-09-06 14:42:24 -07:00
Elliott Hughes
ef19e01f96 Switch to toybox stat on macOS too.
Bug: http://b/139450866
Test: builds
Change-Id: I97af607a311aea5f97544bd3ce1d532d999126ff
2019-09-05 10:01:37 -07:00
Elliott Hughes
5cb822b2cf Switch to toybox find for darwin and linux.
Test: treehugger
Change-Id: Id311b403f50a76e75ed92c2a9e7cc7aa025f2448
2019-08-27 21:47:05 -07:00
Elliott Hughes
f1ff226cd4 Switch most of the darwin tools to toybox.
Bug: http://b/139450866
Test: treehugger
Test: "works for me on my machine" (last week)
Change-Id: I5dd00e6fda134b06967651007afb7565ce4f9e24
2019-08-27 15:17:32 -07:00
Treehugger Robot
2569cc5026 Merge "We have md5sum/sha1sum/sha256sum/sha512sum on Darwin now too." 2019-08-27 03:17:07 +00:00