Commit graph

166 commits

Author SHA1 Message Date
Dan Willemsen
184901135c Revert "Revert "Revert "Revert "Add path interposer""""
This reverts commit 09f4540d66.

Fixes the raw call to net.Listen in the tests to go through the listen()
helper and use the long socket path fallbacks.

Removes the use of timeouts from the tests -- the behaviors being tested
did not rely on timeouts, so removing them will reduce the flakiness if
the build is heavily loading the machine at the same time the test is
running.

Also fixes some potential nil pointer dereferences.

Test: OUT_DIR=<really long> m blueprint_tools
Test: `while .../soong-ui-build-paths/test/test; do sleep 0.01; done` with a build running
Change-Id: I16d44be7517bc415f1c808284088f4ba40df3bfa
2018-05-31 14:59:33 -07:00
Dan Willemsen
09f4540d66 Revert "Revert "Revert "Add path interposer"""
This reverts commit c59a92cb1a.

Reason for revert: tests are broken with long OUT_DIRs
They're directly calling net.Listen, and not using the fallback
for long socket names.

Change-Id: Id14cbd499fd9b36c6926b7552d3554340cb0916c
2018-05-25 23:26:00 +00:00
Dan Willemsen
c59a92cb1a Revert "Revert "Add path interposer""
This reverts commit 96c957ae20.

Fixes issues on some machines where the socket in TMPDIR ended up with a
unix domain socket pathname over 107 characters long, which Go will
reject due to underlying limitations in the system calls. If this
happens, we'll fall back to opening the directory, then using
/proc/self/fd/#/<file>, or manually creating a similar symlink in /tmp.

Also fixes some issues on Mac where os.Executable returns the symlink
instead of the underlying file, sending a message over a unix domain
socket will block if the reader isn't reading, and sandboxing was
preventing us from running `ps`.

Test: m blueprint_tools
Test: m blueprint_tools on mac
Change-Id: Ib19ccfe10cb0a79f1476fb1d5cd20ed0495be367
2018-05-25 14:07:00 -07:00
Dan Willemsen
96c957ae20 Revert "Add path interposer"
This reverts commit a14704c12b.

Reason for revert: breaking builds

Change-Id: I920b0ff41823428a0baf59b3fd82cbcc766babca
2018-05-19 00:54:13 +00:00
Dan Willemsen
a14704c12b Add path interposer
This will allow us to track (and eventually limit) the commands that the
build references via $PATH. These are mostly implicit dependencies on
the host system -- for Linux, we assume something similar to Ubuntu
14.04 with a few extra packages, but this will let us better define
that.

This will not catch uses of tools with absolute paths (/bin/bash, etc),
but most uses shouldn't be relying on absolute path names anyways.

Adds ~400ms on the first startup, ~140ms on subsequent runs, and
overhead of a few ms for every forwarded execution.

Test: m
Test: build/soong/build_test.bash
Test: Add `gcc --version`, TEMPORARY_DISABLE_PATH_RESTRICTIONS=true m
Change-Id: Id68cbb1c8ceef65bbbb10751e83722c7662d2351
2018-05-18 13:24:36 -07:00
Jaekyun Seok
f6307ccae1 Delete product in installClean
Bug: 79780604
Test: succeeded building and tested with taimen
Change-Id: I11c4ecb839a28d852e0d5faf762afda33166ad48
2018-05-16 12:25:41 +09:00
Dan Willemsen
fe8b6453a7 Handle readonly directories in tmpdir
Some tests make their temporary directories readonly. If they fail or
crash before cleaning up, they could leave these readonly directories
behind with files in them. os.RemoveAll fails with an error in this
case, and we can't start the build until they're removed.

Test: `m blueprint_tools` to run the new go tests
Change-Id: I761f96579e96167ebfd98c6cca59765bd50536ec
2018-05-14 19:44:54 -07:00
Dan Willemsen
81759847b2 Add --kati_stats to cleanspec processing
So that we can see where the time is going. Also removes the obsolete
ASAN_OPTIONS, which moved to kati, but I forgot to update this instance.

Test: m nothing; check out/soong.log
Change-Id: I0c4066bad20fc2dc22b389f4c973d10dca554ee3
2018-05-02 17:44:36 +00:00
android-build-prod (mdb)
d08726ac76 Merge "Pass TARGET_DEVICE_DIR from dumpvars to later kati runs" 2018-05-02 17:35:34 +00:00
Dan Willemsen
6ab79db2e6 Pass TARGET_DEVICE_DIR from dumpvars to later kati runs
The find commands used to locate the BoardConfig.mk can't be optimized
by Kati, so we're currently spending ~125ms three times during every
build (dumpvars, cleanspec, and the main kati run). Preserve the value
of TARGET_DEVICE_DIR from the dumpvars run so that we only need to run
the find commands once.

Bug: 78020936
Test: out/build-taimen.ninja is identical
Test: out/soong.log shows that we're not running these finds again
Change-Id: Iee56b454c3661de2b58c161169218ecaf2135398
2018-05-02 00:06:28 -07:00
Dan Willemsen
567851c9cc Save list of AndroidProducts.mk files
Bug: 78020936
Test: diff out/.module_paths/AndroidProducts.mk.list with existing find
      results
Change-Id: Ie09cc18b926b21d6d24a4cfc570686aed2893e00
2018-05-01 22:57:53 -07:00
Dan Willemsen
ebfe33ad80 Remove more problematic env vars
I got a report of a user exporting CDPATH in their shell and causing
strange issues in the build. We should never need this value (and it
probably shouldn't be exported from the shell either -- that was their
workaround).

NDK_ROOT causes issues with Android.mk files thinking that they're
building with the ndk (like external/googletest/googletest/Android.mk).

Bug: 78933670
Test: CDPATH=. NDK_ROOT=test m; check out/soong.log
Change-Id: Icce43d7e31ed5e5e1fb7a4e37fd4dfbf421af4b1
2018-05-01 10:07:50 -07:00
Dan Willemsen
3d60b115a0 Allow boards to turn overriding commands into an error
Many board still have this problem, but if we can switch some over,
we'll prevent global problems and have the ability to clean boards up
one-by-one.

Bug: 77611511
Test: lunch aosp_arm-eng; m nothing
Test: lunch aosp_marlin-eng; m nothing
Test: build_test on all downstream branches
Change-Id: I78bee44adde2059d01188658b9050927748d2028
2018-04-05 16:23:10 -07:00
Colin Cross
f46e37f5f7 Fix format string issues
Fix issues caught by go vet.

Test: m checkbuild
Change-Id: Ib8d740457c15432dabe1575a6707726ddaf93084
2018-03-28 15:54:52 -07:00
Dan Willemsen
32a669b01b Use our own TMPDIR
Some people have run into issues where /tmp is on a smaller partition
than their main source tree, and if it fills up (or only has a few GB
free), their builds can start failing.

There may also be a subset of people whose /tmp may be slower than their
out directory -- if they put their src and out trees on a SSD, but keep
/tmp on a spinning HDD.

Bug: 71755844
Test: check out/soong.log for TMPDIR
Test: do a build, watch out/soong/.temp
Change-Id: Ib749ec736ad0f6c8ce7453861a91b3e99ddbd92d
2018-03-08 19:42:00 -08:00
Dan Willemsen
d9429e6ff2 Handle Scanner errors in kati output filter
If Kati ever wrote a line over 64k characters, the scanner in soong_ui
would error, and Kati would hang trying to write into the pipe. Now if
the Scanner errors out, fall back to copying directly from the pipe to
the output.

Test: Add $(warning $(shell find frameworks)), does not hang
Change-Id: I86412ad2f53f2fe3cbda901ee673abb904d56d3c
2018-03-06 22:08:27 -08:00
Nan Zhang
2e6a4ff3be Let Soong_UI to handle build_date.txt file
Soong_UI will update timestamp to build_date.txt, and export variables
to kati/ninja.

Test: m -j32
Bug: b/70351683
Change-Id: I153897afdf2d3f39a32d757d4c3ae7515caea52d
2018-02-15 11:45:53 -08:00
Dan Willemsen
e7945d76ac Make GOROOT consistent
We were previously setting GOROOT to "prebuilts/go/linux-x86" during the
ninja executions when we were running Soong. But we can also run Soong
during the main ninja execution, were GOROOT was unset. When the GOROOT
was unset, the default GOROOT in our Go installation is
"./prebuilts/go/linux-x86" (note the extra ./).

This would cause g.bootstrap.goRoot to change between some soong runs,
causing us to rebuild all go programs (and anything depending on them)
more often than necessary.

So instead, keep GOROOT undefined when running Soong. Everything that
matters is using runtime.GOROOT(), which will fall back to the default.

Continue setting $GOROOT for bootstrap.bash, otherwise it fails when
there is no system provided go binary. What we give bootstrap.bash
doesn't really matter, since we don't actually use the blueprint wrapper
in Android.

Test: m blueprint_tools; touch bionic/libc/tzcode/new.c;
      m blueprint_tools <doesn't rebuild everything>
Change-Id: I82f30c7c3b5d25e5cbf28fe37a97fdb776c4a164
2018-01-23 22:56:17 -08:00
Dan Willemsen
ed86952403 Normalize LANG / LC_* environment
This ensures that the current locale supports UTF-8, and that we're
getting a consistent (but still supported by the system) locale for
every configuration except user-facing messages. This should eliminate
any reproducibility problems around sorting, formatting, etc for all
built products, while still showing localized error messages where
available.

Bug: 71573630
Test: LANG=es_ES LANGUAGE=es: m   (check env in soong.log)
Change-Id: If33311899eaed8c44573113ee35c5a71cee503a0
2018-01-19 21:46:21 +00:00
Treehugger Robot
7d3e0709a5 Merge "Remove armv5te" 2018-01-12 05:41:11 +00:00
Dan Willemsen
0c51851f08 Apply kati output rewriting to dumpvars
This way we strip out ANSI codes when using dumb terminals. It's likely
overkill to use katiRewriteOutput, but we should rarely see any output
on stderr while dumping variables. This also lets us turn on kati_stats.

Bug: 71729611
Test: lunch missing-eng             (colored error)
Test: lunch missing-eng 2>&1 | cat  (non-colored)
Test: TERM=dumb lunch missing-eng   (non-colored)
Change-Id: Ic63fd42d82a4a64e5c68aecd9ae0f242a0d703f1
2018-01-09 02:24:17 -08:00
Dan Willemsen
e401a8315e Remove armv5te
This architecture only existed for unbundled use, but even the NDK is
removing support in their r17 release, so just remove support for it.

Test: build/soong/build_test.bash -only-soong
Change-Id: I4bd23babf567128d2d242cbdee3311abb198dd7c
2018-01-08 15:48:25 -08:00
Tobias Thierer
18099fd856 Build with OpenJDK 9 -target 1.8 by default (attempt 4).
This is cherry-picked from attempt 3, which was reverted
due to http://b/70862583.

Before this CL topic, the build toolchain for .java source files
used OpenJDK 8, targeting 1.8 (v52 class files) by default.

This CL topic switches the default to OpenJDK 9, but still
targeting 1.8 (v52 class files) by default. If USE_ERROR_PRONE
is set to true, then the default remains OpenJDK 8.

Code in the Android platform should generally be unaffected,
but if host tools that are now compiled and run using
OpenJDK 9 are causing problems for your team, then let me
know.

To manually switch back to the old behavior for now (continue
using OpenJDK 8), run this command in your shell:
export EXPERIMENTAL_USE_OPENJDK9=false

Bug: 69449021
Test: Treehugger
Test: art/test/testrunner/run_build_test_target.py -j40 art-interpreter
Test: dalvik/dx/tests/run-all-tests

(cherry picked from commit 13f23a2753)

Change-Id: I57abae73f9bdb21ef004a5118ff0e4ef70418ed9
Merged-in: Iac78122f58df0ebbb55134d55021ce6c57351b5f
2018-01-04 11:07:38 +00:00
Colin Cross
8d6395c09d Move android/soong/fs to android/soong/finder/fs
The fs package is specific to finder, move it inside finder.

Bug: 70897635
Test: m checkbuild
Change-Id: Ie705f064a832141702a8e87fd59ed75c01018504
2017-12-22 13:56:17 -08:00
Tobias Thierer
e59aeff5c4 Make ANDROID_JAVA{8,9}_HOME available to config.mk.
Previously, these variables were exported only by makevars.go,
but those values are not available to config.mk. This CL adds
the variable to ui/build/config.go, which also makes it
available to config.mk.

Test: Treehugger
Bug: 70862583
Bug: 70521453
Change-Id: Ib54660e4b08ab751265b30004630cf1bb8c0041b
2017-12-20 23:14:37 +00:00
Tobias Thierer
8b6faf6c8e Merge "Revert "Build with OpenJDK 9 -target 1.8 by default except for errorprone."" 2017-12-11 22:57:51 +00:00
Jeff Gaston
592faab77d Merge "Clearer error if no Android.bp is found" 2017-12-11 21:10:38 +00:00
Tobias Thierer
e05f3a5f3f Revert "Build with OpenJDK 9 -target 1.8 by default except for errorprone."
After this revert CL topic, the default toolchain for Android
goes back to being OpenJDK 8.

This revert is being prepared ahead of time in case of
any problems with CL topic:
https://r.android.com/#/q/topic:bug69449021_attempt2

Bug: 69449021
Test: Treehugger
Change-Id: I9ead8d569226bd487baee3c6d5be9ec7033eb56a
2017-12-11 14:54:10 +00:00
Tobias Thierer
13f23a2753 Build with OpenJDK 9 -target 1.8 by default except for errorprone.
(This is cherry-picked from the first attempt to submit this CL
topic, which was reverted after 3 hours because of bug 70286093;
robolectric 3.{1.1,4.2} now stick with OpenJDK 8 to avoid that bug).

Before this CL topic, the build toolchain for .java source files
used OpenJDK 8, targeting 1.8 (v52 class files) by default.

This CL topic switches the default to OpenJDK 9, but still
targeting 1.8 (v52 class files) by default. If USE_ERROR_PRONE
is set to true, then the default remains OpenJDK 8.

Code in the Android platform should generally be unaffected,
but if host tools that are now compiled and run using
OpenJDK 9 are causing problems for your team, then let me
know.

To manually switch back to the old behavior for now (continue
using OpenJDK 8), run this command in your shell:
export EXPERIMENTAL_USE_OPENJDK9=false

Bug: 69449021
Test: Treehugger
Test: Running robolectric tests succeeds on internal-master
      after cherry-picking this CL topic, using the command
      line from http://b/70286093#comment1
(cherry picked from commit 0ae8b548af)

Change-Id: Ide6a7e55126d919a44f89ef8e0bd14fb12ff470e
2017-12-11 14:53:52 +00:00
Jeff Gaston
02ae4decb7 Clearer error if no Android.bp is found
Bug: 70036990
Test: touch .out-dir && m -j nothing 2>&1 | grep "must not exist"
Change-Id: Idefa70dafa53bca17134e99185d2c524671bae25
2017-12-08 14:39:37 -08:00
Jeff Gaston
29e959dd7f Search for files named Blueprints only under build/blueprint
to enable other subprojects to use their own instance of
Blueprint in other directories.

Files named Android.bp are still autodetected throughout the
tree like previously.

Bug: 64363847
Test: mkdir -p subdir \
      && echo "syntax error" > Blueprints \
      && m -j nothing

Test: build/soong/scripts/diff_build_graphs.sh \
      --products=aosp_arm \
      'build/soong:work^' 'build/soong:work'

Change-Id: I8199f12b68dc1699bf44682b86169def37b53a5e
2017-12-07 13:00:51 -08:00
Tobias Thierer
c7efdd9b36 Merge "Revert "Default to building with OpenJDK 9 -target 1.8."" 2017-12-06 14:00:47 +00:00
Tobias Thierer
cf8d83de82 Merge "Build with OpenJDK 9 -target 1.8 by default except for errorprone." 2017-12-06 11:05:31 +00:00
Tobias Thierer
7c05949407 Revert "Default to building with OpenJDK 9 -target 1.8."
After this revert CL topic, the default toolchain for Android
goes back to being OpenJDK 8.

This revert is being prepared ahead of time in case of
any problems with original change topic,
https://r.android.com/#/q/topic:bug_69449021

Bug: 69449021
Test: Treehugger

This reverts commit 0ae8b548af.

Change-Id: Ief7646a94f1a264085cd299b4327d244b78a1537
2017-12-05 02:10:55 +00:00
Tobias Thierer
0ae8b548af Build with OpenJDK 9 -target 1.8 by default except for errorprone.
Before this CL topic, the build toolchain for .java source files
used OpenJDK 8, targeting 1.8 (v52 class files) by default.

This CL topic switches the default to OpenJDK 9, but still
targeting 1.8 (v52 class files) by default. If USE_ERROR_PRONE
is set to true, then the default remains OpenJDK 8.

Code in the Android platform should generally be unaffected,
but if host tools that are now compiled and run using
OpenJDK 9 are causing problems for your team, then let me
know.

To manually switch back to the old behavior for now (continue
using OpenJDK 8), run this command in your shell:
export EXPERIMENTAL_USE_OPENJDK9=false

Bug: 69449021
Test: Treehugger
Test: "make core-oj", checked that compilation now uses
      OpenJDK 9 javac -target 1.8
Test: Checked that this is still compiled using OpenJDK 8.
      export EXPERIMENTAL_USE_OPENJDK9=false
      make core-oj

Change-Id: Ic87e9bb2a2e5da0ff13a2e51845b5365901c1507
2017-12-05 02:07:30 +00:00
Jeff Gaston
088e29ed38 Revert "Revert "Soong support for namespaces""
This mostly reverts commit 178d5fefc0
and mostly reapplies change I6d3e52ef62c4cabe85b9a135a54de0e1a6aab29c .

Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
      --products=aosp_arm \
      'build/blueprint:work^ build/soong:work^' \
      'build/blueprint:work build/soong:work'
      # and see that the only changes were:
      # 1. adding some new files
      # 2. changing some line numbers
Test: m -j nothing # which runs unit tests

Change-Id: I32baae00277a547fdcdd1c2219fe6625ee0e45d7
2017-12-04 17:44:29 -08:00
Jeff Gaston
178d5fefc0 Revert "Soong support for namespaces"
This reverts commit 63a250a336.

Reason for revert: Some failures:
  namespace_test.go:648: dir1/Blueprints:2:4: a namespace must be the first module in the file 
  such as New Build Breakage: aosp-master/build_test @ 4475274

Change-Id: I1b5db8eb934e51ff22241bfca44199d886b1393b
2017-11-30 00:43:43 +00:00
Jeff Gaston
a8063afc16 Merge "Soong support for namespaces" 2017-11-29 23:01:39 +00:00
Treehugger Robot
b8160761f7 Merge "Revert "Revert "Test for dangling rules in make checkbuild""" 2017-11-29 21:09:48 +00:00
Jeff Gaston
63a250a336 Soong support for namespaces
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
      --products=aosp_arm \
      'build/blueprint:work^ build/soong:work^' \
      'build/blueprint:work build/soong:work'
      # and see that the only changes were:
      # 1. adding some new files
      # 2. changing some line numbers
Test: m -j nothing # which runs unit tests

Change-Id: I6d3e52ef62c4cabe85b9a135a54de0e1a6aab29c
2017-11-29 13:03:40 -08:00
Treehugger Robot
8763169bdb Merge "Add TEST_MAPPING to finder's cache" 2017-11-29 19:16:50 +00:00
Colin Cross
37193495cf Revert "Revert "Test for dangling rules in make checkbuild""
This reverts commit fb941913a3.

Reapplies I4933187e8b72f2ef0c32d18ffea756e2c6fa417c with fixes
to disable the check for mac builds, where many modules are
disabled and the check is just going to cause more problems.

Test: m checkbuild
Change-Id: If6712c90ececd5d015fcdcdeefe0c3d4f5590711
2017-11-25 05:26:10 +00:00
Dan Shi
cc3d9b304e Add TEST_MAPPING to finder's cache
Build target test_mapping will create a tarball of all TEST_MAPPING
files in source code based on the cached TEST_MAPPING.list

Bug: 69678490
Test: m -j blueprint_tools && cat out/.module_paths/TEST_MAPPING.list
Change-Id: Iba3479363137609d6631cf1851a8ce4613078f61
2017-11-22 15:35:09 -08:00
Tobias Thierer
a7cdd156a6 Support EXPERIMENTAL_USE_OPENJDK9=false.
This fourth possible value currently has the same semantics
as a default/unset EXPERIMENTAL_USE_OPENJDK9, but allows
people to explicitly switch back to the old semantics when
the default changes.

Test: make showcommands core-oj (in environments with
      EXPERIMENTAL_USE_OPENJDK9 set to "", "1.8", and "true").
Bug: 38177295

Change-Id: I25accf14344a05349a6e97572d7c2c1f6a7f2063
2017-11-17 13:14:11 +00:00
Colin Cross
fb941913a3 Revert "Test for dangling rules in make checkbuild"
This reverts commit 7709a05770.

Reason for revert: Broke mac checkbuilds

Change-Id: If71ec9198ca8943d90ce9d5e3a277faf239e38a5
2017-11-13 23:38:57 +00:00
Colin Cross
7709a05770 Test for dangling rules in make checkbuild
Get a list of leaf nodes in the dependency graph
from ninja, and make sure none of them are in the
output directory.  This ensures that there are no
rules that depend on a file in the output directory
that doesn't have rule to generate it.  The check
will catch a common set of build failures where
a rule to generate a file is deleted (either by
deleting a module in an Android.mk file, or by
modifying the build system incorrectly).  These
failures are often not caught by a local incremental
build because the previously built files are still
present in the output directory.

Bug: 36843214
Bug: 68062417
Test: manual
Change-Id: I4933187e8b72f2ef0c32d18ffea756e2c6fa417c
2017-11-13 12:47:42 -08:00
Jeff Gaston
d3e141de80 Pass results of Finder into Blueprint
Bug: 64363847
Test: m -j

Change-Id: I79db8c524af6e77c35a0199ec1876e5eb94e8971
2017-11-08 23:48:44 +00:00
Dan Willemsen
02c4c1d3f0 Fix mac build
Test: `m nothing` on mac
Change-Id: Ibc377134dbcc5d40263697ef1e79fc31b16e0d1d
2017-11-08 12:50:38 -08:00
Dan Willemsen
d9e8f0a95a Setup java paths in soong_ui
This way config.mk no longer needs to check which java is in PATH and
fix it. It'll be consistent for all build steps under soong_ui.

Also unify handling of ANDROID_JAVA_HOME / JAVA_HOME with
OVERRIDE_ANDROID_JAVA_HOME / EXPERIMENTAL_USE_OPENJDK9.

Test: m nothing
Test: build/soong/soong_ui.bash --make-mode nothing (w/o envsetup.sh)
Test: aosp_arm ninja files are the same before/after
Test: before/after ninja files match with OVERRIDE_ANDROID_JAVA_HOME
Test: before/after ninja files match with EXPERIMENTAL_USE_OPENJDK9
Change-Id: Icdb65093d9c346524074de239a4f895e4230a24d
2017-11-08 00:44:47 -08:00
Dan Willemsen
69b18ed167 Silence error during microfactory trace import
This may happen if there's been another call to microfactory before
we've read the log file. Just ignore the error and continue.

Test: Run `while true; do get_build_var TARGET_PRODUCT; done` in parallel
Change-Id: I670144f4637281ebf1299dbd9151dd9819c363e3
2017-10-18 15:09:16 -07:00
Dan Willemsen
ca8feb385c Synchronize file rotation
Create a lock file during log rotation so that multiple processes won't
step on each other.

Test: Run `while true; do get_build_var TARGET_PRODUCT; done` in parallel
Test: m blueprint_tools
Change-Id: I7144cd42aca47c694487ddae44713f82665ed81e
2017-10-18 15:09:08 -07:00
Treehugger Robot
21dc570e5f Merge "Fix soong_ui with empty arguments" 2017-10-18 05:30:23 +00:00
Dan Willemsen
6ac63ef003 Fix soong_ui with empty arguments
Test: build/soong/soong_ui.bash --make-mode "" nothing
Change-Id: I8bbe947078eaa00976811ce1b9b34e1a7c23f97f
2017-10-17 20:35:34 -07:00
Dan Willemsen
051133ba54 Add --dumpvar[s]-mode to replace config.mk uses
build/soong/soong_ui.bash --dumpvars-mode \
    --vars="..." \
    --abs-vars="..." \
    --var-prefix="..." \
    --abs-var-prefix="..."

is similar to the previous:

  CALLED_FROM_SETUP=true BUILD_SYSTEM=build/core \
  make -f build/core/config.mk dump-many-vars \
    DUMP_MANY_VARS="..." \
    DUMP_MANY_ABS_VARS="..." \
    DUMP_VAR_PREFIX="..." \
    DUMP_ABS_VAR_PREFIX="..."

and

  build/soong/soong_ui.bash --dumpvar-mode [--abs] VAR

is similar to the previous:

  CALLED_FROM_SETUP=true BUILD_SYSTEM=build/core \
  make -f build/core/config.mk dumpvar-[abs-]-VAR

But uses soong_ui and ckati, so that we use a consistent make parser and
sandboxing configurations.

One major output difference between the pure make implementation and
this one is that report_config in Go is implemented using embedded
newlines in single quotes, while the make implementation uses `` with
embedded echo commands. This seems to work fine for both bash and zsh,
and report_config isn't meant to be machine-parsed anyways.

Test: build/soong/soong_ui.bash --dumpvar-mode report_config
Test: build/soong/soong_ui.bash --dumpvar-mode TARGET_DEVICE
Test: build/soong/soong_ui.bash --dumpvar-mode --abs PRODUCT_OUT
Test: build/soong/soong_ui.bash --dumpvar-mode --abs ALL_PRODUCTS
Test: build/soong/soong_ui.bash --dumpvars-mode --vars="report_config TARGET_DEVICE" --abs-vars="ALL_PRODUCTS"
Test: build/soong/soong_ui.bash --dumpvars-mode --vars=TARGET_DEVICE --abs-vars=PRODUCT_OUT --var-prefix=v_ --abs-var-prefix=a_
Change-Id: I0fbd0732bbf6fcfcd24084cf3c830a91a4b6bfc2
2017-10-17 12:57:41 -07:00
Dan Willemsen
8122bd58ba Revert "Always run asan ckati on the build servers"
This reverts commit a3e6c520ac. I fixed
a few heavy stack users, and we've got better debugging.

Bug: 36182021
Test: m dist
Change-Id: Iefd8589185720e44c40a2e0ceff5fd1438211b86
2017-10-14 00:43:13 +00:00
Dan Willemsen
75d2c170b4 Always record kati stats into soong log
Pass --dump-stats to ckati, but filter out the '*kati*' lines so that
they only end up in our verbose output. That way we've always got access
to the statistics.

Bug: 36182021
Test: m nothing; cat out/soong.log
Change-Id: Iaf7a814fc67f3e475c913faf69924a7f4e2ae3b3
2017-10-14 00:42:30 +00:00
Dan Willemsen
b2e6c2e571 Switch product configs from make to ckati
This speeds up dumping make variables from ~380ms using make to ~220ms
using ckati. It also means that we're consistently using the same parser
for builds (with the same .KATI_READONLY/etc extensions).

envsetup.sh (lunch) / other scripts still use make, changing those to go
through soong_ui will be a future change.

Test: m clean; m nothing
Test: USE_GOMA=true m nothing
Test: m PRODUCT-aosp_x86-sdk
Test: m APP-Calculator
Test: build/soong/build_test.bash -only-config  (on AOSP and internal master)
Change-Id: I6ca554de8de4955fb869001d06d29969b75751cc
2017-10-12 18:11:08 -07:00
Dan Willemsen
4dc4e14d15 Remove (TARGET|HOST)_BUILD_TYPE path modifications
They don't really affect anything general in android except for the
output path.

Bug: 65453318
Test: TARGET_BUILD_TYPE=debug m
Change-Id: I98c17fefae61f9f7936e184bd1f8441086bb7d23
2017-09-08 21:47:31 +00:00
Dan Willemsen
59fdf96fcf Split CleanSpec processing out of main Kati run
Instead of pairing the reading of CleanSpec.mk files with the reading of
Android.mk files, split them into separate, individually cachable steps.
This way we only read Android.mk files once after a clean/sync. We'll
still read the CleanSpec.mk files multiple times, but that's
significantly faster than reading all the Android.mk files.

This adds about 50ms if kati doesn't need to reread the CleanSpec.mk
files. Reading all the CleanSpec.mk files takes about a second.

Bug: 35970961
Test: m clean; m nothing; m nothing
Test: Add CleanSpec.mk line, see it executed.
Change-Id: I83bad15c50709510959d5b8b673a907b8aa7de82
2017-08-23 01:53:33 +00:00
Dan Willemsen
e27ed543bf Stop setting ASAN_OPTIONS for kati
Kati sets detect_leaks=0 by default now.

Test: unset ASAN_OPTIONS; SANITIZE_HOST=address m nothing
Change-Id: I0cbf04ae9ed40de520abca3fb35c65e7f543682d
2017-08-21 20:54:06 -07:00
Dan Willemsen
99a75cd2a5 Obsolete bootstrap.bash and ./soong wrappers
We can call directly into the blueprint bootstrap.bash using values that
soong_ui has already calculated.

Instead of calling into blueprint.bash, build minibp with microfactory,
and directly run ninja. This allows us to get individual tracing data
from each component.

Test: m -j blueprint_tools
Test: m clean; m -j blueprint_tools
Change-Id: I2239943c9a8a3ad6e1a40fa0dc914421f4b5202c
2017-08-18 10:13:22 -07:00
Jeff Gaston
b64fc1cab5 Run the Finder and make its results available to Kati
The Finder runs roughly 200ms faster than findleaves.py in aosp,
and runs roughly 400ms faster in internal master.

Bug: 64363847
Test: m -j

Change-Id: I62db8dacc90871e913576fe2443021fb1749a483
2017-08-16 14:56:00 -07:00
Jeff Gaston
f1fd45e784 Revert "Revert "Cacheable, multithreaded finder.""
Bug: 62455338
Test: m -j

This reverts commit d1abeb9d98.

Change-Id: I9f73031636157511b5f1c6ce8a205e9bc91669ff
2017-08-10 12:20:30 -07:00
Jeff Gaston
c3f32070dc Merge "Revert "Cacheable, multithreaded finder."" 2017-08-10 01:21:09 +00:00
Jeff Gaston
d1abeb9d98 Revert "Cacheable, multithreaded finder."
This reverts commit b6d161bf16.

Reason for revert: New Build Breakage: aosp-master/sdk_mac @ 4260825

Change-Id: I8bda8c50c5e5c9f84621d11a4c15b168833bcd21
2017-08-10 01:16:34 +00:00
Jeff Gaston
c17e048bad Merge "Cacheable, multithreaded finder." 2017-08-09 22:18:13 +00:00
Jeff Gaston
df4a081292 Move build system documentation
Bug: 62201421

Test: make help
Change-Id: I7d7b448d3f736e171ebfbdceee46eef7de748742
2017-08-09 19:40:01 +00:00
Dan Willemsen
e0879fc3ae Add --skip-make to replace Soong's blueprint wrapper with soong_ui
This way we only have one way to start a build, which always has logging
/ tracing / etc, even if we don't need Kati.

There's two ways to use this:

As a direct replacement for mkdir out; cd out; ../bootstrap.bash;
./soong -- as long as --skip-make is always passed, we'll never run
Kati, and Soong will run outside of it's "make" mode. This preserves
most of the speed, and allows full user control over the Soong
configuration.

A (experimental, dangerous) way to temporarily bypass the product
variable and kati steps of a build. As long as a user is sure that
nothing has changed from the last build, and they know exactly which
Ninja targets they want to build (which may not be the same as the
arguments normally passed to 'm'), this can lead to shorter build
startup times.

Test: rm -rf out; m --skip-make libc
Test: rm -rf out; m libc; m --skip-make libc
Test: rm -rf out; mkdir out; cd out; ../bootstrap.bash; ./soong libc
Test: build/soong/scripts/build-ndk-prebuilts.sh
Change-Id: Ic0f91167b5779dba3f248a379fbaac67a75a946e
2017-08-08 22:49:28 -07:00
Jeff Gaston
b6d161bf16 Cacheable, multithreaded finder.
It can find every Android.bp in internal master in
about 2.5 sec the first time and 0.3 sec subsequent times

Bug: 62455338

Test: m -j blueprint_tools # which runs the unit tests

Test: m -j blueprint_tools && \
  out/soong/host/linux-x86/bin/finder \
    -v
    --db /tmp/mydb \
    --names Android.mk \
    --prune-files .android-out-dir \
    --exclude-dirs .git,.repo \
    . \
    >/tmp/finder-log 2>&1

Change-Id: I5ab2650459a1dae0d5d076faf411ec2d053c743d
2017-08-08 14:19:55 -07:00
Treehugger Robot
98be5836de Merge "Remove java version checks" 2017-07-25 19:55:48 +00:00
Dan Willemsen
05f17764c3 Use a unified ninja builddir
Instead of calling SetNinjaBuildDir, pass it to bootstrap.bash, so that
the bootstrap package can set it consistently during bootstrapping and
normal execution.

Bug: 63720725
Test: m -j nothing
Test: mkdir o; ../bootstrap.bash; ./soong
Change-Id: Ica88d2d5f1461b5be49bfe6316c6ec4ef4d89d49
2017-07-24 15:58:20 -07:00
Colin Cross
3f837b9d90 Remove java version checks
The build now uses a prebuilt javac binary distributed with the
source, checking the host java version is no longer required.

Reapplies Ieef83481d0e6d68371bbd8a422f870ffe2bbb428 after updating
some more manifests to include the prebuilts.

Test: m -j checkbuild
Bug: 62956999
Change-Id: I774dbb75c8ae2ff1a990423a3827c77886e44779
(cherry picked from commit c6f07f00da)
2017-07-20 15:14:43 +00:00
Colin Cross
6650414087 Revert "Remove java version checks"
This reverts commit c6f07f00da.

Change-Id: Ibcb52ce2aecf20ac741ed9beec27c2b4b07f8b66
2017-07-19 22:42:59 +00:00
Colin Cross
c6f07f00da Remove java version checks
The build now uses a prebuilt javac binary distributed with the
source, checking the host java version is no longer required.

Test: m -j checkbuild
Bug: 62956999
Change-Id: Ieef83481d0e6d68371bbd8a422f870ffe2bbb428
2017-07-14 14:01:29 -07:00
Dan Willemsen
cae59bc29d Add microfactory tracing
Test: m clean; load out/build.trace.gz in chrome://tracing
Test: m -j nothing; load again
Change-Id: I67d4e006a4bdde593b54c20d6d93a48278fe696d
2017-07-13 15:09:44 -07:00
Dan Willemsen
c40e10bd6b Drop make flags in soong_ui
MAKELEVEL and MAKEFLAGS were are both unset by makeparallel when invoked
from Make. In preparation for removing the Make wrapper (and
makeparallel), strip them in soong_ui as well.

Add MFLAGS for completeness.

Test: m -j
Change-Id: I029f9464d11b432e979b32ede360450f4facb038
2017-07-11 15:23:55 -07:00
Dan Willemsen
091525e15c Parse environment options from the command line
Make supports specifying all types of variables on the command line
(using =, :=, +=, and other variable references. When running soong_ui
through make/makeparallel these all effectively become environment
variables.

So in preparation to remove the Make wrapper, support a simplified form
of this syntax, roughly equivalent to what the shell supports if
specified before the command (<NAME>=<VALUE>).

Test: m -j blueprint_tools
Change-Id: I08fa2b86710f282e619b0cc324a3e5bbaf62d26e
2017-07-11 15:23:51 -07:00
Dan Willemsen
9b58749f30 Support more ways to pass -j/-k
In preparation to remove Make/makeparallel from soong_ui startup, we
need to preserve compatibility with the different ways that make
supports the -j option.

Nothing changes unless Make/makeparallel is removed from the startup.
Once that is removed, not specifying a -j value will be equivalent to
'-j' instead of '-j1', like Ninja. A value will also be supported when
specifying -k, like Ninja (though specifying it alone will be equivalent
to '-k 0').

Test: m -j blueprint_tools
Change-Id: I9d5d59bedd4f6e5cca76bdb4cd47e0b5b7d523f0
2017-07-11 14:51:26 -07:00
Dan Willemsen
418420e49d Turn find emulator warnings into errors
Test: build/soong/build_test.bash (multiproduct_kati)
Change-Id: Ieb65a457c8c1ed4bff44a01aa2cc11203172ba9b
Merged-In: Ieb65a457c8c1ed4bff44a01aa2cc11203172ba9b
Merged-In: I70b696525a90c64cc05ad4caa1c3647d57ec9e9e
(cherry picked from commit dc5b70f97c)
(cherry picked from commit e8a25d5577)
(cherry picked from commit 14b3f7f12b)
2017-06-21 12:44:18 -07:00
Jeff Gaston
a6697e8b7f Only check for ninja stuckness after it's been running for a bit
Bug: 62580037
Test: rm -rf out && m -j showcommands | grep "ninja may be stuck" || echo ok

Change-Id: I8ff1bd216b5f8349ce9e06e5465a9f8d0663f8c0
2017-06-13 14:15:02 -07:00
Jeff Gaston
efc1b412f1 Have Soong try to enforce that genrules declare all their outputs.
This causes Soong to put the outputs of each genrule into a temporary
location and copy the declared outputs back to the output directory.
This gets the process closer to having an actual sandbox.

Bug: 35562758
Test: make

Change-Id: I8048fbf1a3899a86fb99d71b60669b6633b07b3e
2017-06-09 17:57:18 +00:00
Jeff Gaston
809cc6f58a Run 'pstree' if ninja_log hasn't updated recently
This doesn't catch all the possible causes of timeouts,
(like if Ninja is only partially stuck or if Kati is stuck)
but it should clarify some causes of stuckness

Bug: 62065855
Test: m -j showcommands NINJA_HEARTBEAT_INTERVAL=500ms
Change-Id: I73a792ae91873b19d7b336166a2d47f37c549906
2017-06-06 16:33:51 -07:00
Dan Willemsen
a18660d366 Have installclean remove android-info.txt again
I removed *.txt from the installclean list yesterday, but it turns out
that android-info.txt being removed was the only thing triggering
system.prop to be rebuilt on the incremental build servers. So continue
android-info.txt so that the version information in system.prop is
correct.

I'll follow this up with a more complicated change to make the
system.prop generation depend on the build number/versions more
directly, but that's a more complicated change since we need to change
how the build number file is written.

Bug: 62252890
Test: m ...intermediates/system.prop; m installclean; m ...intermediates/system.prop
Change-Id: I25dd8ddd99f7b2c927a8cbfb7f1aca1dd3e82a82
2017-06-01 22:49:47 +00:00
Dan Willemsen
e427a40356 Stop installcleaning $PRODUCT_OUT/*.{ini,txt,xlb}
For *.ini/*.txt files we should be recreating them as the instructions
change, so we shouldn't need to remove them. The biggest reason to
remove them is that the auto-installclean was removing
$PRODUCT_OUT/build_fingerprint.txt just after Kati wrote it out, causing
the fingerprint to be blank on builds where the auto installclean code
ran.

There's no reference to xlb in our tree, and it's been around since the
beginning of git history, so remove it.

Bug: 62224537
Test: lunch aosp_arm-eng; m nothing;
      lunch aosp_arm-userdebug; m -j nothing;
      check out/target/product/generic/build_fingerprint.txt
Change-Id: Id391af5c7d95ea78e4e68fae294d8295fccb1964
2017-05-31 17:26:34 -07:00
Dan Willemsen
977fd8f1fa Merge "Support Make rewriting APP-*" 2017-05-26 19:51:56 +00:00
Tobias Thierer
9847add454 Merge "Allow version 9 toolchains for EXPERIMENTAL_USE_OPENJDK9" 2017-05-26 12:08:38 +00:00
Tobias Thierer
849bb6bf99 Allow version 9 toolchains for EXPERIMENTAL_USE_OPENJDK9
By default, the Android build enforces an OpenJDK 8 toolchain,
whose name contains the strings "openjdk" and "1.8".

After this CL, the check can be changed to enforce a toolchain
name starting with "9" and without the need for "openjdk" having
to occur in the name.

This experimental new check can be enabled by running:
  export EXPERIMENTAL_USE_OPENJDK9=true
To switch back to the standard check, run:
  unset EXPERIMENTAL_USE_OPENJDK9

Test: make ANDROID_COMPILE_WITH_JACK=false checkbuild tests \
      && make checkbuild tests
      (with OpenJDK 8u45 toolchain on the PATH)
Test: make EXPERIMENTAL_USE_OPENJDK9=true \
      ANDROID_COMPILE_WITH_JACK=false checkbuild
      (with jdk 9-ea+170 toolchain on the PATH)
Bug: 38177295

Change-Id: I75de3e23fe0b7f41eb6dd3f55dadd3fa3c3383bd
2017-05-26 12:45:25 +01:00
Dan Willemsen
04a16c7dcf Support Make rewriting APP-*
Test: ALLOW_MISSING_DEPENDENCIES=true m -j APP-Calculator
Change-Id: If5a0b60a0dbc615b6f47ea222616f8d34950ba0c
2017-05-25 22:18:57 -07:00
Jeff Gaston
3615fe8665 Revert "Revert "Disallow multiple build executions in parallel""
Bug: 36698966
Test: m -j & m -j # (run two builds at once)

This reverts commit be9cd1192b.

Change-Id: Ia15c0bf156a6de54da054eab8134f36bae0823ca
2017-05-24 13:22:19 -07:00
Tobias Thierer
442b7e6dc9 Merge "Drop build support for LEGACY_USE_JAVA7." 2017-05-24 10:24:53 +00:00
Tobias Thierer
8b4319c3aa Drop build support for LEGACY_USE_JAVA7.
This flag allowed for building with an OpenJDK 7 toolchain. It was
used for build bot builds that now work with an OpenJDK 8 toolchain.
Hence, this feature is no longer required. This CL drops it.

Bug: 27583810
Test: Treehugger build succeeds.
Change-Id: Iba9cf12cff2597fa70aa9999e8a6bda190f29f3c
2017-05-23 14:20:03 +01:00
Dan Willemsen
f052f78b40 Move dataclean and installclean to soong_ui
This speeds things up a little bit, but the major win is that we don't
need to recursively call into the build during the auto installclean.

Test: m -j installclean
Test: m -j dataclean
Change-Id: I09ded8dbd8a2015c7848bc0042dfd3adad1b31c9
2017-05-18 15:48:25 -07:00
Jeff Gaston
be9cd1192b Revert "Disallow multiple build executions in parallel"
This reverts commit 7db68be5e1.

Reason for revert: postsubmit build encountered a failure

Change-Id: I3c7304114f712d78f79afb1b9d84dc2eadcbc876
2017-05-18 00:34:48 +00:00
Jeff Gaston
7db68be5e1 Disallow multiple build executions in parallel
Bug: 36698966
Test: run two 'make -j' executions in parallel in each of two terminals

Change-Id: Ief91391fe1f0fdbf9655cc0ba1298e353396d705
2017-05-16 17:47:09 -07:00
Treehugger Robot
e121bbbb3b Merge changes I9512642d,I6548889c,I8db5198f
* changes:
  Add build_test.sh, split common parts of soong_ui.bash
  Allow specifying a build variant
  Improve multiproduct_kati output
2017-05-16 19:45:08 +00:00
Dan Willemsen
02781d59bb Move auto installclean to soong_ui
This way kati won't need to be run as often (either initially, or when
switching products with the same device).

Bug: 35970961
Test: m clean; m -j blueprint_tools; m -j blueprint_tools; m -j blueprint_tools
Test: lunch aosp_arm-eng; m -j blueprint_tools; lunch full-eng; m -j blueprint_tools; <repeat>
Change-Id: Ie9fca3c8f1dd412459ea47c7090c7c5fdb0bcf6e
2017-05-15 16:26:21 -07:00
Dan Willemsen
db8457cfec Move version checking from Make into soong_ui
When kati keeps state around, it has to regenerate the ninja file every
time the state is changed. So move the java version checking into
soong_ui, where we can parallelize it with other operations instead of
only checking it occasionally.

Bug: 35970961
Test: Put java7 in PATH, m -j
Test: Put java8-google in PATH, m -j
Test: Put a space in TOP, m -j
Test: OUT_DIR=<case-preserving fs> m -j
Test: OUT_DIR=<path with space> m -j
Test: DIST_DIR=<path with sapce> m -j
Change-Id: I3245c8dd6d856240d17d54cb05d593dc9df71a27
2017-05-15 16:21:33 -07:00
Dan Willemsen
a4e43a77d8 Improve multiproduct_kati output
It now uses the same output style as ninja, overwriting status lines in
smart terminals.

Test: multiproduct_kati
Test: multiproduct_kati | cat
Change-Id: I8db5198ffdc5ebc5503241ac492379753d92978e
2017-05-15 15:24:10 -07:00
Dan Willemsen
0b73b4bc37 Move clean/clobber to soong_ui
So that we don't have to load up all the makefile state just to remove
the output directory.

Starting from a completely empty out directory:
 kati:    16s
 soong_ui: 2.0s

From a minimal out directory (m -j blueprint_tools):
 kati:     3.8s
 soong_ui: 0.4s

Test: m -j clean
Test: m -j clobber
Change-Id: Ibeec6fbfa29387750342a752a55336caca4b3992
2017-05-15 14:46:04 -07:00
Dan Willemsen
02f3add3a3 Clean OUT_DIR
Otherwise Make and soong_ui disagree about where the soong.variables
file should be. For example, when "OUT_DIR=out/":

Error dumping make vars: Failed to parse make line: "make: Nothing to be done for `out/soong/soong.variables'."

We would need to pass out//soong/soong.variables into make, since it
doesn't attempt to match clean/unclean paths.

Bug: 37553659
Test: OUT_DIR=out/ m -j
Change-Id: Iebeea4868b96b11e8b08735f272e22be5db1d701
2017-05-12 21:12:13 +00:00