As part of a future change to sandbox the build on Linux, the real
username will be switching to "nobody", and the hostname will be
switching to "android-build".
The USER environment variable will reflect the sandboxed value, so for
the build properties that want the external USER, they'll need to use
BUILD_USERNAME.
Similarly, BUILD_HOSTNAME will reflect the real value, while the
`hostname` tool will return "android-build"
Bug: 122270019
Test: check build.prop
Change-Id: I99604b9488732a63690b256dc4dd7894d369a32c
So that ninja produces an error instead of just a warning when a dep
file is not produced.
Bug: 121058584
Test: check build logs for "depfile is missing" warning
Test: treehugger
Change-Id: I1cbaba866eaf293495c3c0b2b174190bcb2b0f9a
Test: Dumped the text formated based metrics file to out dir,
and checked the file.
Bug: b/63815990
Change-Id: Iff476f72a0be74eb53b6b26ef468d11c0f24a404
This way we don't appear hung at:
No need to regenerate ninja file
Change-Id: I8dbdaa2c1b1c5a6a73187d0e6061f363b62e10c9
Fixes: 122251150
Test: m nothing
Prepend the toybox symlink directory to $PATH, and stop generating
symlinks into out/.path for tools used from toybox.
We don't (yet?) have toybox prebuilts for darwin, so apply the above only to Linux, and preserve existing behavior on Darwin.
Test: check the build banner for uname results
Test: m
Change-Id: I37fc380381e65a628cdc131d462fd4441eacfe9d
Stop writing out ninja files, as they become very large, especially when
multiplied by the number of defined products.
Test: treehugger
Change-Id: Id9529d14040acb72a0188e58b5db2911f142071e
DIST_DIR is controlled by soong_ui, and the make dumpvars code no longer
has access to it. So instead, handle some dumpvar requests inside
soong_ui itself.
Test: get_build_var DIST_DIR
Test: get_build_var OUT_DIR
Test: get_build_var BUILD_SYSTEM
Test: build/soong/soong_ui.bash --dumpvars-mode --vars="DIST_DIR
BUILD_SYSTEM"
Change-Id: Id3bcb8b0748db67c83d0e42d1ae5be564c5eb2f7
We're only using it to distribute files in case of failure, which isn't
well supported currently, but can be handled for now by using the
DIST_DIR environment variable during the command execution.
This was at least one cause that we'd be re-running Soong during every
build server build, as the DIST_DIR values are unique.
Test: m dist
Change-Id: Ibd5e6b6c46695350de80b745bfb6a6aa685033a0
We've got a prebuilt of make in prebuilts/build-tools, and we build
xmllint from source.
Test: No logs on the build servers in the last 24 hours
Change-Id: Ib60a7b5feaa31097071b67ebf6beb7f0b9a9f81b
The idea is that we'd move the installation and packaging tasks over to
it, using data from Soong & the Kati reading Android.mk files.
This would allow us to make more fundamental changes about how we
package things without having to adjust makefiles throughout the tree.
Possible use cases:
* Moving some information from Soong's Android.mk output to a file read
by the packaging step may allow us to read the Android.mk files less
often, speeding up builds.
* Refactoring our current two-stage ASAN builds to run the Kati build
step twice, writing into different object directories, then have a
single packaging step that reads both outputs. Soong already has the
capability of writing out a single ninja file with all the asan
combinations.
* Running two build steps, one building the system-related modules
using a "generic" device configuration, and one building the vendor
modules using a specific device configuration. This could enforce a
GSI/mainline system vs vendor split in a single build invocation.
* If all installation is through this tool, it will be much easier to
track what should no longer be installed on an incremental build,
reducing the need for installclean.
* Changing PRODUCT_PACKAGES should be a much faster operation, which
means we could keep track of local additions to the images. Then
`mma` would be more persistent, instead of installing something once,
then never updating it again.
Eventually we plan on switching from Kati to something Go-based, but
this is a more incremental approach while we clean up everything else.
Currently, this just moves the dist-for-goal handling over to the
packaging step, so that we don't need to read Android.mk files when
DIST_DIR changes, or we switch between dist vs not.
Bug: 116968624
Bug: 117463001
Test: m nothing
Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
Set the default value if one isn't provided.
Test: `get_build_var DIST_DIR` after removing the default setting in envsetup.mk
Change-Id: I0cb310fc65f7747c36de14608b61786ef6863fd1
This is no longer used within Android builds. We don't have any way of
providing extra libraries, or using a hermetic version of perl.
Bug: 117338567
Test: check build logs for the last 48 hours
Change-Id: I06d41a0006a47508a435dbf44bfccc4b385d9f41
Now that we've got prebuilts of make, turn on logging so that
we can find all the users and convert them to the prebuilt.
We also build xmllint as part of the tree, so log uses of that
too.
Test: treehugger
Change-Id: I8a6a5d481dfc709dc0eb3424c51430bc54763c1e
Make the cleanspec & Android.mk runs of kati use much of the same code
and arguments.
Also renames 'Kati' to 'KatiBuild' in many cases, in the the expectation
that we'll have a 'KatiPackage' step in the future.
Use --no_ninja_prelude and move local_pool & _kati_always_build_ into
the combined ninja file. This will reduce the need to re-read makefiles
when Goma is enabled, and it allows us to include more than one
Kati-generated ninja file in the build graph.
Bug: 116968624
Test: build_test on downstream branches
Change-Id: Ibdac689b81f62dc293647fad917d84946f2c3cfa
Check for the presence of Android.mk or CleanSpec.mk, which
are somewhat common problems.
Bug: 113147143
Test: m (with and without files present)
Change-Id: I31cf60c325e7f6c6fce7aec54712c1cb802055c2
All downstream users either mark BUILD_BROKEN_PHONY_TARGETS=true
explicitly, or don't have any warnings.
Test: build_test on downstream branches
Change-Id: I6e9d7ed42da941148827854851657895c2e98649
Build target owners will create a zip of all OWNERS
files in source code based on the cached OWNERS.list
Bug: 114242886
Test: make dist -j owners
Change-Id: I97fa3737e1a2a240a08e072dfb2905c11f28705b
These warnings are usually hit in one of the two conditions:
1. Targets that should be marked .PHONY, but are not.
2. Attempts to define output files in the source tree.
See https://android.googlesource.com/platform/build/+/master/Changes.md#phony_targets
for more information.
Test: multiproduct_kati in all downstream branches
Change-Id: Ibe37cce320612825c22987e94d0740e8c0565b6f
od is used by ninja's inline.sh tool to encode a python script in a C
header.
Test: forrest run of build-tools branch
Change-Id: I87001a02b168ebe183b8b7dc495c67bb43f1b86d
Anything that isn't explicitly marked as Allowed in this list will now
return an error (and log a message) when it is executed.
Test: search all logs from the build server over the last day
Change-Id: I3ceacd9a140097809dde81a8d8979dd2c45f234c
unix2dos is used instead of todos by some build servers in
development/build/tools/patch_windows_sdk.sh
Test: none
Change-Id: I04cb2258f71bebfc8792b7aa95643a45ffe3ec5c
todos is used by development/build/tools/patch_windows_sdk.sh
Test: m win_sdk; check out/soong.log
Change-Id: I5e1a290838cc34261d7f5fc321afe79a8b1a6fbb
These still default to warnings, but if a board decides to set
BOARD_BROKEN_PHONY_TARGETS := false, they'll turn into errors. More
likely I'll just be marking the broken targets as broken, then switching
the logic here to be like the dup rules warnings/errors.
Test: On a build with warnings, try <missing>, "false", and "true"
Change-Id: I041e1ff4618c4114ec43015b6c0ae1b49b36b6f8
Attempting to reduce the number of different spellings we have for
"product services" partition in the codebase.
Bug: 112431447
Test: m
Change-Id: I0a393a1d625e7ea3217d28735a4db709bce32395
Merged-In: I0a393a1d625e7ea3217d28735a4db709bce32395
Setting this will cause our invocation of ninja to fail. Even if we
fixed that, it's likely to run into other differences during the build.
Fixes: 112443729
Test: POSIXLY_CORRECT=1 m nothing
Change-Id: I0315955c9fd9536cc6cb0d7c700eed207f35cc80
lsof is used by goma to find its proxy daemon
ps is used on Mac instead of pstree in soong_ui
Test: none
Change-Id: Ifed7d09d604c00b7eebac1488a6f4658788d6e33
du is used by build/make/tools/releasetools/build_image.py
paste is used by system/core/rootdir/update_and_install_ld_config.mk
xxd is used by device/generic/goldfish/tools/mk_qemu_image.sh
Test: none
Change-Id: I707668578d1babe1b149c2c3f989e5f8be37d897
Buildbot parses failure messages that look like ninja errors, but this
error doesn't look like a ninja error. We used to output this via
stderr, which buildbot would fall back to, but that had synchronization
issues between stderr and stdout.
So switch this over to use the status API, marking this action as failed
when necessary, which will display a ninja-like error that buildbot can
parse.
Bug: 112007097
Test: add dangling dep, look at output.
Change-Id: Ib2bbba4373ea22ac01af8aa6bbf295296a93e94d
--warn_real_to_phony will print a warning any time that a non-PHONY rule
depends on a PHONY rule. Since PHONY rules are always considered dirty
this would force rebuilds on every incremental build. Real rules should
always depend on rules that have real outputs.
--warn_phony_looks_real will print a warning any time that a PHONY rule
has an output that looks like a file (has as / in it). This is can cause
confusion for users if it's not a real file, and it shouldn't be marked
as PHONY if it will be run every time (since that will slow down
incremental builds).
Test: run build_test on downstream branches, see few warnings
Change-Id: Id135c85c991976be33cd48a34eba0caaff1eda2c
All of the existing devices that have problems with this are setting
BUILD_BROKEN_DUP_RULES to true. Switch the behavior to default this to
an error.
Bug: 77611511
Test: check logs of all downstream build_test targets
Test: attempt to add an overriden command to hikey960 (uses default)
Change-Id: Ia83089f035925cf0c9883a1f593ae2da89fee568
Ninja now knows how to write directly to a file (or in our case, a named
pipe). This works around an issue we were seeing on Mac, where Go would
just hang after 50-2000 proto messages. It's also just a simpler
solution.
Bug: 111544015
Test: `m` with updated ninja on both Linux & Mac
Change-Id: Ic91920d83a6d2ea0b79e82b467e2423d78189f12
I've noticed a few instances of interleaved status messages in between
lines in a terminal/Writer.Print call on our build servers. Since
there's a lock protecting everything we write, I've got to assume this
is a stdout vs stderr problem. Ninja had always been outputing to
stdout, except for error messages, which are now marked with FAILED:
like failed actions.
Test: m blueprint_tools
Test: m missing
Change-Id: Idf8320d40694abf212c902c63a9703e4440ffb7a
The extra `defer os.Remove(fifo)` was sometimes racing with the next
instance of ninja, removing the file in between when it was re-created
and used. Since we're always removing the file before creating it, it's
safe to just remove that.
The error message for this failure wasn't all that good either, so move
so use the status Print/Error calls inside the goroutine instead of the
logging ones.
Test: `build/soong/build_test.bash -only-soong` repeatedly
Change-Id: Icfeb6b68802093bd3a07d3e46046ef7d1a89d4a1
This adds a new status package that merges the running of "actions"
(ninja calls them edges) of multiple tools into one view of the current
state, and gives that to a number of different outputs.
For inputs:
Kati's output parser has been rewritten (and moved) to map onto the
StartAction/FinishAction API. A byproduct of this is that the build
servers should be able to extract errors from Kati better, since they
look like the errors that Ninja used to write.
Ninja is no longer directly connected to the terminal, but its output is
read via the protobuf frontend API, so it's just another tool whose
output becomes merged together.
multiproduct_kati loses its custom status routines, and uses the common
one instead.
For outputs:
The primary output is the ui/terminal.Status type, which along with
ui/terminal.Writer now controls everything about the terminal output.
Today, this doesn't really change any behaviors, but having all terminal
output going through here allows a more complicated (multi-line / full
window) status display in the future.
The tracer acts as an output of the status package, tracing all the
action start / finish events. This replaces reading the .ninja_log file,
so it now properly handles multiple output files from a single action.
A new rotated log file (out/error.log, or out/dist/logs/error.log) just
contains a description of all of the errors that happened during the
current build.
Another new compressed and rotated log file (out/verbose.log.gz, or
out/dist/logs/verbose.log.gz) contains the full verbose (showcommands)
log of every execution run by the build. Since this is now written on
every build, the showcommands argument is now ignored -- if you want to
get the commands run, look at the log file after the build.
Test: m
Test: <built-in tests>
Test: NINJA_ARGS="-t list" m
Test: check the build.trace.gz
Test: check the new log files
Change-Id: If1d8994890d43ef68f65aa10ddd8e6e06dc7013a
Using ANDROID_BUILD_TOP and other envsetup.sh varialbes is already
forbidden in makefiles, but some tools run by the build (like
hidl-gen) will use it if it is set. Since they are set by envsetup.sh,
and sourcing envsetup.sh before building is optional, unset them inside
the build to get consistent behavior.
Bug: 79250545
Test: m checkbuild
Change-Id: I3e7d9f01390ccdc34c49115f2f15cd7542d9974b
Make the dangling rules list sorted and unique in order to avoid
very long lists when a dangling rule is referenced many times.
Also prettify the output by indenting the list and printing
"stopping" instead of a blank line for the fatal.
Test: m checkbuild
Change-Id: I8f7c27ae39b59f506b529d9995d90b0d6b9835d1
This matches the change to blueprint's blueprint_impl.bash:
https://github.com/google/blueprint/pull/213
It allows us to run bpglob during the two bootstrap phases, so we're
never running minibp or the primary builder in a later phase than
normal.
Also removes the dependency on out/soong/build.ninja from the main
phase, since it's never generated by the main phase anymore, and is
tripping the dangling dependency checks.
Bug: 73646380
Test: m nothing
Test: rm frameworks/base/core/java/android/content/pm/dex/ArtManager.java
Test: m nothing (soong re-runs)
Change-Id: Ia3952d015ad6091ad5a841f555acda78c9390e84
There are no users of these, other than the PDK, which is already
exempted.
Test: m nothing
Test: Check for warnings in downstream build_test logs
Test: Check for warnings in last two days of continuous build logs
Change-Id: I7c7d0068e928e591355c39d22635af86b8f43f1b
Remove support for compiling with javac from OpenJDK8.
We still target 1.8 by default, and OpenJDK8 prebuilts are still
required for the bootclasspath and running robolectric.
Bug: 38418220
Test: m java
Change-Id: I5686deb0ae4f9927192a039d08adc0117b2605dd
This is useful when determining which devices are setting which values
from the logs, cross-referencing where warnings still occur.
Test: look at out/soong.log
Change-Id: I3e3f0e4c8cbd9ce621b03f5163eff2cccfa8c424
Use the OpenJDK9 javac to run ErrorProne by adding it to
-processorpath and using -Xplugin:ErrorProne.
Bug: 69485063
Test: m RUN_ERROR_PRONE=true javac-check
Change-Id: I0496006b71b70766ef16d57753cbcf037897799c
We've turned off suffix rules with .SUFFIXES, but make them explicit
errors.
Test: build_test on downstream branches
Change-Id: I682ee3eb3a8f1451cd7e16f1e1d59afca5d21a26
pkill and sleep are used in
device/generic/goldfish/tools/emulator_boot_test.sh
pgrep is used by GOMA
bc and pwd are used by kernel builds
Test: none
Change-Id: If7004255b4776fa4409fdd4b0aa6d6617b471317
atree was using the host's strip tool during sdk generation, so expose
the prebuilt for use instead.
`id` was used with a test running minijail0 during the build
`sha1sum` is used by development/build/tools/mk_sdk_repo_xml.sh
`rmdir` was being used to remove some tmp files
Test: m PRODUCT-sdk_arm64-sdk dist sdk_repo
Test: Run other builds through forrest
Change-Id: I49c756a3e9497c0bbd9030754e6aa193a2d989a4
This reverts commit 09f4540d66.
Fixes the raw call to net.Listen in the tests to go through the listen()
helper and use the long socket path fallbacks.
Removes the use of timeouts from the tests -- the behaviors being tested
did not rely on timeouts, so removing them will reduce the flakiness if
the build is heavily loading the machine at the same time the test is
running.
Also fixes some potential nil pointer dereferences.
Test: OUT_DIR=<really long> m blueprint_tools
Test: `while .../soong-ui-build-paths/test/test; do sleep 0.01; done` with a build running
Change-Id: I16d44be7517bc415f1c808284088f4ba40df3bfa
This reverts commit c59a92cb1a.
Reason for revert: tests are broken with long OUT_DIRs
They're directly calling net.Listen, and not using the fallback
for long socket names.
Change-Id: Id14cbd499fd9b36c6926b7552d3554340cb0916c
This reverts commit 96c957ae20.
Fixes issues on some machines where the socket in TMPDIR ended up with a
unix domain socket pathname over 107 characters long, which Go will
reject due to underlying limitations in the system calls. If this
happens, we'll fall back to opening the directory, then using
/proc/self/fd/#/<file>, or manually creating a similar symlink in /tmp.
Also fixes some issues on Mac where os.Executable returns the symlink
instead of the underlying file, sending a message over a unix domain
socket will block if the reader isn't reading, and sandboxing was
preventing us from running `ps`.
Test: m blueprint_tools
Test: m blueprint_tools on mac
Change-Id: Ib19ccfe10cb0a79f1476fb1d5cd20ed0495be367
This will allow us to track (and eventually limit) the commands that the
build references via $PATH. These are mostly implicit dependencies on
the host system -- for Linux, we assume something similar to Ubuntu
14.04 with a few extra packages, but this will let us better define
that.
This will not catch uses of tools with absolute paths (/bin/bash, etc),
but most uses shouldn't be relying on absolute path names anyways.
Adds ~400ms on the first startup, ~140ms on subsequent runs, and
overhead of a few ms for every forwarded execution.
Test: m
Test: build/soong/build_test.bash
Test: Add `gcc --version`, TEMPORARY_DISABLE_PATH_RESTRICTIONS=true m
Change-Id: Id68cbb1c8ceef65bbbb10751e83722c7662d2351
Some tests make their temporary directories readonly. If they fail or
crash before cleaning up, they could leave these readonly directories
behind with files in them. os.RemoveAll fails with an error in this
case, and we can't start the build until they're removed.
Test: `m blueprint_tools` to run the new go tests
Change-Id: I761f96579e96167ebfd98c6cca59765bd50536ec
So that we can see where the time is going. Also removes the obsolete
ASAN_OPTIONS, which moved to kati, but I forgot to update this instance.
Test: m nothing; check out/soong.log
Change-Id: I0c4066bad20fc2dc22b389f4c973d10dca554ee3
The find commands used to locate the BoardConfig.mk can't be optimized
by Kati, so we're currently spending ~125ms three times during every
build (dumpvars, cleanspec, and the main kati run). Preserve the value
of TARGET_DEVICE_DIR from the dumpvars run so that we only need to run
the find commands once.
Bug: 78020936
Test: out/build-taimen.ninja is identical
Test: out/soong.log shows that we're not running these finds again
Change-Id: Iee56b454c3661de2b58c161169218ecaf2135398
