When running ninja stream stdout to the status writer. This improves
the ninja -d explain behavior, and will also allow Soong to be put
into the ninja console pool letting it print timely output to the
console.
Bug: 80165685
Test: NINJA_ARGS="-d explain" m
Change-Id: I40f03fc01d837ad91d400eae378007a7ce807705
The build directory passed in to soong_ui can contain a
symbolic link. soong_ui was not evaluating the build
directory to retrieve the true path, hence failing to
execute the internal soong.
Fixes: b/135995632
Test: Unit test case and executed the scenario in the bug
description.
Change-Id: I5779c6aa3f3183810437dbe2b2d4e40acbafb205
There was one case that did not work:
1) Create dir in root source tree
2) Run "m libc"
The build would fail as it was using the mma logic. A separate
one named BUILD_MODULES was added to allow building specific
modules in any directories using the "m" command.
Bug: b/130049705
Test: unit test cases, ran "m libc" in a tmp directory inside
of the source tree.
Change-Id: I8d23e685a673a311001ee8edd89bd73b662392dd
Everything should be using the prebuilt now.
Bug: 117561006
Test: treehugger
Test: check bass logs for m4 uses
Change-Id: Ic69d715d72d64bb0082e2f2c247113155d8b3de0
The metrics.proto needs to be imported to Google3 in order to unmarshal
the build metrics data from bigstore. Cleaned up the enum names to use
the Camel naming convention and renamed the build_metrics to
soong_metrics to be more specific to soong and to allow quering
soong_metrics files from bigstore.
Bug: b/135280521
Test: lunch andf m -j. Checked if soong_metrics was generated correctly
by using printproto command.
Change-Id: I998c8d05db592e94a653d6ca32250b80df3c9b21
Add the following build actions {BUILD_MODULES_IN_A_DIRECTORY,
BUILD_MODULES_IN_DIRECTORIES} in soong_ui config so the bash code version of
build commands (m, mm, mma, mmm, mmma) in build/make/envsetup.sh can be deprecated.
This is to allow up to date bug fixes on the build commands.
Bug: b/130049705
Test: Unit test cases
Change-Id: I772db1d4e9c1da5273374d1994eb5e8f17cd52f2
terminal.Writer is now just a wrapper around stdio.Stdout() without
any useful functionality. Replace it with stdio.Stdout() as an
io.Writer.
Test: ui/terminal/status_test.go
Change-Id: I5bc5476afdca950b505642f0135a3af9d37fbe24
All users have been removed.
Test: treehugger
Change-Id: I35e55f3ce79c8e68585be120b5dec810563c7a83
Merged-In: I35e55f3ce79c8e68585be120b5dec810563c7a83
- Add SOONG_LOCK_TIMEOUT environmental variable for users to set a
duration (e.g. 1h), which defaults to the current 10 seconds.
- Instead of logging once per second waiting to acquire the lock,
only log once before the first wait
Fixes: 131854539
Test: make; open another tab and make
Change-Id: I0a02a1b071964f74473a5f55034f916526e1b309
Allow install-clean as an alias for installclean and data-clean as
an alias for dataclean.
Test: m install-clean
Change-Id: I9c97d60572a524ad68caecd26d52bfb987468075
This reverts commit e114b3f0a9.
Build broke as follows:
out/soong/host/linux-x86/bin/sbox --sandbox-path out/soong/.temp --output-root out/soong/.intermediates/system/update_engine/ue_unittest_disk_imgs/gen -c 'tar -jxf system/update_engine/sample_images/sample_images.tar.bz2 -C __SBOX_OUT_DIR__/gen disk_ext2_1k.img disk_ext2_4k.img disk_ext2_4k_empty.img disk_ext2_unittest.img' __SBOX_OUT_DIR__/gen/disk_ext2_1k.img __SBOX_OUT_DIR__/gen/disk_ext2_4k.img __SBOX_OUT_DIR__/gen/disk_ext2_4k_empty.img __SBOX_OUT_DIR__/gen/disk_ext2_unittest.img
"bzip2" is not allowed to be used. See https://android.googlesource.com/platform/build/+/master/Changes.md#PATH_Tools for more information.
tar: Child returned status 1
tar: Error is not recoverable: exiting now
sbox command (tar -jxf system/update_engine/sample_images/sample_images.tar.bz2 -C out/soong/.temp/sbox849645882/gen disk_ext2_1k.img disk_ext2_4k.img disk_ext2_4k_empty.img disk_ext2_unittest.img) failed with err "exit status 2"
Change-Id: I3fe7100886ea3b748f5a6b5fc6d2811391bb72a8
There are no remaining users.
Also mark BUILD_BROKEN_ANDROIDMK_EXPORTS and BUILD_BROKEN_ENG_DEBUG_TAGS
as deprecated in scripts/build_broken_logs.go
Test: treehugger
Change-Id: If7892bef1b9001f12a99565f886b395cf1985e70
So that the output of build_test can be parsed to understand which
devices need changes.
Test: read out/soong.log
Change-Id: I40b25527ef72e3f646c0cfe62114b452fa3f27f6
Some people apparently still talk to the network during their build.
Allow this temporarily with a BUILD_BROKEN_USES_NETWORK check.
Bug: 129992021
Test: attempt to talk to the network during the build with and without
this flag.
Change-Id: Ifb967c656aa24c4599e7232d0f1b5a303b5bac52
Wait for the ninja proto processing goroutine to notice the fifo
has closed and exit before continuing.
Test: m nothing
Change-Id: I8cf5f3b8bf6a91496c6d2bbbd3e811eb7f0c9d21
Dump variable in log to track remaining targets which don't set it.
Bug: 122954981
Test: $ grep VNDK out/soong.log
2019/03/07 09:22:04.235604 build/soong/ui/build/dumpvars.go:109: BOARD_VNDK_VERSION current
Change-Id: I34146c752a4bf0d799374bb93cc0d75b0f1b048c
There's more than just toybox in this group now, so let's rename our
variables to something closer to the desired behavior, rather than the
first user.
Test: treehugger
Change-Id: I76d4407792061c8110b194cfe73f1ddc84dbc22f
For ease of investigating the issue with error reports, let me make
the error message more verbose.
Test: make goma_ctl.py raise Exception and run.
Test: I confirmed verbose error message shown.
Change-Id: I04f9a5ca4808746994c41ddce8a5b22a1975fcc2
Signed-off-by: Yoshisato Yanagisawa <yyanagisawa@google.com>
Bug: 124218940
If ANDROID_QUIET_BUILD environment variable is set to '1', do not show 20
lines of the environment variables. In addition, for the failing step show
only the step name and output, omitting the command proper (the verbose log
as well as error.log still contains it).
If build succeeds, the output of the build itself consists of a single
line:
```
> m androidmk
[100% NN/NN] <last command>
```
When it fails, the output does not contain sometimes very long command
line:
```
> m androidmk
[ 97% NN/MM] test androidmk
FAILED: <step>
--- FAIL: TestEndToEnd (0.01s)
androidmk_test.go:1025: failed testcase 'prebuilt_etc_TARGET_OUT_ETC'
input:
include $(CLEAR_VARS)
LOCAL_MODULE := etc.test1
LOCAL_MODULE_CLASS := ETC
LOCAL_MODULE_PATH := $(TARGET_OUT_ETC)/foo/bar
include $(BUILD_PREBUILT)
expected:
prebuilt_etc {
name: "etc.test1",
filename: "foo/bar",
}
got:
prebuilt_etc {
name: "etc.test1",
filename: "foo/bar",
}
FAIL
17:50:53 ninja failed with: exit status 1
```
[The related change in build/make/envsetup.sh suppresses timing display
when the same variable is set.]
Change-Id: I4d3c72457de031ff58a324c2fe98f4c1d10f8239
Test: treehugger
And add a helper script that can help parse the output from
build_test.bash
Test: check for BUILD_BROKEN_ENG_DEBUG_TAGS in soong.log
Test: go run ../build/soong/scripts/build_broken_logs.go *
Change-Id: Idd0fc8b59770dcdbe44eeba262558708a9497f96
We don't want these to be warnings, may as well be errors instead of
turning off the warnings.
Bug: 123583617
Test: treehugger
Change-Id: I6ca518c9647e712426952cb88bdb044d933b23f2
As more phony rules move either to the packaging kati run, or soong, the
existing real-to-phony check failed to notice that they were phony
rules.
So I added --top_level_phony to Make that assumes that all leaf nodes
(source files, or nodes generated by another ninja file generator) that
do not have a '/' in them are phony targets.
The existing phony-looks-real check in Kati should prevent these from
being real generated files, and our source tree shouldn't have any
source files used at the root level.
Test: build_test on downstream branches
Change-Id: I77cff84b536c6cd6402307a9aa5a9c273e72c71f
We've got an internal testcase that uses minijail0 inside an Android.mk
rule. That was failing since we turned on the linux sandbox, as /proc
was mounted read-only, which prevented setting up the uid/gid mappings
for a child namespace.
Fixes: 122985455
Test: treehugger & forrest of breaking build
Change-Id: Ia77a91a7f4eeeb8a24e84075d8272287f5087587
This starts a new network namespace without any connections to the
outside.
Bug: 122270019
Test: USE_GOMA=true m libc
Test: treehugger
Test: add rule to use /usr/bin/wget, fails after this change
Change-Id: Iba262025ce0e4e3bef5c34c817cc678d6c61403b
This really only initializes the sandbox, it does not attempt to change
the view of the filesystem, nor does it turn off networking.
Bug: 122270019
Test: m
Test: trigger nsjail check failure; lunch; m; cat out/soong.log
Test: USE_GOMA=true m libc
Change-Id: Ib291072dcee8247c7a15f5b6831295ead6e4fc22
As suggested in
https://android-review.googlesource.com/c/platform/build/soong/+/839293
I am moving some features in goma.mk to goma.go in Soong UI.
With this CL, let me implement ulimit check to Soong UI.
Test: export USE_GOMA=true
Test: launch aosp_arm-eng
Test: make
Test: the command succeeds if "ulimit -n" and "ulimit -u" are large enough.
Test: Otherwise, it shows error. I confirmed both cases.
Change-Id: I5d7d5ed71f620302a0d635770d1a51a2baab51fd
Signed-off-by: Yoshisato Yanagisawa <yyanagisawa@google.com>
As suggested in b/118390303#comment18, let me run
"goma_ctl.py ensure_start" in soong UI.
Bug: 118390303
Test: stop compiler_proxy, and execute following command:
Test: USE_GOMA=true lunch aosp_arm-eng
Test: and confirms compiler_proxy is running.
Change-Id: I859daae6ae2399c5b6bce2fafd874dfdb7c6aae0
Signed-off-by: Yoshisato Yanagisawa <yyanagisawa@google.com>
As part of a future change to sandbox the build on Linux, the real
username will be switching to "nobody", and the hostname will be
switching to "android-build".
The USER environment variable will reflect the sandboxed value, so for
the build properties that want the external USER, they'll need to use
BUILD_USERNAME.
Similarly, BUILD_HOSTNAME will reflect the real value, while the
`hostname` tool will return "android-build"
Bug: 122270019
Test: check build.prop
Change-Id: I99604b9488732a63690b256dc4dd7894d369a32c
So that ninja produces an error instead of just a warning when a dep
file is not produced.
Bug: 121058584
Test: check build logs for "depfile is missing" warning
Test: treehugger
Change-Id: I1cbaba866eaf293495c3c0b2b174190bcb2b0f9a
Test: Dumped the text formated based metrics file to out dir,
and checked the file.
Bug: b/63815990
Change-Id: Iff476f72a0be74eb53b6b26ef468d11c0f24a404
This way we don't appear hung at:
No need to regenerate ninja file
Change-Id: I8dbdaa2c1b1c5a6a73187d0e6061f363b62e10c9
Fixes: 122251150
Test: m nothing
Prepend the toybox symlink directory to $PATH, and stop generating
symlinks into out/.path for tools used from toybox.
We don't (yet?) have toybox prebuilts for darwin, so apply the above only to Linux, and preserve existing behavior on Darwin.
Test: check the build banner for uname results
Test: m
Change-Id: I37fc380381e65a628cdc131d462fd4441eacfe9d
Stop writing out ninja files, as they become very large, especially when
multiplied by the number of defined products.
Test: treehugger
Change-Id: Id9529d14040acb72a0188e58b5db2911f142071e
DIST_DIR is controlled by soong_ui, and the make dumpvars code no longer
has access to it. So instead, handle some dumpvar requests inside
soong_ui itself.
Test: get_build_var DIST_DIR
Test: get_build_var OUT_DIR
Test: get_build_var BUILD_SYSTEM
Test: build/soong/soong_ui.bash --dumpvars-mode --vars="DIST_DIR
BUILD_SYSTEM"
Change-Id: Id3bcb8b0748db67c83d0e42d1ae5be564c5eb2f7
We're only using it to distribute files in case of failure, which isn't
well supported currently, but can be handled for now by using the
DIST_DIR environment variable during the command execution.
This was at least one cause that we'd be re-running Soong during every
build server build, as the DIST_DIR values are unique.
Test: m dist
Change-Id: Ibd5e6b6c46695350de80b745bfb6a6aa685033a0
We've got a prebuilt of make in prebuilts/build-tools, and we build
xmllint from source.
Test: No logs on the build servers in the last 24 hours
Change-Id: Ib60a7b5feaa31097071b67ebf6beb7f0b9a9f81b
The idea is that we'd move the installation and packaging tasks over to
it, using data from Soong & the Kati reading Android.mk files.
This would allow us to make more fundamental changes about how we
package things without having to adjust makefiles throughout the tree.
Possible use cases:
* Moving some information from Soong's Android.mk output to a file read
by the packaging step may allow us to read the Android.mk files less
often, speeding up builds.
* Refactoring our current two-stage ASAN builds to run the Kati build
step twice, writing into different object directories, then have a
single packaging step that reads both outputs. Soong already has the
capability of writing out a single ninja file with all the asan
combinations.
* Running two build steps, one building the system-related modules
using a "generic" device configuration, and one building the vendor
modules using a specific device configuration. This could enforce a
GSI/mainline system vs vendor split in a single build invocation.
* If all installation is through this tool, it will be much easier to
track what should no longer be installed on an incremental build,
reducing the need for installclean.
* Changing PRODUCT_PACKAGES should be a much faster operation, which
means we could keep track of local additions to the images. Then
`mma` would be more persistent, instead of installing something once,
then never updating it again.
Eventually we plan on switching from Kati to something Go-based, but
this is a more incremental approach while we clean up everything else.
Currently, this just moves the dist-for-goal handling over to the
packaging step, so that we don't need to read Android.mk files when
DIST_DIR changes, or we switch between dist vs not.
Bug: 116968624
Bug: 117463001
Test: m nothing
Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
Set the default value if one isn't provided.
Test: `get_build_var DIST_DIR` after removing the default setting in envsetup.mk
Change-Id: I0cb310fc65f7747c36de14608b61786ef6863fd1
This is no longer used within Android builds. We don't have any way of
providing extra libraries, or using a hermetic version of perl.
Bug: 117338567
Test: check build logs for the last 48 hours
Change-Id: I06d41a0006a47508a435dbf44bfccc4b385d9f41
Now that we've got prebuilts of make, turn on logging so that
we can find all the users and convert them to the prebuilt.
We also build xmllint as part of the tree, so log uses of that
too.
Test: treehugger
Change-Id: I8a6a5d481dfc709dc0eb3424c51430bc54763c1e
Make the cleanspec & Android.mk runs of kati use much of the same code
and arguments.
Also renames 'Kati' to 'KatiBuild' in many cases, in the the expectation
that we'll have a 'KatiPackage' step in the future.
Use --no_ninja_prelude and move local_pool & _kati_always_build_ into
the combined ninja file. This will reduce the need to re-read makefiles
when Goma is enabled, and it allows us to include more than one
Kati-generated ninja file in the build graph.
Bug: 116968624
Test: build_test on downstream branches
Change-Id: Ibdac689b81f62dc293647fad917d84946f2c3cfa
Check for the presence of Android.mk or CleanSpec.mk, which
are somewhat common problems.
Bug: 113147143
Test: m (with and without files present)
Change-Id: I31cf60c325e7f6c6fce7aec54712c1cb802055c2
All downstream users either mark BUILD_BROKEN_PHONY_TARGETS=true
explicitly, or don't have any warnings.
Test: build_test on downstream branches
Change-Id: I6e9d7ed42da941148827854851657895c2e98649
Build target owners will create a zip of all OWNERS
files in source code based on the cached OWNERS.list
Bug: 114242886
Test: make dist -j owners
Change-Id: I97fa3737e1a2a240a08e072dfb2905c11f28705b
These warnings are usually hit in one of the two conditions:
1. Targets that should be marked .PHONY, but are not.
2. Attempts to define output files in the source tree.
See https://android.googlesource.com/platform/build/+/master/Changes.md#phony_targets
for more information.
Test: multiproduct_kati in all downstream branches
Change-Id: Ibe37cce320612825c22987e94d0740e8c0565b6f
od is used by ninja's inline.sh tool to encode a python script in a C
header.
Test: forrest run of build-tools branch
Change-Id: I87001a02b168ebe183b8b7dc495c67bb43f1b86d
Anything that isn't explicitly marked as Allowed in this list will now
return an error (and log a message) when it is executed.
Test: search all logs from the build server over the last day
Change-Id: I3ceacd9a140097809dde81a8d8979dd2c45f234c
unix2dos is used instead of todos by some build servers in
development/build/tools/patch_windows_sdk.sh
Test: none
Change-Id: I04cb2258f71bebfc8792b7aa95643a45ffe3ec5c
todos is used by development/build/tools/patch_windows_sdk.sh
Test: m win_sdk; check out/soong.log
Change-Id: I5e1a290838cc34261d7f5fc321afe79a8b1a6fbb
These still default to warnings, but if a board decides to set
BOARD_BROKEN_PHONY_TARGETS := false, they'll turn into errors. More
likely I'll just be marking the broken targets as broken, then switching
the logic here to be like the dup rules warnings/errors.
Test: On a build with warnings, try <missing>, "false", and "true"
Change-Id: I041e1ff4618c4114ec43015b6c0ae1b49b36b6f8
Attempting to reduce the number of different spellings we have for
"product services" partition in the codebase.
Bug: 112431447
Test: m
Change-Id: I0a393a1d625e7ea3217d28735a4db709bce32395
Merged-In: I0a393a1d625e7ea3217d28735a4db709bce32395
Setting this will cause our invocation of ninja to fail. Even if we
fixed that, it's likely to run into other differences during the build.
Fixes: 112443729
Test: POSIXLY_CORRECT=1 m nothing
Change-Id: I0315955c9fd9536cc6cb0d7c700eed207f35cc80
lsof is used by goma to find its proxy daemon
ps is used on Mac instead of pstree in soong_ui
Test: none
Change-Id: Ifed7d09d604c00b7eebac1488a6f4658788d6e33
du is used by build/make/tools/releasetools/build_image.py
paste is used by system/core/rootdir/update_and_install_ld_config.mk
xxd is used by device/generic/goldfish/tools/mk_qemu_image.sh
Test: none
Change-Id: I707668578d1babe1b149c2c3f989e5f8be37d897
Buildbot parses failure messages that look like ninja errors, but this
error doesn't look like a ninja error. We used to output this via
stderr, which buildbot would fall back to, but that had synchronization
issues between stderr and stdout.
So switch this over to use the status API, marking this action as failed
when necessary, which will display a ninja-like error that buildbot can
parse.
Bug: 112007097
Test: add dangling dep, look at output.
Change-Id: Ib2bbba4373ea22ac01af8aa6bbf295296a93e94d
--warn_real_to_phony will print a warning any time that a non-PHONY rule
depends on a PHONY rule. Since PHONY rules are always considered dirty
this would force rebuilds on every incremental build. Real rules should
always depend on rules that have real outputs.
--warn_phony_looks_real will print a warning any time that a PHONY rule
has an output that looks like a file (has as / in it). This is can cause
confusion for users if it's not a real file, and it shouldn't be marked
as PHONY if it will be run every time (since that will slow down
incremental builds).
Test: run build_test on downstream branches, see few warnings
Change-Id: Id135c85c991976be33cd48a34eba0caaff1eda2c
All of the existing devices that have problems with this are setting
BUILD_BROKEN_DUP_RULES to true. Switch the behavior to default this to
an error.
Bug: 77611511
Test: check logs of all downstream build_test targets
Test: attempt to add an overriden command to hikey960 (uses default)
Change-Id: Ia83089f035925cf0c9883a1f593ae2da89fee568
Ninja now knows how to write directly to a file (or in our case, a named
pipe). This works around an issue we were seeing on Mac, where Go would
just hang after 50-2000 proto messages. It's also just a simpler
solution.
Bug: 111544015
Test: `m` with updated ninja on both Linux & Mac
Change-Id: Ic91920d83a6d2ea0b79e82b467e2423d78189f12
This adds a new status package that merges the running of "actions"
(ninja calls them edges) of multiple tools into one view of the current
state, and gives that to a number of different outputs.
For inputs:
Kati's output parser has been rewritten (and moved) to map onto the
StartAction/FinishAction API. A byproduct of this is that the build
servers should be able to extract errors from Kati better, since they
look like the errors that Ninja used to write.
Ninja is no longer directly connected to the terminal, but its output is
read via the protobuf frontend API, so it's just another tool whose
output becomes merged together.
multiproduct_kati loses its custom status routines, and uses the common
one instead.
For outputs:
The primary output is the ui/terminal.Status type, which along with
ui/terminal.Writer now controls everything about the terminal output.
Today, this doesn't really change any behaviors, but having all terminal
output going through here allows a more complicated (multi-line / full
window) status display in the future.
The tracer acts as an output of the status package, tracing all the
action start / finish events. This replaces reading the .ninja_log file,
so it now properly handles multiple output files from a single action.
A new rotated log file (out/error.log, or out/dist/logs/error.log) just
contains a description of all of the errors that happened during the
current build.
Another new compressed and rotated log file (out/verbose.log.gz, or
out/dist/logs/verbose.log.gz) contains the full verbose (showcommands)
log of every execution run by the build. Since this is now written on
every build, the showcommands argument is now ignored -- if you want to
get the commands run, look at the log file after the build.
Test: m
Test: <built-in tests>
Test: NINJA_ARGS="-t list" m
Test: check the build.trace.gz
Test: check the new log files
Change-Id: If1d8994890d43ef68f65aa10ddd8e6e06dc7013a
Using ANDROID_BUILD_TOP and other envsetup.sh varialbes is already
forbidden in makefiles, but some tools run by the build (like
hidl-gen) will use it if it is set. Since they are set by envsetup.sh,
and sourcing envsetup.sh before building is optional, unset them inside
the build to get consistent behavior.
Bug: 79250545
Test: m checkbuild
Change-Id: I3e7d9f01390ccdc34c49115f2f15cd7542d9974b
Make the dangling rules list sorted and unique in order to avoid
very long lists when a dangling rule is referenced many times.
Also prettify the output by indenting the list and printing
"stopping" instead of a blank line for the fatal.
Test: m checkbuild
Change-Id: I8f7c27ae39b59f506b529d9995d90b0d6b9835d1
This matches the change to blueprint's blueprint_impl.bash:
https://github.com/google/blueprint/pull/213
It allows us to run bpglob during the two bootstrap phases, so we're
never running minibp or the primary builder in a later phase than
normal.
Also removes the dependency on out/soong/build.ninja from the main
phase, since it's never generated by the main phase anymore, and is
tripping the dangling dependency checks.
Bug: 73646380
Test: m nothing
Test: rm frameworks/base/core/java/android/content/pm/dex/ArtManager.java
Test: m nothing (soong re-runs)
Change-Id: Ia3952d015ad6091ad5a841f555acda78c9390e84
There are no users of these, other than the PDK, which is already
exempted.
Test: m nothing
Test: Check for warnings in downstream build_test logs
Test: Check for warnings in last two days of continuous build logs
Change-Id: I7c7d0068e928e591355c39d22635af86b8f43f1b
Remove support for compiling with javac from OpenJDK8.
We still target 1.8 by default, and OpenJDK8 prebuilts are still
required for the bootclasspath and running robolectric.
Bug: 38418220
Test: m java
Change-Id: I5686deb0ae4f9927192a039d08adc0117b2605dd
This is useful when determining which devices are setting which values
from the logs, cross-referencing where warnings still occur.
Test: look at out/soong.log
Change-Id: I3e3f0e4c8cbd9ce621b03f5163eff2cccfa8c424
Use the OpenJDK9 javac to run ErrorProne by adding it to
-processorpath and using -Xplugin:ErrorProne.
Bug: 69485063
Test: m RUN_ERROR_PRONE=true javac-check
Change-Id: I0496006b71b70766ef16d57753cbcf037897799c
We've turned off suffix rules with .SUFFIXES, but make them explicit
errors.
Test: build_test on downstream branches
Change-Id: I682ee3eb3a8f1451cd7e16f1e1d59afca5d21a26
pkill and sleep are used in
device/generic/goldfish/tools/emulator_boot_test.sh
pgrep is used by GOMA
bc and pwd are used by kernel builds
Test: none
Change-Id: If7004255b4776fa4409fdd4b0aa6d6617b471317
atree was using the host's strip tool during sdk generation, so expose
the prebuilt for use instead.
`id` was used with a test running minijail0 during the build
`sha1sum` is used by development/build/tools/mk_sdk_repo_xml.sh
`rmdir` was being used to remove some tmp files
Test: m PRODUCT-sdk_arm64-sdk dist sdk_repo
Test: Run other builds through forrest
Change-Id: I49c756a3e9497c0bbd9030754e6aa193a2d989a4
This reverts commit 09f4540d66.
Fixes the raw call to net.Listen in the tests to go through the listen()
helper and use the long socket path fallbacks.
Removes the use of timeouts from the tests -- the behaviors being tested
did not rely on timeouts, so removing them will reduce the flakiness if
the build is heavily loading the machine at the same time the test is
running.
Also fixes some potential nil pointer dereferences.
Test: OUT_DIR=<really long> m blueprint_tools
Test: `while .../soong-ui-build-paths/test/test; do sleep 0.01; done` with a build running
Change-Id: I16d44be7517bc415f1c808284088f4ba40df3bfa
This reverts commit c59a92cb1a.
Reason for revert: tests are broken with long OUT_DIRs
They're directly calling net.Listen, and not using the fallback
for long socket names.
Change-Id: Id14cbd499fd9b36c6926b7552d3554340cb0916c
This reverts commit 96c957ae20.
Fixes issues on some machines where the socket in TMPDIR ended up with a
unix domain socket pathname over 107 characters long, which Go will
reject due to underlying limitations in the system calls. If this
happens, we'll fall back to opening the directory, then using
/proc/self/fd/#/<file>, or manually creating a similar symlink in /tmp.
Also fixes some issues on Mac where os.Executable returns the symlink
instead of the underlying file, sending a message over a unix domain
socket will block if the reader isn't reading, and sandboxing was
preventing us from running `ps`.
Test: m blueprint_tools
Test: m blueprint_tools on mac
Change-Id: Ib19ccfe10cb0a79f1476fb1d5cd20ed0495be367
This will allow us to track (and eventually limit) the commands that the
build references via $PATH. These are mostly implicit dependencies on
the host system -- for Linux, we assume something similar to Ubuntu
14.04 with a few extra packages, but this will let us better define
that.
This will not catch uses of tools with absolute paths (/bin/bash, etc),
but most uses shouldn't be relying on absolute path names anyways.
Adds ~400ms on the first startup, ~140ms on subsequent runs, and
overhead of a few ms for every forwarded execution.
Test: m
Test: build/soong/build_test.bash
Test: Add `gcc --version`, TEMPORARY_DISABLE_PATH_RESTRICTIONS=true m
Change-Id: Id68cbb1c8ceef65bbbb10751e83722c7662d2351
Some tests make their temporary directories readonly. If they fail or
crash before cleaning up, they could leave these readonly directories
behind with files in them. os.RemoveAll fails with an error in this
case, and we can't start the build until they're removed.
Test: `m blueprint_tools` to run the new go tests
Change-Id: I761f96579e96167ebfd98c6cca59765bd50536ec
So that we can see where the time is going. Also removes the obsolete
ASAN_OPTIONS, which moved to kati, but I forgot to update this instance.
Test: m nothing; check out/soong.log
Change-Id: I0c4066bad20fc2dc22b389f4c973d10dca554ee3
The find commands used to locate the BoardConfig.mk can't be optimized
by Kati, so we're currently spending ~125ms three times during every
build (dumpvars, cleanspec, and the main kati run). Preserve the value
of TARGET_DEVICE_DIR from the dumpvars run so that we only need to run
the find commands once.
Bug: 78020936
Test: out/build-taimen.ninja is identical
Test: out/soong.log shows that we're not running these finds again
Change-Id: Iee56b454c3661de2b58c161169218ecaf2135398
I got a report of a user exporting CDPATH in their shell and causing
strange issues in the build. We should never need this value (and it
probably shouldn't be exported from the shell either -- that was their
workaround).
NDK_ROOT causes issues with Android.mk files thinking that they're
building with the ndk (like external/googletest/googletest/Android.mk).
Bug: 78933670
Test: CDPATH=. NDK_ROOT=test m; check out/soong.log
Change-Id: Icce43d7e31ed5e5e1fb7a4e37fd4dfbf421af4b1
Many board still have this problem, but if we can switch some over,
we'll prevent global problems and have the ability to clean boards up
one-by-one.
Bug: 77611511
Test: lunch aosp_arm-eng; m nothing
Test: lunch aosp_marlin-eng; m nothing
Test: build_test on all downstream branches
Change-Id: I78bee44adde2059d01188658b9050927748d2028
Some people have run into issues where /tmp is on a smaller partition
than their main source tree, and if it fills up (or only has a few GB
free), their builds can start failing.
There may also be a subset of people whose /tmp may be slower than their
out directory -- if they put their src and out trees on a SSD, but keep
/tmp on a spinning HDD.
Bug: 71755844
Test: check out/soong.log for TMPDIR
Test: do a build, watch out/soong/.temp
Change-Id: Ib749ec736ad0f6c8ce7453861a91b3e99ddbd92d
If Kati ever wrote a line over 64k characters, the scanner in soong_ui
would error, and Kati would hang trying to write into the pipe. Now if
the Scanner errors out, fall back to copying directly from the pipe to
the output.
Test: Add $(warning $(shell find frameworks)), does not hang
Change-Id: I86412ad2f53f2fe3cbda901ee673abb904d56d3c
Soong_UI will update timestamp to build_date.txt, and export variables
to kati/ninja.
Test: m -j32
Bug: b/70351683
Change-Id: I153897afdf2d3f39a32d757d4c3ae7515caea52d
We were previously setting GOROOT to "prebuilts/go/linux-x86" during the
ninja executions when we were running Soong. But we can also run Soong
during the main ninja execution, were GOROOT was unset. When the GOROOT
was unset, the default GOROOT in our Go installation is
"./prebuilts/go/linux-x86" (note the extra ./).
This would cause g.bootstrap.goRoot to change between some soong runs,
causing us to rebuild all go programs (and anything depending on them)
more often than necessary.
So instead, keep GOROOT undefined when running Soong. Everything that
matters is using runtime.GOROOT(), which will fall back to the default.
Continue setting $GOROOT for bootstrap.bash, otherwise it fails when
there is no system provided go binary. What we give bootstrap.bash
doesn't really matter, since we don't actually use the blueprint wrapper
in Android.
Test: m blueprint_tools; touch bionic/libc/tzcode/new.c;
m blueprint_tools <doesn't rebuild everything>
Change-Id: I82f30c7c3b5d25e5cbf28fe37a97fdb776c4a164
