This error message it currently being
logged to stdout which goes to build.log file . We want it to be logged to
build_error.log file instead so that downstream systems that need to run
based on the error message will continue to work.
Test: Tested by running `m toybox` and ensuring that FAILED: prefix is
at the beginning of the line.
Bug: b/177221705
Change-Id: Ie0c9acee2739acf149384e33804b5fca3273c24f
Therefore, no need to remove vendor-ramdisk-debug.cpio.lz4
for installclean as it will not be installed to $OUT dir.
Bug: None
Test: make vendorbootimage_debug
Change-Id: Ib7fb9931c55a9bb24e5ca8c4a80bd72f7233c57f
There are few build targets that don't generate an installed file list,
e.g. product-graph. Skip the old installed file cleanup step so that the
func doesn't complain about a lack of the list file.
Fixes: 168105598
Test: m product-graph
Change-Id: Ib7dce6b801979bb565b74d6355143bac23b84fe6
Bazel is executed several times during the execution of soong_build.
For each bazel execution, generate a profile and save under the
BAZEL_METRICS_DIR which is defined in soong_ui.
Bug: b/174479924
Test: * USE_BAZEL_ANALYSIS=1 USE_BAZEL=1 m nothing and checked
if the cquery and graph build bazel profiles were generated.
* Verified that the generated bazel profiles were uploaded
to the local dev metrics pipeline.
Change-Id: I3d20204484dc6c5a1525a5d3eec1d62cfb33535b
Also get Bazel to write real files there (not symlinks) so that the DIST_DIR can be independent.
Test: Manually using e.g. DIST_DIR=/tmp/foo USE_BAZEL=1 m dist
Change-Id: I39d5219500864c9ecc85f356a028e9b5bf2607f4
The upload functionality now supports directories to be uploaded
for metrics purpose.
Bug: b/174479728
Test: Wrote unit test case.
Change-Id: I6906be4c1b7fd76ddf6ff7b94e48fe1c4209c59d
Code refactoring has been done for logs directory logic code since
the bazel metrics directory depends on the log directory. For builds
that did not specify a dist directory, the log directory is under
out directory. With dist, the logs directory is under <dist dir>/logs.
This matters for Android CI builds where the metrics files are
placed under logs directory. With this change, the bazel metrics
directory and corresponding files will be under <dist dir>/logs
directory for Android CI builds.
Bug: b/174479728
Test: * USE_BAZEL=1 m nothing (bazel_metrics dir in out dir)
* m nothing (bazel_metrics dir deleted)
* DIST_DIR=/tmp/build USE_BAZEL=1 m nothing dist (bazel_metrics
is in /tmp/build/logs directory)
Change-Id: Ic9e1ff49a1964fcaaf801bde2c19f33597ca1db4
The existing --skip-make flag disables both the config step and the kati
step in the build. Add support for a --skip-kati flag that skips just
the kati step, and refactor things so that the logic is shared between
these two.
Bug: 174315599
Test: TARGET_PRODUCT=aosp_arm64 soong_ui --make-mode --skip-kati;
(verify soong.variables is regenerated)
Change-Id: I75b1910fc1c12fcda130e37b7bc4c050131c7b33
On top of improving the comments, I also refactored the status checker
to simplify and clarify its use case to be a ninja stuckness checker,
since it doesn't appear to have other purposes.
Test: TH presubmit
Bug: b/173474588
Change-Id: I2cf51a1ebf16071a24a1c13c06c7b1adf60256de
This CL also removes --warn_real_to_phony and --warn_phony_looks_real,
since --werror_real_to_phony and --wrror_phony_looks_real are already
specified.
Test: treehugger
Bug: 173474588
Change-Id: Icbb0f53182d1b912c62f8044f52bb79d985bf868
This removes the need to source bazelenv.sh for USE_BAZEL_ANALYSIS, and
unifies mixed builds to use the checked in tools/bazel and bazelrc.
It also unifies all bazel-related output to be in out/bazel.
Without aosp/1502095, this change still requires toplevel_output_directories to be an empty
list, otherwise there'll be this error:
ERROR: Directories specified with toplevel_output_directories should be
ignored and can not be used as sources.
Test: With aosp/1441774: rm -rf out/ && lunch aosp_cf_x86_auto && USE_BAZEL_ANALYSIS=1 m libc && prebuilts/build-tools/linux-x86/bin/ninja -f out/combined-aosp_cf_x86_auto.ninja -t commands libc | grep bazel-out | wc -l # 2 results
Change-Id: I69b217ec88da531415792bb6e04b6a194ca4718d
This patch adds information behind the Bazel USE_BAZEL=1 execution
pipeline.
Test: m
Bug: 173474588
Change-Id: I54bdd26fb14c8a371de0922d193419112e8c3ece
Ensures that Bazel and Ninja are using the same binaries from PATH, such
as `sort` from the linked bug.
Test: USE_BAZEL=1 m
Test: rm out/soong/apex/depsinfo/new-allowed-deps.txt && USE_BAZEL=1 m
Bug: 161074854
Fixes: 161074854
Change-Id: Icc3e717e6d3cfcecfe90f0fb3a94a86485a11918
Added a tracer named "bazel" in runBazel function to capture topline
metrics when running bazel.
Bug: b/173028918
Test: * USE_BAZEL=1 m nothing and checked soong_metrics file using
printproto command to verify that bazel topline metrics
were captured.
Change-Id: I5cb0728a4da48545dc701cc79627c318d5d9a29d
Generate the bazel profile file that contains the metrics of
executing the ninja build phase of Soong.
Bug: b/173028918
Test: * m nothing and check out dir that there is no bazel_metrics dir.
* USE_BAZEL=1 m nothing and checked
out/bazel_metrics/build_bazel_profile.gz existed.
* m nothing and checked out/bazel_metrics does not exist.
Change-Id: I797d1eecf5c2c260ade8b55dc69c237bfd0fcbd4
$OUT/bazel_metrics directory is created in order to dump the
set of bazel profile files being created from a single build
execution. The directory is created from soong_ui.
Bug: b/173028918
Test: * m nothing and check out dir that there is no bazel_metrics dir.
* USE_BAZEL=1 and checked out/bazel_metrics was created.
* Ran m nothing and verified that out/bazel_metrics directory is
deleted to ensure no metrics is uploaded or collected from the
pipeline.
Change-Id: I1d3c2471065fae6931f35cef7f0f35fd84f76bef
The UseBazel() function will be used in several places to perform
different functionality such as setting up the metrics path.
Bug: b/173028918
Test: m nothing
Change-Id: Ie94073b8f0552ec7528e1cc2f3be74b693dc3135
This ensures that the lunch repo rule always reads the right Ninja file.
Test: USE_BAZEL=1 m
Test: m
Change-Id: I7f670dd538f46c4dae5ea537b857ff04520b60e7
This ensures that USE_BAZEL=1 mode has the same directory structure
under out/ as it does without USE_BAZEL=1.
Test: Manually verified building USE_BAZEL=1 aosp_flame with droid and
libc
Change-Id: I27e810772d1bf04bd7930ea122217b93d2fd48b4
Collect the resource information of each executed process in soong_ui
for metrics analysis.
Bug: b/169453825
Test: m nothing; "lunch 1" && m
Change-Id: I8c7fe019111921c4c59174d133b85439ec825885
This was caught while running the soong_build docs action under Bazel as
ninja executor. Since the config file is not an explicit input to the
action, it's not present in Bazel's execution root during action
execution time, leading to a missing file error. To fix it, add the file
as an input to the action, if it's configured in product variables.
New soong_build statement:
build out/soong/docs/soong_build.html: s.writedocs.soongDocs $
out/.module_paths/Android.bp.list out/soong/soong.variables $
out/soong/dexpreopt.config | out/soong/.bootstrap/bin/soong_build
outDir = out/soong/docs
Test: m
Test: USE_BAZEL=1 m
Bug: 161958189
Fixes: 161958189
Change-Id: I2bf264e848bb29ac91f4196fbe9c4ecf3cea8a2c
There is no need for multiproduct_kati to run the tests for
every product, they don't vary by product config. --skip-soong-tests
can also be used for local development to run soong_build even if
the tests don't pass.
Bug: 156428456
Test: m --skip-soong-tests nothing
Change-Id: I9c00e3d1b6e51d17bb290339c3f124d4d1c9e69f
Several structs and files are named time which records the total
execution of a traceable event. Soon, resource information of an
executed process is collected during a build event so a more generic
name is applied.
Bug: b/169453825
Test: go test and m nothing. Checked the soong_metrics protobuf.
Change-Id: I65a782603530d51018d6cc1192c18cfeef7566a0
Test:
m USE_GOMA=true GOMA_DIR=$(goma_ctl goma_dir) toybox
m FORCE_USE_GOMA=true GOMA_DIR=$(goma_ctl goma_dir) toybox
both result in error message being printed.
Change-Id: I94ccaf217dba71aca990d88d205bad669a49a2f5
This CL adds symlink_outputs to various locations in Soong that creates
actions that creates symlink outputs, and explicitly mark them as such.
Test: m
Bug: 160568334
Change-Id: I322751bada52a9f49011c74731d84761586e03e7
Clean the odm_dlkm and vendor_dlkm staging directories in installclean
to get consistent results in incremental builds on the build servers.
Test: treehugger
Change-Id: Id431d7457d85e0be68437a2034fd544d74cc3709
Collect the total number of CPU cores and available RAM for each build
as part of the system resource info. This is to know what kind of
build machines that the developers are using.
Bug: b/169453825
Test: m nothing and ran printproto on soong_metrics to validate the
data.
Change-Id: I8617bdb9490b37d7689abd27d349af1a651533db
This retriggers soong_build whenever a new bzl, WORKSPACE, or
BUILD.bazel file is changed or added.
Test: Manually verified on bionic/libc genrules with manual changes to
related BUILD/bzl/WORKSPACE files -- these all retrigger builds.
Test: Updated finder_test.go
Change-Id: I634384f88781a6b6db32f5d6bf9c07e179e14c39
Tradefed packages are created in out/host/common/obj/PACKAGING
and should be removed during installclean so that they get an
accurate embedded build number.
Fixes: 170308394
Test: treehugger
Change-Id: Ib7a57377b5c9512584bc93f10da316eadf786979
This reverts commit 323dc60712.
Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests
Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
Prints the following warning messsage:
WARNING:
Setting OUT_DIR to a path other than out may result in slow RBE builds.
See http://go/android_rbe_out_dir for a workaround.
Bug: b/169676232
Change-Id: Id1ad06c4dc672c24373642e0b624833eb6a0dbcf
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.
Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
FORCE_USE_GOMA is needed to be collected as a metric for the
deprecation of GOMA on Android builds.
Bug: b/169423400
Test: m nothing, ran printproto command
Change-Id: I12300389f1f27239d8f21875d1b6e4ad069d95d7
We should override log-dir with the environment variables if its set.
This is needed in our integration tests and generally to allow for
user-overrides.
Test: RBE_log_dir=`pwd`/rbelogs mmma external/boringssl
Change-Id: I3d316a2d60122c42f5c3b5dfede4037362a7b5f1
This is an initial CL that collects the environment variables
USE_GOMA and USE_RBE. This is a needed metric to determine how
many users are currently using GOMA so we can help them migrate
to RBE.
Bug: b/163598127
Test: m nothing and checked soong_metrics
Change-Id: I39536eabad01aafbb3bd7182515d72230b36f658
This hasn't worked for a couple years, and continues to bitrot. Just
remove it.
Adds a bpfix rule so that we can eventually remove the
product_variables.pdk definition, which is now always a no-op.
Test: treehugger
Change-Id: I830b54d419b59f6db1d4617b45e61a78234f57a7
Merged-In: I830b54d419b59f6db1d4617b45e61a78234f57a7
"exec format error" showed up on Mac when running the rbe_test.go.
The generated bash scripts did not have a newline at the end of
the file.
Bug: b/162600461
Fixes: b/162600461
Test: "m blueprint_tools" on Android Mac builder.
Change-Id: I425fa5f428733b6569c3fcdaa312023f9761d020
Disable the error test cases to allow developers to build
AOSP on Mac.
Bug: b/162600461
Test: m nothing
Change-Id: Ic8c6fe04743a8e4fe197a1bb71259036fb749edc
Disable the error test cases to allow developers to build
AOSP on Mac.
Bug: b/162600461
Test: m nothing
Change-Id: Iac4373dcb10710d8a2325eeb3ee20a6eeca6143f
Pass -n to blueprint to make it use validation dependencies for tests,
which will let them run in parallel with soong_build.
Bug: 156428456
Test: m nothing
Change-Id: I706c14330543e2a7728e5dc385976900c67cb2f5
Currently, the build timestamp saved to the build_date_timestamp
field in the MetricsBase metrics protobuf usually comes from a text
file named build_date.txt. It may not be accurate for metrics purposes.
Instead, use the build start time from Soong main UI for a better
timestamp.
Bug: b/140638454
Test: Ran and checked the build_date_timestamp by running the printproto
command.
Change-Id: I4d893fbbf9830ba21911e56ae13bc5272be47ae6
From aosp/1329396, the RBE metrics protobuf file is part of the
metrics uploading process. The RBE metrics protobuf file is
generated by running the bootstrap shutdown command. A new function
named DumpRBEMetrics was written in order to generate the RBE metrics
protobuf file before sending to the uploading process.
Bug: b/140638454
Test: * Unit test cases
* Ran RBE build on my local host and verified the
metrics protobuf file is created.
* Ran non-RBE build after RBE build and verified that
the previous metrics protobuf file was deleted.
Change-Id: I4b8068905cb67c4b8c2d94793917b98974fed707
Help debugging finder issues by disting module_paths/files.db for
later analysis.
Bug: 157656545
Test: treehugger
Change-Id: Idfa4cffe7ef724b399a2667efd487f9c3c76bb6b
Specifically, make Android.bp.list and soong.variables explicit inputs
While this is not a comprehensive list of all inputs
of this action (as the action depends on all blueprint
files in the source tree), this is closer to the truth.
This is a rollforward CL, which was originally rolled
back, as path validation failed for when OUT_DIR was
an absolute path. Validation has now been relaxed.
Test: Manually verified ninja output, checkbuild approved validation
for aosp-crosshatch, and manually ran frameworks/rs/build_rs.py with
patch fix
Change-Id: I4eb0d517f57336dd54eaa4bd31f46df9e93e6da2
Debugging issues on the build servers can be difficult because the
intermediate files are not visible. Gzip ninja file and Makefiles
generated by Soong and the ninja files generated by Kati to the dist
directory, and also copy all of the finder output files.
Bug: 157656545
Test: m dist nothing
Change-Id: I48d75305e551ccae81c7a55721981cf58acd838b
While this is not a comprehensive list of all inputs of this action (as the action
depends on all blueprint files in the source tree), this closer to the truth.
Test: Manually verified ninja output, checkbuild approved validation for aosp-crosshatch
Change-Id: I77a0f612afb025232bee7d3ac31257b808991829
The metrics uploader was currently running on foreground where it
would copy the metrics files in a separate directory and then forked
into the background for the upload process. As a result, running the
lunch command would take a second longer to run since each metrics
uploader run had an average of half a second.
Bug: 140638454
Test: * Wrote and updated unit test cases.
* Set ANDROID_ENABLE_METRICS_UPLOAD to point to the latest
metrics_uploader bash script. Executed the "lunch 1" command
and measured the running time. Executed "m nothing" command
and checked that the metrics were uploaded.
* Ran "lunch 1" and "m nothing" with
ANDROID_ENABLE_METRICS_UPLOAD=""
* Removed oauth from metrics_uploader and ran "m nothing" and
"lunch 1". The oauth Message appeared only to "m nothing"
Change-Id: I13c61e666c8f44613dee291a704cef6a27335188
External applications such as metrics uploader depends on the TMPDIR.
DumpMakeVars is changing the TMPDIR environment variable value. Once
the work is done, TMPDIR directory is deleted. Additional flow of
the tool or external applications can panic since TMPDIR directory
does not exist. Since this is an isolated case, pass the TMPDIR
directory to the command of dumping the makefile variables.
Bug: b/140638454
Test: * Ran lunch command with ANDROID_ENABLE_METRICS_UPLOAD set
and metrics_uploader did not crash.
* m nothing
Change-Id: Ib78a3f8cfb5da48f65bd9bce1511e9abdf3d9cca
Soong now supports the ability to upload metrics to another location
by setting the ANDROID_ENABLE_METRICS_UPLOAD to an uploader that
accepts the upload.proto proto buffer message. When the environment
variable is set, a set of build metrics files (soong_metrics,
rbe_metrics.pb and build_error) is uploaded.
Bug: 140638454
Test: * Wrote unit test cases
* Setup the uploader, built a succcessful and failed aosp_arm-eng
target and monitor the uploading of the metrics.
Change-Id: I76a65739c557dc90345e098ca03119a950ece2d2
Drop down to a single high-mem task when we've got <=16GB, as the system
probably isn't completely free RAM, ninja will be using a couple gigs,
along with whatever "normal" actions will be running concurrently. So
it's unlikely that we can handle two 6-8GB actions along with everything
else.
Also print warnings when we detect <=16GB total RAM, or when we're
running more parallel jobs than than we have GB RAM. These both notify
the user and suggest lowering the `-j` value if they run into problems.
Test: fake totalRAM to [0.5,8,16]GB, checking warning
Test: fake totalRAM to 17GB, `m -j4 nothing` has no warning
Test: `m -j187 nothing` on a 188GB system
Test: `m -j188 nothing` on a 188GB system
Change-Id: Ieb008e9f462d5f40fb65781d94cf116b1caf8446
The stats output will now use the new "DEBUG" message type, which we can
always redirect to verbose.log.gz.
Test: m aprotoc (look in verbose.log.gz)
Change-Id: Ie1b58f12c008ff7d29f11ff7a9807488dba8a504
misc_info.txt's build rule does not contain inputs to tell it to
refresh on an incremental build, so incremental builds (even after
installclean) were reusing this file.
With this change, misc_info.txt is removed by installclean so that an
incremental build generates the file fresh.
Test: # misc_info.txt exists; m installclean; # misc_info.txt gone
Bug: 155930200
Change-Id: Ia319925dad26f98049a988ee1ceb1e20db621353
This was a temporary migration tool, and has been broken ever since the
ninja environment started being restricted. Remove it, since it has
lasted two releases.
Test: TEMPORARY_DISABLE_PATH_RESTRICTIONS=true m
Change-Id: I2e3709525598030f5c24d12fa966e531dfed33ae
A user was trying to use OUT_DIR=/tmp/...-out DIST_DIR=/tmp/...-dist
Test: OUT_DIR=/tmp/test-out DIST_DIR=/tmp/test-dist m nothing
Change-Id: I811cc8e1bda8d766406603df47309a8637cef48c
By the time we do argument parsing, we've already written to $OUT_DIR or
the default, so don't allow overriding the output directory on the
command line.
Test: m OUT_DIR=newdir
Change-Id: I6f8c07cbef3a71d393b766dc5a01ae9c3df8631c
Includes the exit code, real time, user time, system time, and maxrss.
Test: m nothing; check out/soong.log
Change-Id: I93dea9fc1c7fb892150e16e11cae09f2372f4429
This replaces the _kati_always_build_ hack with a ninja feature so that
ninja can actually understand what's happening. This means that we can
turn on some more options and checks around expected output filenames:
* Remove the output file(s) before the command executes
* Error if the output file(s) don't exist after the execution
* Error if the output is a directory
They're turned on immediately during the soong bootstrap runs, as those
run a limited number of rules. The main ninja execution does not remove
the output files, and prints warnings instead of errors for the others.
I'll turn them on as we understand how often those warnings are seen.
Test: m (check for new warnings)
Test: treehugger (check for new warnings)
Change-Id: I7f5c1a361dd1498eb54a2c07a918f3b0aa086e4c
This CL adds a remoteexec package that allows adding a configurable RBE
prefix to the template.
Test: built aosp crosshatch userdebug with and without RBE_CXX_LINKS.
Change-Id: Ica920c3d7f79f2996210b9cbd448126451c1707c
The dist dir is created late in the build process. If a
soong sandbox attempts to mount it before it's created
the sandbox will create an empty file in its place.
Test: lunch aosp_cf_x86_phone-userdebug && make -j dist
Change-Id: Ie6513bf702de6e6322c78753d406d70ea3ccc04d
It's safer to disallow writing to the root filesystem of the sandbox.
Still allow write access to the source, tmp, out and dist directories.
Test: build aosp_cf_x86_64_phone-userdebug
Change-Id: Idf0d3a420669fbf6c3aba24e058f09a314ec6d0e
This makes sure that we don't keep old vendor_boot artifacts around
after a clean.
Signed-off-by: Will McVicker <willmcvicker@google.com>
Fixes: 151957122
Bug: 151949371
Test: touch $OUT/vendor-ramdisk/test-file && make installclean
Change-Id: I357929583f234212a998dceaa31506fce42e0255
If this couldn't read the current user, it would print a generic
message, then proceed to dereference the invalid user struct :(
Provide somewhat reasonable defaults, and better error messages instead.
Test: m nothing
Test: run docker with misconfigured user
Change-Id: I21af77c7d8d1d2d27cb04546667eb1094c62a7a1
We have known problems (docker; distros w/o user namespaces) which mean that we won't be turning this into a fatal error anytime soon. Also remove the bug report link, it's not useful to continue getting the same reports.
Change-Id: I271871d68150417ac938f074d3730cad4518e327
Test: treehugger
Collect the number of modules and variants and some basic statistics
on memory usage inside soong_build by writing out a proto that is
read back in by soong_ui.
Test: examine soong.log
Change-Id: I6926876377a4f6229cf41fdbf166ae03c885ea55
Trying to override the default directory for ccache by using the
CCACHE_DIR environment variable fails unless it is in the allowed
list.
Bug: 149670916
Test: manual
Change-Id: I8e7eea7a5c25d7ea5f0956fafc70d62522f3c4fc
Apparently PackageManager gets confused when the apk disappears, but the
directory still exists.
Test: lunch aosp_arm-eng; m BasicDreams
<remove BasicDreams from PRODUCT_PACKAGES>
m nothing
ls out/target/product/generic/system/app
Change-Id: I486c0ddadde55fad226049d3fb0f09af88117da0
These variables control whether we accept cached-results / not and
whether we should update the cache with locally executed results or not.
I need accpet-cached whitelisted to purge the the RBE cache for the
invalid cache entry we have set for the failure we say yesterday.
Bug: b/148387048
Change-Id: I7344fc083f82e0b7bc11084376a267d19cf30bb8
This CL adds RBE support to javac, r8, and d8 rules which is only
enabled if respective environment variables are set.
Test: an aosp_crosshatch build with and without the new variables.
Change-Id: Ic82f3627944f6a5ee7b9f3228170c2709b1bfcb8
Create a highmem pool based on the total RAM and the number of CPUs,
with an override via the NINJA_HIGHMEM_NUM_JOBS variable. Put
metalava into the highmem pool.
Ninja does not support nested pools, and when goma or RBE is enabled
the maximum ninja parallelism is set very high with local jobs in a
local pool. When both the local pool and highmem pool are enabled,
the total number of local jobs will be as high as the sum of the sizes
of the two pools. Keep the highmem pool limited to 1/16th of the
local pool when remote builds are enabled to try to minimize the
effect while still limiting highmem jobs.
Fixes: 142644983
Test: m nothing, examine pools
Test: m USE_GOMA=true nothing, examine pools
Change-Id: Id79f11f44948992960ac34ecf831dacbe21bd332
Determining the total build time from the existing metrics is
tricky because some of the metrics are overlapping. For example,
the "soong" metric includes "blueprint bootstrap",
"environment check", "minibp", "bpglob", "minibootstrap", and
"bootstrap". Create a new singleton "total" metric that
covers the interesting parts of the build.
Test: m nothing
Change-Id: Iedcf5c9cf0d27252b694d35e6ff66ca1fcf734ac
This relands I12a0f907753fefd1997ab8b4ea2ac331234093cf along with
a fix to blueprint for absolute paths.
Store the current working directory and then change to the root
directory so that all file accesses must go through helpers in
the android package that properly track dependencies.
Change-Id: I24ac485677aa102eec1a2521d16820da6ee1ae77
Fixes: 146437378
Test: m checkbuild
Test: m OUT_DIR=/tmp/out nothing
Store the current working directory and then change to the root
directory so that all file accesses must go through helpers in
the android package that properly track dependencies.
Fixes: 146437378
Test: m checkbuild
Change-Id: I12a0f907753fefd1997ab8b4ea2ac331234093cf
This is used by the art apex test.
Bug: 147197813
Test: m EMMA_INSTRUMENT=true EMMA_INSTRUMENT_FRAMEWORK=true art-check-debug-apex-gen
Change-Id: Id185cd35f16131f2c9a8afeba2e5b87834b0e620
We're passing in ASAN_OPTIONS=detect_leaks=0 on the command line, and
the build started failing after we limited the environment.
Test: forrest of aosp-master/aosp_x86_64-eng
Change-Id: I42c91897c7677e1a249412e5a8bc7bb1edb2f881
Make now exports (partial) lists of what should be installed. Those
lists are incomplete, but if something is removed from the list, we can
remove it from the filesystem so that incremental builds work better.
Test: built-in tests (m blueprint_tools)
Test: remove a module from PRODUCT_PACKAGES, see the print and file removed
Test: change the name of a cts test, see the old one removed from cts
Change-Id: I6bc14c6a5af63383ea265e97ed8b875fa51af0ed
See the change that added ALLOW_NINJA_ENV for more information.
Test: m nothing; check out/soong.log for small list
Test: ALLOW_NINJA_ENV=true m nothing; check out/soong.log
Change-Id: I7761c6a07a7f8b0acee107e9c27c7739dd4b63ab
Ninja does not track changes in environment variables, so we get
potentially incorrect builds when environment variables change during
incremental builds if some action was using one of them.
Add a variable to limit exposure of these variables to ninja, and thus,
all actions run by ninja. Kati and Soong can still read environment
variables, they explicitly track which ones they read so that we can
re-run them appropriately.
This list is just the beginning, there's no good way to detect which
environment variables are currently being used and to pass them through.
So this initial change won't have a behavioral change, and we'll flip
the switch and see what fails or who complains, flipping it off and on
and adding to the list until we can make this always happen.
Also adds a board-specific `BUILD_BROKEN_NINJA_USES_ENV_VARS := ...`
list so that we can temporarily allow board-specific variables until
they're fixed.
Test: check out/soong.log
Test: ALLOW_NINJA_ENV=false m nothing; check out/soong.log
Test: set BUILD_BROKEN_NINJA_USES_ENV_VARS := OLDPWD
ALLOW_NINJA_ENV=false m nothing; check out/soong.log
Change-Id: I08e4834ce12100a577ef7d6a9a21b9e9d345cb93
Colin originally wrote this for the highmem pool:
https://android-review.googlesource.com/c/platform/build/soong/+/1168271
But while that's a larger change, this is useful to just be in the logs,
and for use in multiproduct_kati to better limit the number of
concurrent jobs.
Test: check soong.log on linux and mac
Change-Id: I0518d303a220d775f8d78dba9f660b2954e68e3e
(Although I was the one who added the gzcat reference, I don't believe
we actually use it. And neither my Debian host nor toybox has gzcat --
as opposed to zcat -- anyway. So just quietly drop that.)
Test: treehugger
Change-Id: I95d94dab1daf72faedfa170770e8ebfaa37b70c6
When LOCAL_COPY_HEADER lines are removed, we just removed the copy rule,
we didn't actually remove the old header, so C/C++ files could still
include the old headers during incremental builds.
While we do consider LOCAL_COPY_HEADERS as near-obsolet, and it'll
disappear as we move everything over to Soong, this could produce some
unfortunate incremental build results while people are attempting to
remove them.
Fix this by ensuring that only the files currently listed in
LOCAL_COPY_HEADERS exist in TARGET_OUT_HEADERS after we run Kati.
Test: Remove a LOCAL_COPY_HEADERS entry; see the header removed
Change-Id: I817305703a6996d50490d552623d7df019b608c9
Delay writing the BUILD_DATETIME_FILE until after the out directory
has been created.
Test: cuj_tests
Change-Id: Ice6f34d003f93c26b5d2d0b64f92b11efe16c2d4
This runs an arbitrary shell script for all non-interactive bash sessions, and we've got lots of those.
It was causing problems with Fedora 30, which triggers the `manpath` executable, which isn't available in our $PATH:
https://groups.google.com/d/msg/android-building/WqEyZQuxYtY/zDXl9irOCAAJ
Change-Id: If2f63bbbcc3accb8d239f93fa06823d301d69488
Pass --default_pool=local_pool to kati when RBE or goma is enabled
to put most rules into the local_pool. Specific rules that support
RBE or goma will set .KATI_NINJA_POOL := none to remove themselves
from the local_pool. Passing --default_pool will also disable the
hack in kati that sets the pool based on the presence of the string
"/gomacc" in the command line.
Fixes: 143938974
Test: inspect pools in build-${TARGET-PRODUCT}.ninja for m USE_RBE=true
Test: inspect pools in build-${TARGET-PRODUCT}.ninja for m USE_GOMA=true
Change-Id: I839b2488383fcd63fffd613e25b0b9abcb72b567
Previously, this setting environment variable to false reverted to an
OpenJDK 9 toolchain. This change removes that option.
Test: m core-all-system-modules
Test: zcat out/verbose.log.gz | grep 'prebuilts/jdk/jdk[0-9]*'
Test: EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN=false m core-all-system-modules (fails)
Bug: 131683177
Change-Id: I0d679648e1236925ce3ef7e1652379127e846b00
Prior to this change, the default was to use and OpenJDK 9 toolchain,
with the EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN=true environment
variable available to opt into OpenJDK 11. After this change, the
default is to use OpenJDK 11, with
EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN=false available to opt out.
This change affects:
- The version of javac used.
- The version of javadoc used.
- The version of other tools used during the build process (e.g. jar,
jmod, and jlink).
- For Java bytecode executed on the host, the version of the java
executable used, and so the versions of the JVM and the OpenJDK
libraries in the system module (or bootclasspath).
This change does not affect:
- The Java language level, i.e. the version of Java expected in
source code (the -source option to javac) or the version of Java
bytecode produced (the -target option to javac).
- Anything to do with code execution on the device.
Bug: 131683177
Test: make java droid docs cts tests
Test: zcat out/verbose.log.gz | grep 'prebuilts/jdk/jdk[0-9]*'
Test: make RunBluetoothRoboTests RunCarSettingsLibRoboTests
Test: cts-tradefed help
Test: atest CtsHostTzDataTests
Test: atest CtsLibcoreTestCases
Change-Id: I09dc22f1af4d1f2d7d3b85c08cb0ed9a1105aaca
Merged-In: Iaae89ef25c92ec099575c026fc50e41da4e143e8
A new sysprop neverallow rules are mandatory only for devices launching
with R or later. For devices already launched, neverallow rules can be
relaxed with adding following line to BoardConfig.mk:
BUILD_BROKEN_TREBLE_SYSPROP_NEVERALLOW := true
Bug: 131162102
Test: Set PRODUCT_SHIPPING_API_LEVEL := 30 and try building with
changing some system_public_prop to system_internal_prop
Test: m cts sepolicy_tests
Change-Id: I5e1640f7b43fd47863bc7bd163c2f43b01fa3326
Merged-In: I5e1640f7b43fd47863bc7bd163c2f43b01fa3326
(cherry picked from commit e36f52754f)
Should unbreak the mac build while I work on toybox xargs some more.
Test: suck it and see (since there's no mac presubmit)
Change-Id: I48954f801fc980a1174f93e474d73a24ad12ce66
With this change, setting the environment variable
EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN=true switches from the OpenJDK 9
toolchain to the OpenJDK 11 one (prebuilts/jdk/jdk11).
Since the version of the java.base module has to match the version of
the jlink tool which consumes it, the --module-version argument to the
jmod create call is also switched.
Test: make
Test: EXPERIMENTAL_USE_OPENJDK11_TOOLCHAIN make
Bug: 131683177
Change-Id: I606d22538165f309fe5537bd67a26390b1dd5771
My previous change makes TMPDIR a temporary directory that's removed
before we exit. So instead of starting ckati to get this value to a
directory that no longer exists, return the semi-stable value that we
use during a normal build.
Fixes: 142277335
Test: get_build_var TMPDIR
Change-Id: I37d1219b1fda09f131dee17ef7b91e475d3e39ad
Instead of just using the host $PATH during dumpvars, use our path
configuration from the regular build. But instead of creating a ton of
symlinks to the interposer, just use a small directory of symlinks. This
only takes ~3ms (vs ~300ms), at the expense of error logging. Since we
do just about the same product configuration at the start of the build,
we can just rely on the logging there.
This fixes warnings printed by the Mac build, since we using the host
`date` instead of the toybox version:
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
usage: date [-jnRu] [-d dst] [-r seconds] [-t west] [-v[+|-]val[ymwdHMS]] ...
[-f fmt date | [[[mm]dd]HH]MM[[cc]yy][.ss]] [+format]
TMPDIR was being shared with the main build, but we're allowed to run in
parallel. If a build is started while we're running, our TMPDIR may be
cleared. Instead, create our own temporary directory, which doesn't need
to be a stable value (as we never do caching here).
Fixes: 141893400
Test: (on mac) source build/envsetup.sh; lunch
Change-Id: I0ff536e71dc5649cae26daa0087eb80861704021
log.Fatalln calls os.Exit, which skips deferred functions and
leaves the console with the cursor disabled. Use ctx.Fatalln
instead, which uses a panic to exit and calls the deferred
function to restore the cursor.
Test: DIST_DIR=/tmp/\test\ foo m dist
Change-Id: Ie92297075e37c171d5ba48848a0ddb19652b051c
This commit adds BUILD_BROKEN_PREBUILT_ELF_FILES to
`ui/build/dumpvars.go` so that its value can be printed to
`out/soong.log`.
Bug: 141176116
Test: lunch aosp_crosshatch-userdebug && make && \
grep BUILD_BROKEN_PREBUILT_ELF_FILES out/soong.log
Change-Id: I99600a5b540be2bb93647ee8175b9f9afd1a6eb3
Before this change, host APEX (`.zipapex`) packages would still be
there after `m installclean`.
Test: Run `m installclean` and check that host APEX packages are removed.
Change-Id: I446286ce700f12666be5aa90fbe3bbc648abd280
When removing dependencies from APEXes, the lib files were still there
even after installclean.
Bug: 139016109
Test: run installclean, see apex directories go away :D
Change-Id: I4dad8e3aa0042cdd5bbb76bd6fdd01b025b98495
Otherwise tools built with ASAN can't symbolize their dumps, as
llvm-symbolizer is not allowed to be used from $PATH.
Fixes: 139825736
Test: cherry-pick https://android-review.googlesource.com/c/platform/system/tools/aidl/+/1106532
mmma ASAN_OPTIONS="" SANITIZE_HOST=address system/tools/aidl
Change-Id: Ib1eeec15ce6694d7bde54c7201b3280e38df3a83
The code still needs to exist, but for buildspec.mk files, not the
PRODUCT-* arguments.
Test: none
Change-Id: I60c6044ab2d1627af7e6e2e8831b77d9db12aa55
This is done so that prototype tooling can write logs inside the DIST_DIR.
Test: I wrote out all the environment variables being passed to
goma_ctl.py and the output is https://paste.googleplex.com/6663469954760704. It contains both DIST_DIR and other variables.
Change-Id: Ia5c70e58aa1b87db9777c59cf3e5df1c9e20f08c
Test: Built aosp_arm-user with and without USE_RBE. USE_RBE uses
a proxy script in place of rewrapper.
Change-Id: I5bf008a940513872d70b5b215bd6209f759826ae
If OUT_DIR has been specified through the command line arguments,
it is sanitized first and NewConfig sets the OUT_DIR in the build
environment. When calling OutDir from Config instance, the out
was being sanitized again which is unnecessary.
Bug: b/118730755
Test: Ran unit test cases, "lunch" followed by "m" command.
Change-Id: Ieb4ffa1026c802c76c7369e8cdb3923a4c70f4b3
Previously, any of the build commands (mma, mmma, etc...) would traverse
up the directory path until an Android build file has been found. That
directory path would eventually become the MODULES-IN-<dir name> build
target. The directory where the m* command was executed may contain a
build file in a sub directory. For that case, the build target is the
MODULES-IN-<dir name> where dir is the directory of the executed m*
command since the target has been constructed by the core makefile.
Bug: b/118730755
Test: Wrote and ran unit test cases. Ran "lunch", "m". Went to a
directory in AOSP that had build files in sub directories and
executed mma. Went to a directory that had only build files in
parent directories and executed mma.
Change-Id: I9458cbc4bf0fc25bfb0848e96e92d261442d5cb0
For mmma and mmm, the findBuildFile function in config.go is invoked every time
for specified directories and directories with targets. For directories with
targets, an Android build file must exist in the directory where mmma and mmm
was invoked. There is no need to invoke findBuildFile function as a
simple check of the build file exists in the specified directory.
This is also refactoring the code for b/118730755
Bug: b/118730755
Test: Executed unit test cases through Intellij and executed mmma
command: "mmma external/protobuf:aprotoc external/bzip2".
Change-Id: I5428e3a3c36f77ff740617564b7853705521f29f
Allow running Soong in a headless delve debugger by passing
SOONG_DELVE=<listen addr> in the environment.
Bug: 80165685
Test: SOONG_DELVE=:1234 m nothing
Change-Id: Icfc893c8a8354a9bbc99112d9c83259cb41906d1
When running ninja stream stdout to the status writer. This improves
the ninja -d explain behavior, and will also allow Soong to be put
into the ninja console pool letting it print timely output to the
console.
Bug: 80165685
Test: NINJA_ARGS="-d explain" m
Change-Id: I40f03fc01d837ad91d400eae378007a7ce807705
The build directory passed in to soong_ui can contain a
symbolic link. soong_ui was not evaluating the build
directory to retrieve the true path, hence failing to
execute the internal soong.
Fixes: b/135995632
Test: Unit test case and executed the scenario in the bug
description.
Change-Id: I5779c6aa3f3183810437dbe2b2d4e40acbafb205
There was one case that did not work:
1) Create dir in root source tree
2) Run "m libc"
The build would fail as it was using the mma logic. A separate
one named BUILD_MODULES was added to allow building specific
modules in any directories using the "m" command.
Bug: b/130049705
Test: unit test cases, ran "m libc" in a tmp directory inside
of the source tree.
Change-Id: I8d23e685a673a311001ee8edd89bd73b662392dd
Everything should be using the prebuilt now.
Bug: 117561006
Test: treehugger
Test: check bass logs for m4 uses
Change-Id: Ic69d715d72d64bb0082e2f2c247113155d8b3de0
The metrics.proto needs to be imported to Google3 in order to unmarshal
the build metrics data from bigstore. Cleaned up the enum names to use
the Camel naming convention and renamed the build_metrics to
soong_metrics to be more specific to soong and to allow quering
soong_metrics files from bigstore.
Bug: b/135280521
Test: lunch andf m -j. Checked if soong_metrics was generated correctly
by using printproto command.
Change-Id: I998c8d05db592e94a653d6ca32250b80df3c9b21
Add the following build actions {BUILD_MODULES_IN_A_DIRECTORY,
BUILD_MODULES_IN_DIRECTORIES} in soong_ui config so the bash code version of
build commands (m, mm, mma, mmm, mmma) in build/make/envsetup.sh can be deprecated.
This is to allow up to date bug fixes on the build commands.
Bug: b/130049705
Test: Unit test cases
Change-Id: I772db1d4e9c1da5273374d1994eb5e8f17cd52f2
terminal.Writer is now just a wrapper around stdio.Stdout() without
any useful functionality. Replace it with stdio.Stdout() as an
io.Writer.
Test: ui/terminal/status_test.go
Change-Id: I5bc5476afdca950b505642f0135a3af9d37fbe24
All users have been removed.
Test: treehugger
Change-Id: I35e55f3ce79c8e68585be120b5dec810563c7a83
Merged-In: I35e55f3ce79c8e68585be120b5dec810563c7a83
- Add SOONG_LOCK_TIMEOUT environmental variable for users to set a
duration (e.g. 1h), which defaults to the current 10 seconds.
- Instead of logging once per second waiting to acquire the lock,
only log once before the first wait
Fixes: 131854539
Test: make; open another tab and make
Change-Id: I0a02a1b071964f74473a5f55034f916526e1b309
Allow install-clean as an alias for installclean and data-clean as
an alias for dataclean.
Test: m install-clean
Change-Id: I9c97d60572a524ad68caecd26d52bfb987468075
This reverts commit e114b3f0a9.
Build broke as follows:
out/soong/host/linux-x86/bin/sbox --sandbox-path out/soong/.temp --output-root out/soong/.intermediates/system/update_engine/ue_unittest_disk_imgs/gen -c 'tar -jxf system/update_engine/sample_images/sample_images.tar.bz2 -C __SBOX_OUT_DIR__/gen disk_ext2_1k.img disk_ext2_4k.img disk_ext2_4k_empty.img disk_ext2_unittest.img' __SBOX_OUT_DIR__/gen/disk_ext2_1k.img __SBOX_OUT_DIR__/gen/disk_ext2_4k.img __SBOX_OUT_DIR__/gen/disk_ext2_4k_empty.img __SBOX_OUT_DIR__/gen/disk_ext2_unittest.img
"bzip2" is not allowed to be used. See https://android.googlesource.com/platform/build/+/master/Changes.md#PATH_Tools for more information.
tar: Child returned status 1
tar: Error is not recoverable: exiting now
sbox command (tar -jxf system/update_engine/sample_images/sample_images.tar.bz2 -C out/soong/.temp/sbox849645882/gen disk_ext2_1k.img disk_ext2_4k.img disk_ext2_4k_empty.img disk_ext2_unittest.img) failed with err "exit status 2"
Change-Id: I3fe7100886ea3b748f5a6b5fc6d2811391bb72a8
There are no remaining users.
Also mark BUILD_BROKEN_ANDROIDMK_EXPORTS and BUILD_BROKEN_ENG_DEBUG_TAGS
as deprecated in scripts/build_broken_logs.go
Test: treehugger
Change-Id: If7892bef1b9001f12a99565f886b395cf1985e70
So that the output of build_test can be parsed to understand which
devices need changes.
Test: read out/soong.log
Change-Id: I40b25527ef72e3f646c0cfe62114b452fa3f27f6
Some people apparently still talk to the network during their build.
Allow this temporarily with a BUILD_BROKEN_USES_NETWORK check.
Bug: 129992021
Test: attempt to talk to the network during the build with and without
this flag.
Change-Id: Ifb967c656aa24c4599e7232d0f1b5a303b5bac52
Wait for the ninja proto processing goroutine to notice the fifo
has closed and exit before continuing.
Test: m nothing
Change-Id: I8cf5f3b8bf6a91496c6d2bbbd3e811eb7f0c9d21
Dump variable in log to track remaining targets which don't set it.
Bug: 122954981
Test: $ grep VNDK out/soong.log
2019/03/07 09:22:04.235604 build/soong/ui/build/dumpvars.go:109: BOARD_VNDK_VERSION current
Change-Id: I34146c752a4bf0d799374bb93cc0d75b0f1b048c
There's more than just toybox in this group now, so let's rename our
variables to something closer to the desired behavior, rather than the
first user.
Test: treehugger
Change-Id: I76d4407792061c8110b194cfe73f1ddc84dbc22f
For ease of investigating the issue with error reports, let me make
the error message more verbose.
Test: make goma_ctl.py raise Exception and run.
Test: I confirmed verbose error message shown.
Change-Id: I04f9a5ca4808746994c41ddce8a5b22a1975fcc2
Signed-off-by: Yoshisato Yanagisawa <yyanagisawa@google.com>
Bug: 124218940
If ANDROID_QUIET_BUILD environment variable is set to '1', do not show 20
lines of the environment variables. In addition, for the failing step show
only the step name and output, omitting the command proper (the verbose log
as well as error.log still contains it).
If build succeeds, the output of the build itself consists of a single
line:
```
> m androidmk
[100% NN/NN] <last command>
```
When it fails, the output does not contain sometimes very long command
line:
```
> m androidmk
[ 97% NN/MM] test androidmk
FAILED: <step>
--- FAIL: TestEndToEnd (0.01s)
androidmk_test.go:1025: failed testcase 'prebuilt_etc_TARGET_OUT_ETC'
input:
include $(CLEAR_VARS)
LOCAL_MODULE := etc.test1
LOCAL_MODULE_CLASS := ETC
LOCAL_MODULE_PATH := $(TARGET_OUT_ETC)/foo/bar
include $(BUILD_PREBUILT)
expected:
prebuilt_etc {
name: "etc.test1",
filename: "foo/bar",
}
got:
prebuilt_etc {
name: "etc.test1",
filename: "foo/bar",
}
FAIL
17:50:53 ninja failed with: exit status 1
```
[The related change in build/make/envsetup.sh suppresses timing display
when the same variable is set.]
Change-Id: I4d3c72457de031ff58a324c2fe98f4c1d10f8239
Test: treehugger
And add a helper script that can help parse the output from
build_test.bash
Test: check for BUILD_BROKEN_ENG_DEBUG_TAGS in soong.log
Test: go run ../build/soong/scripts/build_broken_logs.go *
Change-Id: Idd0fc8b59770dcdbe44eeba262558708a9497f96
We don't want these to be warnings, may as well be errors instead of
turning off the warnings.
Bug: 123583617
Test: treehugger
Change-Id: I6ca518c9647e712426952cb88bdb044d933b23f2
As more phony rules move either to the packaging kati run, or soong, the
existing real-to-phony check failed to notice that they were phony
rules.
So I added --top_level_phony to Make that assumes that all leaf nodes
(source files, or nodes generated by another ninja file generator) that
do not have a '/' in them are phony targets.
The existing phony-looks-real check in Kati should prevent these from
being real generated files, and our source tree shouldn't have any
source files used at the root level.
Test: build_test on downstream branches
Change-Id: I77cff84b536c6cd6402307a9aa5a9c273e72c71f
We've got an internal testcase that uses minijail0 inside an Android.mk
rule. That was failing since we turned on the linux sandbox, as /proc
was mounted read-only, which prevented setting up the uid/gid mappings
for a child namespace.
Fixes: 122985455
Test: treehugger & forrest of breaking build
Change-Id: Ia77a91a7f4eeeb8a24e84075d8272287f5087587
This starts a new network namespace without any connections to the
outside.
Bug: 122270019
Test: USE_GOMA=true m libc
Test: treehugger
Test: add rule to use /usr/bin/wget, fails after this change
Change-Id: Iba262025ce0e4e3bef5c34c817cc678d6c61403b
This really only initializes the sandbox, it does not attempt to change
the view of the filesystem, nor does it turn off networking.
Bug: 122270019
Test: m
Test: trigger nsjail check failure; lunch; m; cat out/soong.log
Test: USE_GOMA=true m libc
Change-Id: Ib291072dcee8247c7a15f5b6831295ead6e4fc22
