Commit graph

592 commits

Author SHA1 Message Date
Lukács T. Berki
ddebd2c82a Merge "cd to / before running soong_build ." 2021-03-04 07:07:02 +00:00
Petri Gynther
ac22956dcb Rename vendor-ramdisk to vendor_ramdisk
Rename to get consistent ramdisk directory naming in
out/target/product/<name>:
  debug_ramdisk
  ramdisk
  vendor_debug_ramdisk
  vendor_ramdisk

Test: build and inspect out/target/product/<name>
Change-Id: Ib86e6509bd910e8c95bcb6e3f7bb639ebbfe3758
2021-03-03 00:15:54 -08:00
Lukacs T. Berki
7690c09953 cd to / before running soong_build .
This lets one avoid any decisions as to when to chdir there during its
execution and leads to better sandboxing because the pwd doesn't leak to
init() functions anymore.

Test: Manual.
Change-Id: I1560da8ed3a621249426f9e8908aa890c21e13ba
2021-03-03 09:14:22 +01:00
Colin Cross
fdf28ca403 Merge changes I3e6bc9b5,If9f8fb10
* changes:
  Disable TestSendLog on the buildbots
  Remove global state from VNDK apexes
2021-03-03 01:45:02 +00:00
Colin Cross
6cb1128b54 Disable TestSendLog on the buildbots
TestSendLog sometimes hangs on the buildbot, although it has never
been reproduced locally.  The cause appears to be a unix domain
packet that was sent but never received.  Skip the test in short
mode for now, which will leave it manually runnable but not run
it as part of the build.

Fixes: 120596545
Test: m nothing
Change-Id: I3e6bc9b5f1d9c15dc9a7294d98cbd917d0637c44
2021-03-02 12:26:14 -08:00
Lukacs T. Berki
3243aa51eb Move environment staleness check to soong_ui.
Also delete the now-unnecessary soong_env binary.

Test: Manually. Also checked what happens when a used environment
variable changes.

Change-Id: Ib393e7f444e94819198c6cce4bcd8719d9fd9441
2021-03-02 09:32:49 +01:00
Bob Badour
02040de891 Add LOCAL_LICENSE_KINDS to build/soong
Added SPDX-license-identifier-Apache-2.0 to:
  Android.bp
  android/Android.bp
  android/soongconfig/Android.bp
  androidmk/Android.bp
  apex/Android.bp
  bazel/Android.bp
  bp2build/Android.bp
  bpf/Android.bp
  bpfix/Android.bp
  cc/Android.bp
  cc/config/Android.bp
  cc/libbuildversion/Android.bp
  cc/libbuildversion/tests/Android.bp
  cc/ndk_api_coverage_parser/Android.bp
  cc/ndkstubgen/Android.bp
  cc/symbolfile/Android.bp
  cmd/dep_fixer/Android.bp
  cmd/diff_target_files/Android.bp
  cmd/extract_apks/Android.bp
  cmd/extract_jar_packages/Android.bp
  cmd/extract_linker/Android.bp
  cmd/fileslist/Android.bp
  cmd/host_bionic_inject/Android.bp
  cmd/javac_wrapper/Android.bp
  cmd/merge_zips/Android.bp
  cmd/multiproduct_kati/Android.bp
  cmd/path_interposer/Android.bp
  cmd/pom2bp/Android.bp
  cmd/pom2mk/Android.bp
  cmd/sbox/Android.bp
  cmd/soong_build/Android.bp
  cmd/soong_env/Android.bp
  cmd/soong_ui/Android.bp
  cmd/zip2zip/Android.bp
  cmd/zipsync/Android.bp
  cuj/Android.bp
  dexpreopt/Android.bp
  dexpreopt/dexpreopt_gen/Android.bp
  env/Android.bp
  etc/Android.bp
  filesystem/Android.bp
  finder/Android.bp
  finder/cmd/Android.bp
  genrule/Android.bp
  jar/Android.bp
  java/Android.bp
  java/config/Android.bp
  kernel/Android.bp
  linkerconfig/Android.bp
  linkerconfig/proto/Android.bp
  makedeps/Android.bp
  partner/Android.bp
  phony/Android.bp
  python/Android.bp
  python/tests/Android.bp
  remoteexec/Android.bp
  rust/Android.bp
  rust/config/Android.bp
  scripts/Android.bp
  sdk/Android.bp
  sh/Android.bp
  shared/Android.bp
  symbol_inject/Android.bp
  symbol_inject/cmd/Android.bp
  sysprop/Android.bp
  tradefed/Android.bp
  ui/build/Android.bp
  ui/logger/Android.bp
  ui/metrics/Android.bp
  ui/metrics/proc/Android.bp
  ui/status/Android.bp
  ui/terminal/Android.bp
  ui/tracer/Android.bp
  xml/Android.bp
  zip/Android.bp
  zip/cmd/Android.bp

Added SPDX-license-identifier-Apache-2.0 SPDX-license-identifier-BSD to:
  finder/fs/Android.bp
  third_party/zip/Android.bp

Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m all

Exempt-From-Owner-Approval: janitorial work

Change-Id: Ia47ca14f16b8c9f84f9d533a07e5b00e2c04e8d4
2021-02-06 04:23:21 +00:00
Jingwen Chen
eb76c4319d bp2build: fix running from clean checkout.
bp2build is a Soong mode that returns as soon as the BUILD files are generated. This causes it not generate the build.ninja file, which the Ninja process executing soong_build itself expects to produce as an output. If there isn't an existing build.ninja file generated from a previous build, GENERATE_BAZEL_FILES=true m nothing will fail.

This CL generates the expected files as a workaround, and also makes GENERATE_BAZEL_FILES=true skip the Kati/Ninja steps in soong_ui since they aren't needed.

Test: rm -rf out/ && GENERATE_BAZEL_FILES=true m nothing && m libc

Fixes: 178683777

Change-Id: I2515ef7961682d2be5f096ed24831cc185165a67
2021-01-29 22:33:25 -05:00
Kousik Kumar
1e4d5f3e67 Add FAILED: prefix to RBE Fatal error message
This error message it currently being
logged to stdout which goes to build.log file . We want it to be logged to
build_error.log file instead so that downstream systems that need to run
based on the error message will continue to work.

Test: Tested by running `m toybox` and ensuring that FAILED: prefix is
at the beginning of the line.

Bug: b/177221705
Change-Id: Ie0c9acee2739acf149384e33804b5fca3273c24f
2021-01-26 14:34:30 -05:00
Treehugger Robot
f9be4e07e1 Merge "Moving $OUT/vendor-ramdisk-debug.cpio.lz4 to intermediate dir" 2021-01-24 14:53:43 +00:00
Yo Chiang
d813f12473 Remove GKI artifacts on installclean
Bug: 177495407
Test: m installclean && GKI prebuilts under OUT dir are removed
Change-Id: I05f888376c7f2b253e9966c1a652063a6c4760d9
2021-01-22 00:16:47 +08:00
Bowgo Tsai
ae205c3e32 Moving $OUT/vendor-ramdisk-debug.cpio.lz4 to intermediate dir
Therefore, no need to remove vendor-ramdisk-debug.cpio.lz4
for installclean as it will not be installed to $OUT dir.

Bug: None
Test: make vendorbootimage_debug
Change-Id: Ib7fb9931c55a9bb24e5ca8c4a80bd72f7233c57f
2021-01-20 16:35:31 +08:00
Jaewoong Jung
6e49493dac Skip old file cleanup for non-full builds.
There are few build targets that don't generate an installed file list,
e.g. product-graph. Skip the old installed file cleanup step so that the
func doesn't complain about a lack of the list file.

Fixes: 168105598
Test: m product-graph
Change-Id: Ib7dce6b801979bb565b74d6355143bac23b84fe6
2021-01-05 16:47:04 -08:00
Jaewoong Jung
18aefc1977 Remove unnecessary snake case variables.
Test: m nothing + TreeHugger
Change-Id: I99f7162944daa6c57c6ae4763261e108bb5cb6b1
2020-12-22 12:38:35 -08:00
Patrice Arruda
175500f016 Merge "Read the proc status file when PID is given for metrics purpose." 2020-12-18 04:14:09 +00:00
Patrice Arruda
04157e186f Read the proc status file when PID is given for metrics purpose.
To measure the MaxRSS memory correctly, read the proc status file
under /proc/<pid>/status and extract the MaxRSS value from it. The
implementation is only available for Linux based distributions.

Bug: b/169453825
Test: go test
Change-Id: I32e3068fee7447f9ef5dfb5a8d8dcb6934e0af23
2020-12-17 14:52:14 -08:00
Patrice Arruda
589826bc1e Add and update comments in ui/metrics/metrics.go.
The metrics.go had missing comments and also did minor code refactoring
to be more in golang best practices.

Bug: b/172917718
Test: m nothing.
Change-Id: Id63e4af3f27b582fc13ef3f988ac5c8707fb9adf
2020-12-15 23:36:01 +00:00
Patrice Arruda
5c58b6410c Remove soong_metrics_proto from the import of the protobuf file.
No needed to define soong_metrics_proto since it is defined in
the Android.bp file.

Bug: b/172917718
Test: m nothing
Change-Id: Ib5f4cd6d06c08660803cf070da544e9c2d1dce1f
2020-12-15 19:41:40 +00:00
Patrice Arruda
48d55ad2db Add setup_tools performance metrics to Soong protobuf file.
The setup_tools metrics has been missing since day one and is
needed to track down slowness of setting up the build system.

Bug: b/175706305
Test: m nothing and ran the printproto command to check that the
      setup_tools field has been set.
Change-Id: Idc8a38fa830e0a906170feeede63435715eaa05b
2020-12-15 19:39:09 +00:00
Treehugger Robot
a767196955 Merge "Add and update comments in ui/metrics/event.go" 2020-12-15 15:58:03 +00:00
Patrice Arruda
8a44a37623 Add and update comments in ui/metrics/event.go
The event.go code was refactored to follow the best practices
of golang code style.

Bug: b/172918681
Test: go test ./ui/metrics
Change-Id: Ic57d7cd576a9b146530ae37f03f4f5679d6cffef
2020-12-15 03:01:24 +00:00
Patrice Arruda
05ab2d0731 Enable bazel profiling in soong_build.
Bazel is executed several times during the execution of soong_build.
For each bazel execution, generate a profile and save under the
BAZEL_METRICS_DIR which is defined in soong_ui.

Bug: b/174479924
Test: * USE_BAZEL_ANALYSIS=1 USE_BAZEL=1 m nothing and checked
        if the cquery and graph build bazel profiles were generated.
      * Verified that the generated bazel profiles were uploaded
        to the local dev metrics pipeline.

Change-Id: I3d20204484dc6c5a1525a5d3eec1d62cfb33535b
2020-12-14 18:48:11 +00:00
Treehugger Robot
8574ae773f Merge "Allow Bazel to write to an external DIST_DIR (outside of OUT_DIR)." 2020-12-11 05:16:32 +00:00
Rupert Shuttleworth
3c9f5ac787 Allow Bazel to write to an external DIST_DIR (outside of OUT_DIR).
Also get Bazel to write real files there (not symlinks) so that the DIST_DIR can be independent.

Test: Manually using e.g. DIST_DIR=/tmp/foo USE_BAZEL=1 m dist
Change-Id: I39d5219500864c9ecc85f356a028e9b5bf2607f4
2020-12-11 01:17:58 +00:00
Patrice Arruda
7d235cc24d Allow uploading a directory of metrics files.
The upload functionality now supports directories to be uploaded
for metrics purpose.

Bug: b/174479728
Test: Wrote unit test case.

Change-Id: I6906be4c1b7fd76ddf6ff7b94e48fe1c4209c59d
2020-12-10 21:48:05 +00:00
Treehugger Robot
5839c91074 Merge "Provide an interface for shared paths between Soong and Soong UI." 2020-12-09 17:31:02 +00:00
Rupert Shuttleworth
0bc9a9ad6b Add dependency checker special case for out/build_date.txt.
Test: Future TH presubmit.
Change-Id: I4f80cf12be4aa1a3040b38ac7c6bed593aefd65e
2020-12-08 23:28:25 +00:00
Patrice Arruda
83842d7235 Provide an interface for shared paths between Soong and Soong UI.
Code refactoring has been done for logs directory logic code since
the bazel metrics directory depends on the log directory. For builds
that did not specify a dist directory, the log directory is under
out directory. With dist, the logs directory is under <dist dir>/logs.
This matters for Android CI builds where the metrics files are
placed under logs directory. With this change, the bazel metrics
directory and corresponding files will be under <dist dir>/logs
directory for Android CI builds.

Bug: b/174479728
Test: * USE_BAZEL=1 m nothing (bazel_metrics dir in out dir)
      * m nothing (bazel_metrics dir deleted)
      * DIST_DIR=/tmp/build USE_BAZEL=1 m nothing dist (bazel_metrics
        is in /tmp/build/logs directory)

Change-Id: Ic9e1ff49a1964fcaaf801bde2c19f33597ca1db4
2020-12-08 20:24:14 +00:00
Rupert Shuttleworth
72f72b42cb Print both the implicit and explicit environment variables when running Bazel.
Test: Manually via USE_BAZEL=1 use_rbe m.
Change-Id: I9f0ac8875b47848fdea4ae97d107e4167c4a7f6e
2020-12-08 06:12:51 +00:00
Rupert Shuttleworth
561f2f2d96 Only print the Bazel command line once.
Test: USE_BAZEL=1 m and USE_BAZEL=1 use_rbe m.
Change-Id: I8413e92d91c906f884628bedac6955575cf86a1c
2020-12-07 09:59:13 +00:00
Rupert Shuttleworth
ead7ef6e3b Use lstat instead of stat to avoid permissions issues when creating symlinks.
Also refactor symlink code slightly and add some more error checking.

Test: Manually.
Change-Id: I75119707cdf497e5690ac57abd814c8f18cab1f8
2020-12-07 04:54:18 +00:00
Treehugger Robot
0ec64e5df7 Merge "Print the full Bazel command line (including env) for debugging purposes." 2020-12-07 04:52:37 +00:00
Rupert Shuttleworth
947ed97367 Print the full Bazel command line (including env) for debugging purposes.
Test: Manually ran USE_BAZEL=1 m and USE_BAZEL=1 use_rbe m and looked at the build output.
Change-Id: Idbfdf8b00458fbd4b684cb0c1b58d6515909293c
2020-12-07 04:50:29 +00:00
Treehugger Robot
840708e45b Merge "Remove obsolete environment variables from ninja.go allowlist." 2020-12-05 02:48:42 +00:00
Rupert Shuttleworth
411996ca50 Remove obsolete environment variables from ninja.go allowlist.
Test: N/A.
Change-Id: I717b31f2b68d2cac72cca33f04cfa3b6078cb8eb
2020-12-04 18:42:29 +00:00
Rupert Shuttleworth
ad532f21ab Allow RBE-related environment variables to be visible during Bazel action execution.
Test: Manually.
Change-Id: I1acfdd5f30b2e08288508da8136029e454316622
2020-12-04 11:27:54 +00:00
Anton Hansson
5e5c48b2b5 Add support for skipping just kati
The existing --skip-make flag disables both the config step and the kati
step in the build. Add support for a --skip-kati flag that skips just
the kati step, and refactor things so that the logic is shared between
these two.

Bug: 174315599
Test: TARGET_PRODUCT=aosp_arm64 soong_ui --make-mode --skip-kati;
      (verify soong.variables is regenerated)
Change-Id: I75b1910fc1c12fcda130e37b7bc4c050131c7b33
2020-12-02 10:39:50 +00:00
Treehugger Robot
d74b9c5728 Merge "Minor cleanup of soong.go." 2020-12-02 08:21:59 +00:00
Treehugger Robot
d8c099e12d Merge "Minor cleanup of build.go." 2020-12-02 02:29:08 +00:00
Rupert Shuttleworth
b7d971012a Minor cleanup of soong.go.
Test: TH presubmit.
Change-Id: Ic5b263aa1320d9f70a6749ed12cbdc477acd3067
2020-12-02 00:56:35 +00:00
Rupert Shuttleworth
eeb5caac78 Minor cleanup of build.go.
Test: TH presubmit.
Change-Id: I3a41130b25d39276162850e417f036f9ce6a2ec0
2020-12-02 00:49:03 +00:00
Treehugger Robot
bd8b0d26b2 Merge "Add and update comments in ui/build/upload.go" 2020-11-30 22:01:38 +00:00
Treehugger Robot
b21906cd70 Merge "Minor cleanup of environment.go." 2020-11-25 03:28:33 +00:00
Rupert Shuttleworth
1f304e674d Minor cleanup of cleanbuild.go (and partially build.go).
Test: TH presubmit.
Change-Id: I439c16687b1877d367ff3d0f27e30c9bac97ef58
2020-11-24 14:50:25 +00:00
Rupert Shuttleworth
80115fa6b0 Minor cleanup of environment.go.
Test: TH presubmit.
Change-Id: I89415d6b1219d2204b4fcce4b3a7456250cd0967
2020-11-24 13:16:30 +00:00
Jingwen Chen
cda22c9bb9 Refactor 'in_make' to mean Kati is not skipped.
In Nougat and before, Make wrote a marker file to indicate that
soong_build was invoked from Make to change certain behaviors of Soong
at build time.
https://cs.android.com/android/platform/superproject/+/android-7.1.2_r36:build/core/soong.mk;l=70-73;drc=ae18638b0406ad107b0882a02a13cdd8b92f2a4e

Things have changed, and now soong_build is invoked from soong_ui, which
supports a --skip-make configuration flag:
https://cs.android.com/android/platform/superproject/+/master:build/soong/ui/build/build.go;l=31-33;drc=680387bf1d3ce7cbc77f535be7c42cec411b1687

Thus, the various remnants of 'EmbeddedInMake' and 'inMake'
configuration are misleading, since soong_build is no longer invoked
from Make. This CL refactors all instances to actually mean that
Kati is enabled (not skipped with --skip-make), and will run after
soong_build finishes, so Kati-specific behavior like the AndroidMk
singleton should run.

Test: TH presubmit

Change-Id: I576ab8e54f99f5c8ddf9feaf9a828019b279e266
2020-11-23 00:29:18 -05:00
Treehugger Robot
5898d56912 Merge "soong_ui ninja.go: comment and refactor." 2020-11-22 20:15:46 +00:00
Jingwen Chen
9d1cb491c3 soong_ui ninja.go: comment and refactor.
On top of improving the comments, I also refactored the status checker
to simplify and clarify its use case to be a ninja stuckness checker,
since it doesn't appear to have other purposes.

Test: TH presubmit
Bug: b/173474588
Change-Id: I2cf51a1ebf16071a24a1c13c06c7b1adf60256de
2020-11-21 07:37:28 -05:00
Jingwen Chen
55e0022fc0 Merge "soong_ui path.go: improve comments." 2020-11-20 04:47:51 +00:00
Jingwen Chen
19362b16c3 Merge "soong_ui kati.go: write more comments." 2020-11-20 04:46:18 +00:00