Commit graph

570 commits

Author SHA1 Message Date
Treehugger Robot
8574ae773f Merge "Allow Bazel to write to an external DIST_DIR (outside of OUT_DIR)." 2020-12-11 05:16:32 +00:00
Rupert Shuttleworth
3c9f5ac787 Allow Bazel to write to an external DIST_DIR (outside of OUT_DIR).
Also get Bazel to write real files there (not symlinks) so that the DIST_DIR can be independent.

Test: Manually using e.g. DIST_DIR=/tmp/foo USE_BAZEL=1 m dist
Change-Id: I39d5219500864c9ecc85f356a028e9b5bf2607f4
2020-12-11 01:17:58 +00:00
Patrice Arruda
7d235cc24d Allow uploading a directory of metrics files.
The upload functionality now supports directories to be uploaded
for metrics purpose.

Bug: b/174479728
Test: Wrote unit test case.

Change-Id: I6906be4c1b7fd76ddf6ff7b94e48fe1c4209c59d
2020-12-10 21:48:05 +00:00
Treehugger Robot
5839c91074 Merge "Provide an interface for shared paths between Soong and Soong UI." 2020-12-09 17:31:02 +00:00
Rupert Shuttleworth
0bc9a9ad6b Add dependency checker special case for out/build_date.txt.
Test: Future TH presubmit.
Change-Id: I4f80cf12be4aa1a3040b38ac7c6bed593aefd65e
2020-12-08 23:28:25 +00:00
Patrice Arruda
83842d7235 Provide an interface for shared paths between Soong and Soong UI.
Code refactoring has been done for logs directory logic code since
the bazel metrics directory depends on the log directory. For builds
that did not specify a dist directory, the log directory is under
out directory. With dist, the logs directory is under <dist dir>/logs.
This matters for Android CI builds where the metrics files are
placed under logs directory. With this change, the bazel metrics
directory and corresponding files will be under <dist dir>/logs
directory for Android CI builds.

Bug: b/174479728
Test: * USE_BAZEL=1 m nothing (bazel_metrics dir in out dir)
      * m nothing (bazel_metrics dir deleted)
      * DIST_DIR=/tmp/build USE_BAZEL=1 m nothing dist (bazel_metrics
        is in /tmp/build/logs directory)

Change-Id: Ic9e1ff49a1964fcaaf801bde2c19f33597ca1db4
2020-12-08 20:24:14 +00:00
Rupert Shuttleworth
72f72b42cb Print both the implicit and explicit environment variables when running Bazel.
Test: Manually via USE_BAZEL=1 use_rbe m.
Change-Id: I9f0ac8875b47848fdea4ae97d107e4167c4a7f6e
2020-12-08 06:12:51 +00:00
Rupert Shuttleworth
561f2f2d96 Only print the Bazel command line once.
Test: USE_BAZEL=1 m and USE_BAZEL=1 use_rbe m.
Change-Id: I8413e92d91c906f884628bedac6955575cf86a1c
2020-12-07 09:59:13 +00:00
Rupert Shuttleworth
ead7ef6e3b Use lstat instead of stat to avoid permissions issues when creating symlinks.
Also refactor symlink code slightly and add some more error checking.

Test: Manually.
Change-Id: I75119707cdf497e5690ac57abd814c8f18cab1f8
2020-12-07 04:54:18 +00:00
Treehugger Robot
0ec64e5df7 Merge "Print the full Bazel command line (including env) for debugging purposes." 2020-12-07 04:52:37 +00:00
Rupert Shuttleworth
947ed97367 Print the full Bazel command line (including env) for debugging purposes.
Test: Manually ran USE_BAZEL=1 m and USE_BAZEL=1 use_rbe m and looked at the build output.
Change-Id: Idbfdf8b00458fbd4b684cb0c1b58d6515909293c
2020-12-07 04:50:29 +00:00
Treehugger Robot
840708e45b Merge "Remove obsolete environment variables from ninja.go allowlist." 2020-12-05 02:48:42 +00:00
Rupert Shuttleworth
411996ca50 Remove obsolete environment variables from ninja.go allowlist.
Test: N/A.
Change-Id: I717b31f2b68d2cac72cca33f04cfa3b6078cb8eb
2020-12-04 18:42:29 +00:00
Rupert Shuttleworth
ad532f21ab Allow RBE-related environment variables to be visible during Bazel action execution.
Test: Manually.
Change-Id: I1acfdd5f30b2e08288508da8136029e454316622
2020-12-04 11:27:54 +00:00
Anton Hansson
5e5c48b2b5 Add support for skipping just kati
The existing --skip-make flag disables both the config step and the kati
step in the build. Add support for a --skip-kati flag that skips just
the kati step, and refactor things so that the logic is shared between
these two.

Bug: 174315599
Test: TARGET_PRODUCT=aosp_arm64 soong_ui --make-mode --skip-kati;
      (verify soong.variables is regenerated)
Change-Id: I75b1910fc1c12fcda130e37b7bc4c050131c7b33
2020-12-02 10:39:50 +00:00
Treehugger Robot
d74b9c5728 Merge "Minor cleanup of soong.go." 2020-12-02 08:21:59 +00:00
Treehugger Robot
d8c099e12d Merge "Minor cleanup of build.go." 2020-12-02 02:29:08 +00:00
Rupert Shuttleworth
b7d971012a Minor cleanup of soong.go.
Test: TH presubmit.
Change-Id: Ic5b263aa1320d9f70a6749ed12cbdc477acd3067
2020-12-02 00:56:35 +00:00
Rupert Shuttleworth
eeb5caac78 Minor cleanup of build.go.
Test: TH presubmit.
Change-Id: I3a41130b25d39276162850e417f036f9ce6a2ec0
2020-12-02 00:49:03 +00:00
Treehugger Robot
bd8b0d26b2 Merge "Add and update comments in ui/build/upload.go" 2020-11-30 22:01:38 +00:00
Treehugger Robot
b21906cd70 Merge "Minor cleanup of environment.go." 2020-11-25 03:28:33 +00:00
Rupert Shuttleworth
1f304e674d Minor cleanup of cleanbuild.go (and partially build.go).
Test: TH presubmit.
Change-Id: I439c16687b1877d367ff3d0f27e30c9bac97ef58
2020-11-24 14:50:25 +00:00
Rupert Shuttleworth
80115fa6b0 Minor cleanup of environment.go.
Test: TH presubmit.
Change-Id: I89415d6b1219d2204b4fcce4b3a7456250cd0967
2020-11-24 13:16:30 +00:00
Jingwen Chen
cda22c9bb9 Refactor 'in_make' to mean Kati is not skipped.
In Nougat and before, Make wrote a marker file to indicate that
soong_build was invoked from Make to change certain behaviors of Soong
at build time.
https://cs.android.com/android/platform/superproject/+/android-7.1.2_r36:build/core/soong.mk;l=70-73;drc=ae18638b0406ad107b0882a02a13cdd8b92f2a4e

Things have changed, and now soong_build is invoked from soong_ui, which
supports a --skip-make configuration flag:
https://cs.android.com/android/platform/superproject/+/master:build/soong/ui/build/build.go;l=31-33;drc=680387bf1d3ce7cbc77f535be7c42cec411b1687

Thus, the various remnants of 'EmbeddedInMake' and 'inMake'
configuration are misleading, since soong_build is no longer invoked
from Make. This CL refactors all instances to actually mean that
Kati is enabled (not skipped with --skip-make), and will run after
soong_build finishes, so Kati-specific behavior like the AndroidMk
singleton should run.

Test: TH presubmit

Change-Id: I576ab8e54f99f5c8ddf9feaf9a828019b279e266
2020-11-23 00:29:18 -05:00
Treehugger Robot
5898d56912 Merge "soong_ui ninja.go: comment and refactor." 2020-11-22 20:15:46 +00:00
Jingwen Chen
9d1cb491c3 soong_ui ninja.go: comment and refactor.
On top of improving the comments, I also refactored the status checker
to simplify and clarify its use case to be a ninja stuckness checker,
since it doesn't appear to have other purposes.

Test: TH presubmit
Bug: b/173474588
Change-Id: I2cf51a1ebf16071a24a1c13c06c7b1adf60256de
2020-11-21 07:37:28 -05:00
Jingwen Chen
55e0022fc0 Merge "soong_ui path.go: improve comments." 2020-11-20 04:47:51 +00:00
Jingwen Chen
19362b16c3 Merge "soong_ui kati.go: write more comments." 2020-11-20 04:46:18 +00:00
Jingwen Chen
b1d30d63c5 soong_ui kati.go: write more comments.
This CL also removes --warn_real_to_phony and --warn_phony_looks_real,
since --werror_real_to_phony and --wrror_phony_looks_real are already
specified.

Test: treehugger
Bug: 173474588

Change-Id: Icbb0f53182d1b912c62f8044f52bb79d985bf868
2020-11-18 22:16:08 -05:00
Jingwen Chen
6a9bfa122a soong_ui path.go: improve comments.
Test: th presubmit
Bug: 173587901
Change-Id: Ic30fa88ee84cbc48860a7500223432c720039810
2020-11-18 22:13:23 -05:00
Jingwen Chen
db892410f7 Merge "soong_ui finder.go: improve comments/documentation." 2020-11-18 07:53:56 +00:00
Jingwen Chen
27a8719b0f Merge "Integrate bazelenv.sh environment variables into soong_ui environment." 2020-11-18 04:53:22 +00:00
Jingwen Chen
096a3bf08f soong_ui finder.go: improve comments/documentation.
Test: m
Bug: 173474588
Change-Id: I947a34b3d9a5b2cd36fba293b55860a20a11d40b
2020-11-17 19:48:31 -05:00
Jingwen Chen
7c6089ad95 Integrate bazelenv.sh environment variables into soong_ui environment.
This removes the need to source bazelenv.sh for USE_BAZEL_ANALYSIS, and
unifies mixed builds to use the checked in tools/bazel and bazelrc.

It also unifies all bazel-related output to be in out/bazel.

Without aosp/1502095, this change still requires toplevel_output_directories to be an empty
list, otherwise there'll be this error:

ERROR: Directories specified with toplevel_output_directories should be
ignored and can not be used as sources.

Test: With aosp/1441774: rm -rf out/ && lunch aosp_cf_x86_auto && USE_BAZEL_ANALYSIS=1 m libc && prebuilts/build-tools/linux-x86/bin/ninja -f out/combined-aosp_cf_x86_auto.ninja -t commands libc | grep bazel-out | wc -l # 2 results
Change-Id: I69b217ec88da531415792bb6e04b6a194ca4718d
2020-11-17 18:58:03 -05:00
Jingwen Chen
7a5391a979 soong_ui's bazel.go: document/comment with tiny refactoring.
This patch adds information behind the Bazel USE_BAZEL=1 execution
pipeline.

Test: m
Bug: 173474588
Change-Id: I54bdd26fb14c8a371de0922d193419112e8c3ece
2020-11-17 03:46:23 -05:00
Patrice Arruda
92dc64f9c8 Add and update comments in ui/build/upload.go
Bug: b/172918852
Test: go test .
Change-Id: I63ca307e3935fc5a91a550889601397300ba395e
2020-11-16 16:29:16 -08:00
Jingwen Chen
3a4b58dc5d Set PATH and --action_env=PATH for the Bazel build in soong_ui.
Ensures that Bazel and Ninja are using the same binaries from PATH, such
as `sort` from the linked bug.

Test: USE_BAZEL=1 m
Test: rm out/soong/apex/depsinfo/new-allowed-deps.txt && USE_BAZEL=1 m
Bug: 161074854
Fixes: 161074854
Change-Id: Icc3e717e6d3cfcecfe90f0fb3a94a86485a11918
2020-11-16 04:47:14 -05:00
Patrice Arruda
33e7f6a0ca Remove unnecessary break and default statements in metrics.go file.
break and default statements are not needed inside of a switch
statement.

Bug: b/173028918
Test: USE_BAZEL m nothing
Change-Id: I14aa8cbae3a9b3c859f147c5134600f1e75c6a89
2020-11-13 13:15:09 -08:00
Patrice Arruda
b7cf9ba175 Add tracer to runBazel function.
Added a tracer named "bazel" in runBazel function to capture topline
metrics when running bazel.

Bug: b/173028918
Test: * USE_BAZEL=1 m nothing and checked soong_metrics file using
        printproto command to verify that bazel topline metrics
	were captured.
Change-Id: I5cb0728a4da48545dc701cc79627c318d5d9a29d
2020-11-13 13:14:51 -08:00
Patrice Arruda
18cb70d645 Enable bazel metrics from ninja build.
Generate the bazel profile file that contains the metrics of
executing the ninja build phase of Soong.

Bug: b/173028918
Test: * m nothing and check out dir that there is no bazel_metrics dir.
      * USE_BAZEL=1 m nothing and checked
        out/bazel_metrics/build_bazel_profile.gz existed.
      * m nothing and checked out/bazel_metrics does not exist.
Change-Id: I797d1eecf5c2c260ade8b55dc69c237bfd0fcbd4
2020-11-13 11:37:06 -08:00
Patrice Arruda
af880da0c4 Define the bazel_metrics directory in $OUT.
$OUT/bazel_metrics directory is created in order to dump the
set of bazel profile files being created from a single build
execution. The directory is created from soong_ui.

Bug: b/173028918
Test: * m nothing and check out dir that there is no bazel_metrics dir.
      * USE_BAZEL=1 and checked out/bazel_metrics was created.
      * Ran m nothing and verified that out/bazel_metrics directory is
        deleted to ensure no metrics is uploaded or collected from the
	pipeline.

Change-Id: I1d3c2471065fae6931f35cef7f0f35fd84f76bef
2020-11-13 11:16:21 -08:00
Patrice Arruda
0c1c456643 Define UseBazel in Config
The UseBazel() function will be used in several places to perform
different functionality such as setting up the metrics path.

Bug: b/173028918
Test: m nothing
Change-Id: Ie94073b8f0552ec7528e1cc2f3be74b693dc3135
2020-11-13 11:16:21 -08:00
Jingwen Chen
a26ac3caee Export Ninja file paths to lunch repo rule from bazel.go
This ensures that the lunch repo rule always reads the right Ninja file.

Test: USE_BAZEL=1 m
Test: m

Change-Id: I7f670dd538f46c4dae5ea537b857ff04520b60e7
2020-11-12 08:01:33 +00:00
Rupert Shuttleworth
f8ae317d3e Allow extra startup and build args to be passed to Bazel.
For example: USE_BAZEL=1 BAZEL_STARTUP_ARGS="--batch" m

Test: Manually.
Change-Id: I76fa8e17d4a9ccb8dda9fca049113f434dacabbe
2020-11-10 02:06:30 +00:00
Jingwen Chen
8024c95ecc Add TODO bug number to stop hardcoding 'droid'
Test: N/A
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: I0d60cc4254b3a358085d89e443bd3bfc33b68ed3
2020-11-09 07:03:56 +00:00
Patrice Arruda
bb42c44e71 Merge "Collect useful resource information of each executed process in soong_ui." 2020-11-05 17:17:43 +00:00
Christopher Parsons
fde53fff96 Merge "After bazel execution, symlink results in out/" 2020-11-05 14:54:47 +00:00
Jingwen Chen
70b7eeed1c Move hardcoded bazel flags from soong_ui to common.bazelrc.
Test: USE_BAZEL=1 m
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: Ib675dba8bfaf615bf3752a4bb4531b818526f267
2020-11-05 03:31:28 -05:00
Chris Parsons
c09495bf96 After bazel execution, symlink results in out/
This ensures that USE_BAZEL=1 mode has the same directory structure
under out/ as it does without USE_BAZEL=1.

Test: Manually verified building USE_BAZEL=1 aosp_flame with droid and
libc

Change-Id: I27e810772d1bf04bd7930ea122217b93d2fd48b4
2020-11-04 20:56:14 -05:00
Patrice Arruda
ca221f3e6d Collect useful resource information of each executed process in soong_ui.
Collect the resource information of each executed process in soong_ui
for metrics analysis.

Bug: b/169453825
Test: m nothing; "lunch 1" && m
Change-Id: I8c7fe019111921c4c59174d133b85439ec825885
2020-11-04 23:35:43 +00:00