Commit graph

466 commits

Author SHA1 Message Date
Treehugger Robot
7d7cae85ab Merge "Soong: Add build_error proto raw file to $(OUT_DIR)." 2019-06-24 22:59:33 +00:00
Colin Cross
bbeb2c1a50 Merge "Color long running durations when using action table output" 2019-06-24 21:38:51 +00:00
Elliott Hughes
9b74dfe96c Merge "Switch to toybox tar." 2019-06-24 21:24:53 +00:00
Patrice Arruda
297cebad26 Soong: Add build_error proto raw file to $(OUT_DIR).
Soong_ui produces a build_error raw file to $(OUT_DIR) where
the file contains a list of build action errors. Each build action
error represents an error raised by a command. The build_error file
is populated if there was a build error.

Bug: b/132969697
Test: Ran m for successful build. Introduced a broken build change,
      ran m and verified using printproto that build_error was
      generated successfully.

Change-Id: I690ca1778b4e56f144a3173ba1d16d8494c32c15
2019-06-24 13:02:35 -07:00
Colin Cross
5137de0bdd Color long running durations when using action table output
When printing the action table, color the duration of commands that
have been running for 30 seconds yellow, and commands that have been
running for 60 seconds red.

Test: manual
Change-Id: I61cb21b0dae10811d345cb9f62cd07915cfc69ee
2019-06-24 10:49:11 -07:00
Treehugger Robot
45cdb71290 Merge "Soong: Add BUILD_MODULES build action in soong_ui." 2019-06-21 03:11:34 +00:00
Patrice Arruda
3928206ecc Soong: Add BUILD_MODULES build action in soong_ui.
There was one case that did not work:
  1) Create dir in root source tree
  2) Run "m libc"

The build would fail as it was using the mma logic. A separate
one named BUILD_MODULES was added to allow building specific
modules in any directories using the "m" command.

Bug: b/130049705
Test: unit test cases, ran "m libc" in a tmp directory inside
      of the source tree.

Change-Id: I8d23e685a673a311001ee8edd89bd73b662392dd
2019-06-20 16:35:12 -07:00
Elliott Hughes
63e70530d3 Merge "Allow gzcat as an alias for gzip's -dc mode." 2019-06-20 21:03:11 +00:00
Elliott Hughes
e671f5a397 Allow gzcat as an alias for gzip's -dc mode.
This gets us closer to toybox tar (and toybox gzip/gzcat).

Test: treehugger
Change-Id: If934270439b258eae55cdd3e97ac0e83b1c79c70
2019-06-19 14:02:02 -07:00
Dan Willemsen
b01a74fbea Disallow system m4
Everything should be using the prebuilt now.

Bug: 117561006
Test: treehugger
Test: check bass logs for m4 uses
Change-Id: Ic69d715d72d64bb0082e2f2c247113155d8b3de0
2019-06-19 12:51:00 -07:00
Dan Willemsen
c69d715d72 Use prebuilt m4 instead of system m4
Bug: 117561006
Test: treehugger
Change-Id: I588d867d95cc52963ce764018a9c09b26973c8c3
2019-06-19 12:50:32 -07:00
Treehugger Robot
fedc47141e Merge "Support an action table that shows longest running actions" 2019-06-19 06:41:28 +00:00
Treehugger Robot
4b1f4fe138 Merge "Use SIGWINCH to update terminal size" 2019-06-18 21:23:43 +00:00
Treehugger Robot
eec8d3aee3 Merge "Soong: Fix the package name in metrics.proto file" 2019-06-17 23:42:33 +00:00
Elliott Hughes
3b584b7861 Switch to toybox tar.
Test: treehugger
Change-Id: I4c561186c14cc44782a9e5c83d4d9c002addb3f6
2019-06-17 16:39:35 -07:00
Elliott Hughes
2d10e3f54e Merge "Revert "Revert "Switch to toybox timeout.""" 2019-06-17 23:35:16 +00:00
Elliott Hughes
0e868d0391 Revert "Revert "Switch to toybox timeout.""
This reverts commit 50e91c1a7a.

Reason for revert: SIGRTMIN+2 support added.

Change-Id: I3cb251a102e427335315c4931e23a69b95253ee7
2019-06-17 16:45:15 +00:00
Patrice Arruda
0cc5b21516 Soong: Fix the package name in metrics.proto file
The metrics.proto needs to be imported to Google3 in order to unmarshal
the build metrics data from bigstore. Cleaned up the enum names to use
the Camel naming convention and renamed the build_metrics to
soong_metrics to be more specific to soong and to allow quering
soong_metrics files from bigstore.

Bug: b/135280521
Test: lunch andf m -j. Checked if soong_metrics was generated correctly
      by using printproto command.

Change-Id: I998c8d05db592e94a653d6ca32250b80df3c9b21
2019-06-14 15:27:46 -07:00
Treehugger Robot
49cd4954a0 Merge "soong_ui: Add build actions commands in soong_ui." 2019-06-13 23:14:56 +00:00
Patrice Arruda
1384822a63 soong_ui: Add build actions commands in soong_ui.
Add the following build actions {BUILD_MODULES_IN_A_DIRECTORY,
BUILD_MODULES_IN_DIRECTORIES} in soong_ui config so the bash code version of
build commands (m, mm, mma, mmm, mmma) in build/make/envsetup.sh can be deprecated.
This is to allow up to date bug fixes on the build commands.

Bug: b/130049705
Test: Unit test cases
Change-Id: I772db1d4e9c1da5273374d1994eb5e8f17cd52f2
2019-06-13 13:15:12 -07:00
Colin Cross
3dac80e519 Support an action table that shows longest running actions
If SOONG_UI_TABLE_HEIGHT is set, enable a new smart terminal display
that prints the normal scrolling build history in the top region of
the screen and an action table of the longest currently running
actions in the bottom region of the screen.  This provides better
visibility into which are the longest running actions and when the
build parallelism is very low.

Test: manual
Change-Id: I677d7b6b008699febd259110d7f9e0f98d80c535
2019-06-13 11:26:06 -07:00
Colin Cross
4355ee64a9 Use SIGWINCH to update terminal size
Instead of reading the terminal size on every status update, register
for SIGWINCH to read and store the size when it changes.

This reapplies I555ad21a31a2c924ab0ca681e0c8f00df42a370a with a fix
for a race condition in TestSmartStatusOutputWidthChange.

Test: status_test.go
Change-Id: If342cb4cc8e4ed57af9e3bb417758348c9c41247
2019-06-13 11:26:06 -07:00
Colin Cross
4bc5e802db Revert "Use SIGWINCH to update terminal size"
This reverts commit 49036be407.

Reason for revert: TestSmartStatusOutputWidthChange is flaky

Change-Id: Ie5231dbbab2887ce4c4697f3fe6e52cfd6f4dd17
2019-06-13 18:10:45 +00:00
Colin Cross
49036be407 Use SIGWINCH to update terminal size
Instead of reading the terminal size on every status update, register
for SIGWINCH to read and store the size when it changes.

Test: status_test.go
Change-Id: I555ad21a31a2c924ab0ca681e0c8f00df42a370a
2019-06-12 21:30:56 -07:00
Colin Cross
00bdfd8476 Make status line bold
Bolding the status line provides differentiation between output
of each command.

Test: status_test.go
Change-Id: I9d46761e69c5af0a0aa86c7921e121cfd2a3fc82
2019-06-12 21:30:56 -07:00
Colin Cross
e0df1a36b2 Move all output through StatusOutput
Write log output through StatusOutput so that the status implementation
can synchronize it with its own output.

Test: status_test.go
Change-Id: I917bdeeea4759a12b6b4aa6d6d86ee18a2771723
2019-06-12 21:30:56 -07:00
Colin Cross
097ed2a37c Remove terminal.Writer
terminal.Writer is now just a wrapper around stdio.Stdout() without
any useful functionality.  Replace it with stdio.Stdout() as an
io.Writer.

Test: ui/terminal/status_test.go
Change-Id: I5bc5476afdca950b505642f0135a3af9d37fbe24
2019-06-12 21:30:56 -07:00
Colin Cross
ce525350f4 Move smart and dumb terminals into separate implementations
Support for smart and dumb terminals are implemented in writer.go,
which makes dumb terminals much more complicated than necessary.
Move smart and dumb terminals into two separate implementations
of StatusOutput, with common code moved into a shared formatter
class.

Test: not yet
Change-Id: I59bbdae479f138b46cd0f03092720a3303e8f0fe
2019-06-12 21:30:56 -07:00
Colin Cross
dde49cb687 Add tests for status output
Test: status_test.go
Change-Id: If3febd8fdacb0e70716d0520a41c982bd6474720
2019-06-12 21:30:56 -07:00
Elliott Hughes
bc31b9190a Switch to toybox date.
Test: treehugger
Change-Id: I558870092c29946419b9e15c33a6ee8ee7c6088f
2019-06-11 10:09:08 -07:00
Lokesh Gidra
50e91c1a7a Revert "Switch to toybox timeout."
This reverts commit e9f471635f.

Reason for revert: Multiple art-test failures

Change-Id: I550864c23485a438889ab73c7e4a877cd730cabb
2019-06-08 00:46:09 +00:00
Elliott Hughes
c1bfeed81b Switch to toybox sed (again).
The pathological case has been fixed, so let's see if it sticks this
time.

Test: treehugger
Change-Id: I043d823b05dd8796764e008c5e16e3563b80cec8
2019-06-07 13:10:04 -07:00
Elliott Hughes
e9f471635f Switch to toybox timeout.
Test: treehugger
Change-Id: Ieea84deac5d0ccef0f3d4949aa1ba870d406e482
2019-06-06 21:41:48 -07:00
Dan Willemsen
edff034366 Remove BUILD_BROKEN_ANDROIDMK_EXPORTS
Bug: 73959648
Test: treehugger
Change-Id: I932cc3acc6345f8b1ad8e395235489ae8275e2db
2019-05-25 17:13:55 -07:00
Dan Willemsen
7101d44442 Remove BUILD_BROKEN_ENG_DEBUG_TAGS
All users have been removed.

Test: treehugger
Change-Id: I35e55f3ce79c8e68585be120b5dec810563c7a83
Merged-In: I35e55f3ce79c8e68585be120b5dec810563c7a83
2019-05-25 22:48:12 +00:00
Amin Shaikh
de730781d9 Make soong lock file sync time configurable..
- Add SOONG_LOCK_TIMEOUT environmental variable for users to set a
duration (e.g. 1h), which defaults to the current 10 seconds.
- Instead of logging once per second waiting to acquire the lock,
only log once before the first wait

Fixes: 131854539
Test: make; open another tab and make
Change-Id: I0a02a1b071964f74473a5f55034f916526e1b309
2019-05-24 17:10:48 -04:00
Elliott Hughes
18e0d5e0f0 Go back to the host sed until we have a faster prebuilt.
Bug: https://issuetracker.google.com/131747477
Test: treehugger
Change-Id: I120fe1200216afdb492bd1353848d12a4be1f496
2019-05-09 13:59:01 -07:00
Dan Willemsen
7e52fa9f33 Don't dump BUILD_BROKEN_USES_BUILD_[HOST_]SHARED_TEST_LIBRARY
These have been errors for years, and are now marked as obsolete.

Test: treehugger
Change-Id: I8055ab689ce5085df763378ad40d9b9aaa7ca85b
2019-05-08 12:38:49 -07:00
Treehugger Robot
6ff39230e8 Merge "Support install-clean and data-clean" 2019-05-03 23:38:40 +00:00
Colin Cross
806fd94bed Support install-clean and data-clean
Allow install-clean as an alias for installclean and data-clean as
an alias for dataclean.

Test: m install-clean
Change-Id: I9c97d60572a524ad68caecd26d52bfb987468075
2019-05-03 13:38:57 -07:00
Elliott Hughes
b0e7320fbc Merge "Allow the host expr." 2019-05-03 17:24:00 +00:00
Elliott Hughes
3a653f419b Allow the host expr.
(Kernel builds require GNU unary + extension.)

Bug: https://issuetracker.google.com/131747477
Test: treehugger
Change-Id: I404db44d2ad0503d571d6f554a20445ab36aa7de
2019-05-01 12:52:25 -07:00
Elliott Hughes
6b3a0ddf0b xz: force use of the prebuilt.
Also comment why we can't yet do the same for bzip2.

Test: treehugger
Change-Id: I34d5bc0076f5790fcb01c185ef6d08824a4f6142
2019-04-30 22:20:43 -07:00
Elliott Hughes
3d87302a6d Revert "bzip2/xz: use the prebuilts."
This reverts commit e114b3f0a9.

Build broke as follows:

out/soong/host/linux-x86/bin/sbox --sandbox-path out/soong/.temp --output-root out/soong/.intermediates/system/update_engine/ue_unittest_disk_imgs/gen -c 'tar -jxf system/update_engine/sample_images/sample_images.tar.bz2 -C __SBOX_OUT_DIR__/gen disk_ext2_1k.img disk_ext2_4k.img disk_ext2_4k_empty.img disk_ext2_unittest.img' __SBOX_OUT_DIR__/gen/disk_ext2_1k.img __SBOX_OUT_DIR__/gen/disk_ext2_4k.img __SBOX_OUT_DIR__/gen/disk_ext2_4k_empty.img __SBOX_OUT_DIR__/gen/disk_ext2_unittest.img 
"bzip2" is not allowed to be used. See https://android.googlesource.com/platform/build/+/master/Changes.md#PATH_Tools for more information. 
tar: Child returned status 1 
tar: Error is not recoverable: exiting now 
sbox command (tar -jxf system/update_engine/sample_images/sample_images.tar.bz2 -C out/soong/.temp/sbox849645882/gen disk_ext2_1k.img disk_ext2_4k.img disk_ext2_4k_empty.img disk_ext2_unittest.img) failed with err "exit status 2" 

Change-Id: I3fe7100886ea3b748f5a6b5fc6d2811391bb72a8
2019-04-30 21:55:23 +00:00
Elliott Hughes
e114b3f0a9 bzip2/xz: use the prebuilts.
We added the prebuilts a while ago, but forgot to clean this up.

Test: treehugger
Change-Id: I8920f0c9d77951332a663750102ceeabcb565970
2019-04-29 16:23:08 -07:00
Dan Willemsen
2fbb7571bf Obsolete BUILD_BROKEN_DUP_COPY_HEADERS
There are no remaining users.

Test: treehugger
Change-Id: If8719480f73a8fa009d9733479390d2abd4021b1
2019-04-19 11:22:27 -07:00
Dan Willemsen
6097746769 Obsolete BUILD_BROKEN_PHONY_TARGETS
There are no remaining users.

Also mark BUILD_BROKEN_ANDROIDMK_EXPORTS and BUILD_BROKEN_ENG_DEBUG_TAGS
as deprecated in scripts/build_broken_logs.go

Test: treehugger
Change-Id: If7892bef1b9001f12a99565f886b395cf1985e70
2019-04-19 11:16:52 -07:00
Dan Willemsen
368e55671b Dump make deprecation status to logs
So that the output of build_test can be parsed to understand which
devices need changes.

Test: read out/soong.log
Change-Id: I40b25527ef72e3f646c0cfe62114b452fa3f27f6
2019-04-17 14:44:33 -07:00
Dan Willemsen
25e6f09c06 Add BUILD_BROKEN_USES_NETWORK
Some people apparently still talk to the network during their build.
Allow this temporarily with a BUILD_BROKEN_USES_NETWORK check.

Bug: 129992021
Test: attempt to talk to the network during the build with and without
      this flag.
Change-Id: Ifb967c656aa24c4599e7232d0f1b5a303b5bac52
2019-04-09 10:23:49 -07:00
Elliott Hughes
c2dbadc363 Add toybox seq.
Bug: https://issuetracker.google.com/129992021
Test: builds
Change-Id: I63d5b3fd1c71b771e995638094ee4310435105ee
2019-04-08 09:19:45 -07:00
Colin Cross
1aeb049a54 Panic if logging is attempted after Close
Attempt to catch places where logs are truncated by panicing if
logging is attempted after Close.

Test: m nothing
Change-Id: If670f20d08832ed65b63af5589b548e9815f2f0d
2019-03-21 21:17:04 -07:00
Colin Cross
b98d3bcf4e Wait for ninja proto processing to finish when exiting
Wait for the ninja proto processing goroutine to notice the fifo
has closed and exit before continuing.

Test: m nothing
Change-Id: I8cf5f3b8bf6a91496c6d2bbbd3e811eb7f0c9d21
2019-03-21 21:17:04 -07:00
Steven Moreland
0002028c09 dumpvars += BOARD_VNDK_VERSION
Dump variable in log to track remaining targets which don't set it.

Bug: 122954981
Test: $ grep VNDK out/soong.log
2019/03/07 09:22:04.235604 build/soong/ui/build/dumpvars.go:109: BOARD_VNDK_VERSION current

Change-Id: I34146c752a4bf0d799374bb93cc0d75b0f1b048c
2019-03-07 09:28:35 -08:00
Dan Willemsen
0f0214699e Use python / python2.7 prebuilts
Bug: 117811537
Test: treehugger
Change-Id: I35b335edcfa202a45f55f588a643c62c2ced0f26
2019-02-18 19:34:00 +00:00
Elliott Hughes
47de2a2f52 Switch sed(1) to toybox.
Test: treehugger
Change-Id: I575ae9db8d94cbec811d0029691df98373f8fe35
2019-02-15 15:49:12 -08:00
Dan Willemsen
733547d522 Use prebuilt awk on Darwin too
Test: build on mac
Change-Id: I25d21c0df0b9023d6c43b11dde60e6066f107576
2019-02-14 20:11:26 -08:00
Dan Willemsen
9121973130 Stop calling our host $PATH prebuilts toybox
There's more than just toybox in this group now, so let's rename our
variables to something closer to the desired behavior, rather than the
first user.

Test: treehugger
Change-Id: I76d4407792061c8110b194cfe73f1ddc84dbc22f
2019-02-14 20:07:02 -08:00
Yoshisato Yanagisawa
02eaccafbd More verbose message on goma_ctl.py ensure_start failure.
For ease of investigating the issue with error reports, let me make
the error message more verbose.

Test: make goma_ctl.py raise Exception and run.
Test: I confirmed verbose error message shown.
Change-Id: I04f9a5ca4808746994c41ddce8a5b22a1975fcc2
Signed-off-by: Yoshisato Yanagisawa <yyanagisawa@google.com>
Bug: 124218940
2019-02-13 10:49:10 +09:00
Elliott Hughes
a0f23ad233 Merge "Switch to one-true-awk." 2019-02-13 00:56:35 +00:00
Sasha Smundak
c0c9ef9964 Provide an option to reduce build's verbosity.
If ANDROID_QUIET_BUILD environment variable is set to '1', do not show 20
lines of the environment variables. In addition, for the failing step show
only the step name and output, omitting the command proper (the verbose log
as well as error.log still contains it).
If build succeeds, the output of the build itself consists of a single
line:
```
> m androidmk
[100% NN/NN] <last command>
```

When it fails, the output does not contain sometimes very long command
line:
```
> m androidmk
[ 97% NN/MM] test androidmk
FAILED: <step>
--- FAIL: TestEndToEnd (0.01s)
    androidmk_test.go:1025: failed testcase 'prebuilt_etc_TARGET_OUT_ETC'
        input:
        include $(CLEAR_VARS)
        LOCAL_MODULE := etc.test1
        LOCAL_MODULE_CLASS := ETC
        LOCAL_MODULE_PATH := $(TARGET_OUT_ETC)/foo/bar
        include $(BUILD_PREBUILT)

        expected:
        prebuilt_etc {
            name: "etc.test1",
            filename: "foo/bar",
        }

        got:
        prebuilt_etc {
            name: "etc.test1",
            filename: "foo/bar",

        }

FAIL
17:50:53 ninja failed with: exit status 1
```
[The related change in build/make/envsetup.sh suppresses timing display
when the same variable is set.]

Change-Id: I4d3c72457de031ff58a324c2fe98f4c1d10f8239
Test: treehugger
2019-02-07 12:21:45 -08:00
Elliott Hughes
cc74d59a80 Switch to one-true-awk.
Test: treehugger
Change-Id: I297d21f5a463c4d3070335c7533972397014f171
2019-02-06 18:06:48 +00:00
Dan Willemsen
01f0a05361 Add BUILD_BROKEN_ENG_DEBUG_TAGS to soong.log
And add a helper script that can help parse the output from
build_test.bash

Test: check for BUILD_BROKEN_ENG_DEBUG_TAGS in soong.log
Test: go run ../build/soong/scripts/build_broken_logs.go *
Change-Id: Idd0fc8b59770dcdbe44eeba262558708a9497f96
2019-02-05 15:15:16 -08:00
Dan Willemsen
dcac078c91 Turn on more warnings during CleanSpec processing
We don't want these to be warnings, may as well be errors instead of
turning off the warnings.

Bug: 123583617
Test: treehugger
Change-Id: I6ca518c9647e712426952cb88bdb044d933b23f2
2019-02-04 20:58:25 -08:00
Treehugger Robot
de53d15b0f Merge "Increase max line length for KatiReader bufio.Scanner" 2019-01-31 01:46:07 +00:00
Colin Cross
75ec1ec13f Increase max line length for KatiReader bufio.Scanner
bufio.Scanner defaults to a relatively small max line length of
64kB.  Kati output may have long lines, use a max length of 2MB
instead.

Fixes: 123590367
Test: m checkbuild
Change-Id: Ia869c5c2526621244f0065736e0d5fd9a088df06
2019-01-30 23:19:23 +00:00
Dan Willemsen
ef661b7f18 Fix runtime panics being suppressed
fatalLog was matching runtime errors, and essentially hiding them.

Test: m blueprint_tools
Change-Id: Ib48e7e142fc096998bc14b21fb717392adcff0ec
2019-01-28 20:00:01 -08:00
Treehugger Robot
ee04139154 Merge "Add --top_level_phony to catch more real-to-phony problems" 2019-01-26 05:08:15 +00:00
Elliott Hughes
48d716c367 Switch cp(1) to toybox.
Test: treehugger
Change-Id: I908e1d6aa2aa7016432ac7ff4abc56e8163bad8a
2019-01-25 22:29:58 +00:00
Dan Willemsen
cc62890866 Add --top_level_phony to catch more real-to-phony problems
As more phony rules move either to the packaging kati run, or soong, the
existing real-to-phony check failed to notice that they were phony
rules.

So I added --top_level_phony to Make that assumes that all leaf nodes
(source files, or nodes generated by another ninja file generator) that
do not have a '/' in them are phony targets.

The existing phony-looks-real check in Kati should prevent these from
being real generated files, and our source tree shouldn't have any
source files used at the root level.

Test: build_test on downstream branches
Change-Id: I77cff84b536c6cd6402307a9aa5a9c273e72c71f
2019-01-24 15:53:06 -08:00
Elliott Hughes
d5fdd3a134 Switch hostname(1) to toybox.
Test: treehugger
Change-Id: I52908a14da86ed8f162150bb48f72e141e6f060c
2019-01-24 15:18:06 -08:00
Elliott Hughes
d2c476cd2d Revert "Switch to toybox hostname(1)."
This reverts commit b115b81095.

Reason for revert: http://b/123123255

Change-Id: I0f2bfebf1189fdc71e093538a8b228dd0e0b6946
2019-01-19 18:55:54 +00:00
Elliott Hughes
44f0d4e1fc Switch to toybox which(1).
Test: treehugger
Change-Id: Ic14692b90c2b1246c1c8f72d00d6e0a5223aaf70
2019-01-19 17:43:06 +00:00
Elliott Hughes
db96fc8744 Switch to toybox pgrep(1), pkill(1), and ps(1).
Test: treehugger
Change-Id: Ic357b317e0233a626c89c38df152c3361ee8ea82
2019-01-19 17:41:53 +00:00
Elliott Hughes
b115b81095 Switch to toybox hostname(1).
Test: treehugger
Change-Id: Ibf89209befc8564b308c4b26bf08cef721938099
2019-01-18 15:11:22 -08:00
Elliott Hughes
46e68636f8 Switch to toybox wc(1).
Test: treehugger
Change-Id: I14d22406c1a462d4075a8538e549a8d0f5b88e8d
2019-01-18 08:28:54 -08:00
Elliott Hughes
c97a91183d Switch to toybox md5sum/sha1sum/sha256sum/sha512sum.
Test: treehugger
Change-Id: Icc8c3d5765d73f191e47adfe0a9d68727d075e50
2019-01-17 12:37:40 -08:00
Elliott Hughes
3aa6533f39 Merge "Switch to toybox mv(1)." 2019-01-17 18:32:08 +00:00
Dan Willemsen
3a4dbd651f Fix nested minijail0 execution
We've got an internal testcase that uses minijail0 inside an Android.mk
rule. That was failing since we turned on the linux sandbox, as /proc
was mounted read-only, which prevented setting up the uid/gid mappings
for a child namespace.

Fixes: 122985455
Test: treehugger & forrest of breaking build
Change-Id: Ia77a91a7f4eeeb8a24e84075d8272287f5087587
2019-01-16 23:04:16 -08:00
Elliott Hughes
6064792393 Switch to toybox mv(1).
Test: treehugger
Change-Id: If4e53cc8d088c7b79399ae8e10601c9b22bd118d
2019-01-16 21:54:30 +00:00
Elliott Hughes
c0cf454a63 Switch to toybox rm(1).
Test: treehugger
Change-Id: Ib31182ca7ce827396476699ac9d1c636cdd324fc
2019-01-16 17:44:29 +00:00
Dan Willemsen
24024eafee Effectively disable network access during the build
This starts a new network namespace without any connections to the
outside.

Bug: 122270019
Test: USE_GOMA=true m libc
Test: treehugger
Test: add rule to use /usr/bin/wget, fails after this change
Change-Id: Iba262025ce0e4e3bef5c34c817cc678d6c61403b
2019-01-15 16:58:27 -08:00
Dan Willemsen
63663c6bc9 Implement linux sandboxing with nsjail
This really only initializes the sandbox, it does not attempt to change
the view of the filesystem, nor does it turn off networking.

Bug: 122270019
Test: m
Test: trigger nsjail check failure; lunch; m; cat out/soong.log
Test: USE_GOMA=true m libc
Change-Id: Ib291072dcee8247c7a15f5b6831295ead6e4fc22
2019-01-15 13:47:31 -08:00
Elliott Hughes
713ae39f83 Move to toybox sort(1).
Test: treehugger
Change-Id: Ide649bbe9a73ab0e51d9bf4242d5c3c9172a85d6
2019-01-14 21:56:58 +00:00
Elliott Hughes
61456d1515 Switch to toybox readlink(1).
Test: treehugger
Change-Id: Ie1f6be5c2f273809a2691e506fc34c6840c942dd
2019-01-14 20:23:10 +00:00
Elliott Hughes
6712c0eb87 Switch to toybox xargs(1).
Test: treehugger
Change-Id: I8db68577498c1f6c0e1fdff951bc8972184b2754
2019-01-14 17:25:29 +00:00
Elliott Hughes
f1f016eec1 Move mktemp(1) to toybox.
Test: clean build
Change-Id: I7b02c7ee8dafb504c90d3af63698a09627556947
2019-01-12 17:51:43 +00:00
Elliott Hughes
c8a454c620 Switch to toybox getconf(1).
Test: treehugger
Change-Id: I7b43673b6a349116dd3e100f5e7eef704244e7ef
2019-01-12 17:40:18 +00:00
Elliott Hughes
fa6c8a1426 Switch to toybox stat(1).
Test: treehugger
Change-Id: Iebda993046935c17b1e37129a530f6eab9166018
2019-01-12 01:25:07 +00:00
Elliott Hughes
84af3624bb Merge "Revert "Move sed(1) to toybox."" 2019-01-12 00:09:12 +00:00
Elliott Hughes
de62ce15bc Revert "Move sed(1) to toybox."
This reverts commit e259a66d12.

Reason for revert: http://b/122744241

Change-Id: Id0e86ab79df2356fe12a2659fe39d02469ed9280
2019-01-12 00:07:40 +00:00
Treehugger Robot
e61fca299f Merge "Move sed(1) to toybox." 2019-01-11 22:53:10 +00:00
Elliott Hughes
cfc095251e Merge "Revert "Switch to toybox patch(1)."" 2019-01-11 22:36:12 +00:00
Elliott Hughes
e44c412220 Revert "Switch to toybox patch(1)."
This reverts commit e6df3b80a6.

Reason for revert: http://b/122739027

Change-Id: I5a55835fbb068e3129a69caf887611cd705613f7
2019-01-11 22:20:42 +00:00
Elliott Hughes
99363f2567 Merge "Switch to toybox patch(1)." 2019-01-11 21:14:17 +00:00
Treehugger Robot
b3fd876d81 Merge "Turn on ninja error for missing dep files" 2019-01-11 20:50:37 +00:00
Elliott Hughes
e259a66d12 Move sed(1) to toybox.
Test: treehugger
Change-Id: I338cffd5de27af0a93d0f379d021da5ccb2e5721
2019-01-11 17:15:36 +00:00
Elliott Hughes
e6df3b80a6 Switch to toybox patch(1).
Test: treehugger
Change-Id: Ia20e4132d5cb0e374ae6d97c57d94dfb5caa2c06
2019-01-11 16:35:40 +00:00
Elliott Hughes
81dd1686aa Merge "Switch to toybox ls(1)." 2019-01-11 16:34:38 +00:00
Yoshisato Yanagisawa
0c517bd9e4 Fatal error on insufficient resource to use Goma in Soong UI.
As suggested in
https://android-review.googlesource.com/c/platform/build/soong/+/839293
I am moving some features in goma.mk to goma.go in Soong UI.

With this CL, let me implement ulimit check to Soong UI.

Test: export USE_GOMA=true
Test: launch aosp_arm-eng
Test: make
Test: the command succeeds if "ulimit -n" and "ulimit -u" are large enough.
Test: Otherwise, it shows error.  I confirmed both cases.
Change-Id: I5d7d5ed71f620302a0d635770d1a51a2baab51fd
Signed-off-by: Yoshisato Yanagisawa <yyanagisawa@google.com>
2019-01-11 13:29:10 +09:00
Elliott Hughes
0b11f6590c Switch to toybox ls(1).
Test: treehugger
Change-Id: I7abb1f657e02ccf2eaaff465adfb28114f66875e
2019-01-10 16:55:58 -08:00
Elliott Hughes
3a1fd8e0a1 Switch to toybox ln(1).
Test: treehugger
Change-Id: Ia52e73e87a6c4648285d2340908013cd8528d221
2019-01-10 21:43:11 +00:00
Elliott Hughes
0bd4ab5334 Merge "Switch to toybox expr(1)." 2019-01-10 21:40:37 +00:00
Treehugger Robot
20a8856c8f Merge "Add BUILD_USERNAME and BUILD_HOSTNAME" 2019-01-10 21:29:47 +00:00
Elliott Hughes
6491f8df60 Switch to toybox expr(1).
Test: treehugger
Change-Id: Ib015c70841504d74fd0272a8ca6ecb3c617fb759
2019-01-10 17:04:57 +00:00
Elliott Hughes
e1184be0c0 Switch to toybox echo(1).
Test: treehugger
Change-Id: Ie40cce5113f4b9bd17ec28c572952aacf1057804
2019-01-10 06:25:25 +00:00
Elliott Hughes
c33514e3b6 Revert "Switch to toybox cp(1)."
This reverts commit 9886695763.

Reason for revert: http://b/122623408
Bug: http://b/122623408

Change-Id: Ib9ec92f9dd6f9a0db7f598b10241b0474e46aba0
2019-01-10 06:03:25 +00:00
Treehugger Robot
cd21d46df8 Merge "goma_ctl.py ensure_start in Soong UI." 2019-01-10 03:55:22 +00:00
Yoshisato Yanagisawa
2cb0e5df53 goma_ctl.py ensure_start in Soong UI.
As suggested in b/118390303#comment18, let me run
"goma_ctl.py ensure_start" in soong UI.

Bug: 118390303
Test: stop compiler_proxy, and execute following command:
Test: USE_GOMA=true lunch aosp_arm-eng
Test: and confirms compiler_proxy is running.
Change-Id: I859daae6ae2399c5b6bce2fafd874dfdb7c6aae0
Signed-off-by: Yoshisato Yanagisawa <yyanagisawa@google.com>
2019-01-10 10:14:16 +09:00
Elliott Hughes
e0e244130f Switch to toybox du(1).
Test: treehugger
Change-Id: Id66705f35e82ed25af6dbd5a131ce90ceba60912
2019-01-09 09:39:02 -08:00
Elliott Hughes
9886695763 Switch to toybox cp(1).
Test: treehugger
Change-Id: I91eddd95abc03f76d4da7fdc1e70c5cf314e6127
2019-01-08 16:31:54 -08:00
Elliott Hughes
2837d75ffd Merge "Switch to toybox chmod(1)." 2019-01-09 00:27:35 +00:00
Dan Willemsen
71edc8b848 Add BUILD_USERNAME and BUILD_HOSTNAME
As part of a future change to sandbox the build on Linux, the real
username will be switching to "nobody", and the hostname will be
switching to "android-build".

The USER environment variable will reflect the sandboxed value, so for
the build properties that want the external USER, they'll need to use
BUILD_USERNAME.

Similarly, BUILD_HOSTNAME will reflect the real value, while the
`hostname` tool will return "android-build"

Bug: 122270019
Test: check build.prop
Change-Id: I99604b9488732a63690b256dc4dd7894d369a32c
2019-01-07 23:00:54 +00:00
Dan Willemsen
f793933054 Turn on ninja error for missing dep files
So that ninja produces an error instead of just a warning when a dep
file is not produced.

Bug: 121058584
Test: check build logs for "depfile is missing" warning
Test: treehugger
Change-Id: I1cbaba866eaf293495c3c0b2b174190bcb2b0f9a
2019-01-07 18:57:04 +00:00
Nan Zhang
17f2767724 Generate build timing metrics to proto format file
Test: Dumped the text formated based metrics file to out dir,
and checked the file.
Bug: b/63815990

Change-Id: Iff476f72a0be74eb53b6b26ef468d11c0f24a404
2019-01-04 15:54:01 -08:00
Dan Willemsen
7f30c076d5 Print a status message when starting ninja
This way we don't appear hung at:

  No need to regenerate ninja file

Change-Id: I8dbdaa2c1b1c5a6a73187d0e6061f363b62e10c9
Fixes: 122251150
Test: m nothing
2019-01-02 12:50:49 -08:00
Elliott Hughes
97295bd391 Switch to toybox chmod(1).
Test: treehugger
Change-Id: Iccb70941906d9cf8b004303a6935b7df1217babf
2018-12-17 14:55:10 -08:00
Elliott Hughes
9add81ed0f Move cut(1) to toybox.
Test: treehugger
Change-Id: I07dd7a45ee56178a72deaffc6c4796c3ec53f564
2018-12-11 09:39:48 -08:00
Elliott Hughes
734a780217 Move touch(1) to toybox.
Test: treehugger
Change-Id: If31fe1fdc0150d07d5421702cbac8e142f522802
2018-12-07 18:30:52 -08:00
Elliott Hughes
cc85770628 Move mkdir(1) to toybox.
Test: treehugger
Change-Id: I5fcbf33acb93cdc279a900586b96e0232c4aea87
2018-12-06 22:30:45 -08:00
Elliott Hughes
ed46164d03 Move head(1) to toybox.
Test: clean build
Change-Id: I264c320b4a63e9b0f440791dc20ff99ec910036a
2018-12-07 02:36:16 +00:00
Elliott Hughes
2ebfd495cf Move dirname(1) to toybox.
Test: treehugger
Change-Id: Ie1ffbfb900803940620dd11a7ed6bfda76bee86c
2018-12-06 08:57:31 -08:00
Elliott Hughes
5172a8b50e Move to toybox unix2dos(1).
This removes the need for the confusingly named "todos".

Test: treehugger
Change-Id: Id8931deb00a06d54262b2803a00d13bd4de88f12
2018-12-05 19:42:43 -08:00
Elliott Hughes
6646904a47 Move to toybox cmp(1).
Test: treehugger
Change-Id: Idf5c9f00d0698aca6ceab2956c7dc49ad7c604e1
2018-12-05 10:03:31 -08:00
Elliott Hughes
6141a7d7ff Move to toybox basename(1).
Test: treehugger
Change-Id: I29f210d86a0dada8a40eca1919e2f6f876d50bcd
2018-12-04 13:42:46 -08:00
Elliott Hughes
5728fb536f Move setsid(1) to toybox.
Test: treehugger
Change-Id: I8caf9dbbb671218379c309e698f23e7c964f1a18
2018-12-03 10:14:52 -08:00
Elliott Hughes
1ff2b5d34c Move tee(1) to toybox.
Test: treehugger
Change-Id: Idf7fc37d6e84c295410f71dfc1ad9e460d7f7461
2018-11-30 12:36:44 -08:00
Elliott Hughes
f3f3b99bb3 Move xxd(1) to toybox.
Test: treehugger
Change-Id: I906b0c94da5b65dc387ae253492356ca78f93fd2
2018-11-29 22:04:16 -08:00
Elliott Hughes
15f2dfe482 Move paste(1) to toybox.
Test: treehugger
Change-Id: Id5e8e841406357d8301198eb433b96d2f1ba02d9
2018-11-29 11:36:51 -08:00
Elliott Hughes
716fa36dd9 Move rmdir(1) to toybox.
Test: treehugger
Change-Id: Ibb43f09c0cfcda631025ef0fea4730519a3b6fe6
2018-11-28 14:14:35 -08:00
Elliott Hughes
10412ec50b Move pwd(1) to toybox.
Test: treehugger
Change-Id: I656e322724d13273bf1099c6cc311a520aa4173e
2018-11-27 18:04:34 -08:00
Elliott Hughes
d50d40fe07 Move tail(1) to toybox.
Test: treehugger
Change-Id: If421513911208db4542c4767168ce50b7b837af8
2018-11-27 16:29:17 -08:00
Elliott Hughes
505b76136b Move od(1) to toybox.
Test: treehugger
Change-Id: I19587d3045992fa8f60743ef358c0fd050b43dc0
2018-11-27 09:00:21 -08:00
Elliott Hughes
bb19518a54 Move env(1) to toybox.
Test: clean build
Change-Id: I52770c022f7d34fb83c990f485b0ca3ea29e97ee
2018-11-15 13:55:56 -08:00
Elliott Hughes
6bdad22f7b Move comm(1) to toybox.
Test: clean build
Change-Id: I2d78388ef7211af6d517702c97f3b95ed2f30a46
2018-11-14 20:49:52 -08:00
Elliott Hughes
2907ceaae2 Move sleep(1) to toybox.
Test: clean build
Change-Id: I724278466c4d1367add79f949d1869e1e50a192d
2018-11-14 14:30:56 -08:00
Elliott Hughes
7a907c9291 Move uniq(1) to toybox.
Test: clean build
Change-Id: I4819098fd298dbc4f3e424e437a448e1fb1baeaf
2018-11-13 22:12:30 -08:00
Elliott Hughes
5d593238e2 Move cat(1) to toybox.
Test: builds
Change-Id: I9a367c3204891518994bfe889072dd1cf9aed490
2018-11-13 13:08:13 -08:00
Elliott Hughes
9b370f5ada Move id(1) and whoami(1) to toybox.
Test: builds
Change-Id: I32347c94438c394eb31f5af9f51cee11ad0ef247
2018-11-11 10:57:58 -08:00
Dan Willemsen
417be1fa9f Start using the toybox prebuilt
Prepend the toybox symlink directory to $PATH, and stop generating
symlinks into out/.path for tools used from toybox.

We don't (yet?) have toybox prebuilts for darwin, so apply the above only to Linux, and preserve existing behavior on Darwin.

Test: check the build banner for uname results
Test: m
Change-Id: I37fc380381e65a628cdc131d462fd4441eacfe9d
2018-10-30 23:25:35 -07:00
Treehugger Robot
a758cda418 Merge "Speed up build_test" 2018-10-30 20:27:42 +00:00
Dan Willemsen
f99915f569 Speed up build_test
Stop writing out ninja files, as they become very large, especially when
multiplied by the number of defined products.

Test: treehugger
Change-Id: Id9529d14040acb72a0188e58b5db2911f142071e
2018-10-29 10:43:37 -07:00
Elliott Hughes
be3cfa514a Disallow "runalarm" and allow the more standard "timeout".
Bug: N/A
Test: builds
Change-Id: Id8fe8ee2569b7410c66f448513cce6ff57458ffb
2018-10-27 08:09:18 -07:00
Dan Willemsen
6f03752422 Allow DIST_DIR to be read via dumpvars
DIST_DIR is controlled by soong_ui, and the make dumpvars code no longer
has access to it. So instead, handle some dumpvar requests inside
soong_ui itself.

Test: get_build_var DIST_DIR
Test: get_build_var OUT_DIR
Test: get_build_var BUILD_SYSTEM
Test: build/soong/soong_ui.bash --dumpvars-mode --vars="DIST_DIR
BUILD_SYSTEM"
Change-Id: Id3bcb8b0748db67c83d0e42d1ae5be564c5eb2f7
2018-10-21 09:26:55 -07:00
Dan Willemsen
2d31a44b8a Stop using DIST_DIR in Soong
We're only using it to distribute files in case of failure, which isn't
well supported currently, but can be handled for now by using the
DIST_DIR environment variable during the command execution.

This was at least one cause that we'd be re-running Soong during every
build server build, as the DIST_DIR values are unique.

Test: m dist
Change-Id: Ibd5e6b6c46695350de80b745bfb6a6aa685033a0
2018-10-20 21:33:41 -07:00
Dan Willemsen
d4ad1da3d7 Merge "Add a Kati-based packaging step" 2018-10-20 22:50:10 +00:00
Dan Willemsen
41cabfbbf4 Remove make and xmllint from the allowed tools list
We've got a prebuilt of make in prebuilts/build-tools, and we build
xmllint from source.

Test: No logs on the build servers in the last 24 hours
Change-Id: Ib60a7b5feaa31097071b67ebf6beb7f0b9a9f81b
2018-10-19 12:29:46 -07:00
Dan Willemsen
fb1271a52b Add a Kati-based packaging step
The idea is that we'd move the installation and packaging tasks over to
it, using data from Soong & the Kati reading Android.mk files.

This would allow us to make more fundamental changes about how we
package things without having to adjust makefiles throughout the tree.
Possible use cases:

* Moving some information from Soong's Android.mk output to a file read
  by the packaging step may allow us to read the Android.mk files less
  often, speeding up builds.

* Refactoring our current two-stage ASAN builds to run the Kati build
  step twice, writing into different object directories, then have a
  single packaging step that reads both outputs. Soong already has the
  capability of writing out a single ninja file with all the asan
  combinations.

* Running two build steps, one building the system-related modules
  using a "generic" device configuration, and one building the vendor
  modules using a specific device configuration. This could enforce a
  GSI/mainline system vs vendor split in a single build invocation.

* If all installation is through this tool, it will be much easier to
  track what should no longer be installed on an incremental build,
  reducing the need for installclean.

* Changing PRODUCT_PACKAGES should be a much faster operation, which
  means we could keep track of local additions to the images. Then
  `mma` would be more persistent, instead of installing something once,
  then never updating it again.

Eventually we plan on switching from Kati to something Go-based, but
this is a more incremental approach while we clean up everything else.

Currently, this just moves the dist-for-goal handling over to the
packaging step, so that we don't need to read Android.mk files when
DIST_DIR changes, or we switch between dist vs not.

Bug: 116968624
Bug: 117463001
Test: m nothing
Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-10-19 09:55:00 -07:00
Dan Willemsen
d50e89f389 Always ensure that DIST_DIR is set
Set the default value if one isn't provided.

Test: `get_build_var DIST_DIR` after removing the default setting in envsetup.mk
Change-Id: I0cb310fc65f7747c36de14608b61786ef6863fd1
2018-10-16 17:49:25 -07:00
Dan Willemsen
e9e20dd3e4 Remove perl from the allowed PATH tools list
This is no longer used within Android builds. We don't have any way of
providing extra libraries, or using a hermetic version of perl.

Bug: 117338567
Test: check build logs for the last 48 hours
Change-Id: I06d41a0006a47508a435dbf44bfccc4b385d9f41
2018-10-09 19:51:30 -07:00
Dan Willemsen
6afcd9aab9 Log uses of make and xmllint within the Android Build
Now that we've got prebuilts of make, turn on logging so that
we can find all the users and convert them to the prebuilt.

We also build xmllint as part of the tree, so log uses of that
too.

Test: treehugger
Change-Id: I8a6a5d481dfc709dc0eb3424c51430bc54763c1e
2018-10-09 23:57:45 +00:00
Elliott Hughes
ecdeb1e705 Remove sum(1) from the list of approved binaries.
No-one should be using BSD sum in 2018.

Bug: N/A
Test: treehugger
Change-Id: I9aa313e111217a2d73ba41eaf49035d8293b5785
2018-10-05 19:59:58 +00:00
Dan Willemsen
3eec9c57ca Switch perl to logging
To find the users and see if we can get rid of it.

Test: treehugger
Change-Id: I149f8daee6e21b35840ea1788de7f3e310eb5fba
2018-10-04 23:26:12 +00:00
Dan Willemsen
2997123f73 Refactor Kati logic
Make the cleanspec & Android.mk runs of kati use much of the same code
and arguments.

Also renames 'Kati' to 'KatiBuild' in many cases, in the the expectation
that we'll have a 'KatiPackage' step in the future.

Use --no_ninja_prelude and move local_pool & _kati_always_build_ into
the combined ninja file. This will reduce the need to re-read makefiles
when Goma is enabled, and it allows us to include more than one
Kati-generated ninja file in the build graph.

Bug: 116968624
Test: build_test on downstream branches
Change-Id: Ibdac689b81f62dc293647fad917d84946f2c3cfa
2018-09-28 23:09:24 -07:00
Treehugger Robot
c736cda13f Merge "Add OWNERS to finder's cache" 2018-09-26 21:37:54 +00:00
Anton Hansson
ecf0f10d3d Sanity check the tree before building.
Check for the presence of Android.mk or CleanSpec.mk, which
are somewhat common problems.

Bug: 113147143
Test: m (with and without files present)
Change-Id: I31cf60c325e7f6c6fce7aec54712c1cb802055c2
2018-09-19 22:48:48 +01:00
Dan Willemsen
1456958c83 Default BUILD_BROKEN_PHONY_TARGETS to false
All downstream users either mark BUILD_BROKEN_PHONY_TARGETS=true
explicitly, or don't have any warnings.

Test: build_test on downstream branches
Change-Id: I6e9d7ed42da941148827854851657895c2e98649
2018-09-10 13:06:43 -07:00
Simran Basi
df98cd7aa4 Add OWNERS to finder's cache
Build target owners will create a zip of all OWNERS
files in source code based on the cached OWNERS.list

Bug: 114242886
Test: make dist -j owners

Change-Id: I97fa3737e1a2a240a08e072dfb2905c11f28705b
2018-09-06 15:55:52 -07:00
Dan Willemsen
25a56186d9 Pass --writable to kati
These warnings are usually hit in one of the two conditions:

1. Targets that should be marked .PHONY, but are not.
2. Attempts to define output files in the source tree.

See https://android.googlesource.com/platform/build/+/master/Changes.md#phony_targets
for more information.

Test: multiproduct_kati in all downstream branches
Change-Id: Ibe37cce320612825c22987e94d0740e8c0565b6f
2018-09-03 12:32:34 -07:00
Dan Willemsen
090ec58af5 Add tee to the allowed PATH tools list
tee is used by art/build/Android.gtest.mk during testing

Change-Id: I45993366622694edef5494d9e74b8465a15a8795
2018-08-31 08:13:13 +00:00
Dan Willemsen
2d31ace93e Add od to allowed PATH tools
od is used by ninja's inline.sh tool to encode a python script in a C
header.

Test: forrest run of build-tools branch
Change-Id: I87001a02b168ebe183b8b7dc495c67bb43f1b86d
2018-08-30 16:53:02 -07:00
Dan Willemsen
8125d2a2a2 Start enforcing the $PATH restrictions
Anything that isn't explicitly marked as Allowed in this list will now
return an error (and log a message) when it is executed.

Test: search all logs from the build server over the last day
Change-Id: I3ceacd9a140097809dde81a8d8979dd2c45f234c
2018-08-30 06:20:26 +00:00
Dan Willemsen
6fa18f4e10 Add unix2dos as an allowed PATH tool
unix2dos is used instead of todos by some build servers in
development/build/tools/patch_windows_sdk.sh

Test: none
Change-Id: I04cb2258f71bebfc8792b7aa95643a45ffe3ec5c
2018-08-29 10:26:47 -07:00
Tom Cherry
27a509b017 Merge "Add $PRODUCT_OUT/ramdisk to installclean" 2018-08-29 17:05:40 +00:00
Treehugger Robot
de6eac1d8b Merge "Add todos to the allowed PATH list" 2018-08-28 23:56:13 +00:00
Dan Willemsen
7a08bc5467 Add todos to the allowed PATH list
todos is used by development/build/tools/patch_windows_sdk.sh

Test: m win_sdk; check out/soong.log
Change-Id: I5e1a290838cc34261d7f5fc321afe79a8b1a6fbb
2018-08-28 00:18:02 -07:00
Treehugger Robot
42ff416da3 Merge "Add fuser to allowed PATH tools" 2018-08-28 01:31:33 +00:00
Dan Willemsen
b4d0144a6f Add fuser to allowed PATH tools
fuser is used by goma

Test: none
Change-Id: I9acf4dd062c3b767a1688e4f236a96d5f2b8a7f6
2018-08-27 16:45:23 -07:00
Dan Willemsen
d8aa39d1c3 Support turning phony warnings into errors
These still default to warnings, but if a board decides to set
BOARD_BROKEN_PHONY_TARGETS := false, they'll turn into errors. More
likely I'll just be marking the broken targets as broken, then switching
the logic here to be like the dup rules warnings/errors.

Test: On a build with warnings, try <missing>, "false", and "true"
Change-Id: I041e1ff4618c4114ec43015b6c0ae1b49b36b6f8
2018-08-27 15:01:03 -07:00
Tom Cherry
7803a011d1 Add $PRODUCT_OUT/ramdisk to installclean
Bug: 79173823
Test: installclean removes this directory
Change-Id: I56dabba29064709112caad6549af7eacbe8469e8
2018-08-23 11:12:29 -07:00
Aaron Kling
d236e0e4f5 Add C.utf8 to supported locales
Several Linux distros (e.g. Fedora) , use C.utf8 instead of C.UTF-8

Test: builds
Change-Id: I7e3dd73ec89bc34d7f2dc1834082cc90f9521ad1
2018-08-21 18:06:58 +02:00
Yifan Hong
7bbacf3478 Merge "s/product-services/product_services/g" 2018-08-21 01:07:56 +00:00
Dario Freni
95cf767918 s/product-services/product_services/g
Attempting to reduce the number of different spellings we have for
"product services" partition in the codebase.

Bug: 112431447
Test: m
Change-Id: I0a393a1d625e7ea3217d28735a4db709bce32395
Merged-In: I0a393a1d625e7ea3217d28735a4db709bce32395
2018-08-20 17:46:57 +00:00
Dan Willemsen
00fcb26f2f Remove POSIXLY_CORRECT from the build environment
Setting this will cause our invocation of ninja to fail. Even if we
fixed that, it's likely to run into other differences during the build.

Fixes: 112443729
Test: POSIXLY_CORRECT=1 m nothing
Change-Id: I0315955c9fd9536cc6cb0d7c700eed207f35cc80
2018-08-15 15:37:07 -07:00
Dan Willemsen
42740f2c78 Add lsof and ps to the allowed tool lists
lsof is used by goma to find its proxy daemon
ps is used on Mac instead of pstree in soong_ui

Test: none
Change-Id: Ifed7d09d604c00b7eebac1488a6f4658788d6e33
2018-08-15 10:14:40 -07:00
Dan Willemsen
14eae19b1c Add du, paste, xxd to allowed PATH tools
du is used by build/make/tools/releasetools/build_image.py
paste is used by system/core/rootdir/update_and_install_ld_config.mk
xxd is used by device/generic/goldfish/tools/mk_qemu_image.sh

Test: none
Change-Id: I707668578d1babe1b149c2c3f989e5f8be37d897
2018-08-14 23:08:56 -07:00
Dan Willemsen
d2e231a147 Fix parsing of dangling dep check
Buildbot parses failure messages that look like ninja errors, but this
error doesn't look like a ninja error. We used to output this via
stderr, which buildbot would fall back to, but that had synchronization
issues between stderr and stdout.

So switch this over to use the status API, marking this action as failed
when necessary, which will display a ninja-like error that buildbot can
parse.

Bug: 112007097
Test: add dangling dep, look at output.
Change-Id: Ib2bbba4373ea22ac01af8aa6bbf295296a93e94d
2018-08-02 12:06:24 -07:00
Dario Freni
77d51c6e06 Add product-services to installclean
Test: m installclean
Bug: 80741439
Change-Id: I55f1e4fbe66430625fe32b724bbf21034c956eff
2018-07-31 18:29:49 +01:00
Treehugger Robot
e029b28d97 Merge "Turn on phony checks" 2018-07-24 17:58:11 +00:00
Dan Willemsen
f880b58970 Turn on phony checks
--warn_real_to_phony will print a warning any time that a non-PHONY rule
depends on a PHONY rule. Since PHONY rules are always considered dirty
this would force rebuilds on every incremental build. Real rules should
always depend on rules that have real outputs.

--warn_phony_looks_real will print a warning any time that a PHONY rule
has an output that looks like a file (has as / in it). This is can cause
confusion for users if it's not a real file, and it shouldn't be marked
as PHONY if it will be run every time (since that will slow down
incremental builds).

Test: run build_test on downstream branches, see few warnings
Change-Id: Id135c85c991976be33cd48a34eba0caaff1eda2c
2018-07-23 23:09:05 -07:00
Dan Willemsen
9dc69a43b0 Turn overriding command warnings into errors by default
All of the existing devices that have problems with this are setting
BUILD_BROKEN_DUP_RULES to true. Switch the behavior to default this to
an error.

Bug: 77611511
Test: check logs of all downstream build_test targets
Test: attempt to add an overriden command to hikey960 (uses default)
Change-Id: Ia83089f035925cf0c9883a1f593ae2da89fee568
2018-07-24 04:54:06 +00:00
Dan Willemsen
59339a29e1 Fix go vet issues
Test: go vet ./...
Change-Id: Ifb936ccc5e2b5a2c3fcbbbcb54f680e2973ea1b3
2018-07-22 21:18:45 -07:00
Dan Willemsen
027366751d Use --frontend_file for ninja instead of cat
Ninja now knows how to write directly to a file (or in our case, a named
pipe). This works around an issue we were seeing on Mac, where Go would
just hang after 50-2000 proto messages. It's also just a simpler
solution.

Bug: 111544015
Test: `m` with updated ninja on both Linux & Mac
Change-Id: Ic91920d83a6d2ea0b79e82b467e2423d78189f12
2018-07-17 17:54:31 -07:00
Dan Willemsen
77fe361175 Strip escape characters when TERM=dumb
Bug: 111447427
Test: m nothing
Test: TERM=dumb m nothing
Test: m nothing | tee log
Change-Id: I041ec8a60e633393a145996d9ba7b5dae2880b06
2018-07-16 10:56:10 -07:00
Dan Willemsen
f78a73444e Move all status output to stdout
I've noticed a few instances of interleaved status messages in between
lines in a terminal/Writer.Print call on our build servers. Since
there's a lock protecting everything we write, I've got to assume this
is a stdout vs stderr problem. Ninja had always been outputing to
stdout, except for error messages, which are now marked with FAILED:
like failed actions.

Test: m blueprint_tools
Test: m missing
Change-Id: Idf8320d40694abf212c902c63a9703e4440ffb7a
2018-07-12 21:59:49 -07:00
Dan Willemsen
9611199b94 Fix race condition and logging
The extra `defer os.Remove(fifo)` was sometimes racing with the next
instance of ninja, removing the file in between when it was re-created
and used. Since we're always removing the file before creating it, it's
safe to just remove that.

The error message for this failure wasn't all that good either, so move
so use the status Print/Error calls inside the goroutine instead of the
logging ones.

Test: `build/soong/build_test.bash -only-soong` repeatedly
Change-Id: Icfeb6b68802093bd3a07d3e46046ef7d1a89d4a1
2018-07-12 18:37:38 -07:00
Treehugger Robot
ecc71f8a49 Merge "Add a unified status reporting UI" 2018-07-12 23:14:44 +00:00
Dan Willemsen
b82471ad6d Add a unified status reporting UI
This adds a new status package that merges the running of "actions"
(ninja calls them edges) of multiple tools into one view of the current
state, and gives that to a number of different outputs.

For inputs:

Kati's output parser has been rewritten (and moved) to map onto the
StartAction/FinishAction API. A byproduct of this is that the build
servers should be able to extract errors from Kati better, since they
look like the errors that Ninja used to write.

Ninja is no longer directly connected to the terminal, but its output is
read via the protobuf frontend API, so it's just another tool whose
output becomes merged together.

multiproduct_kati loses its custom status routines, and uses the common
one instead.

For outputs:

The primary output is the ui/terminal.Status type, which along with
ui/terminal.Writer now controls everything about the terminal output.
Today, this doesn't really change any behaviors, but having all terminal
output going through here allows a more complicated (multi-line / full
window) status display in the future.

The tracer acts as an output of the status package, tracing all the
action start / finish events. This replaces reading the .ninja_log file,
so it now properly handles multiple output files from a single action.

A new rotated log file (out/error.log, or out/dist/logs/error.log) just
contains a description of all of the errors that happened during the
current build.

Another new compressed and rotated log file (out/verbose.log.gz, or
out/dist/logs/verbose.log.gz) contains the full verbose (showcommands)
log of every execution run by the build. Since this is now written on
every build, the showcommands argument is now ignored -- if you want to
get the commands run, look at the log file after the build.

Test: m
Test: <built-in tests>
Test: NINJA_ARGS="-t list" m
Test: check the build.trace.gz
Test: check the new log files
Change-Id: If1d8994890d43ef68f65aa10ddd8e6e06dc7013a
2018-07-12 14:15:31 -07:00
Colin Cross
7f09c40c49 Unset envsetup.sh variables inside build
Using ANDROID_BUILD_TOP and other envsetup.sh varialbes is already
forbidden in makefiles, but some tools run by the build (like
hidl-gen) will use it if it is set.  Since they are set by envsetup.sh,
and sourcing envsetup.sh before building is optional, unset them inside
the build to get consistent behavior.

Bug: 79250545
Test: m checkbuild
Change-Id: I3e7d9f01390ccdc34c49115f2f15cd7542d9974b
2018-07-11 15:18:29 -07:00
Colin Cross
63b4e0f5c1 Sort and uniqify dangling rules list
Make the dangling rules list sorted and unique in order to avoid
very long lists when a dangling rule is referenced many times.

Also prettify the output by indenting the list and printing
"stopping" instead of a blank line for the fatal.

Test: m checkbuild
Change-Id: I8f7c27ae39b59f506b529d9995d90b0d6b9835d1
2018-07-09 20:13:52 +00:00
Dan Willemsen
5af1cbe776 Ensure bpglob is built early
This matches the change to blueprint's blueprint_impl.bash:
https://github.com/google/blueprint/pull/213

It allows us to run bpglob during the two bootstrap phases, so we're
never running minibp or the primary builder in a later phase than
normal.

Also removes the dependency on out/soong/build.ninja from the main
phase, since it's never generated by the main phase anymore, and is
tripping the dangling dependency checks.

Bug: 73646380
Test: m nothing
Test: rm frameworks/base/core/java/android/content/pm/dex/ArtManager.java
Test: m nothing   (soong re-runs)
Change-Id: Ia3952d015ad6091ad5a841f555acda78c9390e84
2018-07-06 12:48:01 -07:00
Dan Willemsen
2bc456eff9 Switch implicit rules into errors
There are no users of these, other than the PDK, which is already
exempted.

Test: m nothing
Test: Check for warnings in downstream build_test logs
Test: Check for warnings in last two days of continuous build logs
Change-Id: I7c7d0068e928e591355c39d22635af86b8f43f1b
2018-06-21 21:41:35 -07:00
Colin Cross
5c733856bd Merge "Always use OpenJDK9 for building" 2018-06-21 21:27:16 +00:00
Colin Cross
8eb0ad15bf Merge "Run ErrorProne in OpenJDK9" 2018-06-21 20:47:41 +00:00
Colin Cross
997262f506 Always use OpenJDK9 for building
Remove support for compiling with javac from OpenJDK8.
We still target 1.8 by default, and OpenJDK8 prebuilts are still
required for the bootclasspath and running robolectric.

Bug: 38418220
Test: m java
Change-Id: I5686deb0ae4f9927192a039d08adc0117b2605dd
2018-06-21 12:25:50 -07:00
Dan Willemsen
b58f120496 Dump more BUILD_BROKEN_* values out to log
This is useful when determining which devices are setting which values
from the logs, cross-referencing where warnings still occur.

Test: look at out/soong.log
Change-Id: I3e3f0e4c8cbd9ce621b03f5163eff2cccfa8c424
2018-06-21 10:12:53 -07:00
Colin Cross
6654810f37 Run ErrorProne in OpenJDK9
Use the OpenJDK9 javac to run ErrorProne by adding it to
-processorpath and using -Xplugin:ErrorProne.

Bug: 69485063
Test: m RUN_ERROR_PRONE=true javac-check
Change-Id: I0496006b71b70766ef16d57753cbcf037897799c
2018-06-20 21:03:16 -07:00
Dan Willemsen
fa42f3c10d Turn implicit rules into warnings
Test: build_test in downstream branches
Change-Id: I853e3fa11793cc76ad568b74f7dc31c3d1c5582b
2018-06-20 16:50:45 -07:00
Dan Willemsen
d368d6ff78 Turn suffix rules into errors
We've turned off suffix rules with .SUFFIXES, but make them explicit
errors.

Test: build_test on downstream branches
Change-Id: I682ee3eb3a8f1451cd7e16f1e1d59afca5d21a26
2018-06-15 21:53:18 -07:00
Dan Willemsen
16dbb39afd PATH: Allow more tools
pkill and sleep are used in
device/generic/goldfish/tools/emulator_boot_test.sh

pgrep is used by GOMA

bc and pwd are used by kernel builds

Test: none
Change-Id: If7004255b4776fa4409fdd4b0aa6d6617b471317
2018-06-06 11:33:59 -07:00
Dan Willemsen
f6d300662e Allow more PATH tools, expose HOST_STRIP
atree was using the host's strip tool during sdk generation, so expose
the prebuilt for use instead.

`id` was used with a test running minijail0 during the build

`sha1sum` is used by development/build/tools/mk_sdk_repo_xml.sh

`rmdir` was being used to remove some tmp files

Test: m PRODUCT-sdk_arm64-sdk dist sdk_repo
Test: Run other builds through forrest
Change-Id: I49c756a3e9497c0bbd9030754e6aa193a2d989a4
2018-06-01 12:48:41 -07:00