Commit graph

202 commits

Author SHA1 Message Date
Nan Zhang
17f2767724 Generate build timing metrics to proto format file
Test: Dumped the text formated based metrics file to out dir,
and checked the file.
Bug: b/63815990

Change-Id: Iff476f72a0be74eb53b6b26ef468d11c0f24a404
2019-01-04 15:54:01 -08:00
Dan Willemsen
7f30c076d5 Print a status message when starting ninja
This way we don't appear hung at:

  No need to regenerate ninja file

Change-Id: I8dbdaa2c1b1c5a6a73187d0e6061f363b62e10c9
Fixes: 122251150
Test: m nothing
2019-01-02 12:50:49 -08:00
Elliott Hughes
9add81ed0f Move cut(1) to toybox.
Test: treehugger
Change-Id: I07dd7a45ee56178a72deaffc6c4796c3ec53f564
2018-12-11 09:39:48 -08:00
Elliott Hughes
734a780217 Move touch(1) to toybox.
Test: treehugger
Change-Id: If31fe1fdc0150d07d5421702cbac8e142f522802
2018-12-07 18:30:52 -08:00
Elliott Hughes
cc85770628 Move mkdir(1) to toybox.
Test: treehugger
Change-Id: I5fcbf33acb93cdc279a900586b96e0232c4aea87
2018-12-06 22:30:45 -08:00
Elliott Hughes
ed46164d03 Move head(1) to toybox.
Test: clean build
Change-Id: I264c320b4a63e9b0f440791dc20ff99ec910036a
2018-12-07 02:36:16 +00:00
Elliott Hughes
2ebfd495cf Move dirname(1) to toybox.
Test: treehugger
Change-Id: Ie1ffbfb900803940620dd11a7ed6bfda76bee86c
2018-12-06 08:57:31 -08:00
Elliott Hughes
5172a8b50e Move to toybox unix2dos(1).
This removes the need for the confusingly named "todos".

Test: treehugger
Change-Id: Id8931deb00a06d54262b2803a00d13bd4de88f12
2018-12-05 19:42:43 -08:00
Elliott Hughes
6646904a47 Move to toybox cmp(1).
Test: treehugger
Change-Id: Idf5c9f00d0698aca6ceab2956c7dc49ad7c604e1
2018-12-05 10:03:31 -08:00
Elliott Hughes
6141a7d7ff Move to toybox basename(1).
Test: treehugger
Change-Id: I29f210d86a0dada8a40eca1919e2f6f876d50bcd
2018-12-04 13:42:46 -08:00
Elliott Hughes
5728fb536f Move setsid(1) to toybox.
Test: treehugger
Change-Id: I8caf9dbbb671218379c309e698f23e7c964f1a18
2018-12-03 10:14:52 -08:00
Elliott Hughes
1ff2b5d34c Move tee(1) to toybox.
Test: treehugger
Change-Id: Idf7fc37d6e84c295410f71dfc1ad9e460d7f7461
2018-11-30 12:36:44 -08:00
Elliott Hughes
f3f3b99bb3 Move xxd(1) to toybox.
Test: treehugger
Change-Id: I906b0c94da5b65dc387ae253492356ca78f93fd2
2018-11-29 22:04:16 -08:00
Elliott Hughes
15f2dfe482 Move paste(1) to toybox.
Test: treehugger
Change-Id: Id5e8e841406357d8301198eb433b96d2f1ba02d9
2018-11-29 11:36:51 -08:00
Elliott Hughes
716fa36dd9 Move rmdir(1) to toybox.
Test: treehugger
Change-Id: Ibb43f09c0cfcda631025ef0fea4730519a3b6fe6
2018-11-28 14:14:35 -08:00
Elliott Hughes
10412ec50b Move pwd(1) to toybox.
Test: treehugger
Change-Id: I656e322724d13273bf1099c6cc311a520aa4173e
2018-11-27 18:04:34 -08:00
Elliott Hughes
d50d40fe07 Move tail(1) to toybox.
Test: treehugger
Change-Id: If421513911208db4542c4767168ce50b7b837af8
2018-11-27 16:29:17 -08:00
Elliott Hughes
505b76136b Move od(1) to toybox.
Test: treehugger
Change-Id: I19587d3045992fa8f60743ef358c0fd050b43dc0
2018-11-27 09:00:21 -08:00
Elliott Hughes
bb19518a54 Move env(1) to toybox.
Test: clean build
Change-Id: I52770c022f7d34fb83c990f485b0ca3ea29e97ee
2018-11-15 13:55:56 -08:00
Elliott Hughes
6bdad22f7b Move comm(1) to toybox.
Test: clean build
Change-Id: I2d78388ef7211af6d517702c97f3b95ed2f30a46
2018-11-14 20:49:52 -08:00
Elliott Hughes
2907ceaae2 Move sleep(1) to toybox.
Test: clean build
Change-Id: I724278466c4d1367add79f949d1869e1e50a192d
2018-11-14 14:30:56 -08:00
Elliott Hughes
7a907c9291 Move uniq(1) to toybox.
Test: clean build
Change-Id: I4819098fd298dbc4f3e424e437a448e1fb1baeaf
2018-11-13 22:12:30 -08:00
Elliott Hughes
5d593238e2 Move cat(1) to toybox.
Test: builds
Change-Id: I9a367c3204891518994bfe889072dd1cf9aed490
2018-11-13 13:08:13 -08:00
Elliott Hughes
9b370f5ada Move id(1) and whoami(1) to toybox.
Test: builds
Change-Id: I32347c94438c394eb31f5af9f51cee11ad0ef247
2018-11-11 10:57:58 -08:00
Dan Willemsen
417be1fa9f Start using the toybox prebuilt
Prepend the toybox symlink directory to $PATH, and stop generating
symlinks into out/.path for tools used from toybox.

We don't (yet?) have toybox prebuilts for darwin, so apply the above only to Linux, and preserve existing behavior on Darwin.

Test: check the build banner for uname results
Test: m
Change-Id: I37fc380381e65a628cdc131d462fd4441eacfe9d
2018-10-30 23:25:35 -07:00
Treehugger Robot
a758cda418 Merge "Speed up build_test" 2018-10-30 20:27:42 +00:00
Dan Willemsen
f99915f569 Speed up build_test
Stop writing out ninja files, as they become very large, especially when
multiplied by the number of defined products.

Test: treehugger
Change-Id: Id9529d14040acb72a0188e58b5db2911f142071e
2018-10-29 10:43:37 -07:00
Elliott Hughes
be3cfa514a Disallow "runalarm" and allow the more standard "timeout".
Bug: N/A
Test: builds
Change-Id: Id8fe8ee2569b7410c66f448513cce6ff57458ffb
2018-10-27 08:09:18 -07:00
Dan Willemsen
6f03752422 Allow DIST_DIR to be read via dumpvars
DIST_DIR is controlled by soong_ui, and the make dumpvars code no longer
has access to it. So instead, handle some dumpvar requests inside
soong_ui itself.

Test: get_build_var DIST_DIR
Test: get_build_var OUT_DIR
Test: get_build_var BUILD_SYSTEM
Test: build/soong/soong_ui.bash --dumpvars-mode --vars="DIST_DIR
BUILD_SYSTEM"
Change-Id: Id3bcb8b0748db67c83d0e42d1ae5be564c5eb2f7
2018-10-21 09:26:55 -07:00
Dan Willemsen
2d31a44b8a Stop using DIST_DIR in Soong
We're only using it to distribute files in case of failure, which isn't
well supported currently, but can be handled for now by using the
DIST_DIR environment variable during the command execution.

This was at least one cause that we'd be re-running Soong during every
build server build, as the DIST_DIR values are unique.

Test: m dist
Change-Id: Ibd5e6b6c46695350de80b745bfb6a6aa685033a0
2018-10-20 21:33:41 -07:00
Dan Willemsen
d4ad1da3d7 Merge "Add a Kati-based packaging step" 2018-10-20 22:50:10 +00:00
Dan Willemsen
41cabfbbf4 Remove make and xmllint from the allowed tools list
We've got a prebuilt of make in prebuilts/build-tools, and we build
xmllint from source.

Test: No logs on the build servers in the last 24 hours
Change-Id: Ib60a7b5feaa31097071b67ebf6beb7f0b9a9f81b
2018-10-19 12:29:46 -07:00
Dan Willemsen
fb1271a52b Add a Kati-based packaging step
The idea is that we'd move the installation and packaging tasks over to
it, using data from Soong & the Kati reading Android.mk files.

This would allow us to make more fundamental changes about how we
package things without having to adjust makefiles throughout the tree.
Possible use cases:

* Moving some information from Soong's Android.mk output to a file read
  by the packaging step may allow us to read the Android.mk files less
  often, speeding up builds.

* Refactoring our current two-stage ASAN builds to run the Kati build
  step twice, writing into different object directories, then have a
  single packaging step that reads both outputs. Soong already has the
  capability of writing out a single ninja file with all the asan
  combinations.

* Running two build steps, one building the system-related modules
  using a "generic" device configuration, and one building the vendor
  modules using a specific device configuration. This could enforce a
  GSI/mainline system vs vendor split in a single build invocation.

* If all installation is through this tool, it will be much easier to
  track what should no longer be installed on an incremental build,
  reducing the need for installclean.

* Changing PRODUCT_PACKAGES should be a much faster operation, which
  means we could keep track of local additions to the images. Then
  `mma` would be more persistent, instead of installing something once,
  then never updating it again.

Eventually we plan on switching from Kati to something Go-based, but
this is a more incremental approach while we clean up everything else.

Currently, this just moves the dist-for-goal handling over to the
packaging step, so that we don't need to read Android.mk files when
DIST_DIR changes, or we switch between dist vs not.

Bug: 116968624
Bug: 117463001
Test: m nothing
Change-Id: Idec5ac6f7c7475397ba0fb65bd3785128a7517df
2018-10-19 09:55:00 -07:00
Dan Willemsen
d50e89f389 Always ensure that DIST_DIR is set
Set the default value if one isn't provided.

Test: `get_build_var DIST_DIR` after removing the default setting in envsetup.mk
Change-Id: I0cb310fc65f7747c36de14608b61786ef6863fd1
2018-10-16 17:49:25 -07:00
Dan Willemsen
e9e20dd3e4 Remove perl from the allowed PATH tools list
This is no longer used within Android builds. We don't have any way of
providing extra libraries, or using a hermetic version of perl.

Bug: 117338567
Test: check build logs for the last 48 hours
Change-Id: I06d41a0006a47508a435dbf44bfccc4b385d9f41
2018-10-09 19:51:30 -07:00
Dan Willemsen
6afcd9aab9 Log uses of make and xmllint within the Android Build
Now that we've got prebuilts of make, turn on logging so that
we can find all the users and convert them to the prebuilt.

We also build xmllint as part of the tree, so log uses of that
too.

Test: treehugger
Change-Id: I8a6a5d481dfc709dc0eb3424c51430bc54763c1e
2018-10-09 23:57:45 +00:00
Elliott Hughes
ecdeb1e705 Remove sum(1) from the list of approved binaries.
No-one should be using BSD sum in 2018.

Bug: N/A
Test: treehugger
Change-Id: I9aa313e111217a2d73ba41eaf49035d8293b5785
2018-10-05 19:59:58 +00:00
Dan Willemsen
3eec9c57ca Switch perl to logging
To find the users and see if we can get rid of it.

Test: treehugger
Change-Id: I149f8daee6e21b35840ea1788de7f3e310eb5fba
2018-10-04 23:26:12 +00:00
Dan Willemsen
2997123f73 Refactor Kati logic
Make the cleanspec & Android.mk runs of kati use much of the same code
and arguments.

Also renames 'Kati' to 'KatiBuild' in many cases, in the the expectation
that we'll have a 'KatiPackage' step in the future.

Use --no_ninja_prelude and move local_pool & _kati_always_build_ into
the combined ninja file. This will reduce the need to re-read makefiles
when Goma is enabled, and it allows us to include more than one
Kati-generated ninja file in the build graph.

Bug: 116968624
Test: build_test on downstream branches
Change-Id: Ibdac689b81f62dc293647fad917d84946f2c3cfa
2018-09-28 23:09:24 -07:00
Treehugger Robot
c736cda13f Merge "Add OWNERS to finder's cache" 2018-09-26 21:37:54 +00:00
Anton Hansson
ecf0f10d3d Sanity check the tree before building.
Check for the presence of Android.mk or CleanSpec.mk, which
are somewhat common problems.

Bug: 113147143
Test: m (with and without files present)
Change-Id: I31cf60c325e7f6c6fce7aec54712c1cb802055c2
2018-09-19 22:48:48 +01:00
Dan Willemsen
1456958c83 Default BUILD_BROKEN_PHONY_TARGETS to false
All downstream users either mark BUILD_BROKEN_PHONY_TARGETS=true
explicitly, or don't have any warnings.

Test: build_test on downstream branches
Change-Id: I6e9d7ed42da941148827854851657895c2e98649
2018-09-10 13:06:43 -07:00
Simran Basi
df98cd7aa4 Add OWNERS to finder's cache
Build target owners will create a zip of all OWNERS
files in source code based on the cached OWNERS.list

Bug: 114242886
Test: make dist -j owners

Change-Id: I97fa3737e1a2a240a08e072dfb2905c11f28705b
2018-09-06 15:55:52 -07:00
Dan Willemsen
25a56186d9 Pass --writable to kati
These warnings are usually hit in one of the two conditions:

1. Targets that should be marked .PHONY, but are not.
2. Attempts to define output files in the source tree.

See https://android.googlesource.com/platform/build/+/master/Changes.md#phony_targets
for more information.

Test: multiproduct_kati in all downstream branches
Change-Id: Ibe37cce320612825c22987e94d0740e8c0565b6f
2018-09-03 12:32:34 -07:00
Dan Willemsen
090ec58af5 Add tee to the allowed PATH tools list
tee is used by art/build/Android.gtest.mk during testing

Change-Id: I45993366622694edef5494d9e74b8465a15a8795
2018-08-31 08:13:13 +00:00
Dan Willemsen
2d31ace93e Add od to allowed PATH tools
od is used by ninja's inline.sh tool to encode a python script in a C
header.

Test: forrest run of build-tools branch
Change-Id: I87001a02b168ebe183b8b7dc495c67bb43f1b86d
2018-08-30 16:53:02 -07:00
Dan Willemsen
8125d2a2a2 Start enforcing the $PATH restrictions
Anything that isn't explicitly marked as Allowed in this list will now
return an error (and log a message) when it is executed.

Test: search all logs from the build server over the last day
Change-Id: I3ceacd9a140097809dde81a8d8979dd2c45f234c
2018-08-30 06:20:26 +00:00
Dan Willemsen
6fa18f4e10 Add unix2dos as an allowed PATH tool
unix2dos is used instead of todos by some build servers in
development/build/tools/patch_windows_sdk.sh

Test: none
Change-Id: I04cb2258f71bebfc8792b7aa95643a45ffe3ec5c
2018-08-29 10:26:47 -07:00
Tom Cherry
27a509b017 Merge "Add $PRODUCT_OUT/ramdisk to installclean" 2018-08-29 17:05:40 +00:00
Treehugger Robot
de6eac1d8b Merge "Add todos to the allowed PATH list" 2018-08-28 23:56:13 +00:00