Commit graph

289 commits

Author SHA1 Message Date
Dan Willemsen
9609ad9c21 Add sharding support to multiproduct_kati
This is so that we can split the build_test build on large branches into
multiple builds, each testing a subset of the products.

Test: m blueprint_tools  (runs the go unit tests)
Test: build/soong/build_test.bash --only-config
      vs
Test: build/soong/build_test.bash --only-config --shard-count=2
Test: build/soong/build_test.bash --only-config --shard-count=2 --shard=2

Change-Id: I40ccc1aa477bc0ffa74ff564d155068509be18f0
2019-12-05 19:54:40 -08:00
Colin Cross
e94272d0b3 Silence trailing "1 warning" message from javac when warning is silenced
soong_javac_wrapper is silencing a useless warning:
warning: [options] bootstrap class path not set in conjunction with -source 1.9
but recent versions of javac have started also printing:
1 warning

Read the warning count, subtract the number of silenced warnings, and
reprint it if the non-silenced warning count is nonzero.

Fixes: 144118634
Test: javac_wrapper_test.go
Change-Id: Ie1d0a978188ab7b1c41027f718a1274608628123
2019-11-11 21:48:33 +00:00
Colin Cross
c0b9f6b9d2 Force dumpvars into dumb terminal mode and move log files
Force soong_ui in dumpvars mode to use a dumb terminal so that it
doesn't print an empty status table.  Also move the logs to files
that are prefixed with "dumpvars-" so that they don't rotate the
real build logs.

Fixes: 139307849
Test: lunch
Change-Id: If92421af07731fedc9ca42670f309e95922a2224
2019-09-23 14:24:57 -07:00
Colin Cross
b5f6dc47ee Fix lint error in format string
Test: go test
Change-Id: I8b7df9222db1fc88938cdfa43a6505f2da854876
2019-09-11 17:06:02 +00:00
Dan Willemsen
f80ef34c80 Merge "Rewrite depfile from sbox to stay reproducible"
am: e2ce551909

Change-Id: I2a51549dfb4419b9b6444d33104d35e655c82f4f
2019-09-04 12:38:05 -07:00
Dan Willemsen
c89b6f1981 Rewrite depfile from sbox to stay reproducible
sbox will generate a random directory for the output root, and most
tools will encode that directory name in the output target of the
depfile.

So embed the library from dep_fixer into sbox so that it can rewrite the
output filename to a static (reproducible) value. Ninja doesn't care
what that value is, so it's just "outputfile".

Also fix up rule_builder to actually tell sbox about the depfile.

Test: mmma system/iorap; check the contents of:
out/soong/.intermediates/system/iorap/libiorap-binder/android_arm_armv7-a-neon_core_static/gen/aidl/system/iorap/binder/com/google/android/startop/iorap/IIorap.cpp.d

Change-Id: I3640a2e8b0c034f143a35e398a8418a6d621b265
2019-08-29 14:47:40 -07:00
Alexander Smundak
73d9ff761f Merge "Allow reading zipfile list from a file."
am: 02ed5df4ac

Change-Id: I2888ac8c6b03bcfc09739e5329bc872d92a2e9a1
2019-08-07 10:16:20 -07:00
Sasha Smundak
1459a9259e Allow reading zipfile list from a file.
Android build generates 180K+ zip files that are to be merged and the
result passed to the indexing pipeline to generate the source code
cross-reference. Their names cannot be passed on the command line,
hence the need for this change. That required extensive changes to
the code as it no longer can be assumed that all the input files can
be kept open.

Bug: 121267023
Test: internal + treehugger + combine index files
Change-Id: I0a829f96ff7187ad967fb6b8cede387501ec57cc
2019-08-06 14:23:35 -07:00
Dan Willemsen
57d5786d59 Merge "Remove support for ONE_SHOT_MAKEFILE"
am: d778b93168

Change-Id: I32ef15cf94df3524e4e805327d4af246c95226a6
2019-08-01 14:31:53 -07:00
Dan Willemsen
ce41e943be Remove support for ONE_SHOT_MAKEFILE
Test: treehugger
Change-Id: Icdbe62bcb7bc4717228c5d974962b939d8eafee6
2019-07-31 18:11:27 -07:00
Dan Willemsen
12efb0533e Merge "Deprecate ONE_SHOT_MAKEFILE"
am: 1d73e5e566

Change-Id: Ie7ec9690ada5333a04bae34b7be2b1921863e0fc
2019-07-29 13:19:26 -07:00
Dan Willemsen
c6360837bb Deprecate ONE_SHOT_MAKEFILE
mm is now doing the same thing as mma. Now print a deprecation message
for other tools that may be setting ONE_SHOT_MAKEFILE directly (or users
that are still using a very old envsetup.sh).

Test: mm  # does not print
Test: ONE_SHOT_MAKEFILE=bionic m
Change-Id: I98221657024d824a61c0d721d5c7dd1a2ea2fb1e
2019-07-25 14:12:38 -07:00
Tony Mak
6f716787d1 Merge "Allow other module types in androidx"
am: 6edff69513

Change-Id: I225050ef2c10850eee20820d4602399f5402557b
2019-07-22 14:36:55 -07:00
Tony Mak
817850063b Allow other module types in androidx
1. Update pom2bp to support "host_support" device module.
2. Allow other module type in support library

(The intention is to support the room annotation processor)

Test: m

Bug: 137918883

Change-Id: I9a8fbcff903cc395d5dc38430774202ce389d6bc
Merged-In: I9a8fbcff903cc395d5dc38430774202ce389d6bc
(cherry picked from commit bd7c2f9a1d)
2019-07-22 13:18:48 -07:00
Patrice Arruda
fe6a621684 Merge "soong-ui: Redirect mmm to mmma."
am: 216e315a34

Change-Id: If9a777fd46c06d06fedeee558d68653c72a726df
2019-07-19 04:50:34 -07:00
Treehugger Robot
216e315a34 Merge "soong-ui: Redirect mmm to mmma." 2019-07-18 21:58:24 +00:00
Patrice Arruda
0cacc0468f Merge "soong-ui: Redirect mm to mma."
am: 80b562eb5b

Change-Id: I4f24fd0a055c7d7e80ff61d4c5ed1cd695ea4d6e
2019-07-18 14:06:45 -07:00
Patrice Arruda
a9e1215870 soong-ui: Redirect mmm to mmma.
This is the first part of deprecating the mmm command. mmm will simply
be redirected to mmma by building the list of modules with their
dependencies.

Bug: b/135187558
Test: Ran mmm external/libusb:libusb.
Change-Id: I50cf0d6a07197556ab6130e612561c6d5a3fb31e
2019-07-18 09:43:20 -07:00
Patrice Arruda
2274102f34 soong-ui: Redirect mm to mma.
This is the first part of deprecating the mm command. mm will simply
be redirected to mma by building the modules and their dependencies.

Bug: b/135187558
Test: Ran mm in external/protobuf directory.
Change-Id: Idad40f89e3768aeb459172f632a44fd7b0a201bc
2019-07-18 09:41:19 -07:00
Justin Yun
9321804a11 Rename product_services to system_ext
am: d5f6c8261a

Change-Id: Ifd77d7d729addae59b2b26a8cbdd2ebefffcabae
2019-07-09 02:14:59 -07:00
Justin Yun
d5f6c8261a Rename product_services to system_ext
Bug: 134359158
Test: build and check if system_ext.img is created
Change-Id: Ice3229baf97a94c24a6eea22e7d4571941d7f843
2019-07-09 08:57:05 +00:00
Colin Cross
cd48600321 Merge changes Icfc893c8,I40f03fc0
am: f67e1bee4a

Change-Id: I0b9cf1e3df4ef7ef46e73a5dc6d363fe918c29b7
2019-07-03 22:55:03 -07:00
Colin Cross
aa812d122c Allow debugging with SOONG_DELVE=<listen addr>
Allow running Soong in a headless delve debugger by passing
SOONG_DELVE=<listen addr> in the environment.

Bug: 80165685
Test: SOONG_DELVE=:1234 m nothing
Change-Id: Icfc893c8a8354a9bbc99112d9c83259cb41906d1
2019-07-03 16:33:31 -07:00
Colin Cross
a73c8a4e61 Merge "Compute critical path when ninja finishes" am: 7c9b4cf144
am: 7b4a190539

Change-Id: I1f3869fa3d5afb437bab8a4bf77ea03934c1d4ab
2019-06-26 16:53:05 -07:00
Colin Cross
7b6245388c Compute critical path when ninja finishes
Keep a running map of the critical path to each edge as it
finishes.  At the end of the build, find the edge with the
longest critical path and print the path to out/soong.log.

Test: critical_path_test.go
Change-Id: Ie01d26b068b768156b166bff00cdc3273e8124ca
2019-06-26 11:31:17 -07:00
Patrice Arruda
55ac3ddb21 Merge "Soong: Add build_error proto raw file to $(OUT_DIR)." am: 7d7cae85ab
am: 9f9778cff0

Change-Id: I7d2ee16ad31446360142295c61562cff467f9d19
2019-06-24 16:43:43 -07:00
Patrice Arruda
297cebad26 Soong: Add build_error proto raw file to $(OUT_DIR).
Soong_ui produces a build_error raw file to $(OUT_DIR) where
the file contains a list of build action errors. Each build action
error represents an error raised by a command. The build_error file
is populated if there was a build error.

Bug: b/132969697
Test: Ran m for successful build. Introduced a broken build change,
      ran m and verified using printproto that build_error was
      generated successfully.

Change-Id: I690ca1778b4e56f144a3173ba1d16d8494c32c15
2019-06-24 13:02:35 -07:00
Patrice Arruda
4b557d8f86 Merge "Soong: Add BUILD_MODULES build action in soong_ui." am: 45cdb71290
am: d009ef35b8

Change-Id: I0216045ba79f0cc26a4de5883589f52a65d9919c
2019-06-20 20:59:32 -07:00
Patrice Arruda
3928206ecc Soong: Add BUILD_MODULES build action in soong_ui.
There was one case that did not work:
  1) Create dir in root source tree
  2) Run "m libc"

The build would fail as it was using the mma logic. A separate
one named BUILD_MODULES was added to allow building specific
modules in any directories using the "m" command.

Bug: b/130049705
Test: unit test cases, ran "m libc" in a tmp directory inside
      of the source tree.

Change-Id: I8d23e685a673a311001ee8edd89bd73b662392dd
2019-06-20 16:35:12 -07:00
Patrice Arruda
5dbf869760 Soong: Use ctx.Writer in flag usage for dumpVar and dumpVars in soong_ui. am: db4c2f115e
am: e6fe9edbef

Change-Id: Ifcd0332f5f055633e4284c3387beea84cedf86a7
2019-06-18 15:52:12 -07:00
Patrice Arruda
db4c2f115e Soong: Use ctx.Writer in flag usage for dumpVar and dumpVars in soong_ui.
ctx.Writer is recommended stream to use for standard io.

Fixes: b/135487952
Test: Lunch and executed ./out/soong_ui --dumpvar-mode and
--dumpvars-mode to check the usage is output to the console.

Change-Id: I38351ebdb05f49f92d6b4bfb0423092299d1c5ea
2019-06-18 20:32:01 +00:00
Patrice Arruda
268a702f06 Merge "soong_ui: Add --build-mode flag in soong_ui" am: 2f93a2e07b
am: 9123c55392

Change-Id: Ib7ca530857ad469da3e634b7115a308b6d0d1972
2019-06-17 21:25:36 -07:00
Patrice Arruda
b7b2282fd1 soong_ui: Add --build-mode flag in soong_ui
The --build-mode flag is a new flag in soong_ui that accepts a build
action (m, mm, mmm, mma, mmma), the directory where the build action
is occuring and an optional list of build arguments and invokes the
build option. This is to deprecate the build actions behavior in
envsetup.sh and start using the one defined in soong_ui.

Bug: b/130049705
Test: Ran lunch and ran all the build action commands in soong_ui.
      Ran a bunch of invalid commands such as no Android.mk file in
      a specific directory, invalid target name, etc...

Change-Id: I88e9e59f47f8c5cdc872fb83a0f3f0a0919885b6
2019-06-17 17:21:32 -07:00
Patrice Arruda
10b659a60f Merge "Soong: Fix the package name in metrics.proto file" am: eec8d3aee3
am: 33989c88c3

Change-Id: Ife7e9c2ddeeeda40976c23f8ba07ba4bee53935c
2019-06-17 17:12:42 -07:00
Patrice Arruda
0cc5b21516 Soong: Fix the package name in metrics.proto file
The metrics.proto needs to be imported to Google3 in order to unmarshal
the build metrics data from bigstore. Cleaned up the enum names to use
the Camel naming convention and renamed the build_metrics to
soong_metrics to be more specific to soong and to allow quering
soong_metrics files from bigstore.

Bug: b/135280521
Test: lunch andf m -j. Checked if soong_metrics was generated correctly
      by using printproto command.

Change-Id: I998c8d05db592e94a653d6ca32250b80df3c9b21
2019-06-14 15:27:46 -07:00
Colin Cross
ceec6a2295 Merge changes I555ad21a,I9d46761e,I917bdeee,I5bc5476a,I59bbdae4, ... am: 6ebe07def7
am: 1337decdff

Change-Id: I2f27851391ad68137cc66133de575d75a3c83a74
2019-06-14 01:30:19 -07:00
Colin Cross
e0df1a36b2 Move all output through StatusOutput
Write log output through StatusOutput so that the status implementation
can synchronize it with its own output.

Test: status_test.go
Change-Id: I917bdeeea4759a12b6b4aa6d6d86ee18a2771723
2019-06-12 21:30:56 -07:00
Colin Cross
097ed2a37c Remove terminal.Writer
terminal.Writer is now just a wrapper around stdio.Stdout() without
any useful functionality.  Replace it with stdio.Stdout() as an
io.Writer.

Test: ui/terminal/status_test.go
Change-Id: I5bc5476afdca950b505642f0135a3af9d37fbe24
2019-06-12 21:30:56 -07:00
Dan Willemsen
7488174f2c Merge "Refactor .aar resource compilation" am: 588857b418
am: 2758050dc4

Change-Id: I98dfbdbef71972a040fd53f9e77eb321fb6f5e0e
2019-06-10 21:16:55 -07:00
Dan Willemsen
304cfec778 Refactor .aar resource compilation
Instead of extracting the aar, and passing the res directory to another rule
(along with a dependency to a different known file), don't keep the
extracted res directory, and pass the aar directly to the resource
compliation rule, which will extract just the res directory.

I need this for my RBE experiments, where non-listed output files won't
exist in other rules.

Test: m
Change-Id: I99074381052cbcebb6a402484abae9ab2e40284f
2019-05-29 16:54:43 -07:00
Patrice Arruda
d2e1149cc5 resolve merge conflicts of a3898089ba to qt-dev-plus-aosp
Bug: None
Test: Ran soong_ui.bash which compiles the soong_ui.
Change-Id: I0b8083e3e5ee69c8cbf17443fbc038ec7b5a0b10
2019-05-06 19:24:32 -07:00
Treehugger Robot
565ee6ed87 Merge "Soong: Refactor the soong_ui arguments processing to be more like bazel." 2019-05-07 00:49:21 +00:00
Patrice Arruda
a5c2542f68 Soong: Refactor the soong_ui arguments processing to be more like bazel.
Currently, the command line argments is being processed in multiple
places. In the main soong_ui arguments, there are several if statements
that checks if a specific argument was specified in order to execute
the requested operation. This does not scale well when adding or
removing a command in the near future. In order to support the build
commands (m, mma, etc...) from the envsetup.sh in soong_ui, a refactor
was required in order to add a command rather quickly and to have the
flexibiity to unit test the command. The soong_ui arguments format is
as follows:

soong_ui <command> [<arg 1> <arg 2> ... <arg n>]

The <command> is a specific operation to be executed. The arguments
after the command are processed by the command itself.

Below is the list of changes made in this commit:

 * Created a new command infrastructure that allows adding
   or deprecating a command easily.

 * Fixed a bug when running ./soong_ui.bash directly would cause
   a panic due to index being out of range for args list.

Bug: b/130049705
Test: Below is the list of testing done on this commit:
    * Ran lunch and verified that the output is the same as the the output
      without the modifications. lunch indirectly runs soong_ui passing
      in the --dumpvar-mode (to read makefile variables such as
      TARGET_PRODUCT) and --dumpvars-mode (to build the build cache).
    * Ran ./soong_ui.bash directly (with unsupported flags and no flags)
      and the proper message appeared that soong native UI is not yet
      available.
    * Ran m, mm, mmm, mma, mmma commands.
    * Ran the make installclean command:
         make -j80 TARGET_PRODUCT=aosp_arm64 TARGET_BUILD_VARIANT=eng dist DIST_DIR=/tmp/helloworld installclean
    * Ran ./out/soong_ui -j80 --make-mode PRODUCT-aosp_arm64-eng dist checkbuild tests

Change-Id: Iee4de7ec0fa4661206fda8ae1fe6fa4487d9bb22
Merged-In: Iee4de7ec0fa4661206fda8ae1fe6fa4487d9bb22
2019-05-06 14:05:52 -07:00
Colin Cross
f41ed9ae63 Merge "Remove services.art from known_nondeterminism.whitelist" am: 42603c4ab5
am: 828a488720

Change-Id: Idbdeec76c0d8207890afbf94ea20ebac4c3d5379
2019-04-30 15:24:25 -07:00
Colin Cross
497d4c24b1 Remove services.art from known_nondeterminism.whitelist
Bug: 120039850
Test: none
Change-Id: I70d5243f427a4da5c8126be6b8ebc90221133ad3
2019-04-29 17:36:19 +00:00
Dan Willemsen
7d155094be Merge "pom2bp: Implement -static-deps to match pom2mk" am: 80ae6816ae
am: 9181e08738

Change-Id: I331a7bb1cd82315f673d9007c2240da423864034
2019-04-22 20:17:35 -07:00
Dan Willemsen
52c90d868e pom2bp: Implement -static-deps to match pom2mk
This changes the default behavior of pom2bp to match pom2mk, and not
require pre-extraction of all of the manifests.

Test: convert an internal pom2mk user to pom2bp
Change-Id: I08120cce1f923ef28519563ffd772032b2e500ed
2019-04-21 21:46:42 -07:00
Dan Willemsen
8a4c0da74d Add -jetifier to pom2mk and pom2bp am: 7fdab6eeec
am: fd9fa1a56e

Change-Id: Ifaa39849ef2d858dd63745e2a8a0b9e5587bbd66
2019-04-21 02:47:34 -07:00
Dan Willemsen
7fdab6eeec Add -jetifier to pom2mk and pom2bp
Test: use pom2mk -jetifier to replace some hand-modified makefiles
Change-Id: I2a1988d0120fee1edf2d85c8b3294041840efe7c
2019-04-20 21:47:14 -07:00
Paul Duffin
0018e7f48b Merge "Rename -extra-deps to --extra-static-libs and add --extra-libs" am: 38b122021a
am: 2b9f534fba

Change-Id: I1a607c72ad7f1a71b7dcc0aa216beee2b868776f
2019-04-17 07:32:57 -07:00