Commit graph

605 commits

Author SHA1 Message Date
Treehugger Robot
789093a9ad Merge "Direct Bazel builds from m." 2021-06-21 08:16:21 +00:00
Jingwen Chen
c63677b3c9 Direct Bazel builds from m.
This CL adds support to bp2build/Soong to dump a BUILD file under
out/soong/soong_injection/targets containing alias targets to their real
targets for every converted Soong module, regardless of whether they are
handcrafted or generated.

Test: TH
Change-Id: Ic1816fda5d019c395301618134fac68b3057d752
2021-06-21 06:12:34 +00:00
Colin Cross
9cfe6119fe Use a linker script for host bionic embedded linker sections
Use an implicit linker script instead of flags in a file to specify
the locations of the host bionic embedded linker and to prevent it
from being stripped.

Test: build and run host bionic binary
Change-Id: I64e12118d33c67bab5e657cbc3ea8cde8f0fd7e6
2021-06-11 18:02:22 -07:00
Colin Cross
009f3df380 Give extracted linker sections pretty names
Replace .linker.sect0, etc. with .linker, .linker.text, .linker.data
and .linker.data.rel.ro by extracting the name of the first section
in each program header.

Test: build and run host bionic binary
Change-Id: I25107547536a3a3963fdeb311c45a7ee53c0bc45
2021-06-11 17:57:09 -07:00
Colin Cross
f04eb99acc Stop injecting symbols into host bionic binaries
The host bionic bootstrapping no longer needs an injected symbol.
Replace host_bionic_inject with host_bionic_verify that validates
the resulting binary, and add it as a validation dependency of the
binary.

Test: build and run host bionic binary
Change-Id: I3e303d2a164b6eef851bdc8075e6ee456c05b0a8
2021-06-11 15:22:41 -07:00
Anton Hansson
5a7861a272 Various cleanup in soong_ui to aid new feature
- Rename the "BuildX" variables to "RunX"
- Remove redundant comments
- Inline all the "what to do" based on config in build.go
- Inline some constants only used in one place

Bug: 189187214
Test: m nothing
Test: build/soong/build_test.bash
Change-Id: I111a69e642212d7938d4971283545e0d9acbb01a
Merged-In: I111a69e642212d7938d4971283545e0d9acbb01a
(cherry picked from commit d274ea9196)
2021-06-07 14:34:40 +01:00
Colin Cross
7cbf1d68ec Merge "Fix propagating EMPTY_NINJA_FILE from multiproduct_kati to minibp" 2021-06-01 21:55:25 +00:00
Colin Cross
f3bdbcbea3 Fix propagating EMPTY_NINJA_FILE from multiproduct_kati to minibp
I6dca478f356f56a8aee1e457d71439272351390b replaced calling
build/blueprint/bootstrap.bash with running minibp directly,
but didn't propagate the EMPTY_NINJA_FILE environment variable
to minibp.  Since everything that uses EMPTY_NINJA_FILE is
executed directly from soong_ui now, replace the EMPTY_NINJA_FILE
environment variable with a config bool and pass it to minibp.

Bug: 189148777
Test: build/soong/build_test.bash --dist --incremental --shard-count=300 --shard=1 && du out/multiproduct
Change-Id: I4d64275ce02c5d68948012f71ac4dc3795af9e85
2021-06-01 11:43:55 -07:00
Rupert Shuttleworth
e03bb61072 Use Finder for getPathsToIgnoredBuildFiles() to try and speed up builds with slow hard drives.
NOTE: This relies on https://chromium-review.googlesource.com/c/angle/angle/+/2906252 being merged into AOSP first

Test: bazel build //bionic/... //external/... //frameworks/... //system/...

Change-Id: Id552941274ffbc9955dfcd8f545a58f1e3368845
2021-05-26 16:11:27 -04:00
Dan Willemsen
5ac072371e Merge "Add go2bp tool" 2021-05-26 17:03:35 +00:00
Jingwen Chen
4fabaf52b0 bp2build/b: exit early in GENERATE_BAZEL_FILES=1.
This CL fixes a typo in writeFakeNinjaFile to correctly write
a fake out/soong/build.ninja and its depfile. It also modifies bootstrap
phase to *not* run the main soong build phase (which takes more than a
minute) if GENERATE_BAZEL_FILES=1.

This change has the side effect that `GENERATE_BAZEL_FILES=1 m nothing`
no longer generates the real build.ninja, which is fine because one
shouldn't be using GENERATE_BAZEL_FILES=1 for that anyway (or, use
USE_BAZEL_ANALYSIS=1). This change has no effect on mixed builds.

Time on a change to Soong or any Android.bp files:

Before: bp2build_workspace_marker (~20 seconds) + build.ninja (1 min)
After: bp2build_workspace_marker (~20 seconds)

Time on the second of two consecutive `b build //bionic/...`: 2.070s

Test: TH
Test: Soong integration tests
Change-Id: I43720641815994caba97b8d165d7c3fc254cbd06
2021-05-25 03:42:38 +00:00
Dan Willemsen
beb51ac64c Add go2bp tool
This tool automatically writes Android.bp files based on information
discovered from Go, similar to pom2bp or Bazel's Gazelle. This
implementation is fairly limited, but is usable enough to import
libraries like protobuf without a lot of manual effort.

Change-Id: Iecc145bc7ab9b9e9c3739f77b694b3f4bc2c6396
2021-05-24 16:06:52 -07:00
Rupert Shuttleworth
0096079bbc Make bazel build //external/... work by only allowing "approved" existing BUILD files in the Bazel workspace.
Test: bazel build //bionic/... //external/... //frameworks/... //system/...

Test: ./build/bazel/scripts/run_presubmits.sh

Change-Id: I91865ca87c6535053e7a14d2526ff3ce0991bfea
2021-05-17 06:34:53 -04:00
Andrew Walbran
75bba11bbc Output machine-readable file with all soong keywords.
Test: make, looked at output
Change-Id: Id0427e1a4f04515493c4dadb9e2856518e3a08dd
2021-05-11 11:10:25 +00:00
Colin Cross
71d6ab6827 Fix concurrency issues in Test_runWithTimeout
Use a concurrency-safe writer in runWithTimeout to avoid data races
on the bytes.Buffer passed in during tests.

Bug: 181095653
Fixes: 187149270
Test: Test_runWithTimeout
Test: go test -race ./cmd/run_with_timeout
Change-Id: I57a889765cb9ee7b42983f0906313e0c2d1e414e
2021-05-04 09:15:30 -07:00
Colin Cross
9b6bcc6bff Add a wrapper command to detect timeouts
Add a command that can be used to wrap actions with a timeout, and
optionally run an extra debugging command on timeout.

Bug: 181095653
Test: run_with_timeout_test.go
Change-Id: I91df5c3fb5277968717815a4ad4612113766dab1
2021-04-30 14:14:45 -07:00
Rupert Shuttleworth
2a4fc3ecdc Generate BUILD files for every directory that has an Android.bp file.
Test: Added an integration test
Test: bazel build --package_path=out/soong/workspace //bionic/...

Change-Id: Ie34bd23ab3c5428e6c9c9919e5fb6fcb4e709adc
2021-04-21 07:15:50 -04:00
Christopher Parsons
4e78814200 Merge "Subsume INTEGRATED_BP2BUILD into other env modes" 2021-04-20 23:33:40 +00:00
Chris Parsons
ec1a3dc6f8 Subsume INTEGRATED_BP2BUILD into other env modes
This refactors bazel-build mode determination logic in soong_ui so it's
clearer which of three possible modes are being used in a given
invocation (NO_BAZEL, GENERATE_BUILD_FILES, or MIXED_BUILDS).

Test: bootstrap tests
Change-Id: I41d2baebf8d560c2cc42db8daa8b936101d453e3
2021-04-20 19:25:48 -04:00
Colin Cross
14d09018eb Merge "Update for sharded globs" 2021-04-20 16:16:40 +00:00
Lukács T. Berki
3784448c9e Merge "Create a synthetic Bazel workspace." 2021-04-20 10:44:08 +00:00
Lukacs T. Berki
b353cca496 Create a synthetic Bazel workspace.
It's under out/soong/workspace and is a symlink forest that merges BUILD
files generated by bp2build and the source tree.

bazel-* symlinks and $OUTDIR are currently ignored.

Test: Presubmits.
Change-Id: If14df23bd0f6a6f2d7d1140f95213f2aee66d652
2021-04-20 10:00:02 +02:00
Colin Cross
b5fa2646d8 Merge changes Iefe133ce,I893f3dd0,I604a11c9
* changes:
  Fix metalava api baseline update command
  sbox: print failing command line before output
  sbox: best-effort copy output files on failure
2021-04-19 15:34:57 +00:00
Colin Cross
4258a39bd1 sbox: print failing command line before output
The full command line run inside sbox can be very long, and printing it
after the errors printed by the failing command can hide the error
messages.  Buffer the output of the command and print the failing command
line before the output if it fails.

Bug: 185516277
Test: m out/soong/.intermediates/frameworks/base/system-api-stubs-docs-non-updatable/android_common/metalava/api_lint.timestamp with lint error
Change-Id: I893f3dd01f1baf195e182111c5c49e92eb82f3b0
2021-04-16 14:55:12 -07:00
Colin Cross
fc2d842dfd sbox: best-effort copy output files on failure
Error messages printed by failing commands may reference output files
that were created by the command, for example printing a command line
to copy and paste to update a baseline file.  Copy output files in the
sandbox to their final locations, ignoring missing files, so that the
messages are valid.

Bug: 185516277
Test: m out/soong/.intermediates/frameworks/base/system-api-stubs-docs-non-updatable/android_common/metalava/api_lint.timestamp with lint error
Change-Id: I604a11c9b54e409ca5bc5c016cd04b3133f74a60
2021-04-15 19:50:02 -07:00
Colin Cross
cb33a00992 Update for sharded globs
Update the path bootstrap_test.sh uses to check for bpglob reruns
to follow the changes made for sharding globs.

Use bootstrap.GlobFileListFiles to get dependencies during bpbuild.

Test: tests/bootstrap_test.sh
Bug: 159845846
Change-Id: Ibaa7c4360881ed6b666a811bf34b19ea0cdcafe9
2021-04-15 11:03:00 -07:00
Lukacs T. Berki
6124c9b36e Do not pass the list of deps to RunBlueprint.
It was only appended to the return value, which can be done by the
caller just as well.

Test: Presubmit.
Change-Id: I962696e0dbd4c3496a0159d01d2a911675fd4217
2021-04-15 15:34:11 +02:00
Lukacs T. Berki
d518e1a407 Make bp2build be more correct.
It now handles adding .bp files and changing globs.

In order to do this, depfiles are now written separately from RunBlueprint.

This is necessary due to the confluence of a number of seemingly
unrelated factors:

1. The glob filelist dependencies are discovered in globSingleton
2. Singletons need to be registered because otherwise singleton module
   types panic
3. Singletons don't work because they require mutators bp2build does not
   run

Due to (1), we would need to run the glob singleton. However, due to (2)
and (3), we can't run singletons and have to run Blueprint with
StopBeforeGeneratingBuildActions, which is when the build actions
writing glob files would be generated. So what happens is:

1. When bp2build is run, the glob singleton is disabled
2. At the end of bp2build, the list of glob files is artifically added
   to the depfile of the workspace marker file
3. When build.ninja is generated, the Ninja file containing the glob
   list file is written by the now-active glob singleton

Test: Presubmits.
Change-Id: I3c5898d8c57c554a93520276c64a952afc912dbe
2021-04-15 13:06:16 +02:00
Lukacs T. Berki
f8e2428c5d Allow running bp2build as part of a regular build.
This is done by setting the INTEGRATED_BP2BUILD environment variable
when invoking the build.

Even though the name of the marker file insinuates that a Bazel
workspace is already created, this is not the case yet.

An issue that remains is that a .d file is not written for the marker
file so it won't be rebuilt if a .bp file changes. Fixing this requires
delicate surgery because writing the .d file is the result of delicate
interplay between Soong and Blueprint.

There are also a number of semi-related fixes:

- The name of soong.environment.{used,available} is now on the command
  line of soong_build (soong_docs is still special cased because its
  command line in the Ninja file is taken from the os.Args of
  soong_build so it's not trivial to remove the --{available,used}_env
  from it
- bp2build writes a separate soong.environment.used file
- I had to call SetAllowMissingDependencies() separately when creating
  the android.Context for bp2build so that bp2build runs in the
  integration tests (it was not obvious how not to do this)
- Fixed a number of integration tests where a command with an expected
  exit code of 1 was used as the last one in a test case, thereby
  breaking the test suite

Test: Presubmits.
Change-Id: Ibeb61c26022cf801dcb98505b4039151b3409873
2021-04-15 08:46:07 +02:00
Lukacs T. Berki
53b2f36bd6 Delete android/env.go .
Properly plumb the available environment to the configuration instead of
going through the global variable originalEnv.

Test: Presubmit.
Change-Id: Ia1ea753d5e72c02a0dcaa4c0b43bd6e24fc47cec
2021-04-13 08:10:01 +02:00
Lukacs T. Berki
745380ce23 Remove the dependency on a number of env vars.
This is so that the way soong_build is invoked is simpler, which is in
turn useful so that it can be invoked multiple times within a single
build, which in turn will be used to invoke bp2build routinely (as
opposed to manually when needed)

Test: Presubmits.
Change-Id: Iddaebb05ff7bcedc0db8273192bb31284b739920
2021-04-12 12:07:44 +02:00
Anton Hansson
da42a5a209 Merge "Revert^2 "Set min_sdk_version for java imports in pom2bp"" 2021-04-07 10:13:05 +00:00
Anton Hansson
ebfbad2969 Revert^2 "Set min_sdk_version for java imports in pom2bp"
f99ba27101

Change-Id: Ic88670182029ace2e6dcdcae2713cb48bb0f59cd
2021-04-06 19:21:34 +00:00
Jack He
4ff6cc91c8 Merge "Revert "Set min_sdk_version for java imports in pom2bp"" 2021-04-06 18:57:57 +00:00
Jack He
f99ba27101 Revert "Set min_sdk_version for java imports in pom2bp"
Revert "Give pom2bp -default-min-sdk-version 24 param"

Revert submission 1662079-axd

Bug: 184656328
Reason for revert: Break build
Reverted Changes:
I2f4dc7585:Update androidx min_sdk_version in allowed_deps
I5cb8c736d:Give pom2bp -default-min-sdk-version 24 param
I3b029315a:Set min_sdk_version for java imports in pom2bp

Change-Id: Ic97cbd1714f6113188ad4647ce71105f9641fad7
2021-04-06 18:52:50 +00:00
Anton Hansson
85822e9b5f Merge "Set min_sdk_version for java imports in pom2bp" 2021-04-06 11:46:30 +00:00
Lukacs T. Berki
97bb9f18ea Add dumping the module graph in JSON format.
Usage: SOONG_DUMP_MODULE_GRPH_JSON=<filename> m nothing

Test: The new test case in bootstrap_test.sh .
Change-Id: I69005a75d47dff915d27187645d0cd1cbb3467ef
2021-04-06 08:45:58 +02:00
Lukács T. Berki
c5b3a1722c Merge "Reorganize soong_build/main.go ." 2021-04-01 19:05:31 +00:00
Lukacs T. Berki
6790ebcc59 Reorganize soong_build/main.go .
This makes it more easy to see what's happening under which modes. Modes
are:

- Regular builds
- Mixed mode builds
- bp2build
- soong_docs

Test: Presubmit + bootstrap_test.sh
Change-Id: I06e3fcb84986b919be1cd9eb456531259ede1183
2021-04-01 18:01:14 +02:00
Anton Hansson
5d8652fcaf Set min_sdk_version for java imports in pom2bp
Previously min_sdk_version was only set for *_library build rules, and
not for *_import targets. This made min_sdk_version default to
sdk_version, which is usually "current", which is not correct for these
libraries.

Test: pom2bp -default-min_sdk_version 24 for androidx
Change-Id: I3b029315abeb559e66efad847ae664efb79f4509
2021-04-01 11:49:35 +00:00
Colin Cross
85920c9389 Merge "Fix multiple copies of read-only files in sbox" 2021-04-01 01:09:45 +00:00
Colin Cross
607c0b795c Fix multiple copies of read-only files in sbox
Sbox preserves the permissions of input files when copying them into the
sandbox.  A read-only file copied into the sandbox multiple times causes
a permission denied error on the second write.  Building in Bazel results
in more read-only files, which triggers the issue on existing sbox rules
with duplicate input files.  Remove the destination file when copying if
it exists.

Bug: 184113103
Test: m USE_BAZEL=true
Change-Id: I7edf92d82b766100e3cbbd90d22428269d7d0167
2021-03-31 13:17:42 -07:00
Anton Hansson
c29f076768 Add pom2bp param for default min_sdk_version
Previously 24 was hardcoded (for somewhat unclear reasons). Update
the tool to accept specifying the correct value on the cmdline.

Test: pom2bp -default-min-sdk-version 24
Change-Id: I39c23dd4bca91684a2a68d7b963cf3cb66eb1f33
2021-03-29 16:45:52 +01:00
Colin Cross
e55bd423df Pass rsp files into sbox and rewrapper
The current implementation causes inputs listed in an rsp file used with
sbox to be duplicated 3 times in the build.ninja file; once as a
dependency of the rule, once in the rspfile_content field of the rule
with the paths rewritten to be relative to the sandbox, and once in the
rule to write the sbox manifest.  When RBE is enabled it also gets a
fourth copy in the list of files to be treated as inputs by rewrapper.

Reduce this to a single copy by using "$in" for the rspfile_content so
that the files only have to be listed in the input dependencies of the
rule, and then add support to sbox to rewrite the rsp file while copying
it into the sandbox, and pass it to rewrapper as well.

Test: m lint-check
Change-Id: I3f46f61119508d39a8bb231c99fc130153fb6f04
2021-03-25 11:06:45 -07:00
Colin Cross
fd708b5651 Move response file handling to a separate package
sbox is going to need to read and write response files, move ReadRspFile
to its own package.

Test: response_test.go
Change-Id: Iecb5486b4aaeb2531828743ad8ef784df675e18e
2021-03-25 11:06:45 -07:00
Jingwen Chen
95c6eb3959 Merge "bp2build: add allowlist for package-level conversions." 2021-03-24 21:38:07 +00:00
Lukacs T. Berki
c99c947c88 Make null builds always be null builds.
Previously, soong.environment.used was written after build.ninja and if
the amount of time that passed between the two was long enough, Ninja
would decide that build.ninja is older than soong.environment.used and
rebuild it.

Test: test_null_build in bootstrap_test.sh in a loop.
Change-Id: I5467da487e8e8f2646644b8a7fb9549b9ff18276
2021-03-24 10:51:05 +01:00
Jingwen Chen
12b4c2706d bp2build: add allowlist for package-level conversions.
This CL adds the support for specifying lists of directories in
build/soong/android/bazel.go, which are then written into
out/soong/bp2build/MANIFEST. Using this configuration,
modules/directories can either default to bp2build_available: true or
false, while still retaining the ability to opt-in or out at the module level.

It also ensures that ConvertWithBp2Build returns true iff the module
type has a registered bp2build converter.

Test: go tests
Test: demo.sh full
Test: TreeHugger presubmits for bp2build and mixed builds.

Change-Id: I0e0f6f4b1b2ec045f2f1c338f7084defc5d23a55
2021-03-24 02:27:19 -04:00
Lukacs T. Berki
f0b3b94bb3 Make GENERATE_BAZEL_FILES=true correct.
This is achieved by writing soong.environment.used in Main() instead of
as a side effect of a singleton. This makes a difference because build
actions are not generated when GENERATE_BAZEL_FILES=true is set,
therefore the side effect did not happen.

Arguably, Main() is made worse by this change, but I don't want to
tackle the problem of readably determining which mode soong_build is
running in in this change.

Test: Presubmits + the additional test.
Change-Id: I66af2429aedf008762173eaaa55b828b4cf4328b
2021-03-23 15:18:22 +01:00
Colin Cross
a5d09762a6 Merge changes Idcbe2046,I09616402,Ide2f1ed9
* changes:
  Fix sbox in chdir mode
  Check RuleBuilder temporaries for path errors
  Fix WriteFileRule escaping
2021-03-18 22:28:11 +00:00
Lukacs T. Berki
d7ce8402d8 Remove GeneratingPrimaryBuilder().
It's replaced with a flag in bootstrap.Args .

Test: "m nothing" (presubmits take a full workday these days)
Change-Id: Ia0bfa2091013e389890e583c559006077806af99
2021-03-17 14:08:50 +01:00
Lukacs T. Berki
d1e3f1ff08 Embed minibp into soong_ui.
This requires linking Blueprint into soong_ui. It lets us avoid the
complicated dance of Ninja files and shell scripts: now the information
as to how soong_build is built is passed directly to Blueprint using a
struct that contains all the information the command line arguments used
to contain.

The ability to run Blueprint from the command line is kept (for now).

Some variables in bootstrap/command.go needed public accessor functions
because soong_build reads them. This will be disentangled by moving the
flag parsing to soong_build.

The presence of the flag definitions in Blueprint means that soong_ui
now also accepts them. This is not a problem in practice because they
are ignored and because soong_ui itself is hidden behind a few layers of
shell scripts.

Test: Presubmits + the new bootstrap_test.sh .
Change-Id: I6dca478f356f56a8aee1e457d71439272351390b
2021-03-17 08:35:52 +01:00
Lukács T. Berki
645610e791 Merge "Remove global variables from Blueprint." 2021-03-17 07:08:24 +00:00
Colin Cross
c590ec4bda Fix sbox in chdir mode
When running commands in chdir mode sbox needs to translate the
directory placeholders to paths relative to the sandbox directory
instead of relative to the top of the source tree.

Also translate relative PATH entries into absolute paths so they
are still valid when the current directory is changed.

Bug: 182612695
Test: manual
Change-Id: Idcbe20466888909d423d62788bc9c35f4e03b398
2021-03-16 16:53:23 -07:00
Lukacs T. Berki
89e9a16ebd Remove global variables from Blueprint.
The end goal of this exercise is to remove all mutable global state so
that multiple Blueprint instances can be run in the same process.

Test: Presubmit.
Change-Id: Idb34b0920f2c7d92efb9328ce8a78b7306f89571
2021-03-16 08:40:51 +01:00
Liz Kammer
ba3ea16f14 Use handcrafted build targets in bp2build
If both bp2build_available and label are specified, label will be
preferred.

Initially, we copy the entire BUILD.bazel file. Eventually we may move
this to use bazel query for a more accurate result.

Test: go test *
Test: build/bazel/scripts/milestone-2/demo.sh full
Test: GENERATE_BAZEL_FILES=true m nothing
      edit bionic/libc/tools/BUILD.bazel
      GENERATE_BAZEL_FILES=true m nothing and verify changes picked up
Bug: 180516554
Change-Id: I43025583300e6b10d2c18032cd4a76237b578d59
2021-03-15 13:09:25 -04:00
Lukacs T. Berki
47a9d0cd98 Really fix queryview.
In addition, make the Ninja command nicer by using backticks instead of
backslash-quoting double quotes.

Test: Manual: "m queryview"
Change-Id: I5d6e1d41424e0229bfe08e2e7eaf2fb025ed48c2
2021-03-09 09:37:53 +01:00
Lukacs T. Berki
7d613bfe2c Make it possible to debug soong_ui.
This works by setting the SOONG_UI_DELVE= environment variable to the
port on which soong_ui should accept a Delve connection on.

This is achieved by reusing the Delve execution logic between soong_ui
and soong_build.

Test: Manual.
Change-Id: Id2c1d4b6faac1a4a3918c91030ce2239f7daf54f
2021-03-08 08:27:58 +01:00
Lukacs T. Berki
7690c09953 cd to / before running soong_build .
This lets one avoid any decisions as to when to chdir there during its
execution and leads to better sandboxing because the pwd doesn't leak to
init() functions anymore.

Test: Manual.
Change-Id: I1560da8ed3a621249426f9e8908aa890c21e13ba
2021-03-03 09:14:22 +01:00
Lukacs T. Berki
3243aa51eb Move environment staleness check to soong_ui.
Also delete the now-unnecessary soong_env binary.

Test: Manually. Also checked what happens when a used environment
variable changes.

Change-Id: Ib393e7f444e94819198c6cce4bcd8719d9fd9441
2021-03-02 09:32:49 +01:00
Jingwen Chen
164e0867fc bp2build: add a simple UI to report migration progress.
Sample output:

[bp2build] cc_library_headers: 5 targets
[bp2build] cc_object: 5 targets
[bp2build] filegroup: 4 targets
[bp2build] genrule: 4 targets
[bp2build] sh_binary: 1 targets
[bp2build] Generated 19 total BUILD targets from 39270 Android.bp modules.

This CL adds an additional CodegenMetrics return value to
GenerateBazelTargets calls, which are called from bp2build, queryview,
and their tests. For this UI, we only want to use it for bp2build, and
not queryview or tests, since it's not useful for the former, and can
pollute the CLI for the latter.

Test: build/bazel/scripts/milestone-2/demo.sh
Change-Id: Ic84307a1ed1a25e360c9b23459e5449d932bc2e7
2021-02-19 09:30:11 -05:00
Treehugger Robot
b68036f44b Merge "Add LOCAL_LICENSE_KINDS to build/soong" 2021-02-10 07:18:29 +00:00
Bob Badour
02040de891 Add LOCAL_LICENSE_KINDS to build/soong
Added SPDX-license-identifier-Apache-2.0 to:
  Android.bp
  android/Android.bp
  android/soongconfig/Android.bp
  androidmk/Android.bp
  apex/Android.bp
  bazel/Android.bp
  bp2build/Android.bp
  bpf/Android.bp
  bpfix/Android.bp
  cc/Android.bp
  cc/config/Android.bp
  cc/libbuildversion/Android.bp
  cc/libbuildversion/tests/Android.bp
  cc/ndk_api_coverage_parser/Android.bp
  cc/ndkstubgen/Android.bp
  cc/symbolfile/Android.bp
  cmd/dep_fixer/Android.bp
  cmd/diff_target_files/Android.bp
  cmd/extract_apks/Android.bp
  cmd/extract_jar_packages/Android.bp
  cmd/extract_linker/Android.bp
  cmd/fileslist/Android.bp
  cmd/host_bionic_inject/Android.bp
  cmd/javac_wrapper/Android.bp
  cmd/merge_zips/Android.bp
  cmd/multiproduct_kati/Android.bp
  cmd/path_interposer/Android.bp
  cmd/pom2bp/Android.bp
  cmd/pom2mk/Android.bp
  cmd/sbox/Android.bp
  cmd/soong_build/Android.bp
  cmd/soong_env/Android.bp
  cmd/soong_ui/Android.bp
  cmd/zip2zip/Android.bp
  cmd/zipsync/Android.bp
  cuj/Android.bp
  dexpreopt/Android.bp
  dexpreopt/dexpreopt_gen/Android.bp
  env/Android.bp
  etc/Android.bp
  filesystem/Android.bp
  finder/Android.bp
  finder/cmd/Android.bp
  genrule/Android.bp
  jar/Android.bp
  java/Android.bp
  java/config/Android.bp
  kernel/Android.bp
  linkerconfig/Android.bp
  linkerconfig/proto/Android.bp
  makedeps/Android.bp
  partner/Android.bp
  phony/Android.bp
  python/Android.bp
  python/tests/Android.bp
  remoteexec/Android.bp
  rust/Android.bp
  rust/config/Android.bp
  scripts/Android.bp
  sdk/Android.bp
  sh/Android.bp
  shared/Android.bp
  symbol_inject/Android.bp
  symbol_inject/cmd/Android.bp
  sysprop/Android.bp
  tradefed/Android.bp
  ui/build/Android.bp
  ui/logger/Android.bp
  ui/metrics/Android.bp
  ui/metrics/proc/Android.bp
  ui/status/Android.bp
  ui/terminal/Android.bp
  ui/tracer/Android.bp
  xml/Android.bp
  zip/Android.bp
  zip/cmd/Android.bp

Added SPDX-license-identifier-Apache-2.0 SPDX-license-identifier-BSD to:
  finder/fs/Android.bp
  third_party/zip/Android.bp

Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m all

Exempt-From-Owner-Approval: janitorial work

Change-Id: Ia47ca14f16b8c9f84f9d533a07e5b00e2c04e8d4
2021-02-06 04:23:21 +00:00
Jingwen Chen
7dcc4fcc68 bp2build: fix BUILD file generation incrementality bug.
Before this change, changes to any Android.bp file would not trigger a re-run of bp2build/Soong, since the depfile for Soong's main output, the build.ninja file, didn't contain the list of Android.bp files. bp2build only reran when Soong itself changed, which happens regularly while developing bp2build, but not at all while modifying Android.bp files.

This change correctly dumps the list of all Android.bp files in the tree to the depfile for out/soong/build.ninja, therefore correctly triggering a re-run of bp2build when any Android.bp file changes.

Test: GENERATE_BAZEL_FILES=true m nothing; edit an Android.bp file; GENERATE_BAZEL_FILES=true m nothing -> successfully re-runs and regenerates out/soong/bp2build.

Test: inspect out/soong/build.ninja.d manually
Fixes: 179452534

Change-Id: Iad80c381b3afa38a4d4fafca035e46aa2a1f206a
2021-02-05 01:33:47 -05:00
Jingwen Chen
4d2c0878a3 bp2build: rename method for readability.
Rename GenerateSoongModuleTargets to GenerateBazelTargets to reflect its return type.

Test: TH
Change-Id: I5359c8f4365750790837a9d412946d0b7fae64b7
2021-02-02 07:06:56 -05:00
Jingwen Chen
eb76c4319d bp2build: fix running from clean checkout.
bp2build is a Soong mode that returns as soon as the BUILD files are generated. This causes it not generate the build.ninja file, which the Ninja process executing soong_build itself expects to produce as an output. If there isn't an existing build.ninja file generated from a previous build, GENERATE_BAZEL_FILES=true m nothing will fail.

This CL generates the expected files as a workaround, and also makes GENERATE_BAZEL_FILES=true skip the Kati/Ninja steps in soong_ui since they aren't needed.

Test: rm -rf out/ && GENERATE_BAZEL_FILES=true m nothing && m libc

Fixes: 178683777

Change-Id: I2515ef7961682d2be5f096ed24831cc185165a67
2021-01-29 22:33:25 -05:00
Colin Cross
f04fe9abca multiproduct_kati: increase ram per job to 25GB
multiproduct_kati sometimes passes 20GB per soong_build process,
increase required ram per job to 25GB.

Bug: 178376498
Test: none
Change-Id: I446f52ea7419bc3015d940511af1f03b15eb651e
2021-01-26 22:09:44 +00:00
Jingwen Chen
33832f96e0 bp2build: introduce CodegenMode.
The CodegenMode enum helps to differentiate between bp2build and queryview modes of generating BUILD files from the Soong module graph.

bp2build is used for generating BUILD files from an unconfigured Soong module graph for Soong->Bazel conversion, whereas QueryView is a front-end for querying the existing Soong configured module graph with Bazel query.

Test: go test

Test: TH
Change-Id: I5709f591f1ae08b2770adf272d8ce60c9ee1f46a
2021-01-26 00:30:30 -05:00
Jingwen Chen
7385067640 bp2build: framework for generating BazelTargetModules.
This CL creates the framework necessary for generating
BazelTargetModules from regular Soong Android modules.
BazelTargetModules are code-generated into Bazel targets in BUILD files.

See the follow-up CL for examples of creating filegroup/genrule
BazelTargetModules.

Test: GENERATE_BAZEL_FILES=true m nothing # creates out/soong/bp2build
with no BUILD files, because there are no BazelTargetModules in the
module graph.

Change-Id: I33a96365bd439043b13af6db9e439592e9983188
2021-01-21 22:46:11 -05:00
Jingwen Chen
daa54bcbba Implement code-generation step for bp2build.
Implement bp2build codegen as a discrete step that runs after an
alternatively registered pipeline of mutators, instead of a
presingleton.

bp2build codegen requires a Context that supports VisitAllModules and
PathContext, so this CL also makes a BpToBuildWrapperContext that
conforms to PathContext by adding two method implementations.

Test: GENERATE_BAZEL_FILES=true m nothing && bazel query //... --config=bp2build | wc -l # 31433
Test: m queryview && bazel query //... --config=queryview # 63638

Change-Id: I0dd359746584b228046d2d0ff00895f28f9bdfc3
2021-01-19 11:03:36 -05:00
Colin Cross
2bcb4940bb Merge changes Ib29ede45,I1b2bfdfb
* changes:
  Create LLNDK vendor variants when DeviceVndkVersion is not set
  Add a new SingletonModule type
2021-01-12 20:43:39 +00:00
Colin Cross
9aed5bc715 Add a new SingletonModule type
A SingletonModule is halfway between a Singleton and a Module.  It has
access to visiting other modules via its GenerateSingletonBuildActions
method, but must be defined in an Android.bp file and can also be
depended on like a module.

Bug: 176904285
Test: singleton_module_test.go
Change-Id: I1b2bfdfb3927c1eabf431c53213cb7c581e33ca4
2021-01-07 17:19:27 -08:00
Liz Kammer
2dd9ca422d Refactor queryview.
Splits queryview into queryview and bp2build. The latter runs as a
presingleton (could be converted to a singleton). This prevents needing
to run soong_ui a subsequent time to build the query. Queryview remains
as a separate step to prevent increasing runtime due to this generation
every time Soong runs.

Currently this is running as a presingleton as this gives a translation
of Android.bp files after only LoadHooks have run, no mutators.

Test: go tests
Test: m queryview && bazel query --config=queryview //...
Change-Id: If2ba28c7ef60fbd41f43bda6623d41c8c7d23a1b
2021-01-07 14:34:00 -05:00
Jaewoong Jung
18aefc1977 Remove unnecessary snake case variables.
Test: m nothing + TreeHugger
Change-Id: I99f7162944daa6c57c6ae4763261e108bb5cb6b1
2020-12-22 12:38:35 -08:00
Colin Cross
f2f3d31549 Allow multiple --skip-products and --products arguments to multiproduct_kati
Concatenate multiple comma separated lists passed to --skip-products
and --products.

Test: manual
Change-Id: Ie5e9310e1ebcb9c7fa88e9c8b45fb29f64f9cce7
2020-12-17 19:33:18 +00:00
Rupert Shuttleworth
534f15708f Skip Bazel dist dir migration if not in dist mode or if Bazel dist dir does not exist.
Test: DIST_DIR=/tmp/foo USE_BAZEL=1 m dist and DIST_DIR=/tmp/foo USE_BAZEL=1 m dist installclean
Change-Id: Ibf576d2f3ba506e0f45c30a5dbea69de5c475c9a
2020-12-17 01:01:33 +00:00
Treehugger Robot
8574ae773f Merge "Allow Bazel to write to an external DIST_DIR (outside of OUT_DIR)." 2020-12-11 05:16:32 +00:00
Treehugger Robot
5571f61bc1 Merge changes I394a1b62,I6906be4c
* changes:
  Add bazel metrics directory to the list of metrics files to be uploaded.
  Allow uploading a directory of metrics files.
2020-12-11 02:15:37 +00:00
Rupert Shuttleworth
3c9f5ac787 Allow Bazel to write to an external DIST_DIR (outside of OUT_DIR).
Also get Bazel to write real files there (not symlinks) so that the DIST_DIR can be independent.

Test: Manually using e.g. DIST_DIR=/tmp/foo USE_BAZEL=1 m dist
Change-Id: I39d5219500864c9ecc85f356a028e9b5bf2607f4
2020-12-11 01:17:58 +00:00
Colin Cross
d3b30670f2 Merge changes I1d6750db,Ie2c197bb,I6a51c89a
* changes:
  Add TransitivePackagingSpecs
  Add executable flag to sbox copy requests
  Export files to install as a depset
2020-12-10 21:53:49 +00:00
Patrice Arruda
405640203e Add bazel metrics directory to the list of metrics files to be uploaded.
The contents of the bazel metrics directory (out/bazel_metrics or
<dist dir>/logs/bazel_metrics if dist arg is specified) is added
in the list of files to be uploaded in order to do performance
analysis of bazel builds.

Bug: b/174479728
Test: * m nothing and checked metrics_uploader log file to ensure
        if the metrics is uploaded.
      * USE_BAZEL=1 m nothing and checked if the set of metrics
        files plus the build_bazel_profile.gz is uploaded through
	the metrics_uploader log file.
Change-Id: I394a1b621a06f9acccf154faee6dcd4200fcff4c
2020-12-10 21:48:20 +00:00
Patrice Arruda
83842d7235 Provide an interface for shared paths between Soong and Soong UI.
Code refactoring has been done for logs directory logic code since
the bazel metrics directory depends on the log directory. For builds
that did not specify a dist directory, the log directory is under
out directory. With dist, the logs directory is under <dist dir>/logs.
This matters for Android CI builds where the metrics files are
placed under logs directory. With this change, the bazel metrics
directory and corresponding files will be under <dist dir>/logs
directory for Android CI builds.

Bug: b/174479728
Test: * USE_BAZEL=1 m nothing (bazel_metrics dir in out dir)
      * m nothing (bazel_metrics dir deleted)
      * DIST_DIR=/tmp/build USE_BAZEL=1 m nothing dist (bazel_metrics
        is in /tmp/build/logs directory)

Change-Id: Ic9e1ff49a1964fcaaf801bde2c19f33597ca1db4
2020-12-08 20:24:14 +00:00
Colin Cross
859dfd9240 Add executable flag to sbox copy requests
Add a boolean flag to the sbox proto to request the executable bit
to be set after copying a file.  This will be used for sandboxing
tools when copying a file in from the source tree that would
normally get its executable bit set during installation.

Bug: 124313442
Test: sbox_test.go
Change-Id: Ie2c197bb5183ffc1bf63fd6effd175143cd324d4
2020-12-07 12:27:50 -08:00
Treehugger Robot
741711b34c Merge "Add ranking in docs for exclude_srcs to follow srcs" 2020-12-07 18:27:50 +00:00
Jingwen Chen
946e0b69d2 Merge "Add bp2build mode to soong_build." 2020-12-07 07:16:15 +00:00
Jingwen Chen
4133ce6912 Add bp2build mode to soong_build.
This CL adds a new CONVERT_TO_BAZEL env var, and a bp2build goal. It
reuses the queryview architecture, but registers no mutators before
converting to BUILD files. This gives us a blank slate of the module
graph to work with, and create a shadow/alternate pipeline of mutators
to converge the Bazel BUILD graph to be semantically equivalent with the
Android.bp graph (after apply its current set of mutators).

The command to do so is:

    $ CONVERT_TO_BAZEL=true m bp2build

To not clobber with queryview, the generated files are in
out/soong/bp2build.

Test: CONVERT_TO_BAZEL=true m bp2build && bazel query --config=bp2build //...
Test: m queryview && bazel query --config=queryview //..
Test: soong tests
Test: TH presubmit
Fixes: 174465461
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: I4dd0ccc73abc345d70a50ca2803d6f400cd8c863
2020-12-03 23:36:57 -05:00
Jingwen Chen
97d85c8a61 Merge "Remove soong.config, FileConfigurableOptions, MegaDevice config." 2020-12-03 02:07:35 +00:00
Treehugger Robot
6e25bb9e5f Merge "Soong: use deterministic temp dir names in sbox" 2020-12-01 00:00:30 +00:00
Paul Duffin
af970a2e7a Generalize the handling of dist tags
Previously in order to make use of dist entries with tag properties it
was necessary for each module type to be modified to store the results
of GenerateTaggedDistFiles(module) in the AndroidMkEntry.DistFiles
property. This change generalizes that mechanism to work across all
module types.

This change does improve the behavior of a couple of tests where
previously a dist with tag property was ignored because tagged dist
files were not available now it is used correctly.

Some module types do not implement OutputFileProducer interface and so
they cannot handle dist with tag properties. This change makes it an
error to attempt to do that. That necessitated adding OutputFiles(tag)
method to customModule in queryview_test.go as some of the tests in
that file used dist entries with tag properties.

Follow up changes will remove the code that was made redundant by this.

Test: m nothing
      m dist sdk - before and after this change, compare result to
      make sure that there are no significant differences.
Bug: 174226317
Change-Id: Ifc54d67db10ce0d0fe8179c05b97a2be2113be4e
2020-11-27 15:17:44 +00:00
Ulf Adams
b73daa588d Soong: use deterministic temp dir names in sbox
The sbox tool is used to wrap genrule actions, and run them in a
temporary directory. Afterwards, the outputs are moved into their
proper location in the output tree. However, some tools embed the file
name of the output file (as passed to the tool) into the output file.

For example, the perfetto code generator script uses the output file
name to generate a C header guard when it generates
perfetto_src_base_version_gen_h/gen/perfetto_version.gen.h.

When using remote execution / remote caching, these genrule actions are
run locally whenever the output file doesn't exist, effectively making
them unique (across time and between users).

They then cause cache misses on all actions depending on these output
files as well as on transitive actions. In the above example, this
causes libperfetto.so to differ, which then causes all actions
depending on libperfetto.so to be rerun unnecessarily.

As a fix, this commit changes the sbox tool to use the sha1 hash of the
manifest path. The tool already seems to assume that there are no
concurrent runs of the same tool with the same manifest (otherwise
it would fail flakily depending on exact timing). It seems therefore
safe to use a temporary path deterministically derived from the
manifest path.

Test: ran multiple builds w/ remote execution; observe proper caching
Change-Id: I5b73ffd3b7f85cbb0336dfa1675de7ac0e2fd1a8
2020-11-26 22:11:05 +01:00
Colin Cross
d03797e48a Remove hardlink optmization in sbox
The hardlink optimization attempts to save I/O by hardlinking input
files into the sandbox instead of copying them, but it never works on
a Linux build due to nsjail always making out and the source tree into
separate bind-mounted filesystems.  If the optimization did work it
would actually cause build failures when it hardlinked the relative
bionic/libc/fs_config_generator.py symlink into the sandbox directory
without the target.  Just remove the optimization for now since it
never works.

Test: m checkbuild
Change-Id: I96e8c0c145e7c99958639594edf8a93b69ae90eb
2020-11-25 18:28:58 +00:00
Jingwen Chen
c4d91bceba Remove soong.config, FileConfigurableOptions, MegaDevice config.
No current use cases for soong.config and user-editable local
configuration options for Soong, so let's remove it to reduce the API
surface.

The MegaDevice configuration is used for building every module, for
every cpu variant, for every architecture, but it apparently isn't
used for a while now.

Test: m nothing
Test: TH
Fixes: 174188200
Change-Id: I2550e70ff6f9c8b57e9a7cc517d6a119a032a27a
2020-11-24 23:42:17 -05:00
Colin Cross
38214f5ac5 Merge changes I8af00c8c,Ia526ae89
* changes:
  Reland: Rewrite sbox to use a textproto manifest
  Fix gensrcs subdirs
2020-11-24 19:27:35 +00:00
Treehugger Robot
fbb138cd7e Merge "Refactor queryview templates into a different file." 2020-11-24 07:54:51 +00:00
Colin Cross
e16ce36818 Reland: Rewrite sbox to use a textproto manifest
This relands I3b918a6643cea77199fd39577ef71e34cdeacdb1 with a fix
to create the directory for the output depfile if doesn't exist.

In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.

Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I8af00c8c0b25a92f55a5032fcb525715ae8297c2
2020-11-20 15:45:04 -08:00
Colin Cross
b893f8766c Merge "Revert "Rewrite sbox to use a textproto manifest"" 2020-11-20 18:45:35 +00:00
Colin Cross
619b9ab260 Revert "Rewrite sbox to use a textproto manifest"
This reverts commit 151b9ff0cf.

Reason for revert: broke builds

Change-Id: I69b3b8795d5a36b4fa0debb1af2d433be3c15d6c
2020-11-20 18:44:31 +00:00
Colin Cross
60d06cf8df Merge "Rewrite sbox to use a textproto manifest" 2020-11-20 17:43:29 +00:00
Treehugger Robot
83f738cae2 Merge "zipsync handles symlink" 2020-11-19 07:41:30 +00:00
Jaewoong Jung
157a781ce4 Merge "Add and update comments in env and android_env.go" 2020-11-19 02:19:25 +00:00
Jiyong Park
dc48afd3c5 zipsync handles symlink
This change fixes a bug that zipsync didn't handle symlink correctly;
symlink was extracted as a regular file whose content is the target
path. Fixing the problem by correctly creating the symlink using
os.Symlink.

Bug: N/A
Test: manual

Change-Id: Ib6685c14e1950d1057d89672883cdd9e4879069a
2020-11-19 08:58:02 +09:00
Colin Cross
151b9ff0cf Rewrite sbox to use a textproto manifest
In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.

Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I3b918a6643cea77199fd39577ef71e34cdeacdb1
2020-11-18 13:02:22 -08:00
Jingwen Chen
b24e970c74 Refactor queryview templates into a different file.
Make queryview.go smaller.

Test: m queryview && bazel query --config=queryview //...
Change-Id: I3d3ba688eca6f05d33d1875076e3076ca1cac6ee
2020-11-18 08:54:05 -05:00
Jaewoong Jung
9f98d3f423 Improve soong_ui main.go comments.
And do a minor cleanup while we're there.

Test: TreeHugger
Bug: 173557448
Change-Id: Id1736f82f64c1a0a4b29749f73d3a3724fc501a7
2020-11-17 18:29:58 -08:00
Jaewoong Jung
f2200adccb Add and update comments in env and android_env.go
Bug: 173450116
Test: m nothing
Change-Id: I582a483d11c2ca23fd6e9597e49cdfb7e69152de
2020-11-16 16:01:27 -08:00
Treehugger Robot
ceed6be00d Merge "Pass Config to NewTestContext instead of ctx.Register" 2020-11-14 05:58:58 +00:00
Patrice Arruda
0c1c456643 Define UseBazel in Config
The UseBazel() function will be used in several places to perform
different functionality such as setting up the metrics path.

Bug: b/173028918
Test: m nothing
Change-Id: Ie94073b8f0552ec7528e1cc2f3be74b693dc3135
2020-11-13 11:16:21 -08:00
Chris Parsons
3060ec7731 Stop first pass of mixed bazel builds before writing ninja
This shaves off ~20s, which is ~11%, of `m nothing` with mixed builds.

Test: Manual verification with mixed builds of aosp_flame.
Change-Id: Ief1bc56c92329e899c829fc1374eea9ee9b94fb7
2020-11-12 22:30:46 +00:00
Colin Cross
ae8600b507 Pass Config to NewTestContext instead of ctx.Register
Prepare for using Config when adding singletons by passing
Config to NewTestContext and NewContext instead of to ctx.Register.
This will enable a followup change to store SingletonMakeVarsProviders
registered on the Context in the Config, which is necessary to run
multiple tests in parallel without data races.

Test: all soong tests
Change-Id: Id229629a4e42ff4487d317241673837726c075fc
2020-11-12 10:07:49 -08:00
Liz Kammer
e7211dd37a Add ranking in docs for exclude_srcs to follow srcs
Change-Id: I0c4aa568ed1fb82c116b18217043f19029632545
2020-11-10 18:55:02 +00:00
Jingwen Chen
50f93d2078 Rename bazel overlay to queryview.
Bazel overlay is an experimental feature. This renames the feature to 'queryview' to better describe its purpose, and also move away from the already overloaded 'overlay' term in Android.

Test: m queryview && bazel query --package_path=out/soong/queryview //...
Change-Id: I8b5068c7db46cb61a03a8e87af9c7c9077ebeff9
2020-11-05 19:17:14 -05:00
Jingwen Chen
222ff4d186 bazel overlay: fix regression to "list of string" bpdocs type.
A recent change to Blueprint [1] added the feature to infer the type of
list elements to more than just "list of strings", but this also changed
the text to "list of string" (without s). This CL fixes the bazel
overlay, which uses "list of strings".

Test: m bazel_overlay && bazel query //...  --package_path=out/soong/bazel_overlay

[1]: 764a771917%5E%21/#F1

Change-Id: I7522d15b1ce751807e1881ce1e94f068d226b316
2020-11-05 03:55:20 -05:00
Treehugger Robot
d3e294d8ac Merge "Add --skip-soong-tests argument to soong_ui and use it in multiproduct_kati" 2020-11-02 21:31:31 +00:00
Colin Cross
00a8a3f746 Add --skip-soong-tests argument to soong_ui and use it in multiproduct_kati
There is no need for multiproduct_kati to run the tests for
every product, they don't vary by product config. --skip-soong-tests
can also be used for local development to run soong_build even if
the tests don't pass.

Bug: 156428456
Test: m --skip-soong-tests nothing
Change-Id: I9c00e3d1b6e51d17bb290339c3f124d4d1c9e69f
2020-10-29 14:22:04 -07:00
Patrice Arruda
e92c30dff1 Capture the build command that executed the build system.
We need to capture the user build command to understand what additional
arguments is passed in to Soong. This data is useful to know which
targets are being executed and have statistics on top build targets
are specified.

Bug: b/169453712
Test: ran "m blueprint_tools", "m nothing", "lunch 1" and checked
      in soong_metrics proto.
Change-Id: I7ab94dcf037211f1abffe6afd012b03ffbf99756
2020-10-29 11:01:32 -07:00
Rupert Shuttleworth
680387bf1d Add code to enable USE_BAZEL=1 m functionality in AOSP.
Test: Manually, but builds currently fail due to a recent change to bazel_handler.go.
Change-Id: I9a45a2c73d1755032b11f6b913c03f83a28f63c5
2020-10-26 09:23:24 +00:00
Jingwen Chen
311bd38082 Bazel overlay: write all .bzl files in <bazel_overlay>/build/bazel/overlay_rules directory
This keeps the top level directory clean, and leverages the
new "build/bazel" repo location / package path.

Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery //...

Bug: 171180724

Fixes: 171180724
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: I151b8f3d2177167711e5974e5eabad8d6e910276
2020-10-19 07:59:46 -04:00
Liz Kammer
0e7993e3d2 Clarify error for soong_ui failing to parse args
Test: m soong_ui and give invalid command line args
Change-Id: I7b4479b392d953bd1d87218184c5ed2143568a07
2020-10-15 11:07:13 -07:00
Jingwen Chen
6874dbe104 Stop hardcoding sdk/module_exports dynamic prop types for Bazel overlay.
These sdk/sdk_snapshot/module_exports/module_exports_snapshot properties
are dynamically created using RegisterSdkMemberType [1] and eventually
createDynamicSdkMemberType [2]. All dynamically created sdk member types
are string list types.

Instead of hardcoding the mapping, retrieve them from the registry
instead.

[1]: https://cs.android.com/android/platform/superproject/+/master:build/soong/android/sdk.go;l=489-497;drc=master
[2]: https://cs.android.com/android/platform/superproject/+/master:build/soong/sdk/sdk.go;l=156;drc=master

Bug: 166563303
Bug: 170798271
Fixes: 170798271

Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery //...
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: Ib77f4e0cc14a7b4ec344a2de115682c234927c16
2020-10-14 04:55:00 -04:00
Treehugger Robot
f43eee4b9b Merge "Add warning about out-directory usage with RBE builds" 2020-10-12 15:25:22 +00:00
Colin Cross
405af07859 Revert "Make lots of tests run in parallel"
This reverts commit 323dc60712.

Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests

Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
2020-10-09 18:34:24 -07:00
Christopher Parsons
c8f84809ad Merge "Mixed bazel/soong build prototype for genrule" 2020-10-09 14:27:04 +00:00
Chris Parsons
f3c96efea4 Mixed bazel/soong build prototype for genrule
With this change, bazel_module is a specifiable property on
genrule module definitions. With bazel-enabled mode, soong_build will
defer to Bazel for information on these modules.

source build/soong/bazelenv.sh to enter bazel-enabled mode.

Test: Manually verified on bionic/libc genrules using aosp_cf_x86_phone-userdebug
Change-Id: I3619848186d50be7273a5eba31c79989b981d408
2020-10-08 22:46:23 -04:00
Kousik Kumar
a0a44a84be Add warning about out-directory usage with RBE builds
Prints the following warning messsage:
WARNING:
Setting OUT_DIR to a path other than out may result in slow RBE builds.
See http://go/android_rbe_out_dir for a workaround.

Bug: b/169676232
Change-Id: Id1ad06c4dc672c24373642e0b624833eb6a0dbcf
2020-10-08 02:50:24 -04:00
Colin Cross
323dc60712 Make lots of tests run in parallel
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.

Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
2020-10-06 15:12:22 -07:00
Kousik Kumar
ec47864ca0 Deprecate USE_GOMA and replace with FORCE_USE_GOMA
Bug: b/163361527
Test: Ran builds with USE_GOMA / FORCE_USE_GOMA flags
Change-Id: I67b229fbf52e913b0c4f5e7932021132ddc0598f
2020-09-21 15:35:57 -04:00
Jingwen Chen
d8004efff2 Generate .bzl rule definitions for every module type in Soong, and
surface module properties as attributes.

This CL maps int, bool, string and string_list module props from Soong
modules into their respective Bazel targets.

With this CL, you can now query modules based on their properties. For
example:

$ bazel query 'attr(nocrt, 1, //...)'
$ bazel query 'attr(apex_available, //apex_available:platform, //...)'
$ bazel query //art/dalvikvm:dalvikvm--linux_glibc_x86_64 --output=build | grep compile_multilib

Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery 'attr(apex_available, com.android.runtime, //...)'

Test: soong_build tests

Fixes: 162720644
Fixes: 164320355
Change-Id: Iea8e594b952feccac3281f36dd6bdee8e7d62c3a
2020-09-09 09:51:45 +00:00
Jingwen Chen
ce3d46fe05 Bazel overlay: rename 'deps' to 'module_deps'.
'deps' is a module prop used by some module types, like
prebuilt_build_tool:

https://cs.android.com/android/platform/superproject/+/master:build/soong/android/prebuilt_build_tool.go;l=25-26;drc=751ae87ed29c67629f068a8590af80023a457294

Use `module_deps` to prevent conflicts in the generated overlay
attributes.

Bug: 162720644
Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery
//...
Test: soong_build tests

Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: Ia075a860983960b2d9121d30ead8f808be7e0c5b
2020-08-25 06:11:53 +00:00
Jingwen Chen
81a0b66d93 Merge "Surface module properties as Bazel BUILD target attributes in the Bazel overlay" 2020-08-24 22:59:11 +00:00
Jingwen Chen
69d4cbeb04 Surface module properties as Bazel BUILD target attributes in the Bazel overlay
This patchset changes bazel_overlay to generate soong_module as a macro,
instead of a rule, and generate module properties in the BUILD files as
kwargs to the soong_module macro.

Here's a sample of the new BUILD files with module properties:

bionic/libdl/BUILD.bazel:
https://paste.googleplex.com/6484466996346880?raw

art/build/apex/BUILD.bazel:
https://paste.googleplex.com/5461276001042432?raw

bionic/apex/BUILD.bazel:
https://paste.googleplex.com/4932795173437440?raw

soong_module is now a macro that conditionally expands to underlying
soong_<module type> rules with statically defined attributes. In this
CL, we are starting with a hardcoded filegroup rule definition to
demonstrate the conditional rule loading within the soong_module macro.
If the module_type matches an existing Bazel rule, soong_module forwards
the entire **kwargs into the rule, which Bazel typechecks.

Non-filegroup module types will be expanded into generic_soong_module,
but with the kwargs dropped.

This approach allows us to:

1) Programmtically generate soong_<module type> rules for all module
types available in Soong, together with the statically defined attribute
types in `attrs`.
2) Incrementally migrate and test individual module types from
generic_soong_module to their module rule shims.
3) Swap out the module rule shims to the actual Bazel rules (e.g
cc_library, java_library) and perform attribute manipulation in Starlark
itself.

Example of querying against the 'srcs' attribute in soong_filegroup:

```
$ bazel cquery 'kind(soong_filegroup, //...)' | wc -l
590

$ bazel cquery --output=build 'attr(srcs, "linker.cpp",
kind(soong_filegroup, //bionic/...))'
INFO: Analyzed 3907 targets (0 packages loaded, 0 targets configured).
INFO: Found 3907 targets...
/usr/local/google/home/jingwen/aosp/out/soong/bazel_overlay/bionic/linker/BUILD.bazel:4144:13
soong_filegroup(
  name = "linker_sources",
  generator_name = "linker_sources",
  generator_function = "soong_module",
  generator_location = "bionic/linker/BUILD.bazel:4144:13",
  srcs = ["dlfcn.cpp", "linker.cpp", "linker_block_allocator.cpp",
"linker_dlwarning.cpp", "linker_cfi.cpp", "linker_config.cpp",
"linker_debug.cpp", "linker_gdb_support.cpp", "linker_globals.cpp",
"linker_libc_support.c", "linker_libcxx_support.cpp",
"linker_namespaces.cpp", "linker_logger.cpp",
"linker_mapped_file_fragment.cpp", "linker_phdr.cpp",
"linker_relocate.cpp", "linker_sdk_versions.cpp", "linker_soinfo.cpp",
"linker_tls.cpp", "linker_utils.cpp", "rt.cpp"],
  deps = [],
)
/usr/local/google/home/jingwen/aosp/out/soong/bazel_overlay/soong_module.bzl:32:23
in <toplevel>
```

This CL is known to be lacking the following features, and will be looked at in follow up CLs:

1) Pretty printing reflect.Interface properties, like arch, multilib and
dists.
2) Generating module Bazel rule shims for all module types, instead of
hardcoding them like `soong_filegroup`.

Bug: 162720644
Test: bazel_overlay_test.go (soong build test)
Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery //...
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: Ic1e448887eb540ed15a55bc4090cf75a4d832d41
2020-08-21 11:45:46 +00:00
Ramy Medhat
f668917c25 Add Goma deprecation PSA to soong.
Test: build with Goma
Change-Id: Ie7e146285afd40bc0a1dc17a0b898960f57a118b
2020-08-18 17:29:48 -04:00
Jingwen Chen
5ba7e479d1 Create a AOSP Bazel overlay workspace with Soong
The Bazel overlay is a directory at out/soong/bazel_overlay that
replicates the layout of the AOSP Soong module tree, but as a Bazel
workspace. Each Soong module variant is represented as a BUILD target
created with the `soong_module` rule.

To create this overlay, run `m bazel_overlay`.

A `soong_module` target can depend on other `soong_module` targets.
These dependencies replicate each module's `directDeps` in the Blueprint
graph, just before `PrepareBuildActions`.

This enables users to use bazel query as a way to introspect the Soong
module graph. For example,

- Direct reverse dependencies of //bionic/libc:generated_android_ids in
//bionic/libc/...:

$ bazel query 'rdeps(//bionic/libc/...,
//bionic/libc:generated_android_ids, 1)'
//bionic/libc:libc_bionic_ndk--android_recovery_arm_armv7-a-neon_static
//bionic/libc:libc_bionic_ndk--android_ramdisk_arm_armv7-a-neon_static
//bionic/libc:libc_bionic_ndk--android_arm_armv7-a-neon_static_com.android.runtime
//bionic/libc:libc_bionic_ndk--android_arm_armv7-a-neon_static
//bionic/libc:generated_android_ids

- Why does com.android.runtime depend on lzma?

$ bazel query
'somepath(//bionic/apex:com.android.runtime--android_common_com.android.runtime_image,
//external/lzma/...)'
//bionic/apex:com.android.runtime--android_common_com.android.runtime_image
//bionic/libc/malloc_debug:libc_malloc_debug--android_arm_armv7-a-neon_shared_com.android.runtime
//system/core/libunwindstack:libunwindstack--android_arm_armv7-a-neon_shared_com.android.runtime
//external/lzma/C:liblzma--android_arm_armv7-a-neon_shared_com.android.runtime

- What does the dep graph of //bionic/libc:crtbegin_so look like?

$ bazel query
'deps(//bionic/libc:crtbegin_so--android_arm_armv7-a-neon)'
--output=graph > graph.in && dot -Tpng < graph.in > graph.png

https://photos.app.goo.gl/DfsdoFRNsRjGwTmy8

Test:  croot && m bazel_overlay && cd out/soong/bazel_overlay && bazel
query //... && bazel query 'rdeps(//bionic/libc/...,
//bionic/libc:generated_android_ids, 1)'

Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: I3bf40309bfb2d963bb8a688706385a57ee304c37#
2020-08-06 10:24:21 +00:00
Patrice Arruda
09ef4dc2f7 Update language to comply with inclusive guidance
See https://source.android.com/setup/contribute/respectful-code for
reference

Bug: 161896447
Test: m diff_target_files
Change-Id: I5654ccc5ba9be8cff01689b952d2feaade19ae79
2020-07-31 16:50:52 +00:00
Treehugger Robot
0413d0aa97 Merge "Update language to comply with inclusive guidance" 2020-07-30 18:35:47 +00:00
Patrice Arruda
8da724a64a Update language to comply with inclusive guidance
See https://source.android.com/setup/contribute/respectful-code for
reference

Bug: 161896447
Test: m nothing
Change-Id: Ib54185f2ef9666c7d82afc65fe7fc6bc7fdadfb5
2020-07-28 21:22:12 +00:00
Patrice Arruda
f445ba1788 Inclusive fix: renaming dumb terminal to simple terminal.
Bug: b/161896447
Test: "lunch 1" and "m nothing"
Change-Id: Ifb228c0f0f634477932c9240e57170f7390201f7
2020-07-28 20:25:03 +00:00
Patrice Arruda
73c790f59e Use the build start time from Soong main UI for metrics build timestamp.
Currently, the build timestamp saved to the build_date_timestamp
field in the MetricsBase metrics protobuf usually comes from a text
file named build_date.txt. It may not be accurate for metrics purposes.
Instead, use the build start time from Soong main UI for a better
timestamp.

Bug: b/140638454
Test: Ran and checked the build_date_timestamp by running the printproto
      command.
Change-Id: I4d893fbbf9830ba21911e56ae13bc5272be47ae6
2020-07-14 17:23:23 +00:00
Patrice Arruda
62f1bf2378 Add RBE metrics dump in Soong UI.
From aosp/1329396, the RBE metrics protobuf file is part of the
metrics uploading process. The RBE metrics protobuf file is
generated by running the bootstrap shutdown command. A new function
named DumpRBEMetrics was written in order to generate the RBE metrics
protobuf file before sending to the uploading process.

Bug: b/140638454
Test: * Unit test cases
      * Ran RBE build on my local host and verified the
        metrics protobuf file is created.
      * Ran non-RBE build after RBE build and verified that
        the previous metrics protobuf file was deleted.

Change-Id: I4b8068905cb67c4b8c2d94793917b98974fed707
2020-07-09 21:06:22 +00:00
Liz Kammer
a93fbacdbe Correct typo of defaults for docs property sorting
Test: manual
Change-Id: Iaca7ba51ca658529dacd2a324412964b2732c4b4
2020-07-07 13:47:29 -07:00
Christopher Parsons
0eeff82c98 Merge "Add missing soong_docs inputs" 2020-07-06 22:34:51 +00:00
Jaewoong Jung
11c1e0f94d Output apkcerts file for android_app_set.
Soong and Make have no ways to figure out what splits will be outputted
from a given android_app_set, so it's impossible for them to provide
full PACKAGES.$(LOCAL_MODULE).CERTIFICATE entries, which are required to
build a final apkcerts.txt. This change makes extract_apks produce
apkcerts.txt files for each input modules instead. The Make-side
counterpart of this change merges all local apkcerts.txt into a final
one.

Fixes: 160119159
Test: main_test.go
Test: m apkcerts-list
Merged-In: I321e80fd636a955213761f56a3ac64bfe7f7f7c0
Change-Id: I321e80fd636a955213761f56a3ac64bfe7f7f7c0
2020-07-01 10:03:09 -07:00
Thiébaud Weksteen
5e291c238f Fix bpfmt issues and add bpfmt to preupload checks
Test: m nothing
Change-Id: I10526a33685335bdd26cc7692815133379d4e0f6
2020-06-29 09:14:08 +02:00
Chris Parsons
8f232a2702 Add missing soong_docs inputs
Specifically, make Android.bp.list and soong.variables explicit inputs

While this is not a comprehensive list of all inputs
of this action (as the action depends on all blueprint
files in the source tree), this is closer to the truth.

This is a rollforward CL, which was originally rolled
back, as path validation failed for when OUT_DIR was
an absolute path. Validation has now been relaxed.

Test: Manually verified ninja output, checkbuild approved validation
  for aosp-crosshatch, and manually ran frameworks/rs/build_rs.py with
  patch fix
Change-Id: I4eb0d517f57336dd54eaa4bd31f46df9e93e6da2
2020-06-26 10:36:18 -04:00
Christopher Parsons
cd767eefe5 Revert "Add Android.bp.list and soong.variables to inputs for soong_docs action"
This reverts commit 67eee7cae4.

Reason for revert: Broke renderscript build

Change-Id: Ie26b3c7a78afe44d719f399520e643174109593f
2020-06-18 21:49:48 +00:00
Chris Parsons
67eee7cae4 Add Android.bp.list and soong.variables to inputs for soong_docs action
While this is not a comprehensive list of all inputs of this action (as the action
depends on all blueprint files in the source tree), this closer to the truth.

Test: Manually verified ninja output, checkbuild approved validation for aosp-crosshatch
Change-Id: I77a0f612afb025232bee7d3ac31257b808991829
2020-06-18 12:06:37 -04:00
Patrice Arruda
7cc2074885 Run the metrics uploader in the background.
The metrics uploader was currently running on foreground where it
would copy the metrics files in a separate directory and then forked
into the background for the upload process. As a result, running the
lunch command would take a second longer to run since each metrics
uploader run had an average of half a second.

Bug: 140638454
Test: * Wrote and updated unit test cases.
      * Set ANDROID_ENABLE_METRICS_UPLOAD to point to the latest
        metrics_uploader bash script. Executed the "lunch 1" command
	and measured the running time. Executed "m nothing" command
	and checked that the metrics were uploaded.
      * Ran "lunch 1" and "m nothing" with
        ANDROID_ENABLE_METRICS_UPLOAD=""
      * Removed oauth from metrics_uploader and ran "m nothing" and
        "lunch 1". The oauth Message appeared only to "m nothing"

Change-Id: I13c61e666c8f44613dee291a704cef6a27335188
2020-06-13 00:34:37 +00:00
Treehugger Robot
039b04f4fc Merge "Upload build metrics after a build is completed." 2020-06-01 20:44:54 +00:00
Patrice Arruda
219eef3878 Upload build metrics after a build is completed.
Soong now supports the ability to upload metrics to another location
by setting the ANDROID_ENABLE_METRICS_UPLOAD to an uploader that
accepts the upload.proto proto buffer message. When the environment
variable is set, a set of build metrics files (soong_metrics,
rbe_metrics.pb and build_error) is uploaded.

Bug: 140638454
Test: * Wrote unit test cases
      * Setup the uploader, built a succcessful and failed aosp_arm-eng
        target and monitor the uploading of the metrics.
Change-Id: I76a65739c557dc90345e098ca03119a950ece2d2
2020-06-01 17:29:30 +00:00
Jooyung Han
9a37d4a602 Merge "pom2bp: add apex_available/min_sdk_version" 2020-05-28 09:09:59 +00:00
Martin Stjernholm
321bbe92b1 Explain checkLinker errors a bit more.
For an end user hitting these errors, the most likely reason is that
they have mixed up linker and crtbegin_dynamic.o from different builds
in their out tree.

Test: Build a linux_bionic executable in a tree exhibiting b/157549171.
Test: m nothing
Bug: 157549171
Change-Id: I64bee0848ed61226da6654caf3bafb51cf46d923
2020-05-27 21:31:42 +01:00
Lukács T. Berki
ad37304762 Merge "Call Delve using exec() instead of "dlv attach"." 2020-05-27 08:33:32 +00:00
Jooyung Han
43d3025cd5 pom2bp: add apex_available/min_sdk_version
apex_available/min_sdk_version are required to be available for mainline
modules(APK/APEX).

- java_import/java_library_static
- android_library/android_library_import

Bug: 156996905
Test: prebuilts/sdk/update_prebuilts.sh
Change-Id: I201ebabaa533c5ee19e394ab30ca2fa0e3d7ce0e
2020-05-27 00:37:45 +09:00
Sasha Smundak
827c55f502 Handle universal APK case in extract_apks
Cherrypick of ag/11553550

Bug: 157081961
Test: manual and builtin

Merged-In: I5ac64c845328a54024171da41c369050243462b0
Change-Id: I3ebc4a84501623774b78c9c19628994c1d71dc64
2020-05-20 13:10:59 -07:00
Lukacs T. Berki
a5e0f710c2 Call Delve using exec() instead of "dlv attach".
"dlv attach":

- Spams the terminal
- Requires sysctl -w kernel.yama.ptrace_scope=0
- Apparently, does not allow the debugger to inspect variables

Test: Manual.
Change-Id: I625369effaf5abda2b5d884c8ce5bde7247774eb
2020-05-18 09:50:18 +02:00
Jaewoong Jung
fa00c06187 Add apex_set module.
apex_set takes an .apks file that contains a set of prebuilt apexes with
different configurations. It uses extract_apks to select and install the
best matching one for the current target.

Bug: 153456259
Test: apex_test.go
Test: com.android.media.apks
Change-Id: I1da8bbcf1611b7c580a0cb225856cbd7029cc0a7
2020-05-15 11:02:01 -07:00
Sasha Smundak
7a894a6643 Implement extract_apks
Bug: 152319766
Test: manual and builtin
Change-Id: Ia15d66e86c7bcfd52f5b776173ca1665b68ff438
2020-05-14 19:00:59 -07:00
Dan Willemsen
347ba75679 Remove TEMPORARY_DISABLE_PATH_RESTRICTIONS
This was a temporary migration tool, and has been broken ever since the
ninja environment started being restricted. Remove it, since it has
lasted two releases.

Test: TEMPORARY_DISABLE_PATH_RESTRICTIONS=true m
Change-Id: I2e3709525598030f5c24d12fa966e531dfed33ae
2020-05-04 19:54:57 +00:00
Jaewoong Jung
a86983dc7a Use 0777 instead of FileInfo() for dirs.
Golang's FileInfo() implementation recognizes FAT filesystem directories
correctly, but doesn't actually add executable bits, which essentially
makes them inaccessible. Use 0777 when we know we're dealing with
directories.

(This is a cherry-pick change.)

Fixes: 153207404
Test: manually tested with an aar file with FAT permission bits.
Change-Id: Ie9f5320bc44d24b2f9d54d4036bb1747456655bb
Merged-In: Ie9f5320bc44d24b2f9d54d4036bb1747456655bb
2020-04-06 10:41:40 -07:00
Patrice Arruda
74b43998e7 Add build progress in Soong UI
Soong UI generates a proto file named build_progress.pb in $(OUT_DIR)
output directory that contains build action numbers (how many are executing,
finished and total) during the course of a build. This is for external
systems that invokes the Platform Build Systems and would like to know
the completion status.

Bug: b/150401146
Test: Wrote a bash script that continuously read the
build_progress.pb file and computed the build completed percentage
while building the aosp_arm-eng target. Compared the percentage between
the Soong output console and the one reported by the bash script.

Change-Id: I7c7347bc8e41958093892d8e2731c4f4169937dd
2020-03-19 13:15:05 -07:00
Colin Cross
9403e4b704 Merge "Collect metrics from inside soong_build" 2020-02-28 23:23:29 +00:00
Colin Cross
b72c909899 Collect metrics from inside soong_build
Collect the number of modules and variants and some basic statistics
on memory usage inside soong_build by writing out a proto that is
read back in by soong_ui.

Test: examine soong.log
Change-Id: I6926876377a4f6229cf41fdbf166ae03c885ea55
2020-02-25 20:50:00 +00:00
Bill Peckham
c087be1e8b Automate dependency on inputs of genrule module type.
This change automates one of the best practices for
custom tools. The best practice says "Prefer to list
input files on the command line, otherwise we may not
know to re-run your command when a new input file is
added." [1]

Normally you'd reference these inputs with $(in) or
one of the forms of the $(location*) substituions on
the command line for the custom tool. However, if the
custom tool does not accept the list of inputs on the
command line, the build system could fail to re-run
the custom tool if the list changes.

This change adds a hash of the list of input names to
the sbox command that wraps the custom tool. If the
list of inputs change, the hash will change, and
therefore the sbox command will change, causing ninja
to re-run the custom tool. The hash is visible to
(but ignored by) the sbox command, and hidden from
your custom tool.

[1] https://android.googlesource.com/platform/build/soong/+/refs/heads/master/docs/best_practices.md#custom-build-tools

Test: TestGenruleHashInputs
Bug: 149397658
Change-Id: I18b547ea3c4296ee15bd6150a4778a8f376d80b7
2020-02-19 12:47:35 -08:00
Treehugger Robot
7c78b3c98e Merge "If input zip file cannot opened, print its name." 2020-01-28 01:48:32 +00:00
Colin Cross
8b8bec3b3a Create a highmem pool and put metalava into it
Create a highmem pool based on the total RAM and the number of CPUs,
with an override via the NINJA_HIGHMEM_NUM_JOBS variable.  Put
metalava into the highmem pool.

Ninja does not support nested pools, and when goma or RBE is enabled
the maximum ninja parallelism is set very high with local jobs in a
local pool.  When both the local pool and highmem pool are enabled,
the total number of local jobs will be as high as the sum of the sizes
of the two pools.  Keep the highmem pool limited to 1/16th of the
local pool when remote builds are enabled to try to minimize the
effect while still limiting highmem jobs.

Fixes: 142644983
Test: m nothing, examine pools
Test: m USE_GOMA=true nothing, examine pools
Change-Id: Id79f11f44948992960ac34ecf831dacbe21bd332
2020-01-24 11:44:23 -08:00
Sasha Smundak
6172491b91 If input zip file cannot opened, print its name.
Test: touch empty.zip; merge_zips out.zip empty.zip
(output should contain `empty.zip: zip: not a valid zip file`)

Change-Id: If707876af2301d7ad7d065c004cffb0c2491c7df
2020-01-22 10:21:43 -08:00
Dan Willemsen
1bdbdec5a0 Take into account RAM usage for multiproduct_kati
Apparently we have some instances of Soong taking 16GB of RAM. While
that's a problem to solve on it's own, make multiproduct_kati's
auto-parallelism detection smarter by taking into account the total RAM
on the machine instead of just the number of CPUs.

Bug: 146925549
Test: check soong.log for autodetected parallelism values
Change-Id: Ice34c2cf73ea4f8f235521cbefc8654a79a04eef
2019-12-27 09:54:11 -08:00
Dan Willemsen
9609ad9c21 Add sharding support to multiproduct_kati
This is so that we can split the build_test build on large branches into
multiple builds, each testing a subset of the products.

Test: m blueprint_tools  (runs the go unit tests)
Test: build/soong/build_test.bash --only-config
      vs
Test: build/soong/build_test.bash --only-config --shard-count=2
Test: build/soong/build_test.bash --only-config --shard-count=2 --shard=2

Change-Id: I40ccc1aa477bc0ffa74ff564d155068509be18f0
2019-12-05 19:54:40 -08:00
Colin Cross
e94272d0b3 Silence trailing "1 warning" message from javac when warning is silenced
soong_javac_wrapper is silencing a useless warning:
warning: [options] bootstrap class path not set in conjunction with -source 1.9
but recent versions of javac have started also printing:
1 warning

Read the warning count, subtract the number of silenced warnings, and
reprint it if the non-silenced warning count is nonzero.

Fixes: 144118634
Test: javac_wrapper_test.go
Change-Id: Ie1d0a978188ab7b1c41027f718a1274608628123
2019-11-11 21:48:33 +00:00
Colin Cross
c0b9f6b9d2 Force dumpvars into dumb terminal mode and move log files
Force soong_ui in dumpvars mode to use a dumb terminal so that it
doesn't print an empty status table.  Also move the logs to files
that are prefixed with "dumpvars-" so that they don't rotate the
real build logs.

Fixes: 139307849
Test: lunch
Change-Id: If92421af07731fedc9ca42670f309e95922a2224
2019-09-23 14:24:57 -07:00
Colin Cross
b5f6dc47ee Fix lint error in format string
Test: go test
Change-Id: I8b7df9222db1fc88938cdfa43a6505f2da854876
2019-09-11 17:06:02 +00:00
Dan Willemsen
f80ef34c80 Merge "Rewrite depfile from sbox to stay reproducible"
am: e2ce551909

Change-Id: I2a51549dfb4419b9b6444d33104d35e655c82f4f
2019-09-04 12:38:05 -07:00
Dan Willemsen
c89b6f1981 Rewrite depfile from sbox to stay reproducible
sbox will generate a random directory for the output root, and most
tools will encode that directory name in the output target of the
depfile.

So embed the library from dep_fixer into sbox so that it can rewrite the
output filename to a static (reproducible) value. Ninja doesn't care
what that value is, so it's just "outputfile".

Also fix up rule_builder to actually tell sbox about the depfile.

Test: mmma system/iorap; check the contents of:
out/soong/.intermediates/system/iorap/libiorap-binder/android_arm_armv7-a-neon_core_static/gen/aidl/system/iorap/binder/com/google/android/startop/iorap/IIorap.cpp.d

Change-Id: I3640a2e8b0c034f143a35e398a8418a6d621b265
2019-08-29 14:47:40 -07:00
Alexander Smundak
73d9ff761f Merge "Allow reading zipfile list from a file."
am: 02ed5df4ac

Change-Id: I2888ac8c6b03bcfc09739e5329bc872d92a2e9a1
2019-08-07 10:16:20 -07:00
Sasha Smundak
1459a9259e Allow reading zipfile list from a file.
Android build generates 180K+ zip files that are to be merged and the
result passed to the indexing pipeline to generate the source code
cross-reference. Their names cannot be passed on the command line,
hence the need for this change. That required extensive changes to
the code as it no longer can be assumed that all the input files can
be kept open.

Bug: 121267023
Test: internal + treehugger + combine index files
Change-Id: I0a829f96ff7187ad967fb6b8cede387501ec57cc
2019-08-06 14:23:35 -07:00
Dan Willemsen
57d5786d59 Merge "Remove support for ONE_SHOT_MAKEFILE"
am: d778b93168

Change-Id: I32ef15cf94df3524e4e805327d4af246c95226a6
2019-08-01 14:31:53 -07:00
Dan Willemsen
ce41e943be Remove support for ONE_SHOT_MAKEFILE
Test: treehugger
Change-Id: Icdbe62bcb7bc4717228c5d974962b939d8eafee6
2019-07-31 18:11:27 -07:00
Dan Willemsen
12efb0533e Merge "Deprecate ONE_SHOT_MAKEFILE"
am: 1d73e5e566

Change-Id: Ie7ec9690ada5333a04bae34b7be2b1921863e0fc
2019-07-29 13:19:26 -07:00
Dan Willemsen
c6360837bb Deprecate ONE_SHOT_MAKEFILE
mm is now doing the same thing as mma. Now print a deprecation message
for other tools that may be setting ONE_SHOT_MAKEFILE directly (or users
that are still using a very old envsetup.sh).

Test: mm  # does not print
Test: ONE_SHOT_MAKEFILE=bionic m
Change-Id: I98221657024d824a61c0d721d5c7dd1a2ea2fb1e
2019-07-25 14:12:38 -07:00
Tony Mak
6f716787d1 Merge "Allow other module types in androidx"
am: 6edff69513

Change-Id: I225050ef2c10850eee20820d4602399f5402557b
2019-07-22 14:36:55 -07:00
Tony Mak
817850063b Allow other module types in androidx
1. Update pom2bp to support "host_support" device module.
2. Allow other module type in support library

(The intention is to support the room annotation processor)

Test: m

Bug: 137918883

Change-Id: I9a8fbcff903cc395d5dc38430774202ce389d6bc
Merged-In: I9a8fbcff903cc395d5dc38430774202ce389d6bc
(cherry picked from commit bd7c2f9a1d)
2019-07-22 13:18:48 -07:00
Patrice Arruda
fe6a621684 Merge "soong-ui: Redirect mmm to mmma."
am: 216e315a34

Change-Id: If9a777fd46c06d06fedeee558d68653c72a726df
2019-07-19 04:50:34 -07:00
Treehugger Robot
216e315a34 Merge "soong-ui: Redirect mmm to mmma." 2019-07-18 21:58:24 +00:00
Patrice Arruda
0cacc0468f Merge "soong-ui: Redirect mm to mma."
am: 80b562eb5b

Change-Id: I4f24fd0a055c7d7e80ff61d4c5ed1cd695ea4d6e
2019-07-18 14:06:45 -07:00
Patrice Arruda
a9e1215870 soong-ui: Redirect mmm to mmma.
This is the first part of deprecating the mmm command. mmm will simply
be redirected to mmma by building the list of modules with their
dependencies.

Bug: b/135187558
Test: Ran mmm external/libusb:libusb.
Change-Id: I50cf0d6a07197556ab6130e612561c6d5a3fb31e
2019-07-18 09:43:20 -07:00
Patrice Arruda
2274102f34 soong-ui: Redirect mm to mma.
This is the first part of deprecating the mm command. mm will simply
be redirected to mma by building the modules and their dependencies.

Bug: b/135187558
Test: Ran mm in external/protobuf directory.
Change-Id: Idad40f89e3768aeb459172f632a44fd7b0a201bc
2019-07-18 09:41:19 -07:00
Justin Yun
9321804a11 Rename product_services to system_ext
am: d5f6c8261a

Change-Id: Ifd77d7d729addae59b2b26a8cbdd2ebefffcabae
2019-07-09 02:14:59 -07:00
Justin Yun
d5f6c8261a Rename product_services to system_ext
Bug: 134359158
Test: build and check if system_ext.img is created
Change-Id: Ice3229baf97a94c24a6eea22e7d4571941d7f843
2019-07-09 08:57:05 +00:00
Colin Cross
cd48600321 Merge changes Icfc893c8,I40f03fc0
am: f67e1bee4a

Change-Id: I0b9cf1e3df4ef7ef46e73a5dc6d363fe918c29b7
2019-07-03 22:55:03 -07:00
Colin Cross
aa812d122c Allow debugging with SOONG_DELVE=<listen addr>
Allow running Soong in a headless delve debugger by passing
SOONG_DELVE=<listen addr> in the environment.

Bug: 80165685
Test: SOONG_DELVE=:1234 m nothing
Change-Id: Icfc893c8a8354a9bbc99112d9c83259cb41906d1
2019-07-03 16:33:31 -07:00
Colin Cross
a73c8a4e61 Merge "Compute critical path when ninja finishes" am: 7c9b4cf144
am: 7b4a190539

Change-Id: I1f3869fa3d5afb437bab8a4bf77ea03934c1d4ab
2019-06-26 16:53:05 -07:00
Colin Cross
7b6245388c Compute critical path when ninja finishes
Keep a running map of the critical path to each edge as it
finishes.  At the end of the build, find the edge with the
longest critical path and print the path to out/soong.log.

Test: critical_path_test.go
Change-Id: Ie01d26b068b768156b166bff00cdc3273e8124ca
2019-06-26 11:31:17 -07:00
Patrice Arruda
55ac3ddb21 Merge "Soong: Add build_error proto raw file to $(OUT_DIR)." am: 7d7cae85ab
am: 9f9778cff0

Change-Id: I7d2ee16ad31446360142295c61562cff467f9d19
2019-06-24 16:43:43 -07:00
Patrice Arruda
297cebad26 Soong: Add build_error proto raw file to $(OUT_DIR).
Soong_ui produces a build_error raw file to $(OUT_DIR) where
the file contains a list of build action errors. Each build action
error represents an error raised by a command. The build_error file
is populated if there was a build error.

Bug: b/132969697
Test: Ran m for successful build. Introduced a broken build change,
      ran m and verified using printproto that build_error was
      generated successfully.

Change-Id: I690ca1778b4e56f144a3173ba1d16d8494c32c15
2019-06-24 13:02:35 -07:00
Patrice Arruda
4b557d8f86 Merge "Soong: Add BUILD_MODULES build action in soong_ui." am: 45cdb71290
am: d009ef35b8

Change-Id: I0216045ba79f0cc26a4de5883589f52a65d9919c
2019-06-20 20:59:32 -07:00
Patrice Arruda
3928206ecc Soong: Add BUILD_MODULES build action in soong_ui.
There was one case that did not work:
  1) Create dir in root source tree
  2) Run "m libc"

The build would fail as it was using the mma logic. A separate
one named BUILD_MODULES was added to allow building specific
modules in any directories using the "m" command.

Bug: b/130049705
Test: unit test cases, ran "m libc" in a tmp directory inside
      of the source tree.

Change-Id: I8d23e685a673a311001ee8edd89bd73b662392dd
2019-06-20 16:35:12 -07:00
Patrice Arruda
5dbf869760 Soong: Use ctx.Writer in flag usage for dumpVar and dumpVars in soong_ui. am: db4c2f115e
am: e6fe9edbef

Change-Id: Ifcd0332f5f055633e4284c3387beea84cedf86a7
2019-06-18 15:52:12 -07:00
Patrice Arruda
db4c2f115e Soong: Use ctx.Writer in flag usage for dumpVar and dumpVars in soong_ui.
ctx.Writer is recommended stream to use for standard io.

Fixes: b/135487952
Test: Lunch and executed ./out/soong_ui --dumpvar-mode and
--dumpvars-mode to check the usage is output to the console.

Change-Id: I38351ebdb05f49f92d6b4bfb0423092299d1c5ea
2019-06-18 20:32:01 +00:00
Patrice Arruda
268a702f06 Merge "soong_ui: Add --build-mode flag in soong_ui" am: 2f93a2e07b
am: 9123c55392

Change-Id: Ib7ca530857ad469da3e634b7115a308b6d0d1972
2019-06-17 21:25:36 -07:00
Patrice Arruda
b7b2282fd1 soong_ui: Add --build-mode flag in soong_ui
The --build-mode flag is a new flag in soong_ui that accepts a build
action (m, mm, mmm, mma, mmma), the directory where the build action
is occuring and an optional list of build arguments and invokes the
build option. This is to deprecate the build actions behavior in
envsetup.sh and start using the one defined in soong_ui.

Bug: b/130049705
Test: Ran lunch and ran all the build action commands in soong_ui.
      Ran a bunch of invalid commands such as no Android.mk file in
      a specific directory, invalid target name, etc...

Change-Id: I88e9e59f47f8c5cdc872fb83a0f3f0a0919885b6
2019-06-17 17:21:32 -07:00
Patrice Arruda
10b659a60f Merge "Soong: Fix the package name in metrics.proto file" am: eec8d3aee3
am: 33989c88c3

Change-Id: Ife7e9c2ddeeeda40976c23f8ba07ba4bee53935c
2019-06-17 17:12:42 -07:00