It was confusing because bootstrapping uses two configurations: the
"global" config and the special-cased bootstrap one.
This change merges them.
Test: Presubmits.
Change-Id: I82b482cbe28a343ab6991374b2a28667e1a06b48
I erroneously assumed that the continuous builds don't set DIST_DIR,
whereas they do.
The variable naming is quite questionable, but for now, the important
thing is that the code works and I wanted to do that with the least
amount of change possible.
Also add some more logging:
- Shell environment
- Command line arguments
- Location of logs.zip
- The logs directory
Test: Presubmits.
Change-Id: I651b1045eeef568aa860fd2457fa7799ad68ba38
These are: TopFile and GeneratingPrimaryBuilder.
Also re-shuffle the list of flags to make a bit more sense and finish
the rename of BuildDir and NinjaBuildDir to SoongOutDir and OutDir,
respectively.
Test: Presubmits.
Change-Id: I103ff5f09f1c0d16f695a7da5dea13b55028e33e
Before, we piggybacked on the implementation of regular soong_build and
wrote a fake build.ninja file to satisfy Ninja.
Now, instead, the JSON module graph is a a separate action in the Ninja
output file. This has the pleasant side effect that one can flip back
and forth between generating the JSON file and regular Soong without
loss of incrementality.
Side cleanup: write .d files in a slightly cleaner way.
Test: Presubmits.
Change-Id: Ia853383567b9dd31c53f3bdf56cfc8d517b498ec
These are just out/ and out/soong/ and the old names were quite
confusing.
Test: Presubmits.
Merged-In: I999ca62162062f27e449f83ffb00fa580d4065b8
Merged-In: I2fcd80a9e376a6842a06aa104a9f6f5c5e16f8c7
Change-Id: Ib481d2aac40df8da1b74174dc8ec576c1cb48be0
In particular, the writing of build-globs.ninja and the collection of
glob list files for build.ninja.d is now done outside of the Blueprint
invocation.
This is useful because RunBlueprint() is sometimes called multiple times
in a single soong_build invocation and because now glob processing is
not hidden by the machinery of Blueprint.
Test: Presubmits.
Change-Id: I6eb19b357901e452e6d1e7ce055f8a3bab4a74f4
They used to be defined in Blueprint but since it doesn't have a
separate existence anymore, we can consolidate all command line
arguments here.
Note that Blueprint is called directly from soong_ui, but that happens
by a simple Go function call and not by invoking a separate binary, so
command line arguments are not needed there.
Test: Presubmits.
Change-Id: Ifa2c101ddbe424b76fc5a508d3d41c329e7353ca
The following files are forked:
- build-globs.ninja
- The glob list files
- .d files for the glob list files
- The output file (build.ninja or the bp2build marker file)
This makes bp2build and soong_build not overwrite each other's files
they need for proper incrementality.
Test: Presubmits.
Change-Id: I69d192cbd36ecd9677f46f3fa095dfce6f872227
This fixes null bp2build builds.
When running bp2build, out/soong/.bootstrap/build-globs.ninja wasn't
generated, therefore none of the globs in out/soong/.primary/globs have
actions to generate them. This makes bp2build always re-run on a null
build, because it depends on those non-existent globs in its depfile.
Fix the above by generating a build-globs.ninja file with the globs that
bp2build encounters during conversion, and add tests to verify that null
builds and bp2build -> soong workflows correctly regenerates the
build-globs.ninja file.
Test: soong integration tests for null soong->soong, bp2build->bp2build and bp2build->soong builds.
Fixes: 193650250
Change-Id: Icfbad351dfa85541d1ea451f73e65cac87097926
This is done by moving SetupSignals() to its own little package.
There are a number of tiny little utility packages for soong_ui we might
be better of merging, but that's for another change (maybe)
Test: Presubmits.
Change-Id: I07b0ca98bfb8884ef4223d665e632183b9896a0d
This mostly cuts the dependency from multiproduct_kati to the build
package; the remaining logic is SetupSignals which will be dealt with in
a followup change.
This requires some code duplication (e.g. ensureFileExists() and
detectTotalRAM()).
This removes a Finder invocation added in aosp/452598; it looks like
it's not a measurable performance hit. The quoted half a second does not
matter much in build_test{1,2,3,4} which take an amount of time measured
in hours.
It also makes ANDROID_QUIET_BUILD unsupported for the UI of
multiproduct_kati; the function of that variable seems to be to suppress
the output from build actions, which don't exist in multiproduct_kati so
it should not matter. In addition, I checked whether anyone sets this
flag for multiproduct_kati and it doesn't appear to be the case.
Test: Presubmits.
Change-Id: I1c2901e1ea69c3a35429e6c967a7bba845676ef7
This serves to not link parts of soong_ui (and eventually soong_build)
into a separate, weird binary. This is in turn good because they contain
any number of global variables and no one really thought about what
happens when two instances are executing at the same time in the same
address space.
This comes with a slight performance hit: 5 aosp_* projects build 152
seconds instead of 146. I suppose this is a price worth paying for a
clean design?
Test: presubmits.
Change-Id: I5623dcab2290f0fc392dd2ede597b9794a3d2a4e
Print more messages in run_with_timeout to make it easier to
determine which log statements came before the timeout and the
on_timeout command.
Test: run_with_timeout_test.go
Change-Id: I92ad663529fa03d91185ba382654a73f0eba1ab7
Add __dlwrap_linker and __dlwrap_linker_end symbols for use by the
embedded linker trampoline to determine which load segments are
part of the embedded linker and should not be exposed to the
executable.
Use flags to name the sections to avoid colliding with __dlwrap_linker
when no section name can be determined.
Bug: 190084016
Test: build with musl
Change-Id: I51ccbf9c15a65c6194fa263b7504a598dbf4ad18
This is as part of an effort to move environment variables into config
files so that configs can be tied down to source.
Test:
Ran a build with "ANDROID_BUILD_ENVIRONMENT_CONFIG=googler m" which ran
a non-RBE build (since this is vanilla aosp).
Also ran a build with "ANDROID_BUILD_ENVIRONMENT_CONFIG= m" which ran a
non-RBE build.
https://paste.googleplex.com/5637282880028672
Bug: b/194679562
Change-Id: I416e8da75f84aa2b53995f525cf50501488dc972
Major operations:
* Fix the go_package entries to be consistent, as the new tool
complains when it doesn't have a '/'.
* Regenerate with the new protoc-gen-go tool.
* github.com/golang/protobuf -> google.golang.org/protobuf
* proto.[Un]MarshalText -> prototext.[Un]Marshal
Change-Id: Ie1147bd2457fafb66ba555461b3bf14f0561a25d
Use a note added to linker.s to place the linker in the linker script,
which will work musl, which doesn't have .note.android.ident, while
continuing to work for linux bionic.
Test: m checkbuild
Change-Id: I9cd4ad7c8e8fcb4d725746106d5beaa699a77807
I wrote these flags primarily for build-tools:external/kythe, where we
only use a subset of the Go packages defined in the project.
-limit allows defining the entry points to keep in the Android.bp, and
the utput will omit everything except those entry points and their
dependencies. This allows the kythe Android.gen.bp file to drop from 195
modules to 31 using two flags, without having to manually exclude the
other 160 packages.
-skip-tests does as you'd expect, though ideally it wouldn't be
necessary. We didn't import all of the test-only dependencies for kythe
into the source tree, so we're not able to build them.
Change-Id: I49e9f12dac53a74c6e05725ee41fb7c0ed8b0cbb
This CL adds support to bp2build/Soong to dump a BUILD file under
out/soong/soong_injection/targets containing alias targets to their real
targets for every converted Soong module, regardless of whether they are
handcrafted or generated.
Test: TH
Change-Id: Ic1816fda5d019c395301618134fac68b3057d752
Use an implicit linker script instead of flags in a file to specify
the locations of the host bionic embedded linker and to prevent it
from being stripped.
Test: build and run host bionic binary
Change-Id: I64e12118d33c67bab5e657cbc3ea8cde8f0fd7e6
Replace .linker.sect0, etc. with .linker, .linker.text, .linker.data
and .linker.data.rel.ro by extracting the name of the first section
in each program header.
Test: build and run host bionic binary
Change-Id: I25107547536a3a3963fdeb311c45a7ee53c0bc45
The host bionic bootstrapping no longer needs an injected symbol.
Replace host_bionic_inject with host_bionic_verify that validates
the resulting binary, and add it as a validation dependency of the
binary.
Test: build and run host bionic binary
Change-Id: I3e303d2a164b6eef851bdc8075e6ee456c05b0a8
- Rename the "BuildX" variables to "RunX"
- Remove redundant comments
- Inline all the "what to do" based on config in build.go
- Inline some constants only used in one place
Bug: 189187214
Test: m nothing
Test: build/soong/build_test.bash
Change-Id: I111a69e642212d7938d4971283545e0d9acbb01a
Merged-In: I111a69e642212d7938d4971283545e0d9acbb01a
(cherry picked from commit d274ea9196)
I6dca478f356f56a8aee1e457d71439272351390b replaced calling
build/blueprint/bootstrap.bash with running minibp directly,
but didn't propagate the EMPTY_NINJA_FILE environment variable
to minibp. Since everything that uses EMPTY_NINJA_FILE is
executed directly from soong_ui now, replace the EMPTY_NINJA_FILE
environment variable with a config bool and pass it to minibp.
Bug: 189148777
Test: build/soong/build_test.bash --dist --incremental --shard-count=300 --shard=1 && du out/multiproduct
Change-Id: I4d64275ce02c5d68948012f71ac4dc3795af9e85
This CL fixes a typo in writeFakeNinjaFile to correctly write
a fake out/soong/build.ninja and its depfile. It also modifies bootstrap
phase to *not* run the main soong build phase (which takes more than a
minute) if GENERATE_BAZEL_FILES=1.
This change has the side effect that `GENERATE_BAZEL_FILES=1 m nothing`
no longer generates the real build.ninja, which is fine because one
shouldn't be using GENERATE_BAZEL_FILES=1 for that anyway (or, use
USE_BAZEL_ANALYSIS=1). This change has no effect on mixed builds.
Time on a change to Soong or any Android.bp files:
Before: bp2build_workspace_marker (~20 seconds) + build.ninja (1 min)
After: bp2build_workspace_marker (~20 seconds)
Time on the second of two consecutive `b build //bionic/...`: 2.070s
Test: TH
Test: Soong integration tests
Change-Id: I43720641815994caba97b8d165d7c3fc254cbd06
This tool automatically writes Android.bp files based on information
discovered from Go, similar to pom2bp or Bazel's Gazelle. This
implementation is fairly limited, but is usable enough to import
libraries like protobuf without a lot of manual effort.
Change-Id: Iecc145bc7ab9b9e9c3739f77b694b3f4bc2c6396
Use a concurrency-safe writer in runWithTimeout to avoid data races
on the bytes.Buffer passed in during tests.
Bug: 181095653
Fixes: 187149270
Test: Test_runWithTimeout
Test: go test -race ./cmd/run_with_timeout
Change-Id: I57a889765cb9ee7b42983f0906313e0c2d1e414e
Add a command that can be used to wrap actions with a timeout, and
optionally run an extra debugging command on timeout.
Bug: 181095653
Test: run_with_timeout_test.go
Change-Id: I91df5c3fb5277968717815a4ad4612113766dab1
This refactors bazel-build mode determination logic in soong_ui so it's
clearer which of three possible modes are being used in a given
invocation (NO_BAZEL, GENERATE_BUILD_FILES, or MIXED_BUILDS).
Test: bootstrap tests
Change-Id: I41d2baebf8d560c2cc42db8daa8b936101d453e3
It's under out/soong/workspace and is a symlink forest that merges BUILD
files generated by bp2build and the source tree.
bazel-* symlinks and $OUTDIR are currently ignored.
Test: Presubmits.
Change-Id: If14df23bd0f6a6f2d7d1140f95213f2aee66d652
The full command line run inside sbox can be very long, and printing it
after the errors printed by the failing command can hide the error
messages. Buffer the output of the command and print the failing command
line before the output if it fails.
Bug: 185516277
Test: m out/soong/.intermediates/frameworks/base/system-api-stubs-docs-non-updatable/android_common/metalava/api_lint.timestamp with lint error
Change-Id: I893f3dd01f1baf195e182111c5c49e92eb82f3b0
Error messages printed by failing commands may reference output files
that were created by the command, for example printing a command line
to copy and paste to update a baseline file. Copy output files in the
sandbox to their final locations, ignoring missing files, so that the
messages are valid.
Bug: 185516277
Test: m out/soong/.intermediates/frameworks/base/system-api-stubs-docs-non-updatable/android_common/metalava/api_lint.timestamp with lint error
Change-Id: I604a11c9b54e409ca5bc5c016cd04b3133f74a60
Update the path bootstrap_test.sh uses to check for bpglob reruns
to follow the changes made for sharding globs.
Use bootstrap.GlobFileListFiles to get dependencies during bpbuild.
Test: tests/bootstrap_test.sh
Bug: 159845846
Change-Id: Ibaa7c4360881ed6b666a811bf34b19ea0cdcafe9
It was only appended to the return value, which can be done by the
caller just as well.
Test: Presubmit.
Change-Id: I962696e0dbd4c3496a0159d01d2a911675fd4217
It now handles adding .bp files and changing globs.
In order to do this, depfiles are now written separately from RunBlueprint.
This is necessary due to the confluence of a number of seemingly
unrelated factors:
1. The glob filelist dependencies are discovered in globSingleton
2. Singletons need to be registered because otherwise singleton module
types panic
3. Singletons don't work because they require mutators bp2build does not
run
Due to (1), we would need to run the glob singleton. However, due to (2)
and (3), we can't run singletons and have to run Blueprint with
StopBeforeGeneratingBuildActions, which is when the build actions
writing glob files would be generated. So what happens is:
1. When bp2build is run, the glob singleton is disabled
2. At the end of bp2build, the list of glob files is artifically added
to the depfile of the workspace marker file
3. When build.ninja is generated, the Ninja file containing the glob
list file is written by the now-active glob singleton
Test: Presubmits.
Change-Id: I3c5898d8c57c554a93520276c64a952afc912dbe
This is done by setting the INTEGRATED_BP2BUILD environment variable
when invoking the build.
Even though the name of the marker file insinuates that a Bazel
workspace is already created, this is not the case yet.
An issue that remains is that a .d file is not written for the marker
file so it won't be rebuilt if a .bp file changes. Fixing this requires
delicate surgery because writing the .d file is the result of delicate
interplay between Soong and Blueprint.
There are also a number of semi-related fixes:
- The name of soong.environment.{used,available} is now on the command
line of soong_build (soong_docs is still special cased because its
command line in the Ninja file is taken from the os.Args of
soong_build so it's not trivial to remove the --{available,used}_env
from it
- bp2build writes a separate soong.environment.used file
- I had to call SetAllowMissingDependencies() separately when creating
the android.Context for bp2build so that bp2build runs in the
integration tests (it was not obvious how not to do this)
- Fixed a number of integration tests where a command with an expected
exit code of 1 was used as the last one in a test case, thereby
breaking the test suite
Test: Presubmits.
Change-Id: Ibeb61c26022cf801dcb98505b4039151b3409873
Properly plumb the available environment to the configuration instead of
going through the global variable originalEnv.
Test: Presubmit.
Change-Id: Ia1ea753d5e72c02a0dcaa4c0b43bd6e24fc47cec
This is so that the way soong_build is invoked is simpler, which is in
turn useful so that it can be invoked multiple times within a single
build, which in turn will be used to invoke bp2build routinely (as
opposed to manually when needed)
Test: Presubmits.
Change-Id: Iddaebb05ff7bcedc0db8273192bb31284b739920
Usage: SOONG_DUMP_MODULE_GRPH_JSON=<filename> m nothing
Test: The new test case in bootstrap_test.sh .
Change-Id: I69005a75d47dff915d27187645d0cd1cbb3467ef
This makes it more easy to see what's happening under which modes. Modes
are:
- Regular builds
- Mixed mode builds
- bp2build
- soong_docs
Test: Presubmit + bootstrap_test.sh
Change-Id: I06e3fcb84986b919be1cd9eb456531259ede1183
Previously min_sdk_version was only set for *_library build rules, and
not for *_import targets. This made min_sdk_version default to
sdk_version, which is usually "current", which is not correct for these
libraries.
Test: pom2bp -default-min_sdk_version 24 for androidx
Change-Id: I3b029315abeb559e66efad847ae664efb79f4509
Sbox preserves the permissions of input files when copying them into the
sandbox. A read-only file copied into the sandbox multiple times causes
a permission denied error on the second write. Building in Bazel results
in more read-only files, which triggers the issue on existing sbox rules
with duplicate input files. Remove the destination file when copying if
it exists.
Bug: 184113103
Test: m USE_BAZEL=true
Change-Id: I7edf92d82b766100e3cbbd90d22428269d7d0167
Previously 24 was hardcoded (for somewhat unclear reasons). Update
the tool to accept specifying the correct value on the cmdline.
Test: pom2bp -default-min-sdk-version 24
Change-Id: I39c23dd4bca91684a2a68d7b963cf3cb66eb1f33
The current implementation causes inputs listed in an rsp file used with
sbox to be duplicated 3 times in the build.ninja file; once as a
dependency of the rule, once in the rspfile_content field of the rule
with the paths rewritten to be relative to the sandbox, and once in the
rule to write the sbox manifest. When RBE is enabled it also gets a
fourth copy in the list of files to be treated as inputs by rewrapper.
Reduce this to a single copy by using "$in" for the rspfile_content so
that the files only have to be listed in the input dependencies of the
rule, and then add support to sbox to rewrite the rsp file while copying
it into the sandbox, and pass it to rewrapper as well.
Test: m lint-check
Change-Id: I3f46f61119508d39a8bb231c99fc130153fb6f04
sbox is going to need to read and write response files, move ReadRspFile
to its own package.
Test: response_test.go
Change-Id: Iecb5486b4aaeb2531828743ad8ef784df675e18e
Previously, soong.environment.used was written after build.ninja and if
the amount of time that passed between the two was long enough, Ninja
would decide that build.ninja is older than soong.environment.used and
rebuild it.
Test: test_null_build in bootstrap_test.sh in a loop.
Change-Id: I5467da487e8e8f2646644b8a7fb9549b9ff18276
This CL adds the support for specifying lists of directories in
build/soong/android/bazel.go, which are then written into
out/soong/bp2build/MANIFEST. Using this configuration,
modules/directories can either default to bp2build_available: true or
false, while still retaining the ability to opt-in or out at the module level.
It also ensures that ConvertWithBp2Build returns true iff the module
type has a registered bp2build converter.
Test: go tests
Test: demo.sh full
Test: TreeHugger presubmits for bp2build and mixed builds.
Change-Id: I0e0f6f4b1b2ec045f2f1c338f7084defc5d23a55
This is achieved by writing soong.environment.used in Main() instead of
as a side effect of a singleton. This makes a difference because build
actions are not generated when GENERATE_BAZEL_FILES=true is set,
therefore the side effect did not happen.
Arguably, Main() is made worse by this change, but I don't want to
tackle the problem of readably determining which mode soong_build is
running in in this change.
Test: Presubmits + the additional test.
Change-Id: I66af2429aedf008762173eaaa55b828b4cf4328b
It's replaced with a flag in bootstrap.Args .
Test: "m nothing" (presubmits take a full workday these days)
Change-Id: Ia0bfa2091013e389890e583c559006077806af99
This requires linking Blueprint into soong_ui. It lets us avoid the
complicated dance of Ninja files and shell scripts: now the information
as to how soong_build is built is passed directly to Blueprint using a
struct that contains all the information the command line arguments used
to contain.
The ability to run Blueprint from the command line is kept (for now).
Some variables in bootstrap/command.go needed public accessor functions
because soong_build reads them. This will be disentangled by moving the
flag parsing to soong_build.
The presence of the flag definitions in Blueprint means that soong_ui
now also accepts them. This is not a problem in practice because they
are ignored and because soong_ui itself is hidden behind a few layers of
shell scripts.
Test: Presubmits + the new bootstrap_test.sh .
Change-Id: I6dca478f356f56a8aee1e457d71439272351390b
When running commands in chdir mode sbox needs to translate the
directory placeholders to paths relative to the sandbox directory
instead of relative to the top of the source tree.
Also translate relative PATH entries into absolute paths so they
are still valid when the current directory is changed.
Bug: 182612695
Test: manual
Change-Id: Idcbe20466888909d423d62788bc9c35f4e03b398
The end goal of this exercise is to remove all mutable global state so
that multiple Blueprint instances can be run in the same process.
Test: Presubmit.
Change-Id: Idb34b0920f2c7d92efb9328ce8a78b7306f89571
If both bp2build_available and label are specified, label will be
preferred.
Initially, we copy the entire BUILD.bazel file. Eventually we may move
this to use bazel query for a more accurate result.
Test: go test *
Test: build/bazel/scripts/milestone-2/demo.sh full
Test: GENERATE_BAZEL_FILES=true m nothing
edit bionic/libc/tools/BUILD.bazel
GENERATE_BAZEL_FILES=true m nothing and verify changes picked up
Bug: 180516554
Change-Id: I43025583300e6b10d2c18032cd4a76237b578d59
In addition, make the Ninja command nicer by using backticks instead of
backslash-quoting double quotes.
Test: Manual: "m queryview"
Change-Id: I5d6e1d41424e0229bfe08e2e7eaf2fb025ed48c2
This works by setting the SOONG_UI_DELVE= environment variable to the
port on which soong_ui should accept a Delve connection on.
This is achieved by reusing the Delve execution logic between soong_ui
and soong_build.
Test: Manual.
Change-Id: Id2c1d4b6faac1a4a3918c91030ce2239f7daf54f
This lets one avoid any decisions as to when to chdir there during its
execution and leads to better sandboxing because the pwd doesn't leak to
init() functions anymore.
Test: Manual.
Change-Id: I1560da8ed3a621249426f9e8908aa890c21e13ba
Also delete the now-unnecessary soong_env binary.
Test: Manually. Also checked what happens when a used environment
variable changes.
Change-Id: Ib393e7f444e94819198c6cce4bcd8719d9fd9441
Sample output:
[bp2build] cc_library_headers: 5 targets
[bp2build] cc_object: 5 targets
[bp2build] filegroup: 4 targets
[bp2build] genrule: 4 targets
[bp2build] sh_binary: 1 targets
[bp2build] Generated 19 total BUILD targets from 39270 Android.bp modules.
This CL adds an additional CodegenMetrics return value to
GenerateBazelTargets calls, which are called from bp2build, queryview,
and their tests. For this UI, we only want to use it for bp2build, and
not queryview or tests, since it's not useful for the former, and can
pollute the CLI for the latter.
Test: build/bazel/scripts/milestone-2/demo.sh
Change-Id: Ic84307a1ed1a25e360c9b23459e5449d932bc2e7
Before this change, changes to any Android.bp file would not trigger a re-run of bp2build/Soong, since the depfile for Soong's main output, the build.ninja file, didn't contain the list of Android.bp files. bp2build only reran when Soong itself changed, which happens regularly while developing bp2build, but not at all while modifying Android.bp files.
This change correctly dumps the list of all Android.bp files in the tree to the depfile for out/soong/build.ninja, therefore correctly triggering a re-run of bp2build when any Android.bp file changes.
Test: GENERATE_BAZEL_FILES=true m nothing; edit an Android.bp file; GENERATE_BAZEL_FILES=true m nothing -> successfully re-runs and regenerates out/soong/bp2build.
Test: inspect out/soong/build.ninja.d manually
Fixes: 179452534
Change-Id: Iad80c381b3afa38a4d4fafca035e46aa2a1f206a
bp2build is a Soong mode that returns as soon as the BUILD files are generated. This causes it not generate the build.ninja file, which the Ninja process executing soong_build itself expects to produce as an output. If there isn't an existing build.ninja file generated from a previous build, GENERATE_BAZEL_FILES=true m nothing will fail.
This CL generates the expected files as a workaround, and also makes GENERATE_BAZEL_FILES=true skip the Kati/Ninja steps in soong_ui since they aren't needed.
Test: rm -rf out/ && GENERATE_BAZEL_FILES=true m nothing && m libc
Fixes: 178683777
Change-Id: I2515ef7961682d2be5f096ed24831cc185165a67
multiproduct_kati sometimes passes 20GB per soong_build process,
increase required ram per job to 25GB.
Bug: 178376498
Test: none
Change-Id: I446f52ea7419bc3015d940511af1f03b15eb651e
The CodegenMode enum helps to differentiate between bp2build and queryview modes of generating BUILD files from the Soong module graph.
bp2build is used for generating BUILD files from an unconfigured Soong module graph for Soong->Bazel conversion, whereas QueryView is a front-end for querying the existing Soong configured module graph with Bazel query.
Test: go test
Test: TH
Change-Id: I5709f591f1ae08b2770adf272d8ce60c9ee1f46a
This CL creates the framework necessary for generating
BazelTargetModules from regular Soong Android modules.
BazelTargetModules are code-generated into Bazel targets in BUILD files.
See the follow-up CL for examples of creating filegroup/genrule
BazelTargetModules.
Test: GENERATE_BAZEL_FILES=true m nothing # creates out/soong/bp2build
with no BUILD files, because there are no BazelTargetModules in the
module graph.
Change-Id: I33a96365bd439043b13af6db9e439592e9983188
Implement bp2build codegen as a discrete step that runs after an
alternatively registered pipeline of mutators, instead of a
presingleton.
bp2build codegen requires a Context that supports VisitAllModules and
PathContext, so this CL also makes a BpToBuildWrapperContext that
conforms to PathContext by adding two method implementations.
Test: GENERATE_BAZEL_FILES=true m nothing && bazel query //... --config=bp2build | wc -l # 31433
Test: m queryview && bazel query //... --config=queryview # 63638
Change-Id: I0dd359746584b228046d2d0ff00895f28f9bdfc3
A SingletonModule is halfway between a Singleton and a Module. It has
access to visiting other modules via its GenerateSingletonBuildActions
method, but must be defined in an Android.bp file and can also be
depended on like a module.
Bug: 176904285
Test: singleton_module_test.go
Change-Id: I1b2bfdfb3927c1eabf431c53213cb7c581e33ca4
Splits queryview into queryview and bp2build. The latter runs as a
presingleton (could be converted to a singleton). This prevents needing
to run soong_ui a subsequent time to build the query. Queryview remains
as a separate step to prevent increasing runtime due to this generation
every time Soong runs.
Currently this is running as a presingleton as this gives a translation
of Android.bp files after only LoadHooks have run, no mutators.
Test: go tests
Test: m queryview && bazel query --config=queryview //...
Change-Id: If2ba28c7ef60fbd41f43bda6623d41c8c7d23a1b
Also get Bazel to write real files there (not symlinks) so that the DIST_DIR can be independent.
Test: Manually using e.g. DIST_DIR=/tmp/foo USE_BAZEL=1 m dist
Change-Id: I39d5219500864c9ecc85f356a028e9b5bf2607f4
The contents of the bazel metrics directory (out/bazel_metrics or
<dist dir>/logs/bazel_metrics if dist arg is specified) is added
in the list of files to be uploaded in order to do performance
analysis of bazel builds.
Bug: b/174479728
Test: * m nothing and checked metrics_uploader log file to ensure
if the metrics is uploaded.
* USE_BAZEL=1 m nothing and checked if the set of metrics
files plus the build_bazel_profile.gz is uploaded through
the metrics_uploader log file.
Change-Id: I394a1b621a06f9acccf154faee6dcd4200fcff4c
Code refactoring has been done for logs directory logic code since
the bazel metrics directory depends on the log directory. For builds
that did not specify a dist directory, the log directory is under
out directory. With dist, the logs directory is under <dist dir>/logs.
This matters for Android CI builds where the metrics files are
placed under logs directory. With this change, the bazel metrics
directory and corresponding files will be under <dist dir>/logs
directory for Android CI builds.
Bug: b/174479728
Test: * USE_BAZEL=1 m nothing (bazel_metrics dir in out dir)
* m nothing (bazel_metrics dir deleted)
* DIST_DIR=/tmp/build USE_BAZEL=1 m nothing dist (bazel_metrics
is in /tmp/build/logs directory)
Change-Id: Ic9e1ff49a1964fcaaf801bde2c19f33597ca1db4
Add a boolean flag to the sbox proto to request the executable bit
to be set after copying a file. This will be used for sandboxing
tools when copying a file in from the source tree that would
normally get its executable bit set during installation.
Bug: 124313442
Test: sbox_test.go
Change-Id: Ie2c197bb5183ffc1bf63fd6effd175143cd324d4
This CL adds a new CONVERT_TO_BAZEL env var, and a bp2build goal. It
reuses the queryview architecture, but registers no mutators before
converting to BUILD files. This gives us a blank slate of the module
graph to work with, and create a shadow/alternate pipeline of mutators
to converge the Bazel BUILD graph to be semantically equivalent with the
Android.bp graph (after apply its current set of mutators).
The command to do so is:
$ CONVERT_TO_BAZEL=true m bp2build
To not clobber with queryview, the generated files are in
out/soong/bp2build.
Test: CONVERT_TO_BAZEL=true m bp2build && bazel query --config=bp2build //...
Test: m queryview && bazel query --config=queryview //..
Test: soong tests
Test: TH presubmit
Fixes: 174465461
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: I4dd0ccc73abc345d70a50ca2803d6f400cd8c863
Previously in order to make use of dist entries with tag properties it
was necessary for each module type to be modified to store the results
of GenerateTaggedDistFiles(module) in the AndroidMkEntry.DistFiles
property. This change generalizes that mechanism to work across all
module types.
This change does improve the behavior of a couple of tests where
previously a dist with tag property was ignored because tagged dist
files were not available now it is used correctly.
Some module types do not implement OutputFileProducer interface and so
they cannot handle dist with tag properties. This change makes it an
error to attempt to do that. That necessitated adding OutputFiles(tag)
method to customModule in queryview_test.go as some of the tests in
that file used dist entries with tag properties.
Follow up changes will remove the code that was made redundant by this.
Test: m nothing
m dist sdk - before and after this change, compare result to
make sure that there are no significant differences.
Bug: 174226317
Change-Id: Ifc54d67db10ce0d0fe8179c05b97a2be2113be4e
The sbox tool is used to wrap genrule actions, and run them in a
temporary directory. Afterwards, the outputs are moved into their
proper location in the output tree. However, some tools embed the file
name of the output file (as passed to the tool) into the output file.
For example, the perfetto code generator script uses the output file
name to generate a C header guard when it generates
perfetto_src_base_version_gen_h/gen/perfetto_version.gen.h.
When using remote execution / remote caching, these genrule actions are
run locally whenever the output file doesn't exist, effectively making
them unique (across time and between users).
They then cause cache misses on all actions depending on these output
files as well as on transitive actions. In the above example, this
causes libperfetto.so to differ, which then causes all actions
depending on libperfetto.so to be rerun unnecessarily.
As a fix, this commit changes the sbox tool to use the sha1 hash of the
manifest path. The tool already seems to assume that there are no
concurrent runs of the same tool with the same manifest (otherwise
it would fail flakily depending on exact timing). It seems therefore
safe to use a temporary path deterministically derived from the
manifest path.
Test: ran multiple builds w/ remote execution; observe proper caching
Change-Id: I5b73ffd3b7f85cbb0336dfa1675de7ac0e2fd1a8
The hardlink optimization attempts to save I/O by hardlinking input
files into the sandbox instead of copying them, but it never works on
a Linux build due to nsjail always making out and the source tree into
separate bind-mounted filesystems. If the optimization did work it
would actually cause build failures when it hardlinked the relative
bionic/libc/fs_config_generator.py symlink into the sandbox directory
without the target. Just remove the optimization for now since it
never works.
Test: m checkbuild
Change-Id: I96e8c0c145e7c99958639594edf8a93b69ae90eb
No current use cases for soong.config and user-editable local
configuration options for Soong, so let's remove it to reduce the API
surface.
The MegaDevice configuration is used for building every module, for
every cpu variant, for every architecture, but it apparently isn't
used for a while now.
Test: m nothing
Test: TH
Fixes: 174188200
Change-Id: I2550e70ff6f9c8b57e9a7cc517d6a119a032a27a
This relands I3b918a6643cea77199fd39577ef71e34cdeacdb1 with a fix
to create the directory for the output depfile if doesn't exist.
In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.
Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I8af00c8c0b25a92f55a5032fcb525715ae8297c2
This change fixes a bug that zipsync didn't handle symlink correctly;
symlink was extracted as a regular file whose content is the target
path. Fixing the problem by correctly creating the symlink using
os.Symlink.
Bug: N/A
Test: manual
Change-Id: Ib6685c14e1950d1057d89672883cdd9e4879069a
In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.
Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I3b918a6643cea77199fd39577ef71e34cdeacdb1
The UseBazel() function will be used in several places to perform
different functionality such as setting up the metrics path.
Bug: b/173028918
Test: m nothing
Change-Id: Ie94073b8f0552ec7528e1cc2f3be74b693dc3135
This shaves off ~20s, which is ~11%, of `m nothing` with mixed builds.
Test: Manual verification with mixed builds of aosp_flame.
Change-Id: Ief1bc56c92329e899c829fc1374eea9ee9b94fb7
Prepare for using Config when adding singletons by passing
Config to NewTestContext and NewContext instead of to ctx.Register.
This will enable a followup change to store SingletonMakeVarsProviders
registered on the Context in the Config, which is necessary to run
multiple tests in parallel without data races.
Test: all soong tests
Change-Id: Id229629a4e42ff4487d317241673837726c075fc
Bazel overlay is an experimental feature. This renames the feature to 'queryview' to better describe its purpose, and also move away from the already overloaded 'overlay' term in Android.
Test: m queryview && bazel query --package_path=out/soong/queryview //...
Change-Id: I8b5068c7db46cb61a03a8e87af9c7c9077ebeff9
A recent change to Blueprint [1] added the feature to infer the type of
list elements to more than just "list of strings", but this also changed
the text to "list of string" (without s). This CL fixes the bazel
overlay, which uses "list of strings".
Test: m bazel_overlay && bazel query //... --package_path=out/soong/bazel_overlay
[1]: 764a771917%5E%21/#F1
Change-Id: I7522d15b1ce751807e1881ce1e94f068d226b316
There is no need for multiproduct_kati to run the tests for
every product, they don't vary by product config. --skip-soong-tests
can also be used for local development to run soong_build even if
the tests don't pass.
Bug: 156428456
Test: m --skip-soong-tests nothing
Change-Id: I9c00e3d1b6e51d17bb290339c3f124d4d1c9e69f
We need to capture the user build command to understand what additional
arguments is passed in to Soong. This data is useful to know which
targets are being executed and have statistics on top build targets
are specified.
Bug: b/169453712
Test: ran "m blueprint_tools", "m nothing", "lunch 1" and checked
in soong_metrics proto.
Change-Id: I7ab94dcf037211f1abffe6afd012b03ffbf99756
This keeps the top level directory clean, and leverages the
new "build/bazel" repo location / package path.
Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery //...
Bug: 171180724
Fixes: 171180724
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: I151b8f3d2177167711e5974e5eabad8d6e910276
This reverts commit 323dc60712.
Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests
Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
With this change, bazel_module is a specifiable property on
genrule module definitions. With bazel-enabled mode, soong_build will
defer to Bazel for information on these modules.
source build/soong/bazelenv.sh to enter bazel-enabled mode.
Test: Manually verified on bionic/libc genrules using aosp_cf_x86_phone-userdebug
Change-Id: I3619848186d50be7273a5eba31c79989b981d408
Prints the following warning messsage:
WARNING:
Setting OUT_DIR to a path other than out may result in slow RBE builds.
See http://go/android_rbe_out_dir for a workaround.
Bug: b/169676232
Change-Id: Id1ad06c4dc672c24373642e0b624833eb6a0dbcf
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.
Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
surface module properties as attributes.
This CL maps int, bool, string and string_list module props from Soong
modules into their respective Bazel targets.
With this CL, you can now query modules based on their properties. For
example:
$ bazel query 'attr(nocrt, 1, //...)'
$ bazel query 'attr(apex_available, //apex_available:platform, //...)'
$ bazel query //art/dalvikvm:dalvikvm--linux_glibc_x86_64 --output=build | grep compile_multilib
Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery 'attr(apex_available, com.android.runtime, //...)'
Test: soong_build tests
Fixes: 162720644
Fixes: 164320355
Change-Id: Iea8e594b952feccac3281f36dd6bdee8e7d62c3a
This patchset changes bazel_overlay to generate soong_module as a macro,
instead of a rule, and generate module properties in the BUILD files as
kwargs to the soong_module macro.
Here's a sample of the new BUILD files with module properties:
bionic/libdl/BUILD.bazel:
https://paste.googleplex.com/6484466996346880?raw
art/build/apex/BUILD.bazel:
https://paste.googleplex.com/5461276001042432?raw
bionic/apex/BUILD.bazel:
https://paste.googleplex.com/4932795173437440?raw
soong_module is now a macro that conditionally expands to underlying
soong_<module type> rules with statically defined attributes. In this
CL, we are starting with a hardcoded filegroup rule definition to
demonstrate the conditional rule loading within the soong_module macro.
If the module_type matches an existing Bazel rule, soong_module forwards
the entire **kwargs into the rule, which Bazel typechecks.
Non-filegroup module types will be expanded into generic_soong_module,
but with the kwargs dropped.
This approach allows us to:
1) Programmtically generate soong_<module type> rules for all module
types available in Soong, together with the statically defined attribute
types in `attrs`.
2) Incrementally migrate and test individual module types from
generic_soong_module to their module rule shims.
3) Swap out the module rule shims to the actual Bazel rules (e.g
cc_library, java_library) and perform attribute manipulation in Starlark
itself.
Example of querying against the 'srcs' attribute in soong_filegroup:
```
$ bazel cquery 'kind(soong_filegroup, //...)' | wc -l
590
$ bazel cquery --output=build 'attr(srcs, "linker.cpp",
kind(soong_filegroup, //bionic/...))'
INFO: Analyzed 3907 targets (0 packages loaded, 0 targets configured).
INFO: Found 3907 targets...
/usr/local/google/home/jingwen/aosp/out/soong/bazel_overlay/bionic/linker/BUILD.bazel:4144:13
soong_filegroup(
name = "linker_sources",
generator_name = "linker_sources",
generator_function = "soong_module",
generator_location = "bionic/linker/BUILD.bazel:4144:13",
srcs = ["dlfcn.cpp", "linker.cpp", "linker_block_allocator.cpp",
"linker_dlwarning.cpp", "linker_cfi.cpp", "linker_config.cpp",
"linker_debug.cpp", "linker_gdb_support.cpp", "linker_globals.cpp",
"linker_libc_support.c", "linker_libcxx_support.cpp",
"linker_namespaces.cpp", "linker_logger.cpp",
"linker_mapped_file_fragment.cpp", "linker_phdr.cpp",
"linker_relocate.cpp", "linker_sdk_versions.cpp", "linker_soinfo.cpp",
"linker_tls.cpp", "linker_utils.cpp", "rt.cpp"],
deps = [],
)
/usr/local/google/home/jingwen/aosp/out/soong/bazel_overlay/soong_module.bzl:32:23
in <toplevel>
```
This CL is known to be lacking the following features, and will be looked at in follow up CLs:
1) Pretty printing reflect.Interface properties, like arch, multilib and
dists.
2) Generating module Bazel rule shims for all module types, instead of
hardcoding them like `soong_filegroup`.
Bug: 162720644
Test: bazel_overlay_test.go (soong build test)
Test: m bazel_overlay && cd out/soong/bazel_overlay && bazel cquery //...
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: Ic1e448887eb540ed15a55bc4090cf75a4d832d41
The Bazel overlay is a directory at out/soong/bazel_overlay that
replicates the layout of the AOSP Soong module tree, but as a Bazel
workspace. Each Soong module variant is represented as a BUILD target
created with the `soong_module` rule.
To create this overlay, run `m bazel_overlay`.
A `soong_module` target can depend on other `soong_module` targets.
These dependencies replicate each module's `directDeps` in the Blueprint
graph, just before `PrepareBuildActions`.
This enables users to use bazel query as a way to introspect the Soong
module graph. For example,
- Direct reverse dependencies of //bionic/libc:generated_android_ids in
//bionic/libc/...:
$ bazel query 'rdeps(//bionic/libc/...,
//bionic/libc:generated_android_ids, 1)'
//bionic/libc:libc_bionic_ndk--android_recovery_arm_armv7-a-neon_static
//bionic/libc:libc_bionic_ndk--android_ramdisk_arm_armv7-a-neon_static
//bionic/libc:libc_bionic_ndk--android_arm_armv7-a-neon_static_com.android.runtime
//bionic/libc:libc_bionic_ndk--android_arm_armv7-a-neon_static
//bionic/libc:generated_android_ids
- Why does com.android.runtime depend on lzma?
$ bazel query
'somepath(//bionic/apex:com.android.runtime--android_common_com.android.runtime_image,
//external/lzma/...)'
//bionic/apex:com.android.runtime--android_common_com.android.runtime_image
//bionic/libc/malloc_debug:libc_malloc_debug--android_arm_armv7-a-neon_shared_com.android.runtime
//system/core/libunwindstack:libunwindstack--android_arm_armv7-a-neon_shared_com.android.runtime
//external/lzma/C:liblzma--android_arm_armv7-a-neon_shared_com.android.runtime
- What does the dep graph of //bionic/libc:crtbegin_so look like?
$ bazel query
'deps(//bionic/libc:crtbegin_so--android_arm_armv7-a-neon)'
--output=graph > graph.in && dot -Tpng < graph.in > graph.png
https://photos.app.goo.gl/DfsdoFRNsRjGwTmy8
Test: croot && m bazel_overlay && cd out/soong/bazel_overlay && bazel
query //... && bazel query 'rdeps(//bionic/libc/...,
//bionic/libc:generated_android_ids, 1)'
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: I3bf40309bfb2d963bb8a688706385a57ee304c37#
Currently, the build timestamp saved to the build_date_timestamp
field in the MetricsBase metrics protobuf usually comes from a text
file named build_date.txt. It may not be accurate for metrics purposes.
Instead, use the build start time from Soong main UI for a better
timestamp.
Bug: b/140638454
Test: Ran and checked the build_date_timestamp by running the printproto
command.
Change-Id: I4d893fbbf9830ba21911e56ae13bc5272be47ae6
From aosp/1329396, the RBE metrics protobuf file is part of the
metrics uploading process. The RBE metrics protobuf file is
generated by running the bootstrap shutdown command. A new function
named DumpRBEMetrics was written in order to generate the RBE metrics
protobuf file before sending to the uploading process.
Bug: b/140638454
Test: * Unit test cases
* Ran RBE build on my local host and verified the
metrics protobuf file is created.
* Ran non-RBE build after RBE build and verified that
the previous metrics protobuf file was deleted.
Change-Id: I4b8068905cb67c4b8c2d94793917b98974fed707
Soong and Make have no ways to figure out what splits will be outputted
from a given android_app_set, so it's impossible for them to provide
full PACKAGES.$(LOCAL_MODULE).CERTIFICATE entries, which are required to
build a final apkcerts.txt. This change makes extract_apks produce
apkcerts.txt files for each input modules instead. The Make-side
counterpart of this change merges all local apkcerts.txt into a final
one.
Fixes: 160119159
Test: main_test.go
Test: m apkcerts-list
Merged-In: I321e80fd636a955213761f56a3ac64bfe7f7f7c0
Change-Id: I321e80fd636a955213761f56a3ac64bfe7f7f7c0
Specifically, make Android.bp.list and soong.variables explicit inputs
While this is not a comprehensive list of all inputs
of this action (as the action depends on all blueprint
files in the source tree), this is closer to the truth.
This is a rollforward CL, which was originally rolled
back, as path validation failed for when OUT_DIR was
an absolute path. Validation has now been relaxed.
Test: Manually verified ninja output, checkbuild approved validation
for aosp-crosshatch, and manually ran frameworks/rs/build_rs.py with
patch fix
Change-Id: I4eb0d517f57336dd54eaa4bd31f46df9e93e6da2
While this is not a comprehensive list of all inputs of this action (as the action
depends on all blueprint files in the source tree), this closer to the truth.
Test: Manually verified ninja output, checkbuild approved validation for aosp-crosshatch
Change-Id: I77a0f612afb025232bee7d3ac31257b808991829
The metrics uploader was currently running on foreground where it
would copy the metrics files in a separate directory and then forked
into the background for the upload process. As a result, running the
lunch command would take a second longer to run since each metrics
uploader run had an average of half a second.
Bug: 140638454
Test: * Wrote and updated unit test cases.
* Set ANDROID_ENABLE_METRICS_UPLOAD to point to the latest
metrics_uploader bash script. Executed the "lunch 1" command
and measured the running time. Executed "m nothing" command
and checked that the metrics were uploaded.
* Ran "lunch 1" and "m nothing" with
ANDROID_ENABLE_METRICS_UPLOAD=""
* Removed oauth from metrics_uploader and ran "m nothing" and
"lunch 1". The oauth Message appeared only to "m nothing"
Change-Id: I13c61e666c8f44613dee291a704cef6a27335188
Soong now supports the ability to upload metrics to another location
by setting the ANDROID_ENABLE_METRICS_UPLOAD to an uploader that
accepts the upload.proto proto buffer message. When the environment
variable is set, a set of build metrics files (soong_metrics,
rbe_metrics.pb and build_error) is uploaded.
Bug: 140638454
Test: * Wrote unit test cases
* Setup the uploader, built a succcessful and failed aosp_arm-eng
target and monitor the uploading of the metrics.
Change-Id: I76a65739c557dc90345e098ca03119a950ece2d2
For an end user hitting these errors, the most likely reason is that
they have mixed up linker and crtbegin_dynamic.o from different builds
in their out tree.
Test: Build a linux_bionic executable in a tree exhibiting b/157549171.
Test: m nothing
Bug: 157549171
Change-Id: I64bee0848ed61226da6654caf3bafb51cf46d923
apex_available/min_sdk_version are required to be available for mainline
modules(APK/APEX).
- java_import/java_library_static
- android_library/android_library_import
Bug: 156996905
Test: prebuilts/sdk/update_prebuilts.sh
Change-Id: I201ebabaa533c5ee19e394ab30ca2fa0e3d7ce0e
"dlv attach":
- Spams the terminal
- Requires sysctl -w kernel.yama.ptrace_scope=0
- Apparently, does not allow the debugger to inspect variables
Test: Manual.
Change-Id: I625369effaf5abda2b5d884c8ce5bde7247774eb
apex_set takes an .apks file that contains a set of prebuilt apexes with
different configurations. It uses extract_apks to select and install the
best matching one for the current target.
Bug: 153456259
Test: apex_test.go
Test: com.android.media.apks
Change-Id: I1da8bbcf1611b7c580a0cb225856cbd7029cc0a7
This was a temporary migration tool, and has been broken ever since the
ninja environment started being restricted. Remove it, since it has
lasted two releases.
Test: TEMPORARY_DISABLE_PATH_RESTRICTIONS=true m
Change-Id: I2e3709525598030f5c24d12fa966e531dfed33ae
Golang's FileInfo() implementation recognizes FAT filesystem directories
correctly, but doesn't actually add executable bits, which essentially
makes them inaccessible. Use 0777 when we know we're dealing with
directories.
(This is a cherry-pick change.)
Fixes: 153207404
Test: manually tested with an aar file with FAT permission bits.
Change-Id: Ie9f5320bc44d24b2f9d54d4036bb1747456655bb
Merged-In: Ie9f5320bc44d24b2f9d54d4036bb1747456655bb
Soong UI generates a proto file named build_progress.pb in $(OUT_DIR)
output directory that contains build action numbers (how many are executing,
finished and total) during the course of a build. This is for external
systems that invokes the Platform Build Systems and would like to know
the completion status.
Bug: b/150401146
Test: Wrote a bash script that continuously read the
build_progress.pb file and computed the build completed percentage
while building the aosp_arm-eng target. Compared the percentage between
the Soong output console and the one reported by the bash script.
Change-Id: I7c7347bc8e41958093892d8e2731c4f4169937dd
Collect the number of modules and variants and some basic statistics
on memory usage inside soong_build by writing out a proto that is
read back in by soong_ui.
Test: examine soong.log
Change-Id: I6926876377a4f6229cf41fdbf166ae03c885ea55
This change automates one of the best practices for
custom tools. The best practice says "Prefer to list
input files on the command line, otherwise we may not
know to re-run your command when a new input file is
added." [1]
Normally you'd reference these inputs with $(in) or
one of the forms of the $(location*) substituions on
the command line for the custom tool. However, if the
custom tool does not accept the list of inputs on the
command line, the build system could fail to re-run
the custom tool if the list changes.
This change adds a hash of the list of input names to
the sbox command that wraps the custom tool. If the
list of inputs change, the hash will change, and
therefore the sbox command will change, causing ninja
to re-run the custom tool. The hash is visible to
(but ignored by) the sbox command, and hidden from
your custom tool.
[1] https://android.googlesource.com/platform/build/soong/+/refs/heads/master/docs/best_practices.md#custom-build-tools
Test: TestGenruleHashInputs
Bug: 149397658
Change-Id: I18b547ea3c4296ee15bd6150a4778a8f376d80b7
Create a highmem pool based on the total RAM and the number of CPUs,
with an override via the NINJA_HIGHMEM_NUM_JOBS variable. Put
metalava into the highmem pool.
Ninja does not support nested pools, and when goma or RBE is enabled
the maximum ninja parallelism is set very high with local jobs in a
local pool. When both the local pool and highmem pool are enabled,
the total number of local jobs will be as high as the sum of the sizes
of the two pools. Keep the highmem pool limited to 1/16th of the
local pool when remote builds are enabled to try to minimize the
effect while still limiting highmem jobs.
Fixes: 142644983
Test: m nothing, examine pools
Test: m USE_GOMA=true nothing, examine pools
Change-Id: Id79f11f44948992960ac34ecf831dacbe21bd332
Test: touch empty.zip; merge_zips out.zip empty.zip
(output should contain `empty.zip: zip: not a valid zip file`)
Change-Id: If707876af2301d7ad7d065c004cffb0c2491c7df
Apparently we have some instances of Soong taking 16GB of RAM. While
that's a problem to solve on it's own, make multiproduct_kati's
auto-parallelism detection smarter by taking into account the total RAM
on the machine instead of just the number of CPUs.
Bug: 146925549
Test: check soong.log for autodetected parallelism values
Change-Id: Ice34c2cf73ea4f8f235521cbefc8654a79a04eef
This is so that we can split the build_test build on large branches into
multiple builds, each testing a subset of the products.
Test: m blueprint_tools (runs the go unit tests)
Test: build/soong/build_test.bash --only-config
vs
Test: build/soong/build_test.bash --only-config --shard-count=2
Test: build/soong/build_test.bash --only-config --shard-count=2 --shard=2
Change-Id: I40ccc1aa477bc0ffa74ff564d155068509be18f0
soong_javac_wrapper is silencing a useless warning:
warning: [options] bootstrap class path not set in conjunction with -source 1.9
but recent versions of javac have started also printing:
1 warning
Read the warning count, subtract the number of silenced warnings, and
reprint it if the non-silenced warning count is nonzero.
Fixes: 144118634
Test: javac_wrapper_test.go
Change-Id: Ie1d0a978188ab7b1c41027f718a1274608628123
Force soong_ui in dumpvars mode to use a dumb terminal so that it
doesn't print an empty status table. Also move the logs to files
that are prefixed with "dumpvars-" so that they don't rotate the
real build logs.
Fixes: 139307849
Test: lunch
Change-Id: If92421af07731fedc9ca42670f309e95922a2224
sbox will generate a random directory for the output root, and most
tools will encode that directory name in the output target of the
depfile.
So embed the library from dep_fixer into sbox so that it can rewrite the
output filename to a static (reproducible) value. Ninja doesn't care
what that value is, so it's just "outputfile".
Also fix up rule_builder to actually tell sbox about the depfile.
Test: mmma system/iorap; check the contents of:
out/soong/.intermediates/system/iorap/libiorap-binder/android_arm_armv7-a-neon_core_static/gen/aidl/system/iorap/binder/com/google/android/startop/iorap/IIorap.cpp.d
Change-Id: I3640a2e8b0c034f143a35e398a8418a6d621b265
Android build generates 180K+ zip files that are to be merged and the
result passed to the indexing pipeline to generate the source code
cross-reference. Their names cannot be passed on the command line,
hence the need for this change. That required extensive changes to
the code as it no longer can be assumed that all the input files can
be kept open.
Bug: 121267023
Test: internal + treehugger + combine index files
Change-Id: I0a829f96ff7187ad967fb6b8cede387501ec57cc
mm is now doing the same thing as mma. Now print a deprecation message
for other tools that may be setting ONE_SHOT_MAKEFILE directly (or users
that are still using a very old envsetup.sh).
Test: mm # does not print
Test: ONE_SHOT_MAKEFILE=bionic m
Change-Id: I98221657024d824a61c0d721d5c7dd1a2ea2fb1e
1. Update pom2bp to support "host_support" device module.
2. Allow other module type in support library
(The intention is to support the room annotation processor)
Test: m
Bug: 137918883
Change-Id: I9a8fbcff903cc395d5dc38430774202ce389d6bc
Merged-In: I9a8fbcff903cc395d5dc38430774202ce389d6bc
(cherry picked from commit bd7c2f9a1d)
This is the first part of deprecating the mmm command. mmm will simply
be redirected to mmma by building the list of modules with their
dependencies.
Bug: b/135187558
Test: Ran mmm external/libusb:libusb.
Change-Id: I50cf0d6a07197556ab6130e612561c6d5a3fb31e
This is the first part of deprecating the mm command. mm will simply
be redirected to mma by building the modules and their dependencies.
Bug: b/135187558
Test: Ran mm in external/protobuf directory.
Change-Id: Idad40f89e3768aeb459172f632a44fd7b0a201bc
Allow running Soong in a headless delve debugger by passing
SOONG_DELVE=<listen addr> in the environment.
Bug: 80165685
Test: SOONG_DELVE=:1234 m nothing
Change-Id: Icfc893c8a8354a9bbc99112d9c83259cb41906d1
Keep a running map of the critical path to each edge as it
finishes. At the end of the build, find the edge with the
longest critical path and print the path to out/soong.log.
Test: critical_path_test.go
Change-Id: Ie01d26b068b768156b166bff00cdc3273e8124ca
Soong_ui produces a build_error raw file to $(OUT_DIR) where
the file contains a list of build action errors. Each build action
error represents an error raised by a command. The build_error file
is populated if there was a build error.
Bug: b/132969697
Test: Ran m for successful build. Introduced a broken build change,
ran m and verified using printproto that build_error was
generated successfully.
Change-Id: I690ca1778b4e56f144a3173ba1d16d8494c32c15
There was one case that did not work:
1) Create dir in root source tree
2) Run "m libc"
The build would fail as it was using the mma logic. A separate
one named BUILD_MODULES was added to allow building specific
modules in any directories using the "m" command.
Bug: b/130049705
Test: unit test cases, ran "m libc" in a tmp directory inside
of the source tree.
Change-Id: I8d23e685a673a311001ee8edd89bd73b662392dd
ctx.Writer is recommended stream to use for standard io.
Fixes: b/135487952
Test: Lunch and executed ./out/soong_ui --dumpvar-mode and
--dumpvars-mode to check the usage is output to the console.
Change-Id: I38351ebdb05f49f92d6b4bfb0423092299d1c5ea
The --build-mode flag is a new flag in soong_ui that accepts a build
action (m, mm, mmm, mma, mmma), the directory where the build action
is occuring and an optional list of build arguments and invokes the
build option. This is to deprecate the build actions behavior in
envsetup.sh and start using the one defined in soong_ui.
Bug: b/130049705
Test: Ran lunch and ran all the build action commands in soong_ui.
Ran a bunch of invalid commands such as no Android.mk file in
a specific directory, invalid target name, etc...
Change-Id: I88e9e59f47f8c5cdc872fb83a0f3f0a0919885b6
The metrics.proto needs to be imported to Google3 in order to unmarshal
the build metrics data from bigstore. Cleaned up the enum names to use
the Camel naming convention and renamed the build_metrics to
soong_metrics to be more specific to soong and to allow quering
soong_metrics files from bigstore.
Bug: b/135280521
Test: lunch andf m -j. Checked if soong_metrics was generated correctly
by using printproto command.
Change-Id: I998c8d05db592e94a653d6ca32250b80df3c9b21
Write log output through StatusOutput so that the status implementation
can synchronize it with its own output.
Test: status_test.go
Change-Id: I917bdeeea4759a12b6b4aa6d6d86ee18a2771723
terminal.Writer is now just a wrapper around stdio.Stdout() without
any useful functionality. Replace it with stdio.Stdout() as an
io.Writer.
Test: ui/terminal/status_test.go
Change-Id: I5bc5476afdca950b505642f0135a3af9d37fbe24
Instead of extracting the aar, and passing the res directory to another rule
(along with a dependency to a different known file), don't keep the
extracted res directory, and pass the aar directly to the resource
compliation rule, which will extract just the res directory.
I need this for my RBE experiments, where non-listed output files won't
exist in other rules.
Test: m
Change-Id: I99074381052cbcebb6a402484abae9ab2e40284f
Currently, the command line argments is being processed in multiple
places. In the main soong_ui arguments, there are several if statements
that checks if a specific argument was specified in order to execute
the requested operation. This does not scale well when adding or
removing a command in the near future. In order to support the build
commands (m, mma, etc...) from the envsetup.sh in soong_ui, a refactor
was required in order to add a command rather quickly and to have the
flexibiity to unit test the command. The soong_ui arguments format is
as follows:
soong_ui <command> [<arg 1> <arg 2> ... <arg n>]
The <command> is a specific operation to be executed. The arguments
after the command are processed by the command itself.
Below is the list of changes made in this commit:
* Created a new command infrastructure that allows adding
or deprecating a command easily.
* Fixed a bug when running ./soong_ui.bash directly would cause
a panic due to index being out of range for args list.
Bug: b/130049705
Test: Below is the list of testing done on this commit:
* Ran lunch and verified that the output is the same as the the output
without the modifications. lunch indirectly runs soong_ui passing
in the --dumpvar-mode (to read makefile variables such as
TARGET_PRODUCT) and --dumpvars-mode (to build the build cache).
* Ran ./soong_ui.bash directly (with unsupported flags and no flags)
and the proper message appeared that soong native UI is not yet
available.
* Ran m, mm, mmm, mma, mmma commands.
* Ran the make installclean command:
make -j80 TARGET_PRODUCT=aosp_arm64 TARGET_BUILD_VARIANT=eng dist DIST_DIR=/tmp/helloworld installclean
* Ran ./out/soong_ui -j80 --make-mode PRODUCT-aosp_arm64-eng dist checkbuild tests
Change-Id: Iee4de7ec0fa4661206fda8ae1fe6fa4487d9bb22
Merged-In: Iee4de7ec0fa4661206fda8ae1fe6fa4487d9bb22
This changes the default behavior of pom2bp to match pom2mk, and not
require pre-extraction of all of the manifests.
Test: convert an internal pom2mk user to pom2bp
Change-Id: I08120cce1f923ef28519563ffd772032b2e500ed
Allow callers of pom2bp to add dependencies on runtime libraries as
well as static libraries. Renames -extra-deps to --extra-static-libs
to make it clearer what it does.
Bug: 130287656
Test: prebuilts/sdk/update_prebuilts.py --stx 5280039
prebuilts/misc/common/androidx-test/update-from-gmaven.py
Change-Id: I8a3ee8056aa4ed9935a4f47d563ec423672cf38b
This will prevent it from being affected by future css location changes.
Test: m soong_docs
Fixes: 130567020
Change-Id: I80fd7b8a12a32bcea7b270423b8d8f8dc0976177
This essentially allows you to declare that everything in a directory
will be created by the rule, and we'll ensure that your command actually
writes out all of the claimed outputs, and remove any other files that
previously existed in that directory.
Test: built-in tests
Change-Id: I990dce2b3a0d89ebd2736ac1a0cadfb5864c6e73
The dependency type in the pom file is not always accurate, just
add all jar and aar dependencies to both the -nodeps and final
module.
Test: cd prebuilts/sdk/current/androidx && pom2bp -regen Android.bp && mma
Change-Id: I1784b8ebf3485db19405cb353c9bb5fb3ae837d8
android/soong/common was renamed to android/soong/android long
ago, but the pctx package path was still "android/soong/common".
This required all users of rules defined in android/soong/android
to import "android/soong/android" and then
pctx.Import("android/soong/common").
Test: m checkbuild
Change-Id: I20d096522760538f7cfc2bec3d4bfeba99b275d4
* changes:
Add a whitelist for services.art changing between builds
Add diff_target_files tool
Import files from compare_target_files for use in diff_target_files
Some tools used by genrules set the timestamp of the generated file
to an old value. For example, when using tar to extract files it
will set the timestamp of the extracted file to the value stored in
the tar file. This can cause unnecessary rebuilds due to output files
that are older than their inputs. Touch output files in sbox before
moving them into the output location.
Test: m checkbuild && m checkbuild
Change-Id: Idb0e71d2bf0ac3618338371132cc71be74bf9d9d
Add a tool that lists files that have changed between two builds by
comparing the target-files.zip artifacts.
Bug: 121158314
Test: unit tests
Change-Id: I3ca5dd1fdf652dffef923ad7307fd71e49e2e95a
Allow rules built with RuleBuilder to use depfiles. Ninja only
supports a single depfile with single output. If there are
multiple outputs in a rule, move all but the first to implicit
outputs. If multiple depfiles are specified, use new support
in dep_fixer to combine additional depfiles into the first depfile.
Test: rule_builder_test.go
Change-Id: I7dd4ba7fdf9feaf89b3dd2b7abb0e79006e06018
Bring property type strings to the start of property comments. This is
how source.android.com shows variables and paratemeters, and it is also
easier to read, especially when a comment ends with a code example.
Test: m soong_docs
Change-Id: Ic3e9db3a3711495998919ee95bbef6c7cb33b215
To add code format (<pre>...</pre>) support to module type texts.
Fixes: 124332708
Test: m soong_docs
Change-Id: Ibe9d3845a8f952fdef5007893d4af5836809ce1f
It appears that there's a bug in repo where when a linkfile is removed
from the manifest, the symlink remains in the source tree.
Cleaning this up needs to happen before we start scanning the source
tree for Android.bp/Android.mk files, so this has to be done very early
-- way before CleanSpec processing could handle it.
Bug: https://bugs.chromium.org/p/gerrit/issues/detail?id=10583
Test: ln -s missing hardware/qcom/sdm710/Android.bp; m
Change-Id: Ib68f2507ffe58ccdd9fbc88925f8a4f6150f2f7d
Follow the change to blueprint to make *Escape take and return a string
and add *EscapeList that take and return slices of strings. Fix up
a few places that were unnecessarily converting a string to a slice
and back to a string.
Test: m nothing
Change-Id: I3fa87de175522205f36544ef76aa2f04aef1b936
The href base added by I8a8ac0f9ba6c0d57ed83db2155955a95e070a265
doesn't really play nice with anchor links. This fixes the issue.
Fixes: 126550408
Test: m soong_docs
Change-Id: If62411a6391fd8151fa40cb2ecf32c5006797d6c
This adds a script to copy the current page's base url to hrefs in the
doc.
Fixes: 123473548
Test: m soong_docs
Change-Id: I8a8ac0f9ba6c0d57ed83db2155955a95e070a265
Previously the build doc was a gigantic list of modules and properties,
which can be overwhelming to new users. This change breaks it up by
packages, so that it is easier to look up and feels more coherent.
Bug: 123521276
Test: m soong_docs
Change-Id: I1a1a331f4b4deb8782d698e9076098c5a27b9566
Uses more python rather than C++, and skips less of Py_Main.
Test: build/soong/python/tests/runtests.sh
Change-Id: I03997d88e2e16047c96bb4e00e530229c42b3325
It now has module list on the left and the main panel containing
a section for each module. Each section contains the navigable list
of the module's attributes, grouped by attribute's origin.
Test: Manual
Bug: 120512870
Change-Id: Ib62ef58c61daa88950f0934a70ba8ed16f433bf2
bootstrap.ModuleTypeDocs needs a mapping of module types to factories
to support factories that are wrapped in ModuleFactoryAdapter closures.
It also returns ModuleType objects grouped into Package objects.
Bug: 67909957
Test: m soong_docs
Change-Id: I70eac9f0f0e13075580da92d4219792ca0b18fbf
If ANDROID_QUIET_BUILD environment variable is set to '1', do not show 20
lines of the environment variables. In addition, for the failing step show
only the step name and output, omitting the command proper (the verbose log
as well as error.log still contains it).
If build succeeds, the output of the build itself consists of a single
line:
```
> m androidmk
[100% NN/NN] <last command>
```
When it fails, the output does not contain sometimes very long command
line:
```
> m androidmk
[ 97% NN/MM] test androidmk
FAILED: <step>
--- FAIL: TestEndToEnd (0.01s)
androidmk_test.go:1025: failed testcase 'prebuilt_etc_TARGET_OUT_ETC'
input:
include $(CLEAR_VARS)
LOCAL_MODULE := etc.test1
LOCAL_MODULE_CLASS := ETC
LOCAL_MODULE_PATH := $(TARGET_OUT_ETC)/foo/bar
include $(BUILD_PREBUILT)
expected:
prebuilt_etc {
name: "etc.test1",
filename: "foo/bar",
}
got:
prebuilt_etc {
name: "etc.test1",
filename: "foo/bar",
}
FAIL
17:50:53 ninja failed with: exit status 1
```
[The related change in build/make/envsetup.sh suppresses timing display
when the same variable is set.]
Change-Id: I4d3c72457de031ff58a324c2fe98f4c1d10f8239
Test: treehugger
Test: Dumped the text formated based metrics file to out dir,
and checked the file.
Bug: b/63815990
Change-Id: Iff476f72a0be74eb53b6b26ef468d11c0f24a404
The zip file format does support scripts/programs prefixed to the
archive, though many of the offsets are supposed to start from the
beginning of the file. Some tools (python and zipinfo) are able to
read zip files with arbitrary prefixes, but others (libziparchive and
zipdetails) fail.
So pass the file to prefix onto the zip file into merge_zips so that we
can set the offsets from the real start of the file.
Test: m sepolicy_tests (runs the embedded python interpreter)
Test: zipinfo out/host/linux-x86/bin/sepolicy_tests
Test: zipdetails out/host/linux-x86/bin/sepolicy_tests
Change-Id: If73d4c2465581f7de5aa47959284ecf2059df091
Gold uses two PT_LOAD programs: RX, RW
LLD is using three PT_LOAD programs: R, RX, RW
All non-static host bionic executables are linked with LLD, as it's
required in order to embed the linker. The dynamic linker is considered
a static executable for this definition.
When the linker had been built with Gold, we inserted the the RX&RW
loads in the middle of the executable's readonly load, splitting the R
load into two. When this happened, all of the linker's loads existed 1:1
in the final executable.
Now that the linker is built with LLD, we're inserting the loads into
the same place, but LLD merges the first part of the executable's R load
with the linker's R load, so there's not a 1:1 matching.
So just make sure that all of the linker's loads are contained within
the executable's loads (and with the appropriate flags).
Test: m blueprint_tools (runs new tests)
Test: build host bionic binaries
Change-Id: If7868d2ab8980d27ba5ab8994fefe0d6142f215a
Patterns containing multiple globs or a recurisve glob may match
files with the same name in multiple directories. Keep the relative
directories of matches after the path entry containing a glob.
Bug: 117295826
Test: zip2zip_test.go
Change-Id: I5d663e546953af374175837551d23f484d568377
Stop writing out ninja files, as they become very large, especially when
multiplied by the number of defined products.
Test: treehugger
Change-Id: Id9529d14040acb72a0188e58b5db2911f142071e
In order to simplify the wrapper function, and stop using a linker
script, generate a set of flags to pass to LLD. Then run
host_bionic_inject on the linked binary in order to verify the
embedding, and give the wrapper function the address of the original
entry point (_start).
Bug: 31559095
Test: build host bionic with prebuilts/build-tools/build-prebuilts.sh
Change-Id: I53e326050e0f9caa562c6cf6f76c4d0337bb6faf
Don't warn on duplicate files in merge_zips if they have identical
CRC32 and size values.
Test: m checkbuild
Test: merge_zips_test.go
Change-Id: I61336ca4d4d3b7402c24a7abd337bd350fe10930
Make needs to exclude all files that match a filespec except for
those specified in a list. Add a -X argument that overrides the
-x argument to allow files be included that were previously
excluded.
Bug: 69500920
Test: zip2zip_test.go
Change-Id: Icc8aebc788b53bcdde73a39c9c7cf3107a049251
Add -0 flag to convert files in a zip to stored instead of deflated.
Bug: 69500920
Test: zip2zip_test.go
Change-Id: I6c2b10f3b200a53a3339e3c97a78f65192b309ca
Make the cleanspec & Android.mk runs of kati use much of the same code
and arguments.
Also renames 'Kati' to 'KatiBuild' in many cases, in the the expectation
that we'll have a 'KatiPackage' step in the future.
Use --no_ninja_prelude and move local_pool & _kati_always_build_ into
the combined ninja file. This will reduce the need to re-read makefiles
when Goma is enabled, and it allows us to include more than one
Kati-generated ninja file in the build graph.
Bug: 116968624
Test: build_test on downstream branches
Change-Id: Ibdac689b81f62dc293647fad917d84946f2c3cfa
Add test that cover basic command line usage of soong_zip. -D
is not covered yet as the implementation will be replaced with
one that is also more easily testable in the next patch.
Bug: 116751500
Test: zip_test.go
Change-Id: I5a1bcee74ebc9cb3cf332c36f89bc12c0e807ad2
Move the actual per-product builds out of the main function, combining
the product configuration and build sections. This makes it a lot more
readable.
This ends up saving a lot of FDs -- a -only-config build on aosp with 77
products used to require ~500 FDs, it succeeds with a limit of 150 FDs
now. I'm leaving the code to bump our FD limit though, since 2x the
number of internal products is too close to the 1024 limit for comfort.
Bug: 70370883
Test: prlimit -n150:150 build/soong/build_test.bash -only-config
Change-Id: Ia559beadc19deb8a5b9d50af6e0276e846fd8608
This uses a very large amount of intermediates, but can speed up
subsequent multiproduct_kati runs by not re-reading all the makefiles
unless necessary.
Since we'll no longer have the full build output on every run,
explicitly save away a std_full.log of the log from the last time we ran
Kati.
Test: build/soong/build_test.bash -products aosp_arm -incremental
Change-Id: Iaae580c0e49a2024528617d39b2b477e29d99250
Move some of the more complicated conditional logic out to helper
functions. Use the {{- }} syntax to strip previous whitespace to
allow spacing out the remaning conditionals in the templates.
Test: (cd prebuilts/sdk/current/androidx && pom2bp -regen Android.bp)
Change-Id: I766bd0e1837aa04375f322fbe796d923cd99ecde
Allows specifying make goals when using multiproduct_kati, for example:
$ multiproduct_kati -only-config dump-products
This makes dumping products with multiproduct_kati easier than before.
Test: multiproduct_kati -only-config -products=full,aosp_arm64 -out=/tmp/p dump-products
Change-Id: If526bd2ba1b7bbadcb230c020202ef9e421e1852
