A typo caused one of the two files to be compared to itself if they
both had the same size.
Bug: 228496289
Test: Test_filesHaveSameContents
Change-Id: Ie44f32e3c9b8ef725f0d9933c7e701340036710a
Some of the prebuilt ELF files used for bionic heads cause an
unexpected EOF error, ignore unexpected EOF the same way we do
for EOF.
Test: not yet
Change-Id: I267d11b4d12b83ecebedc72a565e148c5e53af6d
There are cc_binary_prebuilt modules in the tree that are shell
scripts, and attempting to extract an elf ID from them results
in an error. Ignore too short files and files missing the elf
magic header the same way we do for elf files without an elf ID.
Bug: 218888599
Test: Test_elfIdentifierFromReaderAt_BadElfFile
Change-Id: If7117925ca2371a8ee61ba3616372f6e9b0fab0e
The command line run inside the sbox sandbox can be very long and
printing it obscures the error message. It is already written to
a script for execution, so point to the script instead of printing
it.
Fixes: 220592170
Test: manual
Change-Id: Icf883344af0f707802b090b4cfb0ef6dcfe0e5c1
Add support to sbox for only writing output files if they are changed
to support enabling restat. Move the clearing of the output directory
into sbox, and when write if changed is enabled delete obsolete output
files without deleting output files that will be replaced.
Enable restat for local metalava rules. Restat for metalava rules
run remotely is still blocked on b/183630617.
Bug: 222095735
Test: TestRuleBuilder_Build
Test: m SystemUI
Change-Id: If8fc47e711f4a6f43947ab69f17cccc8ba6ae61a
Add a symbols_map tool that can extract an identifiying hash from
and elf file or an r8 dictionary. The tool writes the hash to a
textproto, and also supports a merge mode to combine textprotos into
a output file for inclusion in the build artifacts.
Bug: 218888599
Test: m dist
Test: symbols_map_test.go
Change-Id: Icd3ed6e5510e058c92d97c78759e7a4cfcdbb6ca
Metrics are added to a materialized protobuf soong_build_metrics.pb
Recorded events have scoped names and recorded runtimes.
For example, mixedbuilds includes the following events:
mixedbuilds.prepare
mixedbuilds.analyze.write_files
mixedbuild (this is a total of all mixedbuild events)
Test: Manually verified materialized protos for mixedbuild, bp2build,
and legacy build.
Change-Id: I2e72d4502c0373bee0279bb1320b2e4dd586129c
Dumping of RBE metrics should only happen when reproxy was started.
Reproxy is started only when Soong is run in make / build mode and not
in dumpvars mode.
Doc: https://docs.google.com/document/d/1Iqhu5QgozB_rA9fucMHO6X4U6A8e8qFtZyub9j83ICQ/edit#
Test: Ran a build with / without RBE. No metrics dumped during lunch
without RBE and metrics dumping worked as expected with RBE.
Bug: b/197213143
Change-Id: Ic3403331e1f4ac2c7c72fd9169aa7b82a7545c75
I0e293944c5c804b4512fb834f8f18e1d6ecd053c accidentally used
Println instead of Fprintln, causing it to print the current
contents of the buffer instead of adding to it.
Bug: 216192129
Test: manual
Change-Id: I1ddeeef3443c1091cf729328cb8fd74c217432ec
Discarding PT_INTERP in the linker script generated for the embedded
linker allows linking without the --no-dynamic-linker argument.
Bug: 216192129
Test: manual
Change-Id: I0e293944c5c804b4512fb834f8f18e1d6ecd053c
When the environment variable config file is loaded after config object
is initialized, we end up loading the USE_RBE variable in the config
file after the code to start reproxy process is run. This causes a
problem when USE_RBE variable itself is set in the config files. To
avoid this, I've moved the config file loading code to be within the
config package, this makes the main.go file cleaner too.
Test:
Ran a build with USE_RBE:true set in the config json file. The build
fails without this change and succeeds with this change.
Bug: b/209486170
Change-Id: Iab3957b64f4b5456a861057d16ad318f4f78e0cb
This is useful for external users to be able to specify their own config files that can be loaded by soong during startup. In addition, we need this for upcoming changes to incorporate an experiments framework in Soong since the config file will be fetched from CDPush and put into the OUT_DIR folder by the config file fetcher binary.
Note: Once this is merged into internal branch, I'll fully get rid of
the vendor/google/ path from Soong in aosp.
Test:
1. Ran a build in aosp with these changes and no config file was loaded.
2. Ran a build in internal master with these changes and the current
config file inside vendor/google was loaded as expected.
Bug: b/214035335
Change-Id: I9af83687d4eaeee1ffb0f88a750cfeb7c6d2bafb
Tie verbosity of bp2build to env var BP2BUILD_VERBOSE
Test: m bp2build
Test: BP2BUILD_VERBOSE=true m bp2build
Change-Id: Ib8f10ad73ed06c935126a77d18c1d9febc25348f
Extract and install the primary apk normally, and then unzip the rest
of them as a post install command.
Bug: 204136549
Test: app_set_test.go
Change-Id: I17437ff27f49df6bc91bdbbea6173b46c7d3ec4e
Linux has limit of 32 pages for a single argument. Sbox's method of
handling commands as a single large string can run into this limitation.
Write the large command to a script file and then execute that file.
This better accommodates the large commands, and leaves a script in the
temporary directory useful for manual inspection if necessary.
Bug: 177070955
Bug: 174232579
Fixes: 202297224
Test: m
Test: make particular target with very long command exceeding limit.
Change-Id: Ia298fdfd7a759821c37f540deaf800026041e511
This is necessary because GenerateAndroidBuildActions() is effectively a
mutator and therefore can change the data stored in each module.
Sometimes one wants to dump that data in the JSON module graph, and in
order to do so, actions need to be generated.
I briefly entertained removing StopBeforePrepareBuildActions(), but it
incurs a 10-second penalty on "m soong_docs" and "m bp2build" so I
decided to keep it.
Test: Presubmits.
Change-Id: I438aaf981e989dc39a09e8a2ba73f7a08ee55386
Device builds are no longer supported on Mac, but we do support building
various host tools, including the SDK build-tools and platform-tools
packages. These have dependencies on [java] device modules, so we don't
completely disable device modules, only hide them from Make (which makes
them more difficult to trigger from the command line).
Also fix the mac build of multiproduct_kati, so that `m blueprint_tools`
works on Mac.
Bug: 187222815
Test: `m`, `m dist`, etc on Mac
Change-Id: I92f16605d5cd173d431cbcb79081234d45cc6e2e
Soong writes out text protos for the sbox manifests, and if those change
every time soong changes, we trigger rebuilds of significant portions of
the tree. This is very annoying when attempting to iterate on Soong, so
disable this functionality at the soong_build level.
Bug: 200727888
Change-Id: Icf21e1775a4c4200e983c1b8ac66ddd780d7c90c
Note that this is an initial, rather crude attempt at this,
that unblocks generating BUILD files needed to build AOSP android apps.
In particular, note that
- Jetifier step currently ignored
- Visibility set to public for all output targets
- All dependencies are gathered in 'deps', though they are split under various categories in bp files
- Host, host-only notions are ignored for now
Test: Copy first 2 lines of, e.g. prebuilts/sdk/current/support/Android.bp to a BUILD.bazel file and run pom2bp -pom2build -regen BUILD.bazel
Bug: 198433754
Change-Id: I5efa93517e6c319fd828da5b16ba7efd2d90bca8
If SOONG_UI_ANSI_OUTPUT is set to a true value force the simple status
output to keep ANSI codes. This will allow buildbots to collect logs
with ANSI codes in them. Smart status output is not affected as it
always keeps ANSI codes.
Bug: 147310922
Test: manual
Change-Id: I0b78ceebb65125b8e8dafb4787816fb679d3eb3e
Under load sometimes the 1 millisecond timeouts are not run before
a 1 second sleep finishes, or `echo foo` is not run before a 1
second timeoutt. Increase the long sleeps and timeouts to 10 seconds.
This doesn't affect normal test runtime as the command should timeout
after 1 millisecond.
Bug: 199207577
Test: Test_runWithTimeout
Change-Id: I31166850a473b9cde37906a62d026580e7a7bd3f
Support three options for converting modules with unconverted
dependencies
1. (default) Warn when converting a module if it has unconverted deps.
2. Error when encountering a module with unconverted deps. (not hooked
up yet)
Test: build/bazel/ci/bp2build.sh
Test: build/bazel/ci/mixed_libc.sh
Test: BP2BUILD_ERROR_UNCONVERTED=1 build/bazel/ci/bp2build.sh with
unconverted deps -- get appropriate error
Bug: 181155349
Change-Id: Ifaabf0cd2e43e963366dc137159c705294165c3d
This field was missing for Property objects that have non-empty inner
Property objects
Bug: 172797653
Test: m soong_docs
Change-Id: Iee9c66f8b85d68a6b5bf18fd9787143191c4f002
They were a bit repetitive.
In addition, make the environment checks more correct; they could
probably use an integration test or two to make sure that when the
environment changes, exactly those outputs are rebuilt that need to be,
but for now, this is an improvement already.
Test: Presubmits.
Change-Id: Idd79b81ca6975d57d00e5bf4699d266152505ff8
This makes soong_ui the only place where soong_build is invoked, thus
greatly simplifying the conceptual model of the build.
It comes with the slight limitation that now soong_docs (and queryview
and the JSON module graph) are not Make targets anymore, but I suppose
that's an acceptable loss.
The only place where someone depended on soong_docs from a Makefile is
removed in a separate change.
Test: Presubmits.
Change-Id: I3f9ac327725c15d84de725d05e3cdde1da3dcbe2
Its value is a function of the call site, so it doesn't make a lot of
sense to plumb it through the configuration.
Test: Presubmits.
Change-Id: If928b34de075969fd42932212ce9187808cbdf86
The breakage was because:
- aosp/1818236 removed the "rm -fr" invocation from the Ninja file that
removed the old queryview tree
- queryview files were created with permissions 440 to discourage manual
editing
These two together made incremental queryview invocations not be able to
overwrite its output files.
(Of course, it also made them incorrect since old files were lft in place)
Test: Presubmits.
Change-Id: Idaa3e1db82c9eb8325c48e37aca5146e69af7333
It used to be invoked from out/soong/build.ninja, which required two
soong_build invocations one after the other (ne to generate
out/soong/build.ninja, one to generate the queryview workspace). This
was slower and required some shell-quoted-in-ninja-quoted-in-Go .
Test: Presubmits.
Change-Id: Idda79c067606663b66e9f94626fa24f3b5af4114
They both used to return out/soong. Unfortunately, the tests seem to
assume the old behavior, so I had to keep tests inconsistent and I was
scared of dexpreopt, too.
Test: Presubmits.
Change-Id: Ib00c7956fd2cdb28cc11d481c17e5a3ff5ad563a
It was confusing because bootstrapping uses two configurations: the
"global" config and the special-cased bootstrap one.
This change merges them.
Test: Presubmits.
Change-Id: I82b482cbe28a343ab6991374b2a28667e1a06b48
I erroneously assumed that the continuous builds don't set DIST_DIR,
whereas they do.
The variable naming is quite questionable, but for now, the important
thing is that the code works and I wanted to do that with the least
amount of change possible.
Also add some more logging:
- Shell environment
- Command line arguments
- Location of logs.zip
- The logs directory
Test: Presubmits.
Change-Id: I651b1045eeef568aa860fd2457fa7799ad68ba38
These are: TopFile and GeneratingPrimaryBuilder.
Also re-shuffle the list of flags to make a bit more sense and finish
the rename of BuildDir and NinjaBuildDir to SoongOutDir and OutDir,
respectively.
Test: Presubmits.
Change-Id: I103ff5f09f1c0d16f695a7da5dea13b55028e33e
Before, we piggybacked on the implementation of regular soong_build and
wrote a fake build.ninja file to satisfy Ninja.
Now, instead, the JSON module graph is a a separate action in the Ninja
output file. This has the pleasant side effect that one can flip back
and forth between generating the JSON file and regular Soong without
loss of incrementality.
Side cleanup: write .d files in a slightly cleaner way.
Test: Presubmits.
Change-Id: Ia853383567b9dd31c53f3bdf56cfc8d517b498ec
These are just out/ and out/soong/ and the old names were quite
confusing.
Test: Presubmits.
Merged-In: I999ca62162062f27e449f83ffb00fa580d4065b8
Merged-In: I2fcd80a9e376a6842a06aa104a9f6f5c5e16f8c7
Change-Id: Ib481d2aac40df8da1b74174dc8ec576c1cb48be0
In particular, the writing of build-globs.ninja and the collection of
glob list files for build.ninja.d is now done outside of the Blueprint
invocation.
This is useful because RunBlueprint() is sometimes called multiple times
in a single soong_build invocation and because now glob processing is
not hidden by the machinery of Blueprint.
Test: Presubmits.
Change-Id: I6eb19b357901e452e6d1e7ce055f8a3bab4a74f4
They used to be defined in Blueprint but since it doesn't have a
separate existence anymore, we can consolidate all command line
arguments here.
Note that Blueprint is called directly from soong_ui, but that happens
by a simple Go function call and not by invoking a separate binary, so
command line arguments are not needed there.
Test: Presubmits.
Change-Id: Ifa2c101ddbe424b76fc5a508d3d41c329e7353ca
The following files are forked:
- build-globs.ninja
- The glob list files
- .d files for the glob list files
- The output file (build.ninja or the bp2build marker file)
This makes bp2build and soong_build not overwrite each other's files
they need for proper incrementality.
Test: Presubmits.
Change-Id: I69d192cbd36ecd9677f46f3fa095dfce6f872227
This fixes null bp2build builds.
When running bp2build, out/soong/.bootstrap/build-globs.ninja wasn't
generated, therefore none of the globs in out/soong/.primary/globs have
actions to generate them. This makes bp2build always re-run on a null
build, because it depends on those non-existent globs in its depfile.
Fix the above by generating a build-globs.ninja file with the globs that
bp2build encounters during conversion, and add tests to verify that null
builds and bp2build -> soong workflows correctly regenerates the
build-globs.ninja file.
Test: soong integration tests for null soong->soong, bp2build->bp2build and bp2build->soong builds.
Fixes: 193650250
Change-Id: Icfbad351dfa85541d1ea451f73e65cac87097926
This is done by moving SetupSignals() to its own little package.
There are a number of tiny little utility packages for soong_ui we might
be better of merging, but that's for another change (maybe)
Test: Presubmits.
Change-Id: I07b0ca98bfb8884ef4223d665e632183b9896a0d
This mostly cuts the dependency from multiproduct_kati to the build
package; the remaining logic is SetupSignals which will be dealt with in
a followup change.
This requires some code duplication (e.g. ensureFileExists() and
detectTotalRAM()).
This removes a Finder invocation added in aosp/452598; it looks like
it's not a measurable performance hit. The quoted half a second does not
matter much in build_test{1,2,3,4} which take an amount of time measured
in hours.
It also makes ANDROID_QUIET_BUILD unsupported for the UI of
multiproduct_kati; the function of that variable seems to be to suppress
the output from build actions, which don't exist in multiproduct_kati so
it should not matter. In addition, I checked whether anyone sets this
flag for multiproduct_kati and it doesn't appear to be the case.
Test: Presubmits.
Change-Id: I1c2901e1ea69c3a35429e6c967a7bba845676ef7
This serves to not link parts of soong_ui (and eventually soong_build)
into a separate, weird binary. This is in turn good because they contain
any number of global variables and no one really thought about what
happens when two instances are executing at the same time in the same
address space.
This comes with a slight performance hit: 5 aosp_* projects build 152
seconds instead of 146. I suppose this is a price worth paying for a
clean design?
Test: presubmits.
Change-Id: I5623dcab2290f0fc392dd2ede597b9794a3d2a4e
Print more messages in run_with_timeout to make it easier to
determine which log statements came before the timeout and the
on_timeout command.
Test: run_with_timeout_test.go
Change-Id: I92ad663529fa03d91185ba382654a73f0eba1ab7
Add __dlwrap_linker and __dlwrap_linker_end symbols for use by the
embedded linker trampoline to determine which load segments are
part of the embedded linker and should not be exposed to the
executable.
Use flags to name the sections to avoid colliding with __dlwrap_linker
when no section name can be determined.
Bug: 190084016
Test: build with musl
Change-Id: I51ccbf9c15a65c6194fa263b7504a598dbf4ad18
This is as part of an effort to move environment variables into config
files so that configs can be tied down to source.
Test:
Ran a build with "ANDROID_BUILD_ENVIRONMENT_CONFIG=googler m" which ran
a non-RBE build (since this is vanilla aosp).
Also ran a build with "ANDROID_BUILD_ENVIRONMENT_CONFIG= m" which ran a
non-RBE build.
https://paste.googleplex.com/5637282880028672
Bug: b/194679562
Change-Id: I416e8da75f84aa2b53995f525cf50501488dc972
Major operations:
* Fix the go_package entries to be consistent, as the new tool
complains when it doesn't have a '/'.
* Regenerate with the new protoc-gen-go tool.
* github.com/golang/protobuf -> google.golang.org/protobuf
* proto.[Un]MarshalText -> prototext.[Un]Marshal
Change-Id: Ie1147bd2457fafb66ba555461b3bf14f0561a25d
Use a note added to linker.s to place the linker in the linker script,
which will work musl, which doesn't have .note.android.ident, while
continuing to work for linux bionic.
Test: m checkbuild
Change-Id: I9cd4ad7c8e8fcb4d725746106d5beaa699a77807
I wrote these flags primarily for build-tools:external/kythe, where we
only use a subset of the Go packages defined in the project.
-limit allows defining the entry points to keep in the Android.bp, and
the utput will omit everything except those entry points and their
dependencies. This allows the kythe Android.gen.bp file to drop from 195
modules to 31 using two flags, without having to manually exclude the
other 160 packages.
-skip-tests does as you'd expect, though ideally it wouldn't be
necessary. We didn't import all of the test-only dependencies for kythe
into the source tree, so we're not able to build them.
Change-Id: I49e9f12dac53a74c6e05725ee41fb7c0ed8b0cbb
This CL adds support to bp2build/Soong to dump a BUILD file under
out/soong/soong_injection/targets containing alias targets to their real
targets for every converted Soong module, regardless of whether they are
handcrafted or generated.
Test: TH
Change-Id: Ic1816fda5d019c395301618134fac68b3057d752
Use an implicit linker script instead of flags in a file to specify
the locations of the host bionic embedded linker and to prevent it
from being stripped.
Test: build and run host bionic binary
Change-Id: I64e12118d33c67bab5e657cbc3ea8cde8f0fd7e6
Replace .linker.sect0, etc. with .linker, .linker.text, .linker.data
and .linker.data.rel.ro by extracting the name of the first section
in each program header.
Test: build and run host bionic binary
Change-Id: I25107547536a3a3963fdeb311c45a7ee53c0bc45
The host bionic bootstrapping no longer needs an injected symbol.
Replace host_bionic_inject with host_bionic_verify that validates
the resulting binary, and add it as a validation dependency of the
binary.
Test: build and run host bionic binary
Change-Id: I3e303d2a164b6eef851bdc8075e6ee456c05b0a8
- Rename the "BuildX" variables to "RunX"
- Remove redundant comments
- Inline all the "what to do" based on config in build.go
- Inline some constants only used in one place
Bug: 189187214
Test: m nothing
Test: build/soong/build_test.bash
Change-Id: I111a69e642212d7938d4971283545e0d9acbb01a
Merged-In: I111a69e642212d7938d4971283545e0d9acbb01a
(cherry picked from commit d274ea9196)
I6dca478f356f56a8aee1e457d71439272351390b replaced calling
build/blueprint/bootstrap.bash with running minibp directly,
but didn't propagate the EMPTY_NINJA_FILE environment variable
to minibp. Since everything that uses EMPTY_NINJA_FILE is
executed directly from soong_ui now, replace the EMPTY_NINJA_FILE
environment variable with a config bool and pass it to minibp.
Bug: 189148777
Test: build/soong/build_test.bash --dist --incremental --shard-count=300 --shard=1 && du out/multiproduct
Change-Id: I4d64275ce02c5d68948012f71ac4dc3795af9e85
This CL fixes a typo in writeFakeNinjaFile to correctly write
a fake out/soong/build.ninja and its depfile. It also modifies bootstrap
phase to *not* run the main soong build phase (which takes more than a
minute) if GENERATE_BAZEL_FILES=1.
This change has the side effect that `GENERATE_BAZEL_FILES=1 m nothing`
no longer generates the real build.ninja, which is fine because one
shouldn't be using GENERATE_BAZEL_FILES=1 for that anyway (or, use
USE_BAZEL_ANALYSIS=1). This change has no effect on mixed builds.
Time on a change to Soong or any Android.bp files:
Before: bp2build_workspace_marker (~20 seconds) + build.ninja (1 min)
After: bp2build_workspace_marker (~20 seconds)
Time on the second of two consecutive `b build //bionic/...`: 2.070s
Test: TH
Test: Soong integration tests
Change-Id: I43720641815994caba97b8d165d7c3fc254cbd06
This tool automatically writes Android.bp files based on information
discovered from Go, similar to pom2bp or Bazel's Gazelle. This
implementation is fairly limited, but is usable enough to import
libraries like protobuf without a lot of manual effort.
Change-Id: Iecc145bc7ab9b9e9c3739f77b694b3f4bc2c6396
Use a concurrency-safe writer in runWithTimeout to avoid data races
on the bytes.Buffer passed in during tests.
Bug: 181095653
Fixes: 187149270
Test: Test_runWithTimeout
Test: go test -race ./cmd/run_with_timeout
Change-Id: I57a889765cb9ee7b42983f0906313e0c2d1e414e
Add a command that can be used to wrap actions with a timeout, and
optionally run an extra debugging command on timeout.
Bug: 181095653
Test: run_with_timeout_test.go
Change-Id: I91df5c3fb5277968717815a4ad4612113766dab1
This refactors bazel-build mode determination logic in soong_ui so it's
clearer which of three possible modes are being used in a given
invocation (NO_BAZEL, GENERATE_BUILD_FILES, or MIXED_BUILDS).
Test: bootstrap tests
Change-Id: I41d2baebf8d560c2cc42db8daa8b936101d453e3
It's under out/soong/workspace and is a symlink forest that merges BUILD
files generated by bp2build and the source tree.
bazel-* symlinks and $OUTDIR are currently ignored.
Test: Presubmits.
Change-Id: If14df23bd0f6a6f2d7d1140f95213f2aee66d652
The full command line run inside sbox can be very long, and printing it
after the errors printed by the failing command can hide the error
messages. Buffer the output of the command and print the failing command
line before the output if it fails.
Bug: 185516277
Test: m out/soong/.intermediates/frameworks/base/system-api-stubs-docs-non-updatable/android_common/metalava/api_lint.timestamp with lint error
Change-Id: I893f3dd01f1baf195e182111c5c49e92eb82f3b0
Error messages printed by failing commands may reference output files
that were created by the command, for example printing a command line
to copy and paste to update a baseline file. Copy output files in the
sandbox to their final locations, ignoring missing files, so that the
messages are valid.
Bug: 185516277
Test: m out/soong/.intermediates/frameworks/base/system-api-stubs-docs-non-updatable/android_common/metalava/api_lint.timestamp with lint error
Change-Id: I604a11c9b54e409ca5bc5c016cd04b3133f74a60
Update the path bootstrap_test.sh uses to check for bpglob reruns
to follow the changes made for sharding globs.
Use bootstrap.GlobFileListFiles to get dependencies during bpbuild.
Test: tests/bootstrap_test.sh
Bug: 159845846
Change-Id: Ibaa7c4360881ed6b666a811bf34b19ea0cdcafe9
It was only appended to the return value, which can be done by the
caller just as well.
Test: Presubmit.
Change-Id: I962696e0dbd4c3496a0159d01d2a911675fd4217
It now handles adding .bp files and changing globs.
In order to do this, depfiles are now written separately from RunBlueprint.
This is necessary due to the confluence of a number of seemingly
unrelated factors:
1. The glob filelist dependencies are discovered in globSingleton
2. Singletons need to be registered because otherwise singleton module
types panic
3. Singletons don't work because they require mutators bp2build does not
run
Due to (1), we would need to run the glob singleton. However, due to (2)
and (3), we can't run singletons and have to run Blueprint with
StopBeforeGeneratingBuildActions, which is when the build actions
writing glob files would be generated. So what happens is:
1. When bp2build is run, the glob singleton is disabled
2. At the end of bp2build, the list of glob files is artifically added
to the depfile of the workspace marker file
3. When build.ninja is generated, the Ninja file containing the glob
list file is written by the now-active glob singleton
Test: Presubmits.
Change-Id: I3c5898d8c57c554a93520276c64a952afc912dbe
This is done by setting the INTEGRATED_BP2BUILD environment variable
when invoking the build.
Even though the name of the marker file insinuates that a Bazel
workspace is already created, this is not the case yet.
An issue that remains is that a .d file is not written for the marker
file so it won't be rebuilt if a .bp file changes. Fixing this requires
delicate surgery because writing the .d file is the result of delicate
interplay between Soong and Blueprint.
There are also a number of semi-related fixes:
- The name of soong.environment.{used,available} is now on the command
line of soong_build (soong_docs is still special cased because its
command line in the Ninja file is taken from the os.Args of
soong_build so it's not trivial to remove the --{available,used}_env
from it
- bp2build writes a separate soong.environment.used file
- I had to call SetAllowMissingDependencies() separately when creating
the android.Context for bp2build so that bp2build runs in the
integration tests (it was not obvious how not to do this)
- Fixed a number of integration tests where a command with an expected
exit code of 1 was used as the last one in a test case, thereby
breaking the test suite
Test: Presubmits.
Change-Id: Ibeb61c26022cf801dcb98505b4039151b3409873
Properly plumb the available environment to the configuration instead of
going through the global variable originalEnv.
Test: Presubmit.
Change-Id: Ia1ea753d5e72c02a0dcaa4c0b43bd6e24fc47cec
This is so that the way soong_build is invoked is simpler, which is in
turn useful so that it can be invoked multiple times within a single
build, which in turn will be used to invoke bp2build routinely (as
opposed to manually when needed)
Test: Presubmits.
Change-Id: Iddaebb05ff7bcedc0db8273192bb31284b739920
Usage: SOONG_DUMP_MODULE_GRPH_JSON=<filename> m nothing
Test: The new test case in bootstrap_test.sh .
Change-Id: I69005a75d47dff915d27187645d0cd1cbb3467ef
This makes it more easy to see what's happening under which modes. Modes
are:
- Regular builds
- Mixed mode builds
- bp2build
- soong_docs
Test: Presubmit + bootstrap_test.sh
Change-Id: I06e3fcb84986b919be1cd9eb456531259ede1183
Previously min_sdk_version was only set for *_library build rules, and
not for *_import targets. This made min_sdk_version default to
sdk_version, which is usually "current", which is not correct for these
libraries.
Test: pom2bp -default-min_sdk_version 24 for androidx
Change-Id: I3b029315abeb559e66efad847ae664efb79f4509
Sbox preserves the permissions of input files when copying them into the
sandbox. A read-only file copied into the sandbox multiple times causes
a permission denied error on the second write. Building in Bazel results
in more read-only files, which triggers the issue on existing sbox rules
with duplicate input files. Remove the destination file when copying if
it exists.
Bug: 184113103
Test: m USE_BAZEL=true
Change-Id: I7edf92d82b766100e3cbbd90d22428269d7d0167
Previously 24 was hardcoded (for somewhat unclear reasons). Update
the tool to accept specifying the correct value on the cmdline.
Test: pom2bp -default-min-sdk-version 24
Change-Id: I39c23dd4bca91684a2a68d7b963cf3cb66eb1f33
The current implementation causes inputs listed in an rsp file used with
sbox to be duplicated 3 times in the build.ninja file; once as a
dependency of the rule, once in the rspfile_content field of the rule
with the paths rewritten to be relative to the sandbox, and once in the
rule to write the sbox manifest. When RBE is enabled it also gets a
fourth copy in the list of files to be treated as inputs by rewrapper.
Reduce this to a single copy by using "$in" for the rspfile_content so
that the files only have to be listed in the input dependencies of the
rule, and then add support to sbox to rewrite the rsp file while copying
it into the sandbox, and pass it to rewrapper as well.
Test: m lint-check
Change-Id: I3f46f61119508d39a8bb231c99fc130153fb6f04
sbox is going to need to read and write response files, move ReadRspFile
to its own package.
Test: response_test.go
Change-Id: Iecb5486b4aaeb2531828743ad8ef784df675e18e
Previously, soong.environment.used was written after build.ninja and if
the amount of time that passed between the two was long enough, Ninja
would decide that build.ninja is older than soong.environment.used and
rebuild it.
Test: test_null_build in bootstrap_test.sh in a loop.
Change-Id: I5467da487e8e8f2646644b8a7fb9549b9ff18276
This CL adds the support for specifying lists of directories in
build/soong/android/bazel.go, which are then written into
out/soong/bp2build/MANIFEST. Using this configuration,
modules/directories can either default to bp2build_available: true or
false, while still retaining the ability to opt-in or out at the module level.
It also ensures that ConvertWithBp2Build returns true iff the module
type has a registered bp2build converter.
Test: go tests
Test: demo.sh full
Test: TreeHugger presubmits for bp2build and mixed builds.
Change-Id: I0e0f6f4b1b2ec045f2f1c338f7084defc5d23a55
This is achieved by writing soong.environment.used in Main() instead of
as a side effect of a singleton. This makes a difference because build
actions are not generated when GENERATE_BAZEL_FILES=true is set,
therefore the side effect did not happen.
Arguably, Main() is made worse by this change, but I don't want to
tackle the problem of readably determining which mode soong_build is
running in in this change.
Test: Presubmits + the additional test.
Change-Id: I66af2429aedf008762173eaaa55b828b4cf4328b
It's replaced with a flag in bootstrap.Args .
Test: "m nothing" (presubmits take a full workday these days)
Change-Id: Ia0bfa2091013e389890e583c559006077806af99
This requires linking Blueprint into soong_ui. It lets us avoid the
complicated dance of Ninja files and shell scripts: now the information
as to how soong_build is built is passed directly to Blueprint using a
struct that contains all the information the command line arguments used
to contain.
The ability to run Blueprint from the command line is kept (for now).
Some variables in bootstrap/command.go needed public accessor functions
because soong_build reads them. This will be disentangled by moving the
flag parsing to soong_build.
The presence of the flag definitions in Blueprint means that soong_ui
now also accepts them. This is not a problem in practice because they
are ignored and because soong_ui itself is hidden behind a few layers of
shell scripts.
Test: Presubmits + the new bootstrap_test.sh .
Change-Id: I6dca478f356f56a8aee1e457d71439272351390b
When running commands in chdir mode sbox needs to translate the
directory placeholders to paths relative to the sandbox directory
instead of relative to the top of the source tree.
Also translate relative PATH entries into absolute paths so they
are still valid when the current directory is changed.
Bug: 182612695
Test: manual
Change-Id: Idcbe20466888909d423d62788bc9c35f4e03b398
The end goal of this exercise is to remove all mutable global state so
that multiple Blueprint instances can be run in the same process.
Test: Presubmit.
Change-Id: Idb34b0920f2c7d92efb9328ce8a78b7306f89571
If both bp2build_available and label are specified, label will be
preferred.
Initially, we copy the entire BUILD.bazel file. Eventually we may move
this to use bazel query for a more accurate result.
Test: go test *
Test: build/bazel/scripts/milestone-2/demo.sh full
Test: GENERATE_BAZEL_FILES=true m nothing
edit bionic/libc/tools/BUILD.bazel
GENERATE_BAZEL_FILES=true m nothing and verify changes picked up
Bug: 180516554
Change-Id: I43025583300e6b10d2c18032cd4a76237b578d59
In addition, make the Ninja command nicer by using backticks instead of
backslash-quoting double quotes.
Test: Manual: "m queryview"
Change-Id: I5d6e1d41424e0229bfe08e2e7eaf2fb025ed48c2
This works by setting the SOONG_UI_DELVE= environment variable to the
port on which soong_ui should accept a Delve connection on.
This is achieved by reusing the Delve execution logic between soong_ui
and soong_build.
Test: Manual.
Change-Id: Id2c1d4b6faac1a4a3918c91030ce2239f7daf54f
This lets one avoid any decisions as to when to chdir there during its
execution and leads to better sandboxing because the pwd doesn't leak to
init() functions anymore.
Test: Manual.
Change-Id: I1560da8ed3a621249426f9e8908aa890c21e13ba
Also delete the now-unnecessary soong_env binary.
Test: Manually. Also checked what happens when a used environment
variable changes.
Change-Id: Ib393e7f444e94819198c6cce4bcd8719d9fd9441
Sample output:
[bp2build] cc_library_headers: 5 targets
[bp2build] cc_object: 5 targets
[bp2build] filegroup: 4 targets
[bp2build] genrule: 4 targets
[bp2build] sh_binary: 1 targets
[bp2build] Generated 19 total BUILD targets from 39270 Android.bp modules.
This CL adds an additional CodegenMetrics return value to
GenerateBazelTargets calls, which are called from bp2build, queryview,
and their tests. For this UI, we only want to use it for bp2build, and
not queryview or tests, since it's not useful for the former, and can
pollute the CLI for the latter.
Test: build/bazel/scripts/milestone-2/demo.sh
Change-Id: Ic84307a1ed1a25e360c9b23459e5449d932bc2e7
Before this change, changes to any Android.bp file would not trigger a re-run of bp2build/Soong, since the depfile for Soong's main output, the build.ninja file, didn't contain the list of Android.bp files. bp2build only reran when Soong itself changed, which happens regularly while developing bp2build, but not at all while modifying Android.bp files.
This change correctly dumps the list of all Android.bp files in the tree to the depfile for out/soong/build.ninja, therefore correctly triggering a re-run of bp2build when any Android.bp file changes.
Test: GENERATE_BAZEL_FILES=true m nothing; edit an Android.bp file; GENERATE_BAZEL_FILES=true m nothing -> successfully re-runs and regenerates out/soong/bp2build.
Test: inspect out/soong/build.ninja.d manually
Fixes: 179452534
Change-Id: Iad80c381b3afa38a4d4fafca035e46aa2a1f206a
bp2build is a Soong mode that returns as soon as the BUILD files are generated. This causes it not generate the build.ninja file, which the Ninja process executing soong_build itself expects to produce as an output. If there isn't an existing build.ninja file generated from a previous build, GENERATE_BAZEL_FILES=true m nothing will fail.
This CL generates the expected files as a workaround, and also makes GENERATE_BAZEL_FILES=true skip the Kati/Ninja steps in soong_ui since they aren't needed.
Test: rm -rf out/ && GENERATE_BAZEL_FILES=true m nothing && m libc
Fixes: 178683777
Change-Id: I2515ef7961682d2be5f096ed24831cc185165a67
multiproduct_kati sometimes passes 20GB per soong_build process,
increase required ram per job to 25GB.
Bug: 178376498
Test: none
Change-Id: I446f52ea7419bc3015d940511af1f03b15eb651e
The CodegenMode enum helps to differentiate between bp2build and queryview modes of generating BUILD files from the Soong module graph.
bp2build is used for generating BUILD files from an unconfigured Soong module graph for Soong->Bazel conversion, whereas QueryView is a front-end for querying the existing Soong configured module graph with Bazel query.
Test: go test
Test: TH
Change-Id: I5709f591f1ae08b2770adf272d8ce60c9ee1f46a
This CL creates the framework necessary for generating
BazelTargetModules from regular Soong Android modules.
BazelTargetModules are code-generated into Bazel targets in BUILD files.
See the follow-up CL for examples of creating filegroup/genrule
BazelTargetModules.
Test: GENERATE_BAZEL_FILES=true m nothing # creates out/soong/bp2build
with no BUILD files, because there are no BazelTargetModules in the
module graph.
Change-Id: I33a96365bd439043b13af6db9e439592e9983188
Implement bp2build codegen as a discrete step that runs after an
alternatively registered pipeline of mutators, instead of a
presingleton.
bp2build codegen requires a Context that supports VisitAllModules and
PathContext, so this CL also makes a BpToBuildWrapperContext that
conforms to PathContext by adding two method implementations.
Test: GENERATE_BAZEL_FILES=true m nothing && bazel query //... --config=bp2build | wc -l # 31433
Test: m queryview && bazel query //... --config=queryview # 63638
Change-Id: I0dd359746584b228046d2d0ff00895f28f9bdfc3
A SingletonModule is halfway between a Singleton and a Module. It has
access to visiting other modules via its GenerateSingletonBuildActions
method, but must be defined in an Android.bp file and can also be
depended on like a module.
Bug: 176904285
Test: singleton_module_test.go
Change-Id: I1b2bfdfb3927c1eabf431c53213cb7c581e33ca4
Splits queryview into queryview and bp2build. The latter runs as a
presingleton (could be converted to a singleton). This prevents needing
to run soong_ui a subsequent time to build the query. Queryview remains
as a separate step to prevent increasing runtime due to this generation
every time Soong runs.
Currently this is running as a presingleton as this gives a translation
of Android.bp files after only LoadHooks have run, no mutators.
Test: go tests
Test: m queryview && bazel query --config=queryview //...
Change-Id: If2ba28c7ef60fbd41f43bda6623d41c8c7d23a1b
Also get Bazel to write real files there (not symlinks) so that the DIST_DIR can be independent.
Test: Manually using e.g. DIST_DIR=/tmp/foo USE_BAZEL=1 m dist
Change-Id: I39d5219500864c9ecc85f356a028e9b5bf2607f4
The contents of the bazel metrics directory (out/bazel_metrics or
<dist dir>/logs/bazel_metrics if dist arg is specified) is added
in the list of files to be uploaded in order to do performance
analysis of bazel builds.
Bug: b/174479728
Test: * m nothing and checked metrics_uploader log file to ensure
if the metrics is uploaded.
* USE_BAZEL=1 m nothing and checked if the set of metrics
files plus the build_bazel_profile.gz is uploaded through
the metrics_uploader log file.
Change-Id: I394a1b621a06f9acccf154faee6dcd4200fcff4c
Code refactoring has been done for logs directory logic code since
the bazel metrics directory depends on the log directory. For builds
that did not specify a dist directory, the log directory is under
out directory. With dist, the logs directory is under <dist dir>/logs.
This matters for Android CI builds where the metrics files are
placed under logs directory. With this change, the bazel metrics
directory and corresponding files will be under <dist dir>/logs
directory for Android CI builds.
Bug: b/174479728
Test: * USE_BAZEL=1 m nothing (bazel_metrics dir in out dir)
* m nothing (bazel_metrics dir deleted)
* DIST_DIR=/tmp/build USE_BAZEL=1 m nothing dist (bazel_metrics
is in /tmp/build/logs directory)
Change-Id: Ic9e1ff49a1964fcaaf801bde2c19f33597ca1db4
Add a boolean flag to the sbox proto to request the executable bit
to be set after copying a file. This will be used for sandboxing
tools when copying a file in from the source tree that would
normally get its executable bit set during installation.
Bug: 124313442
Test: sbox_test.go
Change-Id: Ie2c197bb5183ffc1bf63fd6effd175143cd324d4
This CL adds a new CONVERT_TO_BAZEL env var, and a bp2build goal. It
reuses the queryview architecture, but registers no mutators before
converting to BUILD files. This gives us a blank slate of the module
graph to work with, and create a shadow/alternate pipeline of mutators
to converge the Bazel BUILD graph to be semantically equivalent with the
Android.bp graph (after apply its current set of mutators).
The command to do so is:
$ CONVERT_TO_BAZEL=true m bp2build
To not clobber with queryview, the generated files are in
out/soong/bp2build.
Test: CONVERT_TO_BAZEL=true m bp2build && bazel query --config=bp2build //...
Test: m queryview && bazel query --config=queryview //..
Test: soong tests
Test: TH presubmit
Fixes: 174465461
Signed-off-by: Jingwen Chen <jingwen@google.com>
Change-Id: I4dd0ccc73abc345d70a50ca2803d6f400cd8c863
Previously in order to make use of dist entries with tag properties it
was necessary for each module type to be modified to store the results
of GenerateTaggedDistFiles(module) in the AndroidMkEntry.DistFiles
property. This change generalizes that mechanism to work across all
module types.
This change does improve the behavior of a couple of tests where
previously a dist with tag property was ignored because tagged dist
files were not available now it is used correctly.
Some module types do not implement OutputFileProducer interface and so
they cannot handle dist with tag properties. This change makes it an
error to attempt to do that. That necessitated adding OutputFiles(tag)
method to customModule in queryview_test.go as some of the tests in
that file used dist entries with tag properties.
Follow up changes will remove the code that was made redundant by this.
Test: m nothing
m dist sdk - before and after this change, compare result to
make sure that there are no significant differences.
Bug: 174226317
Change-Id: Ifc54d67db10ce0d0fe8179c05b97a2be2113be4e
The sbox tool is used to wrap genrule actions, and run them in a
temporary directory. Afterwards, the outputs are moved into their
proper location in the output tree. However, some tools embed the file
name of the output file (as passed to the tool) into the output file.
For example, the perfetto code generator script uses the output file
name to generate a C header guard when it generates
perfetto_src_base_version_gen_h/gen/perfetto_version.gen.h.
When using remote execution / remote caching, these genrule actions are
run locally whenever the output file doesn't exist, effectively making
them unique (across time and between users).
They then cause cache misses on all actions depending on these output
files as well as on transitive actions. In the above example, this
causes libperfetto.so to differ, which then causes all actions
depending on libperfetto.so to be rerun unnecessarily.
As a fix, this commit changes the sbox tool to use the sha1 hash of the
manifest path. The tool already seems to assume that there are no
concurrent runs of the same tool with the same manifest (otherwise
it would fail flakily depending on exact timing). It seems therefore
safe to use a temporary path deterministically derived from the
manifest path.
Test: ran multiple builds w/ remote execution; observe proper caching
Change-Id: I5b73ffd3b7f85cbb0336dfa1675de7ac0e2fd1a8
The hardlink optimization attempts to save I/O by hardlinking input
files into the sandbox instead of copying them, but it never works on
a Linux build due to nsjail always making out and the source tree into
separate bind-mounted filesystems. If the optimization did work it
would actually cause build failures when it hardlinked the relative
bionic/libc/fs_config_generator.py symlink into the sandbox directory
without the target. Just remove the optimization for now since it
never works.
Test: m checkbuild
Change-Id: I96e8c0c145e7c99958639594edf8a93b69ae90eb
No current use cases for soong.config and user-editable local
configuration options for Soong, so let's remove it to reduce the API
surface.
The MegaDevice configuration is used for building every module, for
every cpu variant, for every architecture, but it apparently isn't
used for a while now.
Test: m nothing
Test: TH
Fixes: 174188200
Change-Id: I2550e70ff6f9c8b57e9a7cc517d6a119a032a27a
This relands I3b918a6643cea77199fd39577ef71e34cdeacdb1 with a fix
to create the directory for the output depfile if doesn't exist.
In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.
Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I8af00c8c0b25a92f55a5032fcb525715ae8297c2
This change fixes a bug that zipsync didn't handle symlink correctly;
symlink was extracted as a regular file whose content is the target
path. Fixing the problem by correctly creating the symlink using
os.Symlink.
Bug: N/A
Test: manual
Change-Id: Ib6685c14e1950d1057d89672883cdd9e4879069a
In preparation for more complicated sandboxing that copies tools
and/or inputs into the sandbox directory, make sbox use a textproto
input that describes the commands to be run and the files to copy
in or out of the sandbox.
Bug: 124313442
Test: m checkbuild
Test: rule_builder_test.go
Test: genrule_test.go
Change-Id: I3b918a6643cea77199fd39577ef71e34cdeacdb1
The UseBazel() function will be used in several places to perform
different functionality such as setting up the metrics path.
Bug: b/173028918
Test: m nothing
Change-Id: Ie94073b8f0552ec7528e1cc2f3be74b693dc3135
This shaves off ~20s, which is ~11%, of `m nothing` with mixed builds.
Test: Manual verification with mixed builds of aosp_flame.
Change-Id: Ief1bc56c92329e899c829fc1374eea9ee9b94fb7
Prepare for using Config when adding singletons by passing
Config to NewTestContext and NewContext instead of to ctx.Register.
This will enable a followup change to store SingletonMakeVarsProviders
registered on the Context in the Config, which is necessary to run
multiple tests in parallel without data races.
Test: all soong tests
Change-Id: Id229629a4e42ff4487d317241673837726c075fc
Bazel overlay is an experimental feature. This renames the feature to 'queryview' to better describe its purpose, and also move away from the already overloaded 'overlay' term in Android.
Test: m queryview && bazel query --package_path=out/soong/queryview //...
Change-Id: I8b5068c7db46cb61a03a8e87af9c7c9077ebeff9
A recent change to Blueprint [1] added the feature to infer the type of
list elements to more than just "list of strings", but this also changed
the text to "list of string" (without s). This CL fixes the bazel
overlay, which uses "list of strings".
Test: m bazel_overlay && bazel query //... --package_path=out/soong/bazel_overlay
[1]: 764a771917%5E%21/#F1
Change-Id: I7522d15b1ce751807e1881ce1e94f068d226b316
There is no need for multiproduct_kati to run the tests for
every product, they don't vary by product config. --skip-soong-tests
can also be used for local development to run soong_build even if
the tests don't pass.
Bug: 156428456
Test: m --skip-soong-tests nothing
Change-Id: I9c00e3d1b6e51d17bb290339c3f124d4d1c9e69f
We need to capture the user build command to understand what additional
arguments is passed in to Soong. This data is useful to know which
targets are being executed and have statistics on top build targets
are specified.
Bug: b/169453712
Test: ran "m blueprint_tools", "m nothing", "lunch 1" and checked
in soong_metrics proto.
Change-Id: I7ab94dcf037211f1abffe6afd012b03ffbf99756
