This helps with incrementality a lot: the symlink forest must depend on
almost every directory in the source tree so that if a new file is added
or removed from *anywhere*, it is regenerated.
Previously, we couldn't do this without invoking bp2build, which is
quite wasteful because bp2build takes way more time than the symlink
forest creation, even though we do the latter in a very suboptimal way
at the moment.
This means that if a source file is added or removed (which does not
affect globs), we don't pay the cost of bp2build anymore.
Also refactored symlink_forest.go on the side. Too much state was being
passed around in arguments.
This change reimplements aosp/2263423 ; the semantics of not touching an
output file is the exact same as order-only inputs and the latter is a
bit fewer lines of code.
Test: Presubmits.
Change-Id: I565c580df8a01bacf175d56747c3f50743d4a4d4
test_bp2build_generates_all_buildfiles uses the same directory for
--output_base and --sandbox_tmpfs_path. This is required on the build
servers, but causes issues locally. This commit adds a more helpful
error message when this test fails.
Test: build/soong/tests/bp2build_bazel_test.sh
Change-Id: Ib29a7edf456f8ae39bd429ebb367c8115e02c284
* We are interested mostly in the conversion the `default_applicable_licenses`
attribute, as `default_visibility` cannot be handled until every module's
`visibility` is handled.
* Several referenced license modules had to be manually enabled for
conversion, and likewise a few trivial Android.bp containing only
package and license modules.
* As Bazel allows only a single `package` rule, the package rules in a
manually converted BUILD.bazel files were removed (in
external/protobuf and prebuilts/clang/host/linux-x86 trees).
* The converted package rule is emitted before the `load` statements per
Bazel documentation.
Bug: 190817312
Test: treehugger
Change-Id: If8bf6fee1580177de3bb402923615bcd48923ed2
apex_comparison_test.sh defines a helper function to clean up state on
exit, but in error conditions the call_bazel function is not always
defined.
Test: added an exit 1 statement below trap
Change-Id: I73b0c56c0a4a2d9ed1d4c0bf38b6ac0ac985e716
RBE is already disabled for this test on CI. Locally, the mock checkout
created for the Soong integration tests does not set up RBE properly, so
we should just disable it explicitly so that the integration tests can
be run locally.
Test: build/soong/tests/run_integration_tests.sh
Change-Id: I38875a02b549a9d8c8368d6287c4516518d20e34
apex_diff_test targets were the last remaining dependents of make_injection. Rewriting them allows us to delete make_injection.
Test: CI
Change-Id: I928d8fe136d709b9f819c00c3127af66e720a7db
SC2236: use [[ -n "$var" ]] instead of [[ ! -z "$var" ]]
SC2035: use -- end of options, to accomodate filename w\ hyphens
SC2155: `local -r var=..` for readonly variable shellcheck ignores return value masking
SC2045: prefer glob to iteration over `ls`
Bug: N/A
Test: ran ./run_intergration_tests.sh
Change-Id: Iaf313857ecc417e855afd70c7455841b54b1c363
In the isDir check, there's a possibility that the Stat check fails
because the path is an unresolved symlink. Verify it with Lstat (which
would succeed, since it doesn't follow links), and treat it like a file
if Lstat succeeds.
Test: new integration test
Fixes: 232370097
Change-Id: I9807ca363a5dbdc20639b489b54627bd2cc1ca60
This required the following:
- Adding Platform_base_sdk_extension_version to default soong.variables
- Teaching the symlink tree creation code to understand symlinks
- Making finder.go follow symlinks when requested
Adding yet another knob is unfortunate, but I can't allow that
unconditionally because the Android code base contains a number of
symlinks giving rise to infinite directory trees because they point back
to their parent and this seemed preferable to adding complicated logic
like "follow symlink but if only its fully resolved version does not
point under the source tree".
I could be convinced about the latter, though.
Test: Presubmits.
Change-Id: I453f6b7e5334771f5832c700db00f9d24ed1d82f
Currently, mk2rbc/rbcrun aren't built in certain phases
of the build, which causes the Starlark product configuration
to fail when using get_build_var if a regular build hasn't
been run before.
Bug: 226677850
Test: Manually
Change-Id: If85608eef63be25fddc4d916c82247c13eb0195f
1. Added unit tests for license conversions in the androidmk tool, by
using a mocked filesystem;
2. Added integration tests in a shell script with a mocked filesystem.
Bug: 205615944
Test: m androidmk
Test: bash androidmk_test.sh
Test: TreeHugger
Change-Id: Id3d1f914190b52b82aadda7ea019192787b839d6
We already have a rbc product config ci postsubmit target,
but we want to add a small presubmit check as well.
run_integration_tests.sh is already run on presubmit, so
add it there.
Test: Presubmits
Change-Id: Id6fc282261507fb49a57b578ce2fdf20855dd6fd
It's better not to have state hidden in dotfiles (or rather,
dotdirectories) if we can help.
It's questionable whether the "linux-x86" path segment makes sense since
soong_build only ever runs on one operating system, but I didn't want to
rock the boat now.
Drive-by fixed some quoting fixes in rbc-run. Notably, I didn't wrap
`$@` into double quotes because I don't know whether the lack of double
quotes was intended or not.
Also drive-by fixed the fact that "out/soong" was added twice to the
directory name of bpglob. This turned out not to be a problem because
bpglob doesn't need to be explicitly built: if it's needed, it's
declared as an input of the glob files so it'll be built automatically
(at the cost of the first "null build" not actually being a null build)
Test: Presubmits.
Change-Id: I710d8d16cd8212059a0ca1ee95378505303eed83
--skip-make is deprecated because of its odd behavior around the
out/.soong.kati_enabled file. Replace it with --skip-config --soong-only
instead, which will unambigiously skip running kati.
Bug: 204136549
Test: build/soong/tests/run_integration_tests.sh
Test: build/soong/scripts/build-mainline-modules.sh
Test: build/soong/scripts/build-ndk-prebuilts.sh
Change-Id: I629ee7542687aaafd68eea50346b0cf53eafe1ef
Revert submission 1854375-update-bazel-daecf427ec0bf0e963c324783062c4f5b61ff679
Reason for revert: introduces 503d56eda2, which conflicts with our current cc_binary->cc_shared_library work
Reverted Changes:
I40a511195:Fix test to use --experimental_platform_in_output_...
I05c2c11f3:Update Bazel to https://github.com/bazelbuild/baze...
Ie772d1665:Set --experimental_platform_in_output_dir.
Id66674d72:Update Bazel to https://github.com/bazelbuild/baze...
Change-Id: I0e5a7548feb13c2f3b4deae1921542b15d23a283
This makes soong_ui the only place where soong_build is invoked, thus
greatly simplifying the conceptual model of the build.
It comes with the slight limitation that now soong_docs (and queryview
and the JSON module graph) are not Make targets anymore, but I suppose
that's an acceptable loss.
The only place where someone depended on soong_docs from a Makefile is
removed in a separate change.
Test: Presubmits.
Change-Id: I3f9ac327725c15d84de725d05e3cdde1da3dcbe2
It used to be invoked from out/soong/build.ninja, which required two
soong_build invocations one after the other (ne to generate
out/soong/build.ninja, one to generate the queryview workspace). This
was slower and required some shell-quoted-in-ninja-quoted-in-Go .
Test: Presubmits.
Change-Id: Idda79c067606663b66e9f94626fa24f3b5af4114
This includes the JSON graph generator and bp2build.
Before:
GENERATE_BAZEL_FILES=1 m nothing
GENERATE_JSON_MODULE_GRAPH=1 m nothing
Now:
m json-module-graph
m bp2build
They can now also be combined with other targets or each other.
The longer-term goal is to run "m queryview" and "m soong_docs" using
the same infrastructure. There are two alternate approaches:
1. Call soong_build from within the main Ninja invocation. This requires
two sequential soong_build invocations and is thus slower.
2. Do everything requested in the same soong_build invocation. This
would be faster, but one AFAIU can't tell Ninja that multiple possible
actions can build the same output so that doesn't work.
(1) is somewhat more desirable because soong_docs seems to be built
from build/make/core/main.mk ; I assume that that can be worked around
although I haven't checked where the output of "m soong_docs" goes.
Test: Presubmits.
Change-Id: If5ba36490d9f3f60733e6d6be9286eb2b67c3ff5
Before, we piggybacked on the implementation of regular soong_build and
wrote a fake build.ninja file to satisfy Ninja.
Now, instead, the JSON module graph is a a separate action in the Ninja
output file. This has the pleasant side effect that one can flip back
and forth between generating the JSON file and regular Soong without
loss of incrementality.
Side cleanup: write .d files in a slightly cleaner way.
Test: Presubmits.
Change-Id: Ia853383567b9dd31c53f3bdf56cfc8d517b498ec
This requires some explanation: turns out, Ninja uses the $builddir
variable declared in the top-level Ninja file to figure out where the
command log should go. This resulted in a loss of incrementality between
the manual Ninja invocation in test_null_build_after_docs because they
used different Ninja files and therefore different values of $builddir.
The reason why this was not a problem before is that the output files of
the main Ninja file and the bootstrap one were separate, but with that
changing, this will start to matter.
Test: Presubmits.
Change-Id: I28959cc7d659e698c7694a58326c8731eac060eb
The following files are forked:
- build-globs.ninja
- The glob list files
- .d files for the glob list files
- The output file (build.ninja or the bp2build marker file)
This makes bp2build and soong_build not overwrite each other's files
they need for proper incrementality.
Test: Presubmits.
Change-Id: I69d192cbd36ecd9677f46f3fa095dfce6f872227
This fixes null bp2build builds.
When running bp2build, out/soong/.bootstrap/build-globs.ninja wasn't
generated, therefore none of the globs in out/soong/.primary/globs have
actions to generate them. This makes bp2build always re-run on a null
build, because it depends on those non-existent globs in its depfile.
Fix the above by generating a build-globs.ninja file with the globs that
bp2build encounters during conversion, and add tests to verify that null
builds and bp2build -> soong workflows correctly regenerates the
build-globs.ninja file.
Test: soong integration tests for null soong->soong, bp2build->bp2build and bp2build->soong builds.
Fixes: 193650250
Change-Id: Icfbad351dfa85541d1ea451f73e65cac87097926
In particular, Bazel sometimes creates directories whose files cannot be deleted.
Test: m clean used to fail after b build //system/timezone/apex:com.android.tzdata, but now it works
Test: Added integration test which fails without this change
Change-Id: I08c8feed21c31ec187fe40be513f7eb4865c8ac3
Major operations:
* Fix the go_package entries to be consistent, as the new tool
complains when it doesn't have a '/'.
* Regenerate with the new protoc-gen-go tool.
* github.com/golang/protobuf -> google.golang.org/protobuf
* proto.[Un]MarshalText -> prototext.[Un]Marshal
Change-Id: Ie1147bd2457fafb66ba555461b3bf14f0561a25d
The source tree will eventually be made ReadOnly, and recipes that write
directly to the source tree will fail. Use a pattern-match approach on
the results of stdout/stderr to provide hints to the user in such a
scenario.
If multiple patterns are found in raw output, print error hint
corresponding to first pattern match. first pattern match is chosen
since the failing function will be at the top of the stack, and hence
will be logged first
Test: Wrote a unit test to assert errorhint is added to output.
Wrote an integration test that writes to a file in the source tree
1. When source_tree is RO, the recipe fails and an error hint is printed
to stdout
2. When source tree is RW, the recipe succeeds and no error hint is
printed
Bug: 174726238
Change-Id: Id67b48f8094cdf8a571c239ae469d60464a1e89c
In mixed builds, Bazel operates in <root>/out/soong/workspace, but the real
OUT_DIR is just <root>out/, not <root>/out/soong/workspace/out. This CL
ensures that the correct OUT_DIR is passed to mixed builds's bazel by
making OUT_DIR absolute.
Also add STANDALONE_BAZEL to build_test's Bazel smoke test.
Test: TH
Change-Id: If8b4058d43bf821f5196016f9b0ad38edfe18716
This CL fixes a typo in writeFakeNinjaFile to correctly write
a fake out/soong/build.ninja and its depfile. It also modifies bootstrap
phase to *not* run the main soong build phase (which takes more than a
minute) if GENERATE_BAZEL_FILES=1.
This change has the side effect that `GENERATE_BAZEL_FILES=1 m nothing`
no longer generates the real build.ninja, which is fine because one
shouldn't be using GENERATE_BAZEL_FILES=1 for that anyway (or, use
USE_BAZEL_ANALYSIS=1). This change has no effect on mixed builds.
Time on a change to Soong or any Android.bp files:
Before: bp2build_workspace_marker (~20 seconds) + build.ninja (1 min)
After: bp2build_workspace_marker (~20 seconds)
Time on the second of two consecutive `b build //bionic/...`: 2.070s
Test: TH
Test: Soong integration tests
Change-Id: I43720641815994caba97b8d165d7c3fc254cbd06
Now, if the same file exists in the generated tree and the source tree,
it symlinks in the generated file instead of failing outright.
Drive-by fix: print errors for all conflicts instead of bailing out on
the first one.
Test: Presubmits (including the two new tests)
Change-Id: Ifb5b3fc89b5454d231966bfa4e61c22cd69834f3
This refactors bazel-build mode determination logic in soong_ui so it's
clearer which of three possible modes are being used in a given
invocation (NO_BAZEL, GENERATE_BUILD_FILES, or MIXED_BUILDS).
Test: bootstrap tests
Change-Id: I41d2baebf8d560c2cc42db8daa8b936101d453e3
It's under out/soong/workspace and is a symlink forest that merges BUILD
files generated by bp2build and the source tree.
bazel-* symlinks and $OUTDIR are currently ignored.
Test: Presubmits.
Change-Id: If14df23bd0f6a6f2d7d1140f95213f2aee66d652
I6a98660cfe6a40bcaa978e8d7544cdffad86a7fa was submitted concurrently
with test_glob_noop_incremental and broke the test by making the
first build incremental. Add an rm -rf out to make the test start
from a clean build.
Bug: 185591972
Test: tests/bootstrap_test.sh
Change-Id: I529aaf707b2790e5e1022ba9b2637ed7d2435bd5
Update the path bootstrap_test.sh uses to check for bpglob reruns
to follow the changes made for sharding globs.
Use bootstrap.GlobFileListFiles to get dependencies during bpbuild.
Test: tests/bootstrap_test.sh
Bug: 159845846
Change-Id: Ibaa7c4360881ed6b666a811bf34b19ea0cdcafe9
It now handles adding .bp files and changing globs.
In order to do this, depfiles are now written separately from RunBlueprint.
This is necessary due to the confluence of a number of seemingly
unrelated factors:
1. The glob filelist dependencies are discovered in globSingleton
2. Singletons need to be registered because otherwise singleton module
types panic
3. Singletons don't work because they require mutators bp2build does not
run
Due to (1), we would need to run the glob singleton. However, due to (2)
and (3), we can't run singletons and have to run Blueprint with
StopBeforeGeneratingBuildActions, which is when the build actions
writing glob files would be generated. So what happens is:
1. When bp2build is run, the glob singleton is disabled
2. At the end of bp2build, the list of glob files is artifically added
to the depfile of the workspace marker file
3. When build.ninja is generated, the Ninja file containing the glob
list file is written by the now-active glob singleton
Test: Presubmits.
Change-Id: I3c5898d8c57c554a93520276c64a952afc912dbe
This is done by running a build before the first test case, tarring up
$TOP (including out/) then untarring it for every test case instead of
rebuilding soong_build each time.
The speedup is approximately 2x.
Also add some cute ANSI escape sequences and delineation of individual
test cases.
Test: The aforementioned integration tests.
Change-Id: I6a98660cfe6a40bcaa978e8d7544cdffad86a7fa
This is done by setting the INTEGRATED_BP2BUILD environment variable
when invoking the build.
Even though the name of the marker file insinuates that a Bazel
workspace is already created, this is not the case yet.
An issue that remains is that a .d file is not written for the marker
file so it won't be rebuilt if a .bp file changes. Fixing this requires
delicate surgery because writing the .d file is the result of delicate
interplay between Soong and Blueprint.
There are also a number of semi-related fixes:
- The name of soong.environment.{used,available} is now on the command
line of soong_build (soong_docs is still special cased because its
command line in the Ninja file is taken from the os.Args of
soong_build so it's not trivial to remove the --{available,used}_env
from it
- bp2build writes a separate soong.environment.used file
- I had to call SetAllowMissingDependencies() separately when creating
the android.Context for bp2build so that bp2build runs in the
integration tests (it was not obvious how not to do this)
- Fixed a number of integration tests where a command with an expected
exit code of 1 was used as the last one in a test case, thereby
breaking the test suite
Test: Presubmits.
Change-Id: Ibeb61c26022cf801dcb98505b4039151b3409873
Test that an incremental build with a glob doesn't rerun soong_build, and
only regenerates the globs on the first but not the second incremental build.
Bug: 159845846
Test: tests/bootstrap_test.sh
Change-Id: Ib9e3f2f22422d567253a366d28255e3f39d97ca8
bootstrapping can read extra Android.bp files that could affect
soong_build.
Bug: 159845846
Test: bootstrap_test.sh
Change-Id: Ib053ff9c2ee70638419f122a36438a515640e0b4
- Move them into a new directory
- Split tests by theme and add a library file
- Add a shell script that runs them all
Test: Manually ran build/soong/run_integration_tests.sh .
Change-Id: I758b91d679f41aee47d15472cc02547ce89f6386
