This is a second attempt at aosp/2273288 which got rolled back because
it turned out that:
1. We make ~120K symlinks in AOSP (!), all of which need to be deleted
2. System calls are sometimes slow
3. Golang spawns a new OS-level thread for each blocking system calls to
keep cores busy
All this together means that we sometimes had 10K system calls in
flight, which meant 10K OS-level threads, which is when Go gives up and
says "I created too many threads, please help".
The fix is to move the system calls into a pool of goroutines, which
soon end up on a pool of threads (since they mostly do blocking system
calls)
Test: Presubmits.
Change-Id: Ia9aefff3b0ed373f09bb6c8b2ec1d8b0f00b213b
This makes symlink forest creation ca. 2x faster again, taking 2-3
seconds instead of 5.
Who would have thought that os.RemoveAll() is slow.
Test: Presubmits.
Change-Id: I91e41319c972dbf1113cf723e383c785433c18b9
This makes it take ~5 seconds on AOSP instead of ~10. Frankly, the
speedup is somewhat disappointing but at least the code is not
complicated.
Test: Presubmits.
Change-Id: Icf94d7ca8bd80c458d014f4cf4cc1be7138deaa6
This helps with incrementality a lot: the symlink forest must depend on
almost every directory in the source tree so that if a new file is added
or removed from *anywhere*, it is regenerated.
Previously, we couldn't do this without invoking bp2build, which is
quite wasteful because bp2build takes way more time than the symlink
forest creation, even though we do the latter in a very suboptimal way
at the moment.
This means that if a source file is added or removed (which does not
affect globs), we don't pay the cost of bp2build anymore.
Also refactored symlink_forest.go on the side. Too much state was being
passed around in arguments.
This change reimplements aosp/2263423 ; the semantics of not touching an
output file is the exact same as order-only inputs and the latter is a
bit fewer lines of code.
Test: Presubmits.
Change-Id: I565c580df8a01bacf175d56747c3f50743d4a4d4
bp2build (i.e. bootstrap.ninja#bp2build_workspace_marker) generate $OUT/soong/bp2build/**/BUILD.bazel files
Having them as a dependency would thus make bp2build_workspace_marker stale upon incremental builds (because we don't re-touch the marker file if it already exists)
See Also: https://android-review.googlesource.com/c/platform/build/soong/+/2263423
Bug: b/253450880
Test: NINJA_ARGS='-d explain' m --bazel-mode adbd
repeat and see if bp2build is rerun
prior to this CL, one would see the input /usr/local/google/home/usta/aosp/out/soong/bp2build/build/make/tools/BUILD.bazel trigger bp2build over and over again
Change-Id: I904cd333a5d6ef506fc4019eda7623ef96a1daa3
The merged build files should be added as ninja dependencies,
so we rerun when they're changed.
Fixes: 246552590
Test: m bp2build, m bp2build again and observe it didn't rerun. Then add a comment in external/protobuf/BUILD.bazel, run m bp2build again, and observe it reruns
Change-Id: I26ed035cc0a894500a192f9aa3371fb46519689b
Previous behavior:
- Packge not listed in bp2buildKeepExistingBuildFile:
- Use bp2build generated build file
- Package listed in bp2buildKeepExistingBuildFile:
- Use handcrafted build file even if there were allowlisted bp2build
modules in the same package.
- Package listed in bp2buildKeepExistingBuildFile and a soong module has
a bp2build: { label } attribute:
- Merge the handcrafted and bp2build generated build files
New behavior:
- Packge not listed in bp2buildKeepExistingBuildFile:
- Use bp2build generated build file
- Package listed in bp2buildKeepExistingBuildFile:
- Merge with bp2build generated build file.
Bug: 234167862
Test: ./build/bazel/ci/bp2build.sh
Change-Id: Ifbaf4f8f0f5158b5b2bd6d534eb2311e2e5f399b
In the isDir check, there's a possibility that the Stat check fails
because the path is an unresolved symlink. Verify it with Lstat (which
would succeed, since it doesn't follow links), and treat it like a file
if Lstat succeeds.
Test: new integration test
Fixes: 232370097
Change-Id: I9807ca363a5dbdc20639b489b54627bd2cc1ca60
This required the following:
- Adding Platform_base_sdk_extension_version to default soong.variables
- Teaching the symlink tree creation code to understand symlinks
- Making finder.go follow symlinks when requested
Adding yet another knob is unfortunate, but I can't allow that
unconditionally because the Android code base contains a number of
symlinks giving rise to infinite directory trees because they point back
to their parent and this seemed preferable to adding complicated logic
like "follow symlink but if only its fully resolved version does not
point under the source tree".
I could be convinced about the latter, though.
Test: Presubmits.
Change-Id: I453f6b7e5334771f5832c700db00f9d24ed1d82f
Now, if the same file exists in the generated tree and the source tree,
it symlinks in the generated file instead of failing outright.
Drive-by fix: print errors for all conflicts instead of bailing out on
the first one.
Test: Presubmits (including the two new tests)
Change-Id: Ifb5b3fc89b5454d231966bfa4e61c22cd69834f3
It's under out/soong/workspace and is a symlink forest that merges BUILD
files generated by bp2build and the source tree.
bazel-* symlinks and $OUTDIR are currently ignored.
Test: Presubmits.
Change-Id: If14df23bd0f6a6f2d7d1140f95213f2aee66d652
