There was an incrementality bug in the symlink forest generation that
was recently fixed. However, if the appropriate files don't get touched,
the issue will remain. Add a mechanism to clean old symlink forests
so that when we fix incrementality bugs we can ensure they don't remain
in any old out directories.
Bug: 276349152
Test: Presubmits
Change-Id: I36664dfb0ca7227e3e1f89de859ebccb808c5f15
It was possible for the merged content to end up back in the bp2build
generated file because there was a symlink from the symlink forest to
the bp2build generated file.
Remove the symlink if it exists.
Bug: 276349152
Test: m bp2build, add a handcrafted file in the same folder as a Android.bp file, m bp2build again, check that the symlink forest version is not a symlink
Change-Id: Id64aa3addebcf0c6b1728389f21ae246796aaf8d
This changes bp2build codegen, symlink forest generation, and
soong_build so that they do not rewrite output files if the contents are
unchanged.
Bug: 266983462
Test: m droid
Test: canonical_perf.sh benchmarking
Test: Manually verified that rerunning analysis did not regenerate
out/soong/workspace/prebuilts/sdk/BUILD.bazel unless contents changed
Change-Id: I5ec227df7a32b53c7fa0d741fb1403a51931024b
Previously, symlink forest generation involved removing the entire
symlink forest and recreating it from scratch. With this change,
a) symlinks which need not change are untouched,
b) symlinks pointing to the wrong location are fixed, and
c) symlinks which should no longer exist are removed.
On AOSP on my local machine, this reduces the symlink forest generation
step from 2.5s to 1.1s clean, and 0.6s when a single file is added to
a source directory.
Bug: 257528847
Test: m bp2build, touch `fakefile` under the forest, remove a file
from the source tree, rerun m bp2build. Manually verify the new forest
does not retain the link to the deleted source file, and that fakefile
no longer exists in the forest.
Change-Id: I481371ae487e9419af6a3a4370c552578b07d650
We were looping over a map, and thus processing files
in a non-deterministic order. Some tests check for errors
produced during the symlink forest creation and they would
flake due to this nondeterminism.
Test: m nothing
Change-Id: Ie1a4cf98e341e4a951f110b7d5611bb69c4ddd2b
Brings down time taken to fail from 18s to 13s (on aosp branch)
Test: temporarily coded random failure
Bug: N/A
Change-Id: Ib694eec977293f4dd7054e779d1b82b8cace93f6
This is a second attempt at aosp/2273288 which got rolled back because
it turned out that:
1. We make ~120K symlinks in AOSP (!), all of which need to be deleted
2. System calls are sometimes slow
3. Golang spawns a new OS-level thread for each blocking system calls to
keep cores busy
All this together means that we sometimes had 10K system calls in
flight, which meant 10K OS-level threads, which is when Go gives up and
says "I created too many threads, please help".
The fix is to move the system calls into a pool of goroutines, which
soon end up on a pool of threads (since they mostly do blocking system
calls)
Test: Presubmits.
Change-Id: Ia9aefff3b0ed373f09bb6c8b2ec1d8b0f00b213b
This makes symlink forest creation ca. 2x faster again, taking 2-3
seconds instead of 5.
Who would have thought that os.RemoveAll() is slow.
Test: Presubmits.
Change-Id: I91e41319c972dbf1113cf723e383c785433c18b9
This makes it take ~5 seconds on AOSP instead of ~10. Frankly, the
speedup is somewhat disappointing but at least the code is not
complicated.
Test: Presubmits.
Change-Id: Icf94d7ca8bd80c458d014f4cf4cc1be7138deaa6
This helps with incrementality a lot: the symlink forest must depend on
almost every directory in the source tree so that if a new file is added
or removed from *anywhere*, it is regenerated.
Previously, we couldn't do this without invoking bp2build, which is
quite wasteful because bp2build takes way more time than the symlink
forest creation, even though we do the latter in a very suboptimal way
at the moment.
This means that if a source file is added or removed (which does not
affect globs), we don't pay the cost of bp2build anymore.
Also refactored symlink_forest.go on the side. Too much state was being
passed around in arguments.
This change reimplements aosp/2263423 ; the semantics of not touching an
output file is the exact same as order-only inputs and the latter is a
bit fewer lines of code.
Test: Presubmits.
Change-Id: I565c580df8a01bacf175d56747c3f50743d4a4d4
bp2build (i.e. bootstrap.ninja#bp2build_workspace_marker) generate $OUT/soong/bp2build/**/BUILD.bazel files
Having them as a dependency would thus make bp2build_workspace_marker stale upon incremental builds (because we don't re-touch the marker file if it already exists)
See Also: https://android-review.googlesource.com/c/platform/build/soong/+/2263423
Bug: b/253450880
Test: NINJA_ARGS='-d explain' m --bazel-mode adbd
repeat and see if bp2build is rerun
prior to this CL, one would see the input /usr/local/google/home/usta/aosp/out/soong/bp2build/build/make/tools/BUILD.bazel trigger bp2build over and over again
Change-Id: I904cd333a5d6ef506fc4019eda7623ef96a1daa3
The merged build files should be added as ninja dependencies,
so we rerun when they're changed.
Fixes: 246552590
Test: m bp2build, m bp2build again and observe it didn't rerun. Then add a comment in external/protobuf/BUILD.bazel, run m bp2build again, and observe it reruns
Change-Id: I26ed035cc0a894500a192f9aa3371fb46519689b
Previous behavior:
- Packge not listed in bp2buildKeepExistingBuildFile:
- Use bp2build generated build file
- Package listed in bp2buildKeepExistingBuildFile:
- Use handcrafted build file even if there were allowlisted bp2build
modules in the same package.
- Package listed in bp2buildKeepExistingBuildFile and a soong module has
a bp2build: { label } attribute:
- Merge the handcrafted and bp2build generated build files
New behavior:
- Packge not listed in bp2buildKeepExistingBuildFile:
- Use bp2build generated build file
- Package listed in bp2buildKeepExistingBuildFile:
- Merge with bp2build generated build file.
Bug: 234167862
Test: ./build/bazel/ci/bp2build.sh
Change-Id: Ifbaf4f8f0f5158b5b2bd6d534eb2311e2e5f399b
In the isDir check, there's a possibility that the Stat check fails
because the path is an unresolved symlink. Verify it with Lstat (which
would succeed, since it doesn't follow links), and treat it like a file
if Lstat succeeds.
Test: new integration test
Fixes: 232370097
Change-Id: I9807ca363a5dbdc20639b489b54627bd2cc1ca60
This required the following:
- Adding Platform_base_sdk_extension_version to default soong.variables
- Teaching the symlink tree creation code to understand symlinks
- Making finder.go follow symlinks when requested
Adding yet another knob is unfortunate, but I can't allow that
unconditionally because the Android code base contains a number of
symlinks giving rise to infinite directory trees because they point back
to their parent and this seemed preferable to adding complicated logic
like "follow symlink but if only its fully resolved version does not
point under the source tree".
I could be convinced about the latter, though.
Test: Presubmits.
Change-Id: I453f6b7e5334771f5832c700db00f9d24ed1d82f
Now, if the same file exists in the generated tree and the source tree,
it symlinks in the generated file instead of failing outright.
Drive-by fix: print errors for all conflicts instead of bailing out on
the first one.
Test: Presubmits (including the two new tests)
Change-Id: Ifb5b3fc89b5454d231966bfa4e61c22cd69834f3
It's under out/soong/workspace and is a symlink forest that merges BUILD
files generated by bp2build and the source tree.
bazel-* symlinks and $OUTDIR are currently ignored.
Test: Presubmits.
Change-Id: If14df23bd0f6a6f2d7d1140f95213f2aee66d652
