This required the following:
- Adding Platform_base_sdk_extension_version to default soong.variables
- Teaching the symlink tree creation code to understand symlinks
- Making finder.go follow symlinks when requested
Adding yet another knob is unfortunate, but I can't allow that
unconditionally because the Android code base contains a number of
symlinks giving rise to infinite directory trees because they point back
to their parent and this seemed preferable to adding complicated logic
like "follow symlink but if only its fully resolved version does not
point under the source tree".
I could be convinced about the latter, though.
Test: Presubmits.
Change-Id: I453f6b7e5334771f5832c700db00f9d24ed1d82f
This retriggers soong_build whenever a new bzl, WORKSPACE, or
BUILD.bazel file is changed or added.
Test: Manually verified on bionic/libc genrules with manual changes to
related BUILD/bzl/WORKSPACE files -- these all retrigger builds.
Test: Updated finder_test.go
Change-Id: I634384f88781a6b6db32f5d6bf9c07e179e14c39
This reverts commit 323dc60712.
Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests
Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.
Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
Help debugging finder issues by disting module_paths/files.db for
later analysis.
Bug: 157656545
Test: treehugger
Change-Id: Idfa4cffe7ef724b399a2667efd487f9c3c76bb6b
When finder encountered a symlink pointing to a directory it attempted
to ignore it, but becuase Lstat returns the a *os.FileInfo for the
symlink the child.IsDir() check always returned false. Call Stat
on the symlink to get the *os.FileInfo for the target of the symlink.
Bug: 157656545
Test: finder_test.go
Change-Id: Ie33d2e05d5c435b48e83eaeadf7b3c9816398404
Add a Stat method to finder/fs that will be used by finder to read
the mode of the target of a symlink.
Bug: 157656545
Test: fs_test.go
Change-Id: Ie2b4509b7d11857d9a1685de4477088b91d43c63
Move the filesystem helper functions used by finder_test.go into the
fs package so they can also be used by fs tests.
Bug: 157656545
Test: m checkbuild
Change-Id: I5ac3a1f502d8e2cb7c15dd70ca85a46d20f939a6
ioutil.ReadDir returns []os.FileInfo, which contains information on
each entry in the directory that is only available by calling
os.Lstat on the entry. Finder only the name and type (regular,
directory or symlink) of the files, which on Linux kernels >= 2.6.4
is available in the return values of syscall.Getdents.
Replace ioutil.ReadDir with a call that uses syscall.Getdents
directly and collects the type information from the result.
Testing with:
rm -f /tmp/db && strace -fc finder -names Android.mk,Android.bp,Blueprints,CleanSpec.mk,TEST_MAPPING -exclude-dirs .git,.repo -prune-files .out-dir,.find-ignore -db /tmp/db .
Before:
7.01 52.688304 63 833398 1 lstat
1.90 14.246644 68 210523 getdents64
1.25 9.370471 90 104286 1 openat
After:
3.48 12.201385 117 104286 1 openat
3.06 10.729138 51 210523 getdents64
1.70 5.951892 57 104283 1 lstat
Pros:
Avoids 729115 calls to lstat.
Cons:
Requires copying ~200 lines of finicky buffer parsing code.
Puts all getdents calls (and possibly fallback lstat calls) onto
a non-blocking file descriptor, which will cause it to block a
thread and not just a goroutine.
Only works on Linux and Darwin.
Bug: 70897635
Test: m checkbuild
Change-Id: Iab9f82c38c8675d0b73b4e90540bb9e4d2ee52c1
ioutil.ReadDir returns []os.FileInfo, which contains information on
each entry in the directory that is only available by calling
os.Lstat on the entry. Finder only the name and type (regular,
directory or symlink) of the files, which on Linux kernels >= 2.6.4
is available in the return values of syscall.Getdents.
In preparation for using syscall.Getdents, switch filesystem.ReadDir
to return an interface that only contains the information that will
be available from syscall.Getdents.
Bug: 70897635
Test: m checkbuild
Change-Id: Id2749d709a0f7b5a61abedde68549d4bf208a568
It turns out the os was returning a different permissions error than
os.ErrPermission. Fortunately we can just use os.IsPermission().
Bug: 62455338
Test: mkdir not-permitted && chmod 000 not-permitted && m nothing
Change-Id: Ieb1653b9687ffdaae83184dcdcd4166ddb6a4613
Also change the existing tests to use exactly 2 threads.
Also update some tests to also same number of threads as will
be used by the finder.
Bug: 62455338
Test: m -j # which runs unit tests
Change-Id: I2b9d39680ecddd6d28c951df982ac51c077d2777
Previously there could be an infinite loop in the Finder's db export when
running tests on a system with few cores.
The Finder divides the cache-serialization work amongst cores. On systems with
enough cores, each path in TestFileSystemRoot would get its own core and the
infinite loop in the prefix calculation logic would be avoided. On systems with
fewer cores, the computation of the maximum common path could run forever.
This shouldn't have affected the general usage of the Finder in builds other
than in its unit tests, because the builds don't ask to include the
filesystem root as a path to scan. This should only have affected
finder_test.go which tested the filesystem root anyway.
Bug: 62455338
Test: m -j # which runs unit tests
Change-Id: I7a1467fb32e4538fb96833791180bfbfad5a3911
The Finder runs roughly 200ms faster than findleaves.py in aosp,
and runs roughly 400ms faster in internal master.
Bug: 64363847
Test: m -j
Change-Id: I62db8dacc90871e913576fe2443021fb1749a483
Permissions errors are ignored
Errors on pruned dirs are also ignored
Bug: 62455338
Test: m -j blueprint_tools # which runs unit tests
Change-Id: I8ba85fdd0295deb7dc374a851212e7c850e76b75
This reverts commit b6d161bf16.
Reason for revert: New Build Breakage: aosp-master/sdk_mac @ 4260825
Change-Id: I8bda8c50c5e5c9f84621d11a4c15b168833bcd21
It can find every Android.bp in internal master in
about 2.5 sec the first time and 0.3 sec subsequent times
Bug: 62455338
Test: m -j blueprint_tools # which runs the unit tests
Test: m -j blueprint_tools && \
out/soong/host/linux-x86/bin/finder \
-v
--db /tmp/mydb \
--names Android.mk \
--prune-files .android-out-dir \
--exclude-dirs .git,.repo \
. \
>/tmp/finder-log 2>&1
Change-Id: I5ab2650459a1dae0d5d076faf411ec2d053c743d
