osFs.acquire and osFs.release are surprisingly expensive, using a
combined 345.7s of runtime in an AOSP aosp_cf_x86_64_phone-userdebug
build. They are used to ensure we don't use too many simultaneous
open files, but many of the functions they are called from don't
actually open a file. Remove them from all the stat-based functions
(Exists, IsDir, IsSymlink, Lstat, Stat), and from ReadLink. After
this change the time spent in acquire and release is effectively
zero.
Test: SOONG_PROFILE_CPU=/tmp/cpu.pprof m nothing
Change-Id: Ie5e22e33c61794354821f05ab79ceb4efc3b276c
Mac builds keep running into too many files open. Restrict access of
filesystem to the current limit.
Test: m nothing
Change-Id: I2365da7c641f7c7f5d948396c6862eb3a0d1d8b9
When globbing with following symlinks enabled, treat dangling symlinks
as files instead of erroring.
Bug: 202547639
Test: TestGlobFollowDanglingSymlinks
Change-Id: Ic1b241d3fcf1bc6989cb724d00c2b97fefa8dcdb
There are a few cases that force all globs to be rerun at the beginning
of the build (changes to bpglob or dependencies, second build after a
clean build). The number of globs has gotten high enough that rerunning
them all can have significant overhead to start bpglob for each one.
Replace the per-glob bpglob invocations with sharded invocations using
1024 hash buckets.
Bug: 159845846
Test: glob_test.go
Test: m nothing && m nothing
Test: build/soong/bootstrap_test.sh
Change-Id: Ife1f7a03c8f6b25d1be01531425d8dc2c76d1ea0
bpglob is executed through the rules in build-globs.ninja to determine
whether soong_build needs to rerun. That means when the arguments
accepted by bpglob change it will be called with the old arguments,
then soong_build will rerun and update build-globs.ninja with the new
arguments.
To avoid having to maintain backwards compatibility with old arguments
across the transition, a version argument is used to detect the
transition in order to stop parsing arguments, touch the output file
and exit immediately. The version number in
pathtools.BPGlobArgumentVersion should be manually incremented when
the bpglob argument format changes.
If the version argument is not passed then a version mismatch is assumed.
Bug: 159845846
Test: build/soong/tests/bootstrap_test.sh
Change-Id: Id6742c7acc51af8d0d224d51e47bddea78e6e363
Replace the individual matches and deps return values with a GlobResult
struct. Use the GlobResult to create the file list file in GlobWithDepFile
(used by bpglob) and in the glob singleton, as the generated files must
match exactly so that soong_build is not rerun.
Bug: 159845846
Test: glob_test.go
Change-Id: I2159cc9d85f388073198eac7456e5bf43e813096
ReplaceExtension had an unexpected behaviour when the file did not have
an extension. In certain cases, the final path would be severely
trimmed: out/.intermediates/my_file would become out/.new_extension.
Explicitly handle the case by appending the new extension.
Test: Run checkbuild on Android Soong
Change-Id: Ie27a98845894cfaee5af5e2a02d44168c40ed821
This is an imported pull request from
https://github.com/google/blueprint/pull/345
GitOrigin-RevId: f9166c0e6151499b4b1a23b89b0bc133203a1116
Change-Id: I63f0798177545792440b8a84b04f1090590f1642
Glob was calling IsSymlink and IsDir on each visited directory entry,
which resulted in an lstat and then a stat call on each.
Instead, use lstat when not following symlinks and use stat when
following symlinks, then use the result to check if the entry is a
directory.
Test: glob_test.go
Change-Id: I83d769e2de64ce8221e952e5204d365aeaf47687
OsFs may be asked to read absolute paths if buildDir is absolute.
Check if the path is absolute before prepending srcDir to it.
Bug: 146437378
Test: fs_test.go
Change-Id: I2a67593e9d836ca3e11dc10b81f49a4fb49d2cdf
filepath.Match has always supported hierarchical patterns (a/*),
so pathtools.Match can be significantly simplified by reusing
filepath.Match directly instead of recursively stepping through
the path elements using filepath.Match on each one.
Test: glob_test.go
Change-Id: I8af59ee880f0402609b994922bafb1961fcabcf3
Add some more test cases for pathtools.Match before making
signficant changes to it.
Test: glob_test.go
Change-Id: I2eeb5ebf03fb645a2053852a1a9f4e368d22084a
Stat is used by soong_zip. Add it to the FileSystem interface
and add tests for it.
Test: fs_test.go
Change-Id: I1baa2b27398846a4e55bcf4fa291c62f507a4e9d
Readlink is used by soong_zip. Add it to the FileSystem interface
and add tests for it.
Test: fs_test.go
Change-Id: Ie8ca5cd7cae98a47980a50d2891501fe79fd47a5
Only check that functions that return an *os.SyscallError have
the correct errno, not that they have identical test text.
Test: fs_test.go
Change-Id: Iba050cb0474eaf2643858bcca4e52ba770702c2f
Both *os.File and *bytes.Buffer support all of io.Reader,
io.ReaderAt, io.Seeker and io.Closer. Return a combo interface
so that soong_zip can use the result in an io.SectionReader.
Test: m checkbuild
Change-Id: I31c3ce35e28c52bae20b536b5905de2f8a3d1478
Allow the caller to specify whether symlinks should be followed
(the old behavior) or not.
Test: glob_test.go
Test: fs_test.go
Change-Id: I550dc91b8e6370fb32a9a1cbdcb2edade48bda46
filepath.Walk does not walk symlinks to directories, which leads to
inconsitent behavior with following symlinks. Replace the use of
filepath.Walk in ListDirsRecursive with a helper function that lists
each directory and walks anything for which IsDir reports true, which
includes following symlinks, and then reconstructs the path through
the symlink.
Test: fs_test.go
Test: glob_test.go
Change-Id: Ie4dd0820e9c7c0a124caa65210ce20439a44da16
Add support for specifying symlinks in mock filesystems to prepare
for glob symlink tests.
This patch leaves incorrect behavior by not walking symlinks in
mockFs.ListDirsRecursive, but it matches what osFs does.
Test: fs_test.go
Change-Id: If87a83c00f21e14696faf890b7b09e88b18e95b9
IsDir on a dangling symlink produces ErrNotExist. Manually
check if the file exists but is a symlink to report a better
error.
Test: m checkbuild
Change-Id: I3181e74002436d74ec35a0923635835e561030dd
Globs that match a file that looks like a glob were causing duplicate
results because the prefix match would then re-match the
filename as a wildcard. Add escaping to prevent re-matching.
Also add tests for globs on files with wildcards, tests for
? and [a-z] glob matches supported by filepath.Match, and tests
for escaped wildcard characters.
Test: glob_test.go
Change-Id: Id11a754863979bb36cca0dbd18ea2e76dd1470e3
pathtools.Match was trimming the trailing slash from patterns. Make
it handle a trailing slash by first confirming that both the pattern
and name either both have or both do not have a trailing slash, and
then trimming it from both.
Bug: 111389216
Test: TestMatch in glob_test.go
Change-Id: I743e00c14d885de5b5a060aa9e2b22c81dc7e09d
This makes it easy for users of Glob to detect whether the match is a
file or directory. Doing the check at this level means that the filelist
file used as a dependency will be updated if a directory is replaced
with a file of the same name, or vice versa.
Change-Id: I79ebba39327218bcdcf50b393498306119de9d6c
Export pathtools.Match so that tools that operate on file lists
(like zip listings) can use the same glob format.
Test: m checkbuild
Change-Id: Ic4b90139c381c57551836590da3ea54826b0227a
Recursive globs on MockFs were ending up in an unmocked os.Lstat
call in walkAllDirs. Move walkAllDirs into the FileSystem interface
as ListDirsRecursive.
Also duplicate the glob tests on both the real filesystem and on
MockFs.
Test: glob_test.go
Change-Id: Ia6b6b5eecdd17955a49d684a0fd5e55df05cfe62
Bug: 64600838
Test: mkdir errtest \
&& ln -s /tmp/dontexist errtest/Android.bp \
# and add errtest to ./Android.bp \
&& m nothing \
# and check that the error message mentions a symlink
Change-Id: I841ec12d613f61ccc3396538062bee48c8c1ca27
Patterns that were not wild would return an empty "dirs" list if the
file was found. But then if they were removed, we wouldn't know to
update the glob file and re-run the primary builder.
In this case, instead of adding the final directory into the dirs list,
add the matched files themselves. Due to editors performing atomic
writes (the directory timestamp often gets updated at the same time as
file timestamp) this is probably more efficient. In either case, we're
only re-running the individual glob, which is rather cheap.
Rename startGlob/Glob return name from "dirs" to "deps" since it may
contain files now too.
Add globbing to filesystem mocking so that more code can be tested
against the mock. Also moves the filesystem mock to pathtools,
and renames pathtools.GlobWithExcludes to pathtools.Glob, replacing
the existing pathtools.Glob.
Test: blueprint tests
Change-Id: I722df8121bc870c4a861d7c249245c57dcb607be
Hidden files are often source control related (".git", ".repo", etc) or
editor related (vim's .*.swp), so are not guaranteed to exist, and may
be temporary. The build system shouldn't be using these files by
default.
If the glob pattern explicitly uses "." at the beginning of a path
component, allow returning hidden files for that component. Because of
this behavior, non-wildcard globs remain unchanged.
The one behavior that cannot be handled anymore is including hidden
files in recursive globs.
Change-Id: I583c506e9a18ed2ff7ca011a791165d9582de90f
Bypassing c.glob() and using filepath.Glob() directly for non-glob
paths does not add dependencies on directories that contain missing
files. For optional_subdirs, this means no dependency is added to
rerun the primary builder when an Android.bp file is added to an
optional_subdirs directory. Always use c.glob(), for the non-optional
case it will not insert any dependencies if the file exists (as tested
by glob_test.go's no-wild tests), and if the file doesn't exist the
len(matches) == 0 will error out.
Change-Id: I370479c6e89f5ff590897702e256256a4dca6952
Sometimes os.Stat on a path seems to return nil after filepath.Glob
returned the path as valid. Return the error message to see why.
Bug: 32676828
Test: builds
Change-Id: Ieedddb673b4d641e5de08778febeb3d8ea025c0d
Add globbing with dependency checking to blueprint. Calling
ModuleContext.GlobWithDeps or SingletonContext.GlobWithDeps will return
a list of files that match the globs, while also adding efficient
dependencies to rerun the primary builder if a file that matches the
glob is added or removed.
Also use the globbing support for optional_subdirs=, subdirs= and build=
lines in blueprints files. The globbing slightly changes the behavior
of subname= lines, it no longer falls back to looking for a file called
"Blueprints". Blueprint files that need to include a subdirectory with
a different name can use build= instead of subdir= to directly include
them. The Blueprints file is updated to reset subname="Blueprints" in
case we want to include subdirectories inside blueprint and the primary
builder has changed the subname.
Also adds a new test directory that contains a simple primary builder
tree to test regeneration for globbing, and runs the tests in travis.
Change-Id: I83ce525fd11e11579cc58ba5308d01ca8eea7bc6
If the initial non-wild part of a glob path does not exist, return
the last existing part of the path as a dependency to detect if the
path is created later.
Change-Id: Ib5a39e6830cb386deed26e017279d0aac1bc9a20
Add GlobWithExcludes, which takes a pattern and a list of exclude
patterns, and returns all files that match the pattern but do not
match any exclude patterns.
Change-Id: I8b94b3ba5a37409071b475b9a4035f52f47863f1
Recursive globs are supported by passing ** in any single non-final
path element. For example, path/**/*.java will find all files named
*.java under "path".
Change-Id: Ifebd76f8959289f7d0d378504053c1c6b88cdeed
The directory structure:
a/
a
b/
b
With the glob pattern */a would previously return []string{"a"} for
dirs, but it needs to return []string{"a", "b"} in order to re-run
the generator if a file called "a" is created inside b/.
Rewrite Glob to manually recurse through path elements, only calling
filepath.Glob for a pattern with wilds in the last element of the
path, and add the globbed directory to the dirs list each time.
Also add tests and test data for pathtools.Glob.
Change-Id: Ibbdb2f99809ea0826d4fa82066cf84103005ef57
regexp is overkill for a simple extension replacement, just find
the last '.' and add the new extension after it. Saves 5% cpu time
on one workload.
Change-Id: Ic299c9ec5132d15bbea2c9c7778c000d6fdf509a
Make integrating with go tools easier by putting the blueprint package
files in the top level directory of the git project instead of in a
subdirectory called blueprint.
Change-Id: I35c144c5fe7ddf34e478d0c47c50b2f6c92c2a03
