Commit graph

727 commits

Author SHA1 Message Date
Colin Cross
1be9abc109 Add link dependency on test package archive
The test package archive was being picked up through "-L" + testRoot,
but did not have a dependency on it.  If the *_test.go file was
changed but not the implementation then the tests would not be
rerun.

Test: Add t.Fatal() to a *_test.go file and rebuild.
Change-Id: I32b8c7a0fa1d5b76b1242646887fe937e396e318
2018-10-24 13:13:13 -07:00
Jaewoong Jung
ccc3494088 Make sure all modules have a name. (#226)
Throw an error if run into a module without a name when generating
contexts.

Test: context_test.go
Change-Id: I3976d86d1f15b8ac10a7a38aa42ae277740a8f3b
2018-10-11 13:01:05 -07:00
colincross
d127ba077b
Merge pull request #225 from colincross/test_link_deps
Add missing dependencies when linking tests
2018-10-05 11:28:06 -07:00
Colin Cross
0e58dc0176 Add missing dependencies when linking tests
Linking tests needs the same dependencies that linking binaries
got in https://github.com/google/blueprint/pull/222.

Test: m checkbuild
Change-Id: I33330f3184b8c0fd2bc20b48736c20d6edeaea68
2018-10-05 11:12:53 -07:00
colincross
3f13f07565
Merge pull request #224 from colincross/slice_of_structs
Allow mutated property structs to contain slices of non-strings
2018-10-03 20:25:27 -07:00
colincross
888fe0dd3b
Merge pull request #222 from colincross/go_restat
Enable restat for go binaries
2018-10-03 14:30:12 -07:00
Colin Cross
12392ca0f1 Allow mutated property structs to contain slices of non-strings
Property structs can now contain slices of structs and other
non-strings as long as they are tagged with `blueprint:"mutated"`.

Test: clone_test.go
Test: unpack_test.go
Change-Id: Ib77348dc6e7314a24f17caba10040f7d3ac54e54
2018-10-03 14:29:12 -07:00
Colin Cross
6c92af937e Enable restat for go binaries
Many Android build tools are written in go, and changes to Soong or
Blueprint can cause them to rebuild.  Almost everything in an
Android build is downstream of at least one of these tools, so
they all rebuild too.

Go binaries are static, so their contents will always change if any
of their dependencies change.  Only update output files of go compile
and go link if the contents change, and enable restat for the rules.

Test: m soong_zip && m soong_zip
Test: m soong_zip && touch build/soong/zip/cmd/main.go && m soong_zip
Change-Id: I9267580f644b42b44d43fb0a2674dc5234f127e5
2018-10-02 21:52:50 -07:00
colincross
c54f732470
Merge pull request #221 from colincross/fs_for_soong_zip
Extend pathtools.FileSystem for soong_zip
2018-09-28 14:14:29 -07:00
Colin Cross
3316a5ee54 Add pathtools.Filesystem.Stat for soong_zip
Stat is used by soong_zip.  Add it to the FileSystem interface
and add tests for it.

Test: fs_test.go
Change-Id: I1baa2b27398846a4e55bcf4fa291c62f507a4e9d
2018-09-27 15:54:24 -07:00
Colin Cross
15fbefb87e Add tests for pathtools.FileSystem.Lstat
Test: fs_test.go
Change-Id: I202d639d8db3f0cf11d927cbf9ca5aac073ccc43
2018-09-27 15:54:24 -07:00
Colin Cross
e81b432f09 Add pathtools.FileSystem.Readlink
Readlink is used by soong_zip.  Add it to the FileSystem interface
and add tests for it.

Test: fs_test.go
Change-Id: Ie8ca5cd7cae98a47980a50d2891501fe79fd47a5
2018-09-26 16:54:15 -07:00
Colin Cross
d85b3c78c7 Replace exact errors in tests with errno checking
Only check that functions that return an *os.SyscallError have
the correct errno, not that they have identical test text.

Test: fs_test.go
Change-Id: Iba050cb0474eaf2643858bcca4e52ba770702c2f
2018-09-26 16:54:15 -07:00
Colin Cross
192dbc59c1 Make pathtools.FileSystem.Open return a ReaderAtSeekerCloser
Both *os.File and *bytes.Buffer support all of io.Reader,
io.ReaderAt, io.Seeker and io.Closer.  Return a combo interface
so that soong_zip can use the result in an io.SectionReader.

Test: m checkbuild
Change-Id: I31c3ce35e28c52bae20b536b5905de2f8a3d1478
2018-09-26 16:54:15 -07:00
Colin Cross
daf9de1999 Fix typo in TestMockFs_followSymlinks
c/f was tested twice and d/f was not tested.

Test: fs_test.go
Change-Id: I0fd6ba8ef37a3407f9be98608efed387cca5d7ff
2018-09-26 16:54:15 -07:00
colincross
c955bc8bcc
Merge pull request #220 from colincross/glob_symlinks
Improve glob behavior with symlinks
2018-09-25 15:02:27 -07:00
Colin Cross
e98d0828c8 Add ShouldFollowSymlinks argument to pathtools.Glob
Allow the caller to specify whether symlinks should be followed
(the old behavior) or not.

Test: glob_test.go
Test: fs_test.go
Change-Id: I550dc91b8e6370fb32a9a1cbdcb2edade48bda46
2018-09-24 15:09:32 -07:00
Colin Cross
9e1ff7423b Fix recursive globs through symlinks
filepath.Walk does not walk symlinks to directories, which leads to
inconsitent behavior with following symlinks.  Replace the use of
filepath.Walk in ListDirsRecursive with a helper function that lists
each directory and walks anything for which IsDir reports true, which
includes following symlinks, and then reconstructs the path through
the symlink.

Test: fs_test.go
Test: glob_test.go
Change-Id: Ie4dd0820e9c7c0a124caa65210ce20439a44da16
2018-09-24 15:09:32 -07:00
Colin Cross
c64f26418e Add symlink support to mockFs
Add support for specifying symlinks in mock filesystems to prepare
for glob symlink tests.

This patch leaves incorrect behavior by not walking symlinks in
mockFs.ListDirsRecursive, but it matches what osFs does.

Test: fs_test.go
Change-Id: If87a83c00f21e14696faf890b7b09e88b18e95b9
2018-09-24 15:09:32 -07:00
Colin Cross
d9bea8f909 Add OsFs tests for escaped globs
Test: glob_test.go
Change-Id: I18d5f0c1139c68eafe4993792e8640d2144d49e1
2018-09-21 15:55:06 -07:00
Colin Cross
e3b7ec32c9 Improve error message for globbed dangling symlink
IsDir on a dangling symlink produces ErrNotExist.  Manually
check if the file exists but is a symlink to report a better
error.

Test: m checkbuild
Change-Id: I3181e74002436d74ec35a0923635835e561030dd
2018-09-21 15:55:03 -07:00
colincross
112a732a5a
Merge pull request #218 from colincross/glob_escapes
Fix globs matching files with wildcard characters in the name
2018-09-19 16:45:40 -07:00
colincross
4425194294
Merge pull request #217 from colincross/singleton_panic
Report panic context for SingletonContext.VisitAllModules
2018-09-19 16:45:26 -07:00
Colin Cross
cc1ec0fedc Replace *Escape with *EscapeList
NinjaEscape and ShellEscape operated on lists, which led to
awkward NinjaEscape([]string{s})[0].  Replace NinjaEscape
and ShellEscape with NinjaEscapeList and ShellEscapeList,
and add new NinjaEscape and ShellEscape functions that
operate on a string.

Test: m checkbuild
Change-Id: I283d92cdddc8e0066a300015863a3eab66f77c23
2018-09-19 16:17:10 -07:00
Colin Cross
a470612f97 Fix globs matching files with wildcard characters in the name
Globs that match a file that looks like a glob were causing duplicate
results because the prefix match would then re-match the
filename as a wildcard.  Add escaping to prevent re-matching.

Also add tests for globs on files with wildcards, tests for
? and [a-z] glob matches supported by filepath.Match, and tests
for escaped wildcard characters.

Test: glob_test.go
Change-Id: Id11a754863979bb36cca0dbd18ea2e76dd1470e3
2018-09-19 15:44:40 -07:00
Colin Cross
7f90d170b3 Report panic context for SingletonContext.VisitAllModules
If a panic occurs in SingletonContext.VisitAllModules report the
module that was being visited.

Change-Id: Ia7fc07e2e33e9e3c0297903d3b06b7efb33f0105
2018-09-19 14:03:30 -07:00
Dan Willemsen
0799fad550
Merge pull request #215 from danw/gomod
Add go1.11 go.mod and fix `go vet` issues
2018-07-23 22:17:22 -07:00
Dan Willemsen
734f20c205 Fix issues found by go vet
Change-Id: If3e20a1f394d757554d6a7da5ed4c41fe194f55f
2018-07-21 13:10:25 -07:00
Dan Willemsen
44d336bdb3 Add go.mod for go1.11 module support
This allows using the various go commands without a GOPATH.

Change-Id: I428ad3a4e884b68615e6e73168c5a844bad7c4d4
2018-07-21 13:02:56 -07:00
colincross
46d1070798
Merge pull request #214 from colincross/matchdirs
Support trailing slash in pathtools.Match patterns
2018-07-14 10:12:41 -07:00
Colin Cross
c0c3b0f946 Support trailing slash in pathtools.Match patterns
pathtools.Match was trimming the trailing slash from patterns.  Make
it handle a trailing slash by first confirming that both the pattern
and name either both have or both do not have a trailing slash, and
then trimming it from both.

Bug: 111389216
Test: TestMatch in glob_test.go
Change-Id: I743e00c14d885de5b5a060aa9e2b22c81dc7e09d
2018-07-13 21:20:03 -07:00
Dan Willemsen
f7d50937a2
Merge pull request #213 from danw/bpglob_bootstrap
Run globs during earlier bootstrap phases
2018-07-06 11:35:30 -07:00
Dan Willemsen
ab223a512b Run globs during earlier bootstrap phases
Instead of sometimes re-running minibp/the primary builder during the
next phase, run bpglob earlier to check dependencies.

We've run into issues where the environment is slightly different
between bootstrapping phase and the main build phase. It's also a
problem because our primary builder (Soong) exports information used by
another tool (Kati) that runs in between the bootstrapping phases and
the main phase. When Soong would run in the main phase, it could get out
of sync, and would require the build to be run again.

To do this, add a "subninja" include a build-globs.ninja file to each
build.ninja file. The first time, this will be an empty file, but we'll
always run minibp / the primary builder anyway. When the builder runs,
in addition to writing a dependency file, write out the
build-globs.ninja file with the rules to run bpglob.

Since bpglob may need to be run very early, before it would normally be
built, build it with microfactory.

Change-Id: I89fcd849a8729e892f163d40060ab90b5d4dfa5d
2018-07-06 10:39:38 -07:00
colincross
9cbbb8b91d
Merge pull request #212 from loganchien/stop-mixed-syntax
Emit errors on mixed property syntax
2018-06-25 21:25:59 -07:00
Logan Chien
3deba3df45 Emit errors on mixed property syntax
This commit refines `compat` condition in `parseProperty()` so that a
module definition would either use the new syntax or the old syntax,
but not something in the between, such as

    cc_library {
        name: "bad_example",
        srcs= ["bad.c"],
    }

Test: lunch aosp_arm64-userdebug; make  # runs unit test
Change-Id: If2d3e5d55edccc28d314d99b83b0f54e5c53ac35
2018-06-26 12:20:08 +08:00
colincross
3b3bd2aac0
Merge pull request #211 from colincross/multiple_deps
Prevent duplicate visit calls in WalkDeps
2018-06-21 14:53:13 -07:00
Colin Cross
526e02f0c6 Prevent duplicate visit calls in WalkDeps
WalkDeps was following every possible path through the dependency
tree, which can be enormous.  Modify it to only call visit for
any particular (child, parent) pair once for each direct dependency
by not recursing into child if visitDown returns true but child
has already been visited.

Test: TestWalkDeps, TestWalkDepsDuplicates
Change-Id: Ieef28399bd10e744417cdeb661dfa04fbeb4ec60
2018-06-21 13:41:42 -07:00
colincross
cd0310f225
Merge pull request #210 from colincross/multiple_deps
Allow multiple dependencies on the same module
2018-06-20 15:18:21 -07:00
Colin Cross
9607a9f248 Allow multiple dependencies on the same module
Allow adding multiple dependencies on the same module.  Adds a flag
to Context.walkDeps to determine whether dependencies should be
visited multiple times, and clarifies the behavior of
VisitDepsDepthFirst (continues to visit each module once, even if
there are multiple dependencies on it) and VisitDirectDeps (may
visit a module multiple times).

Test: TestWalkDepsDuplicates, TestVisit
Change-Id: I58afbe90490aca102d242d63e185386e1fe55d73
2018-06-20 14:10:18 -07:00
Dan Willemsen
3a16825a7c
Merge pull request #209 from danw/multiline_indent
Improve indentation for multi-line expressions
2018-05-07 18:11:20 -07:00
Dan Willemsen
d2c8162ca9 Improve indentation for multi-line expressions
When someone used a multiline string:

    cmd: "..." +
         "...",

bpfmt used to print this out as:

    cmd: "..." +
    "...",

This change doesn't do the quote alignment like I see in some of our
user-created instances of this, but it does indent a single level:

    cmd: "..." +
        "...",

Test: unit tests
Test: Compared bpfmt results before and after across AOSP
Change-Id: I61bf790be9d08a187857b2725facf71e8b38e372
2018-05-07 16:15:33 -07:00
Dan Willemsen
1e428e0c05
Merge pull request #208 from danw/microfactory_race
Disable parallel compilation when using race detector
2018-04-30 15:54:15 -07:00
Dan Willemsen
db29636ba2 Disable parallel compilation when using race detector
These options aren't compatible with one another.

Change-Id: I1ec52b8fa8465edd551bcd1c20a9a902a5669e52
2018-04-30 14:44:22 -07:00
colincross
bb58119655
Merge pull request #207 from colincross/nameinterface
Return an error when renaming a module that doesn't exist
2018-04-16 15:23:56 -07:00
Colin Cross
61fa603f92 Return an error when renaming a module that doesn't exist
Misusing Rename was causing a nil pointer derefernece, return
an error if group was not found in s.modules.

Bug: 77922456
Change-Id: I7e7b20b1595d569ae07c755bd29d701e0e5dab78
2018-04-16 14:44:41 -07:00
colincross
148f9f316e
Merge pull request #205 from colincross/goroutine
Reduce maximum goroutine count
2018-04-10 17:28:48 -07:00
colincross
e0d935660b
Merge pull request #206 from colincross/booldefault
Add proptools.BoolDefault and proptools.StringDefault
2018-04-10 17:28:36 -07:00
colincross
4106611b26
Merge pull request #204 from colincross/modify_text
Add Patch and PatchList for making textual changes
2018-04-10 16:54:24 -07:00
Colin Cross
de3ef3a66e Add proptools.BoolDefault and proptools.StringDefault
I get the logic wrong every time I try to make a *bool property
that defaults to true.

Change-Id: Idc409921c3a4baecf611acf348176f261712cb92
2018-04-10 16:51:47 -07:00
Colin Cross
7e72337259 Reduce maximum goroutine count
Running with the data race detector enabled hits a 8192 active
goroutine limit.  Reduce the maximum number of active goroutines
by limiting parallelVisit to 1000 goroutines.  Also rewrite
cloneModule in terms of parallelVisit.

Change-Id: Icdd63648e8e010557b624e12592156490b40adb9
2018-04-10 16:51:29 -07:00