Commit graph

750 commits

Author SHA1 Message Date
colincross
bf8c1d31d9
Merge pull request #239 from colincross/go1.12
Support go 1.12
2019-02-28 19:52:33 -08:00
Colin Cross
16b01609ce Run go 1.11 and go 1.12 in travis
Change-Id: If8d53ca83fb97cc6b68e3e5f4025119bb3ea2502
2019-02-28 14:42:04 -08:00
Colin Cross
f7104c0ad6 Support go 1.12
go 1.12 changed the stack trace of a function call during global
variable initialization to contain a function called
"PKG.init.ializers".  Fix callerName to split the package path
and function name on the first "." after the last "/", and look
for functions called "init.ializers" or functions with the
prefix "init."

Change-Id: Ic2190837a8c4cde075f727bd69dd18d517a0ebc0
2019-02-28 14:38:42 -08:00
colincross
284c742a33
Merge pull request #219 from colincross/escape_list
Replace *Escape with *EscapeList
2019-02-28 11:14:02 -08:00
colincross
02c110735d
Merge pull request #238 from allight/rebase-add-java-services
GetDirectDepWithTag needs to check all tags before panicing
2019-02-27 16:13:57 -08:00
Alex Light
142de248fa GetDirectDepWithTag needs to check all tags before panicing
GetDirectDeWithTag would panic if the first possible directDep did not
have the correct tag. It should check to see if any version of that
dependency has the right tag.

Test: atest CtsJdwpTunnelHostTestCases
Bug: 124507633
Bug: 125933724
Change-Id: I00d269130e9f136a93fd30c58b8fd929372d5b37
2019-02-27 15:34:26 -08:00
colincross
8908a0a806
Merge pull request #237 from colincross/singleton
Allow Context to query Singletons
2019-02-26 10:54:08 -08:00
Colin Cross
9226d6c9f5 Allow Context to query Singletons
Add Context.Singletons and Context.SingletonName to allow primary
builders to query Singletons returned by all registered
SingletonFactories.

Change-Id: Iee85643dd47f7472e6bb5ae8004374bd6ea0419e
2019-02-26 10:15:40 -08:00
asmundak
d444506bf6
Merge pull request #235 from asmundak/collapseduplicate
Export CollapseDuplicateProperties, Soong writedocs uses it
2019-02-17 16:36:43 -08:00
colincross
58edc0fa5c
Merge pull request #236 from colincross/examples
Support examples as tests
2019-02-15 16:39:32 -08:00
Colin Cross
463eab7950 Support examples as tests
go test supports running examples as tests and verifying their
output, add support for running them to gotestmain.

Change-Id: If51abc64467d4701195cefc6db513f4d008794b5
2019-02-15 15:13:37 -08:00
Sasha Smundak
797563bec4 Export CollapseDuplicateProperties, Soong writedocs uses it 2019-02-14 10:58:48 -08:00
Jaewoong Jung
33cfa1c98a
Merge pull request #234 from gyias/bpdoc
bpdoc improvements
2019-02-12 11:32:37 -08:00
Jaewoong Jung
781f6b2896 bpdoc improvements
1. Extract module type documentation.
2. Support primary builder customization of factory function to use for
documentation for each module type.
3. Change the ModuleType list order so that they are grouped by package.

This is basically minor refactoring + readability improvement done on
top of https://github.com/google/blueprint/pull/232.

Change-Id: If7413e5ac23486b85f18d02fb3ba288a38730c32
2019-02-08 15:48:27 -08:00
colincross
3eeabc7991
Merge pull request #231 from colincross/pprof
Performance improvements
2019-01-24 10:32:25 -08:00
Colin Cross
937efab2e6 Use sync.Map for type field cache
The type field cache was using an atomic pointer to an immutable
map, which required copying the entire map each time a cache
entry was added.  Profiling showed that this was a significant
hot spot.  Use sync.Map instead.

Change-Id: Ie7c779c5e9e2be1cd530747d74025dcfd206763a
2019-01-23 13:26:42 -08:00
Colin Cross
de7afaaf74 Write ninja file directly to the output file
Writing the ninja file to a byte buffer causes a significant amount
of time to be spent in memmove when growing the byte slice.  Write
the file directly to disk instead.

This also fixes some unhandled error warnings, which become more
likely when doing disk IO instead of byte buffer writes.

Change-Id: I5094e4c45cab4012713037f60c5a4fb00718f92e
2019-01-23 13:26:42 -08:00
Colin Cross
3a8c025648 Add pprof labels
Add pprof labels to the larger stages of blueprint, including
labels for each mutator.

Change-Id: Ie91daff51811187c1b702a775c05d0b32f7170fc
2019-01-23 13:26:42 -08:00
Dan Willemsen
716137ec38
Merge pull request #230 from smore-lore/master
bpmodify: add argument if it is missing
2018-12-26 16:43:54 -08:00
Steven Moreland
b40aaadbca bpmodify: add argument if it is missing
before, adding to a list which had no entries would do nothing (and not
throw any errors). Now it will create the list and add a single element
as expected.

Change-Id: I8c48a42303f7d9b3741868ad86097e2071ec434a
2018-12-17 18:16:16 -08:00
Dan Willemsen
f8565113e1
Merge pull request #228 from danw/empty_ninja_file
Add --empty-ninja-file for test usecases
2018-10-26 10:00:37 -07:00
Dan Willemsen
dac90d33ca Add --empty-ninja-file for test usecases
In cases that we want to run blueprint-based builds in many
configurations to verify all of the logic works without errors, but
don't care about running the final ninja file, writing it out only
wastes time and disk space. So add a --empty-ninja-file option that
writes out an empty ninja file instead.

Our specific use case (Soong's build_test / multiproduct_kati) runs
Soong several hundred times for different configurations, and the ninja
files are around 1GB, which leads to several hundred gigabytes of disk
writes (and persistent use during incremental generation).

Change-Id: I0198dfb2f744ce22284c05d5214dac2ab5dc9700
2018-10-26 09:57:24 -07:00
colincross
19099e68bb
Merge pull request #227 from colincross/testdeps
Add link dependency on test package archive
2018-10-24 15:04:22 -07:00
Colin Cross
1be9abc109 Add link dependency on test package archive
The test package archive was being picked up through "-L" + testRoot,
but did not have a dependency on it.  If the *_test.go file was
changed but not the implementation then the tests would not be
rerun.

Test: Add t.Fatal() to a *_test.go file and rebuild.
Change-Id: I32b8c7a0fa1d5b76b1242646887fe937e396e318
2018-10-24 13:13:13 -07:00
Jaewoong Jung
ccc3494088 Make sure all modules have a name. (#226)
Throw an error if run into a module without a name when generating
contexts.

Test: context_test.go
Change-Id: I3976d86d1f15b8ac10a7a38aa42ae277740a8f3b
2018-10-11 13:01:05 -07:00
colincross
d127ba077b
Merge pull request #225 from colincross/test_link_deps
Add missing dependencies when linking tests
2018-10-05 11:28:06 -07:00
Colin Cross
0e58dc0176 Add missing dependencies when linking tests
Linking tests needs the same dependencies that linking binaries
got in https://github.com/google/blueprint/pull/222.

Test: m checkbuild
Change-Id: I33330f3184b8c0fd2bc20b48736c20d6edeaea68
2018-10-05 11:12:53 -07:00
colincross
3f13f07565
Merge pull request #224 from colincross/slice_of_structs
Allow mutated property structs to contain slices of non-strings
2018-10-03 20:25:27 -07:00
colincross
888fe0dd3b
Merge pull request #222 from colincross/go_restat
Enable restat for go binaries
2018-10-03 14:30:12 -07:00
Colin Cross
12392ca0f1 Allow mutated property structs to contain slices of non-strings
Property structs can now contain slices of structs and other
non-strings as long as they are tagged with `blueprint:"mutated"`.

Test: clone_test.go
Test: unpack_test.go
Change-Id: Ib77348dc6e7314a24f17caba10040f7d3ac54e54
2018-10-03 14:29:12 -07:00
Colin Cross
6c92af937e Enable restat for go binaries
Many Android build tools are written in go, and changes to Soong or
Blueprint can cause them to rebuild.  Almost everything in an
Android build is downstream of at least one of these tools, so
they all rebuild too.

Go binaries are static, so their contents will always change if any
of their dependencies change.  Only update output files of go compile
and go link if the contents change, and enable restat for the rules.

Test: m soong_zip && m soong_zip
Test: m soong_zip && touch build/soong/zip/cmd/main.go && m soong_zip
Change-Id: I9267580f644b42b44d43fb0a2674dc5234f127e5
2018-10-02 21:52:50 -07:00
colincross
c54f732470
Merge pull request #221 from colincross/fs_for_soong_zip
Extend pathtools.FileSystem for soong_zip
2018-09-28 14:14:29 -07:00
Colin Cross
3316a5ee54 Add pathtools.Filesystem.Stat for soong_zip
Stat is used by soong_zip.  Add it to the FileSystem interface
and add tests for it.

Test: fs_test.go
Change-Id: I1baa2b27398846a4e55bcf4fa291c62f507a4e9d
2018-09-27 15:54:24 -07:00
Colin Cross
15fbefb87e Add tests for pathtools.FileSystem.Lstat
Test: fs_test.go
Change-Id: I202d639d8db3f0cf11d927cbf9ca5aac073ccc43
2018-09-27 15:54:24 -07:00
Colin Cross
e81b432f09 Add pathtools.FileSystem.Readlink
Readlink is used by soong_zip.  Add it to the FileSystem interface
and add tests for it.

Test: fs_test.go
Change-Id: Ie8ca5cd7cae98a47980a50d2891501fe79fd47a5
2018-09-26 16:54:15 -07:00
Colin Cross
d85b3c78c7 Replace exact errors in tests with errno checking
Only check that functions that return an *os.SyscallError have
the correct errno, not that they have identical test text.

Test: fs_test.go
Change-Id: Iba050cb0474eaf2643858bcca4e52ba770702c2f
2018-09-26 16:54:15 -07:00
Colin Cross
192dbc59c1 Make pathtools.FileSystem.Open return a ReaderAtSeekerCloser
Both *os.File and *bytes.Buffer support all of io.Reader,
io.ReaderAt, io.Seeker and io.Closer.  Return a combo interface
so that soong_zip can use the result in an io.SectionReader.

Test: m checkbuild
Change-Id: I31c3ce35e28c52bae20b536b5905de2f8a3d1478
2018-09-26 16:54:15 -07:00
Colin Cross
daf9de1999 Fix typo in TestMockFs_followSymlinks
c/f was tested twice and d/f was not tested.

Test: fs_test.go
Change-Id: I0fd6ba8ef37a3407f9be98608efed387cca5d7ff
2018-09-26 16:54:15 -07:00
colincross
c955bc8bcc
Merge pull request #220 from colincross/glob_symlinks
Improve glob behavior with symlinks
2018-09-25 15:02:27 -07:00
Colin Cross
e98d0828c8 Add ShouldFollowSymlinks argument to pathtools.Glob
Allow the caller to specify whether symlinks should be followed
(the old behavior) or not.

Test: glob_test.go
Test: fs_test.go
Change-Id: I550dc91b8e6370fb32a9a1cbdcb2edade48bda46
2018-09-24 15:09:32 -07:00
Colin Cross
9e1ff7423b Fix recursive globs through symlinks
filepath.Walk does not walk symlinks to directories, which leads to
inconsitent behavior with following symlinks.  Replace the use of
filepath.Walk in ListDirsRecursive with a helper function that lists
each directory and walks anything for which IsDir reports true, which
includes following symlinks, and then reconstructs the path through
the symlink.

Test: fs_test.go
Test: glob_test.go
Change-Id: Ie4dd0820e9c7c0a124caa65210ce20439a44da16
2018-09-24 15:09:32 -07:00
Colin Cross
c64f26418e Add symlink support to mockFs
Add support for specifying symlinks in mock filesystems to prepare
for glob symlink tests.

This patch leaves incorrect behavior by not walking symlinks in
mockFs.ListDirsRecursive, but it matches what osFs does.

Test: fs_test.go
Change-Id: If87a83c00f21e14696faf890b7b09e88b18e95b9
2018-09-24 15:09:32 -07:00
Colin Cross
d9bea8f909 Add OsFs tests for escaped globs
Test: glob_test.go
Change-Id: I18d5f0c1139c68eafe4993792e8640d2144d49e1
2018-09-21 15:55:06 -07:00
Colin Cross
e3b7ec32c9 Improve error message for globbed dangling symlink
IsDir on a dangling symlink produces ErrNotExist.  Manually
check if the file exists but is a symlink to report a better
error.

Test: m checkbuild
Change-Id: I3181e74002436d74ec35a0923635835e561030dd
2018-09-21 15:55:03 -07:00
colincross
112a732a5a
Merge pull request #218 from colincross/glob_escapes
Fix globs matching files with wildcard characters in the name
2018-09-19 16:45:40 -07:00
colincross
4425194294
Merge pull request #217 from colincross/singleton_panic
Report panic context for SingletonContext.VisitAllModules
2018-09-19 16:45:26 -07:00
Colin Cross
cc1ec0fedc Replace *Escape with *EscapeList
NinjaEscape and ShellEscape operated on lists, which led to
awkward NinjaEscape([]string{s})[0].  Replace NinjaEscape
and ShellEscape with NinjaEscapeList and ShellEscapeList,
and add new NinjaEscape and ShellEscape functions that
operate on a string.

Test: m checkbuild
Change-Id: I283d92cdddc8e0066a300015863a3eab66f77c23
2018-09-19 16:17:10 -07:00
Colin Cross
a470612f97 Fix globs matching files with wildcard characters in the name
Globs that match a file that looks like a glob were causing duplicate
results because the prefix match would then re-match the
filename as a wildcard.  Add escaping to prevent re-matching.

Also add tests for globs on files with wildcards, tests for
? and [a-z] glob matches supported by filepath.Match, and tests
for escaped wildcard characters.

Test: glob_test.go
Change-Id: Id11a754863979bb36cca0dbd18ea2e76dd1470e3
2018-09-19 15:44:40 -07:00
Colin Cross
7f90d170b3 Report panic context for SingletonContext.VisitAllModules
If a panic occurs in SingletonContext.VisitAllModules report the
module that was being visited.

Change-Id: Ia7fc07e2e33e9e3c0297903d3b06b7efb33f0105
2018-09-19 14:03:30 -07:00
Dan Willemsen
0799fad550
Merge pull request #215 from danw/gomod
Add go1.11 go.mod and fix `go vet` issues
2018-07-23 22:17:22 -07:00