Commit graph

727 commits

Author SHA1 Message Date
Colin Cross
957b39cba5 Add Patch and PatchList for making textual changes
Patch and PatchList provide an API for making changes to substrings
of a Blueprint file by parsing the AST and using the token positions
to perform text replacements.

Test: modify_test.go
Change-Id: Ibb8993221982b54602ba5a05486198fab8d35a67
2018-04-10 16:50:39 -07:00
Colin Cross
fdeaf881f4 Make End() return the position after the node
End() was previously only used to determine if a comment was within
a Node, so it used the expedient definition of the position of the
last token in the node.  In the next patch it will be used for
capturing substrings of the Blueprint file, so make it point to
the character after the last token instead.

Also add tests for it.

Test: parser_test.go
Change-Id: Icaff3915b41e251ef9d0aad5615021bf37406aee
2018-04-10 16:43:51 -07:00
Dan Willemsen
ff2c5b4cc8
Merge pull request #203 from danw/numcpu_cap
Cap the number of cpus for Go compiles
2018-03-02 16:02:06 -08:00
Dan Willemsen
f75f4e9b7b
Merge pull request #202 from danw/go1.10_goroot
Try to make GOROOT relative in Go 1.10
2018-03-02 15:59:17 -08:00
Dan Willemsen
20c343b89e Cap the number of cpus for Go compiles
When passing "-c" to the Go compiler, any time this value changes, we'd
force all of the Go compiles to rebuild. This could trigger a
substantial portion of the tree to rebuild (anything that transitive
depends on a Go helper tool).

We're running into issues when moving output directories between
multiple GCE machines with different core counts, but are otherwise
identical. This could also hit users moving/mounting disks between
machines, though changes to other host tools can make an impact too.

On my 48-core machine, I get a ~15% benefit from going from -c 1 to -c
48, but also ~12% benefit from going from -c 1 to -c 8. So this will
still let us scale somewhat, but prevent rebuilds when transitioning
between machines that are more likely building Android.
2018-03-02 15:32:22 -08:00
Dan Willemsen
a1e6eeeb44 Try to make GOROOT relative in Go 1.10
In Go 1.10, runtime.GOROOT() will attempt to find the current location
of the go binaries, instead of using the GOROOT_FINAL that was baked
into the binaries. This means that GOROOT() will usually return an
absolute path.

We avoid putting absolute paths into the ninja file, since any change to
the paths would then cause all of the actions including it to rebuild.
Since we've got a decent number of build tools in Android using Go now,
this causes us to rebuild a decent portion of the tree.

Instead of passing the GOROOT around manually in a side channel, just
let the Go 1.10 detection do its thing, and always try to turn the
result into a relative path.
2018-03-02 15:29:35 -08:00
Dan Willemsen
774db4a8aa
Merge pull request #201 from danw/go1.10
Support go1.10
2018-02-27 10:50:20 -08:00
Dan Willemsen
e6d45fe39f Support go1.10
Add stubs for the new testDeps interface functions. Also removes testing
for 1.8.

Change-Id: Ice58cca20658d905df9fb87e822d7861abf60976
2018-02-27 01:49:49 -08:00
Dan Willemsen
77dc4de53e
Merge pull request #200 from danw/glob_dirs
Append / to directories in Glob results
2018-02-26 14:10:21 -08:00
Dan Willemsen
b6c90239d6 Append / to directories in Glob results
This makes it easy for users of Glob to detect whether the match is a
file or directory. Doing the check at this level means that the filelist
file used as a dependency will be updated if a directory is replaced
with a file of the same name, or vice versa.

Change-Id: I79ebba39327218bcdcf50b393498306119de9d6c
2018-02-23 17:21:37 -08:00
colincross
cd3b7aed33
Merge pull request #198 from colincross/glob
Fix glob cache conflict when excludes=nil and excludes=[]string{}
2018-02-23 14:23:37 -08:00
colincross
f12def88ce
Merge pull request #199 from colincross/vet
Fix format issues found by go vet
2018-02-23 14:23:17 -08:00
Colin Cross
c1d878159c Fix format issues found by go vet
Test: m checkbuild
Change-Id: I5135e0daecd20abce9a66631f5f55230168613de
2018-02-23 14:05:03 -08:00
Colin Cross
54cb95a53f Fix glob cache conflict when excludes=nil and excludes=[]string{}
Performing the same glob twice, once with excludes=nil and once
with excludes=[]string{} would hit the same entry in the glob
cache (since the glob filename would be the same), but fail
the verifyGlob check because DeepEqual considers []string(nil)
and []string{} to be different.  Use a manual array check
instead.

Test: glob_test.go
Change-Id: If0d4fe80163a871077b7276e1b4a3e888a4a4898
2018-02-23 14:03:56 -08:00
Dan Willemsen
759bb61dd5
Merge pull request #196 from danw/ninja_deps
Store build.ninja deps in .ninja_deps
2017-12-18 12:26:58 -08:00
Dan Willemsen
2d9b59bc20 Store build.ninja deps in .ninja_deps
We're already going to load the .ninja_deps file, so we may as well use
it. These are the only two files in our build that don't.

Test: dump the .ninja_deps file, see the dependencies
2017-12-18 09:14:16 -08:00
colincross
7b8405f9c0
Merge pull request #195 from colincross/docs
Give primary builder more control over automated documentation
2017-12-11 19:27:42 -08:00
colincross
ee8fe641e9
Merge pull request #193 from colincross/glob
Limit length of glob result file names
2017-12-11 16:33:22 -08:00
Colin Cross
28b2843b78 Allow primary builder to stop bootstrap early
Allow the primary builder to stop bootstrap between
ResolveDependencies and PrepareBuildActions so that it can
use the build graph to write customized documentation.

Change-Id: I67ee5e6e438a0c2c3ec844c4c453d223971e60a2
2017-12-11 16:31:41 -08:00
Colin Cross
5eb116a46e Move auto-generated docs to main stage
Move the docs to the main stage and make a phony rule to build
them.  This will allow the primaray builder to choose not to
build them so that it can use its own rule instead.

Change-Id: Ib788f74717ccdcd85c2e5b6a581cc744d439341a
2017-12-11 16:31:41 -08:00
Colin Cross
92639f3fbb Refactor bpdoc
Separate the extraction of documentation-related info from writing
it out so that the primary builder can choose to write it out in
a customized format.

Change-Id: If2c91d6f13b8cea2e9e1cca75d8c2d684ad7f6df
2017-12-11 16:31:41 -08:00
Colin Cross
c581620795 Fix import list in singleton_ctx.go
Otherwise my editor keeps fixing it for me.

Change-Id: I2795a26b9a4f0dd7bf20edd146fc28425a60e89c
2017-12-11 16:31:41 -08:00
Colin Cross
741e14ebb9 Limit length of glob result file names
Glob names were appending every excluded file, which could result
in too many / characters for ninja to handle.  Use a hash if the
result contains too many / characters.

Change-Id: I88920f0596df49639a6c2b4f8af47ea8fdc1ee5a
2017-12-08 10:12:42 -08:00
Jeff
7a88d0db4d
Merge pull request #192 from mathjeff/propagate-syntax-errors
Fix propagating syntax errors to the user
2017-12-05 17:03:28 -08:00
Jeff
6b60fcefa3
Merge pull request #182 from mathjeff/microfactory-cmd-err
Update microfactory compile error message to show the command that fa…
2017-12-05 16:37:10 -08:00
Jeff Gaston
a7e408af0a prevent file=nil panic if syntax error in Blueprints
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
      'build/blueprint:work^^^' 'build/blueprint:work'
Test: put a syntax error in a file and see that the
      reported error reports the location of the violation

Change-Id: Iaeedb91ea8e816cb8e9ee954f21cd6c6bc4afa48
2017-12-05 16:09:02 -08:00
Jeff Gaston
8fd9578a6a have openAndParse use return values rather than channels
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
      'build/blueprint:work^^^' 'build/blueprint:work'

Change-Id: I941dfa76b94178198994a280eb40bcded891eda1
2017-12-05 15:26:57 -08:00
Jeff Gaston
5800d046c1 rename parseOneAsync to openAndParse
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
      'build/blueprint:work^^^' 'build/blueprint:work'

Change-Id: I56e08716057c3746b01b7a5e43f0223ee1f940db
2017-12-05 15:26:57 -08:00
Jeff
14978f8f9d
Merge pull request #190 from mathjeff/visit-order
Have handleOneFile(child) wait for handleOneFile(parent).
2017-12-04 17:41:54 -08:00
Jeff Gaston
0e90759bfe Make ninja file deterministic even with dup module names
Bug: 65683273
Test: ./build/soong/scripts/diff_build_graphs.sh \
      --products=aosp_arm '' ''

Change-Id: I5e45b2309ba4993ba2180b5f9a4785f31e28d503
2017-12-04 17:16:21 -08:00
Jeff Gaston
f23e36690e Disallow bp 'build' includes to reference other dirs
Bug: 65683273
Test: add 'build = ["sub/nope.bp"]' to an Android.bp; notice the error

Change-Id: Ic0f171f283edda074f65a76029e660dfaab2504b
2017-12-04 17:16:21 -08:00
Jeff Gaston
3c8c3346d2 Pass ModulePath to NameInterface
To allow it to validate that the filename equals Android.bp

Bug: 65683273
Test: m -j nothing # which runs tests
Change-Id: I171dddd102590df558053b615d39c75c00b6ac6e
2017-12-04 17:16:21 -08:00
Jeff Gaston
656870fbca Have handleOneFile(child) wait for handleOneFile(parent).
Bug: 65683273
Test: m -j nothing # which runs unit tests

Change-Id: I850e78ebcdbee68637f66d81fc127f19dd28508a
2017-12-04 17:16:21 -08:00
Jeff
94c2179666
Merge pull request #189 from mathjeff/namespaces
Extract module naming into an interface
2017-11-29 12:49:06 -08:00
Jeff Gaston
d70bf75491 Extract module naming into an interface
in facilitate moving name resolution to Soong

Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
      --products=aosp_arm \
      'build/blueprint:work^' 'build/blueprint:work'
      # and see that the only changes were:
      # 1. adding the name_interface.go file
      # 2. changing some line numbers

Change-Id: Ifa7603cca59b3b3d592f2f146fdafe57012bd4b9
2017-11-29 12:01:09 -08:00
colincross
90d258833a
Merge pull request #188 from colincross/match
Export pathtools.Match
2017-11-22 13:07:33 -08:00
Colin Cross
f9c2e8cc43 Export pathtools.Match
Export pathtools.Match so that tools that operate on file lists
(like zip listings) can use the same glob format.

Test: m checkbuild
Change-Id: Ic4b90139c381c57551836590da3ea54826b0227a
2017-11-22 12:51:59 -08:00
Jeff
fb52eb9a1a
Merge pull request #187 from mathjeff/android-bp-tests
Support files named Android.bp in tests
2017-11-21 12:21:31 -08:00
Jeff Gaston
9f63090a4c Support files named Android.bp in tests
Bug: 65683273
Test: m -j nothing # which runs unit tests

Change-Id: I00862cd9673719424a2b18e347c7f9fe84be2857
2017-11-20 17:03:49 -08:00
colincross
7ab9124b27
Merge pull request #183 from colincross/presingleton
Add presingletons
2017-11-07 14:50:05 -08:00
Colin Cross
5f03f11c0a Add presingletons
PreSingletons use the same interface as Singletons, but run after
parsing all the blueprint files but before running any mutators
or GenerateBuildActions.  They can be used to perform global setup,
including generating rules, that may be referenced by later
mutator or generate passes.

Test: m checkbuild
Change-Id: I4b93425a724e739f5b8843262efb8804b3bf3531
2017-11-07 13:31:33 -08:00
Jeff
f393902342
Merge pull request #168 from mathjeff/file-list
File list
2017-11-06 15:06:15 -08:00
Nan-Zhang
e55004478f
Merge pull request #181 from Nan-Zhang/support-integer
Support parsing int64 number in Blueprint file
2017-11-06 10:02:57 -08:00
Dan Willemsen
c09771a5e9
Merge pull request #169 from danw/buildtags
Make microfactory support go build tags
2017-11-03 14:34:49 -07:00
Nan Zhang
f586544ab7 Support parsing int64 in Blueprint file.
Support int64 number instead of int to be more fixed to bit size so
that the underlying arch won't affect overflow cases. Besides,
refection: func (v Value) Int() int64 always cast to int64 no matter the
input is int, int16, int32. Currently we always treat "-" as negative
sign to bind to next value, and "+" as plus operator to add operands
together.
So we allow:
a = 5 + -4 + 5 or a = -4 + 5
But we don't allow:
a = +5 + 4 + -4 since we don't treat "+" as a positive sign, otherwise,
a = 5 + +5 would exist which looks pretty weird. In the future, we may
want fully support number calculator logic eg, "+"/"-" can be
positive/negative sign or operator, and "(" and ")" will be considered
to group expressions with a higher precedence.

int & uint properties within struct keeps unchanged, which is only
allowed when tagged with 'blueprint:mutated'. We only allow *int64
property instead of int64 property within struct since it does't make
sense to do prepending or appending to int64.

Change-Id: I565e046dbd268af3538aee148cd7300037e56523
2017-11-02 22:10:47 -07:00
Jeff Gaston
4334d39a22 Update microfactory compile error message to show the command that failed
Easier to debug than "Failed to compile: main: exit status 2"

Bug: 68770962
Test: sed -i 's/build.FindSources/files := build.FindSources/' build/soong/cmd/soong_ui/main.go \
      && m -j nothing

Change-Id: If78a4fa1c3999e7c658dce072c05d7d3e23b8683
2017-11-02 13:56:32 -07:00
Jeff Gaston
4fc22f66c2 Make test_tree_tests.sh slightly easier to read
Bug: 64363847
Test: ./tests/test_tree_tests.sh
Change-Id: I40c1bcebf176bf4d70d4bc6339f419881013e0a6
2017-10-30 15:00:19 -07:00
Jeff Gaston
3802ba9f18 No longer follow subdirs in Android.bp files
because there some symlinks named Android.bp that point to
files named Android.bp that contain <subdir> entries that
are supposed to be interpreted relative to the original
symlink. It's more convenient to just skip following subdirs.

Bug: 64363847
Test: make -j
Change-Id: Ib15f1d1c7d511191fd841b1a29ceed559dd6cdbb
2017-10-30 15:00:19 -07:00
Jeff Gaston
c3e2844dfe Support for a custom list of Blueprints files to parse
Bug: 64363847
Test: BLUEPRINT_LIST_FILE=out/.module_paths/Android.bp.list minibp

Change-Id: Id7f8cb1ab3a6684b3f8265d77bb32413957f1c93
2017-10-30 15:00:19 -07:00
Jeff Gaston
cb42130440 Make a copy of Blueprint in tests
because the Finder doesn't follow symlinks
Bug: 64363847
Test: tests/test_tree_tests.sh

Change-Id: I2e4115b6d833e5bfcaa1e20bb9c8e1d827e3f43f
2017-10-30 15:00:19 -07:00