Commit graph

195 commits

Author SHA1 Message Date
Spandan Das
3a495910ad Delete creation of empty glob file
An empty glob file would be created inside the source tree if
soong_build is invoked directly (instead of via soong_ui). This logic is
now moved to soong/ui/build to prevent this behavior

Bug: 187194795
Test: Ran the following command locally for the target
art-target-arm:git_master-art
```
. ./build/envsetup.sh && lunch armv8-eng &&
art/tools/buildbot-build.sh --target
```
Before change glob file is created in $TOP/.bootstrap
After change glob file is created in $TOP/out/soong/.bootstrap

Change-Id: I6745996c940d790d1f1f387ca08aafc6a8a669df
2021-06-15 17:57:03 +00:00
Dan Willemsen
0f0e304d08 Add blueprint_go_packages phony to build all go packages
We have `blueprint_tools` that can build all Go binaries (and run their
associates tests). But if a package is not (yet) used, it won't be
included there. This makes testing the build of new Go packages
difficult.

Test: m blueprint_go_packages
Change-Id: I7a67cf2593fedcf499d40d82c266d3b0d7ebbef2
2021-05-24 11:36:37 -07:00
Colin Cross
9021eef07b Merge "Speed up globs with sharding" 2021-04-20 16:16:40 +00:00
Lukács T. Berki
d04dcc89f4 Merge "Remove the extraNinjaFileDeps arg of RunBlueprint." 2021-04-19 06:46:52 +00:00
Lukács T. Berki
8308aab95e Merge "Use the correct build dir." 2021-04-19 06:46:16 +00:00
Colin Cross
2523698c12 Speed up globs with sharding
There are a few cases that force all globs to be rerun at the beginning
of the build (changes to bpglob or dependencies, second build after a
clean build).  The number of globs has gotten high enough that rerunning
them all can have significant overhead to start bpglob for each one.

Replace the per-glob bpglob invocations with sharded invocations using
1024 hash buckets.

Bug: 159845846
Test: glob_test.go
Test: m nothing && m nothing
Test: build/soong/bootstrap_test.sh
Change-Id: Ife1f7a03c8f6b25d1be01531425d8dc2c76d1ea0
2021-04-15 11:04:11 -07:00
Colin Cross
45222ec3ca Merge "Version bpglob command line arguments" 2021-04-15 16:42:08 +00:00
Lukacs T. Berki
f646afb52f Use the correct build dir.
Args.BuildDir was accidentally hard-wired to "." in one of my previous
changes, and since Blueprint is not really independent of Soong anymore,
it's probably better to fix it in a way that doesn't involve putting
back the almost-but-not-completely redundant "-b" command line argument
to the primary builder invocation.

Test: Presubmits.
Change-Id: I76d064a99be16cd812cb47a6c448fe44b51893dc
2021-04-15 15:46:35 +02:00
Lukacs T. Berki
f4d43ac0d7 Remove the extraNinjaFileDeps arg of RunBlueprint.
The only thing that was done with it was to append it to the list of
dependencies returned, which can just as well be done by the caller.

Test: Presubmits.
Change-Id: Ibedf024de411a5881f19ed0c137a91b0898f1269
2021-04-15 15:05:26 +02:00
Colin Cross
7e6f6b7932 Version bpglob command line arguments
bpglob is executed through the rules in build-globs.ninja to determine
whether soong_build needs to rerun.  That means when the arguments
accepted by bpglob change it will be called with the old arguments,
then soong_build will rerun and update build-globs.ninja with the new
arguments.

To avoid having to maintain backwards compatibility with old arguments
across the transition, a version argument is used to detect the
transition in order to stop parsing arguments, touch the output file
and exit immediately.  The version number in
pathtools.BPGlobArgumentVersion should be manually incremented when
the bpglob argument format changes.

If the version argument is not passed then a version mismatch is assumed.

Bug: 159845846
Test: build/soong/tests/bootstrap_test.sh
Change-Id: Id6742c7acc51af8d0d224d51e47bddea78e6e363
2021-04-14 09:28:22 -07:00
Lukacs T. Berki
98e0efb9a4 Make Blueprint better callable from Go:
- Make the parsed command line arguments accessible from other packages
- Separate writing the depfile from writing the Ninja file
- Extract a method to compute the name of the file list file for globs

Test: Presubmits.
Change-Id: I4d0daf0f3776a9336da305be0108ebf70a314776
2021-04-14 13:47:52 +02:00
Lukacs T. Berki
78df853e07 Make it possible to call the primary builder more than once.
This is accomplished by plumbing PrimaryBuilderInvocation in from
bootstrap.Args.

The confusing removal of "primaryBuilderExtraFlags" from documentation
generation is because it used to (erroneously) refer to the func defined
in command.go and its removal apparently doesn't break anything. I
suspect that this code path is dead within Android, but we'll need to
wait until May to remove it.

Test: Presubmits.
Change-Id: I8763579b82bd79d5ee0f233edfd48327282e401c
2021-04-14 10:28:54 +02:00
Colin Cross
5ce0d78363 Merge "Make Glob return a GlobResult" 2021-04-13 17:01:39 +00:00
Lukacs T. Berki
77ef79b7c4 Move primary builder command line computation.
This makes it possible to invoke the primary builder more than once in a
single Ninja file.

This required adding arguments that were hard-wired in the build.ninja
rule to extraArgs and adding a "set of primary builder invocations"
member to bootstrap.Config . The only command line arguments that remain
hard-wired are those which are expected to be relevant to every
invocation of the primary builder.

As a welcome side effect, I was able to remove a number of random
environment variables that were so far special-cased ($SOONG_DELVE,
$SOONG_DELVE_PATH and $SOONG_OUTDIR). I was also able to move writing
the empty stub ninja glob file to command.go from the bowels of the
Blueprint machinery.

In theory, $TOP and $BUILDER could be removed, too, but this would
require hard-coding the value of $TOP into build.ninja and I don't know
what would break if I did that (it's okay to hard-wired $SOONG_DELVE and
$SOONG_DELVE_PATH because those are only used for debugging and
$SOONG_OUTDIR turned out to be superfluous)

Test: Presubmits.
Change-Id: Idbfd9976c4b270bc3e5a8926c8c760a8534596cf
2021-04-12 14:30:47 +02:00
Colin Cross
67c99255cc Make Glob return a GlobResult
Replace the individual matches and deps return values with a GlobResult
struct.  Use the GlobResult to create the file list file in GlobWithDepFile
(used by bpglob) and in the glob singleton, as the generated files must
match exactly so that soong_build is not rerun.

Bug: 159845846
Test: glob_test.go
Change-Id: I2159cc9d85f388073198eac7456e5bf43e813096
2021-04-08 11:03:17 -07:00
Lukacs T. Berki
a9ec69433d Add an accessor function for the output Ninja file.
Test: Presubmits.
Change-Id: I505dfbbf7f87c6ea6bce71f7ae4ef8ca750d066d
2021-03-24 10:57:13 +01:00
Lukacs T. Berki
a4c074bc25 Add comment about what -N -l does.
This was added in aosp/1642084 and I forgot to address a comment.

Test: "m nothing".
Change-Id: I0fd2f2e8eeaa5b316cf19c3b426f708f2587e60f
2021-03-18 10:03:39 +01:00
Lukacs T. Berki
07a91f072c Add a flag to compile Go code for debugging.
This makes it possible to debug the primary builder without manually
adding the respective flags to the compilation command lines.

Test: "m nothing", "m nothing" with debugging, then "m nothing" again
Change-Id: I7d403af48528a94e5e0033d9a5c1fa9981100cfd
2021-03-17 15:01:51 +01:00
Lukacs T. Berki
b3b9cb6ccc Remove the GeneratingPrimaryBuilder() method.
It's replaced with a flag in bootstrap.Args. It's a little bit ugly that
bootstrap.Main also gained a new Boolean flag in the process. The plan
is to delete that method in favor of directly calling RunBlueprint().

Test: "m nothing" (presubmits take a full workday these days)
Change-Id: Iae3421ae517a90570259b6278f18c3614bb1bb73
2021-03-17 14:08:47 +01:00
Lukacs T. Berki
7ea1c168fe Make it possible to call Blueprint from Go.
In addition to the command line interface, a new struct is created that
holds all the information Blueprint needs to do its job.

Test: Presubmits.
Change-Id: I0d6c924e09e069b3c1ae2000722efb52fd561e79
2021-03-17 08:33:21 +01:00
Lukacs T. Berki
5353744f1d Remove global variables from Blueprint.
The end goal of this exercise is to remove all mutable global state so
that multiple Blueprint instances can be run in the same process.

This is done by plumbing the necessary data through the configuration.

Test: Presubmit.
Change-Id: I040a461ae17705258b882b3a44740a4dd5027a2c
2021-03-17 08:06:38 +01:00
Lukács T. Berki
51868f1577 Merge "Fix quoting issues when generating build.ninja ." 2021-03-10 07:03:48 +00:00
Lukacs T. Berki
0bd3de33fd Fix quoting issues when generating build.ninja .
Some of these were introduced in aosp/1610784 and some others were
present since a good while.

Test: Manual: "m nothing"
Change-Id: I7d5cb56ca2d87cb439354fd1b298c3b8d97d665d
2021-03-09 09:32:54 +01:00
Lukacs T. Berki
7d2e60ece7 Use stageDir() to determine the install directory.
In particular, for Go binary modules.

Test: Manual ("m nothing")
Change-Id: I98d184a00f1a48bc68e4b03747331fd449e37de1
2021-03-08 16:48:38 +01:00
Lukacs T. Berki
122b3ee153 Make debugging easier:
- Add the -N -l command line arguments to Microfactory
- Pass the the Delve arguments to the primary builder on the command line

Test: Manual.

Change-Id: I4034f2d48b3e40d9863529053715183dc3dce1f5
2021-03-03 09:15:04 +01:00
Lukacs T. Berki
f802ffcd12 cd / before generating build.ninja .
This is so that soong_build doesn't know the current directory, thereby
avoiding one way to leak information.

Test: Manual.
Change-Id: I976a8663b11d99c5382726487102df10043c5a61
2021-03-03 09:14:22 +01:00
Colin Cross
c8b9e55289 Increase ninja file writer buffer
Increase the ninja file writer buffer from the default 4k to 16MB.

Test: m checkbuild
Change-Id: Ieb2c82218517b98469ef93f1ea4dd04b5651f7d1
2021-01-21 22:02:30 -08:00
Colin Cross
0335e0900d Use io.StringWriter in ninjaWriter
ninjaWriter repeatedly called io.WriteString() on its writer, which
does a type assertion every time.  Replace its io.Writer with an
io.StringWriter and call WriteString on it directly.

Test: ninja_writer_test.go
Change-Id: Ie073d996a319190242bf6a00af07a13a60d078b5
2021-01-21 22:02:30 -08:00
Colin Cross
7ceeeaf364 Remove unnecessary depFile parameter from GlobFile
GlobRule uses $out.d for the dep file, it doesn't need an extra
parameter.

Test: glob_test.go
Change-Id: Ie753a119c3602c325450909f0240f6eeef9ef243
2020-11-10 12:24:17 -08:00
Chris Parsons
5e834261a9 Create early-exit hook StopBeforeWriteNinja
This allows exiting bootstrap directly before writing ninja files.
This facilitates shorter runtime on integrations which do processing
which do not require ninja file output.

Test: Manually verified on Soong integration use case which involves
  running bootstrap twice in a single program; stopping before ninja
  output reduces runtime by ~20s, or ~11%.
2020-11-05 11:17:32 -05:00
Liz Kammer
764a771917
Identify the type of the list for bpdocs (#326)
* Identify the type of the list for bpdocs

Test: go test bpdoc tests
Test: m soong_docs
Change-Id: I6a4a916e1f72b3fc702da90c32a2eddca70b3bac
2020-11-04 17:20:11 -08:00
Liz Kammer
2068e08a2c Correct bug in generating anonymous nested props
Some anonymous nested properties are missing from property structs,
since setting the property to anonymous is to allow future filtering,
there is no issue if we cannot find the struct.

test: go bpdoc tests
test: m soong_docs
2020-10-05 14:55:16 -07:00
Liz Kammer
5c9fe3834d Add docs for nested and embedded structs
Test: m soong_docs
Test: go bpdoc tests
Change-Id: I99f15405e1a7d4a819f6fb20fda22372afe253e1
2020-09-25 13:20:23 -07:00
Colin Cross
d2458a23bf Enable variants of bootstrap modules
The primary builder may want to create variants of bootstrap
modules if they need to fit in to the primary builder's dependency
graph.  Enable arbitrary variants of bootstrap modules by only
running the module's actions on the primary variant and then
copying the result to any other variants that exist.

Test: m checkbuild
Change-Id: I24b97771bb11faeacab4079ed8cf69aef59da140
2020-09-09 14:32:42 -07:00
Liz Kammer
9810dcd4b9
Update bpdocs to filter nested properties by tags (#312)
* Update bpdocs to filter nested properties by tag

Prior to this change nested properties tagged with `blueprint:"mutated"`
are erroneously included in documentation, this corrects that behavior
and adds testing to verify.

Tested: go test bpdocs tests
Change-Id: I822c9a98276634d2f584d8709e83003824cdffd5
2020-07-28 09:30:46 -07:00
Colin Cross
0cdec99c81 Add flag to use validations for tests
Using validations for tests ensures the tests run without blocking
the critical path.

Change-Id: Icb21a52e96f70d815f7df86882351c13f5575cf5
2020-07-09 14:28:20 -07:00
Dan Willemsen
55252655d2 Stop writing depfiles from Glob singleton
This makes a minimal difference on my AOSP tree (which is on a fast NVME
drive), but it saves ~10-15% of the time in our primary builder
(soong_build) on larger trees on slower drives.

We're always going to run the glob tool at least once outside of the
primary builder, and ninja only loads depfiles created after running the
tool, so this should be a no-op change.

We still need to write the list file (if it has changed), since that's
the file we're adding as a dependency of the primary builder, so the
effective timestamp must be earlier than the build.ninja timestamp.

Change-Id: I872b7581da50e7f2089daa7e248ca05b6703f019
2020-06-24 14:23:22 -07:00
c-parsons
fe2fa04c30
Export ModuleListFile in bootstrap (with fix) (#304)
* Export ModuleListFile in bootstrap (with fix)

* Run gofmt on command.go
2020-06-15 09:03:27 -07:00
Colin Cross
bae4de9f16 Revert "Merge pull request #301 from c-parsons/modulelistfile"
This reverts commit 1160e9826f, reversing
changes made to 3a153ecc7f.
2020-06-11 16:29:43 -07:00
Colin Cross
f89751462c Revert "Always emit rules for tests and add phony to run them"
This reverts commit 63085f9e7c.

It breaks the blueprint tests, and we aren't use it yet, so revert
it for now.

Change-Id: I1ac81371342285e8d57baf15fe3a223d4140c6cf
2020-06-11 16:23:10 -07:00
c-parsons
8690f8f196
Change bootstrap.moduleListFile to be exported 2020-05-27 21:31:07 -04:00
Colin Cross
63085f9e7c Always emit rules for tests and add phony to run them
Emit the rules to build blueprint tests even if runGoTests is not
set, and add a phony rule "blueprint_tests" to run them.  This will
allow Soong to stop running the tests at the beginning of every build
but still run them as part of checkbuild or with a manual
`m blueprint_tests`.

Bug: 156428456
Test: m
Test: m blueprint_tests
Change-Id: If293a0757766d3046e78bf230a1825f15adc68fd
2020-05-12 19:20:26 -07:00
Dan Willemsen
6c6c103612 Actually write an empty ninja file with --empty-ninja-file
Otherwise we'll re-run the primary builder every time this is used. This
triggered my new ninja checks that verify that all commands write their
outputs.

Change-Id: Ic92d1ea57203b953da8025a170b24e6420c5436c
2020-04-19 12:50:48 -07:00
Colin Cross
571f77a60d Remove blueprint:"filter(*)" tag support
The filter tag is unused, replaced with FilterPropertyStruct to
generate a new type at runtime that only contains the filtered
fields.

Test: unpack_test.go
Change-Id: Id91cf99290832094d05426f3263279836f0fea73
2020-01-21 11:49:27 -08:00
Colin Cross
67d0cbed90 Fix missing glob filelists when sandboxed
Sandboxing the primary builder caused the glob filelists to not
be written because they were using a relative path, causing
primary builder reruns on the second build.

Also report errors when writing the filelist files.

Test: m checkbuild
Change-Id: Id1706560d04c85f00f829cfb714967bb8600626f
2020-01-16 10:57:08 -08:00
Colin Cross
1cda3fd3e4 Fix primary builder reruns caused by globs
Globs produce a filelist using restat rules to avoid primary builder
reruns.  On the first execution of the primary builder the filelists
are written directly by the primary builder, so that the second
execution can restat them all but avoid rerunning the primary builder.
This wasn't working because the glob filelists were written after
the build.ninja file, but the build.ninja file depends on the
filelists.  Switching the order avoids the extra rerun.

Test: m nothing && m nothing
Change-Id: Ia8e0924598220d4ff34235907a8de2e3e03632df
2020-01-16 10:38:46 -08:00
Colin Cross
c5fa50e057 Allow primary builder to change working directory
Bug: 146437378
Test: pathtools/fs_test.go
Change-Id: I513ceb9b8b0b4f18223bc34ecad9846fe220709b
2020-01-08 15:54:58 -08:00
Colin Cross
f27c5e470b Move unpackProperties to proptools and export it
Test: unpack_test.go
Change-Id: I145369323bd7d5003a261c13dfb8ed31d0be51b5
2020-01-02 20:32:51 -08:00
Colin Cross
da70fd0b84 Move LoadHooks from Soong to Blueprint
Move LoadHooks from Blueprint and run them during ParseBlueprintsFiles.
This will allow them to add scoped module types that are visible to the
rest of the Blueprints file.  Requires passing the config object to
ParseBlueprintsFiles.

Test: all blueprint tests
Change-Id: Ia2a2c9a0223d5458bfd48bd22ebed0fdbd0156c6
2020-01-02 20:32:51 -08:00
Colin Cross
16fec72dfc Move outputs of bootstrap modules in main stage to separate directory
Don't write to $buildDir/.bootstrap in the ninja file generated for
the main stage.

Test: examine main stage build.ninja file
Change-Id: I9c6183c9a434ff11d44cdd4f9844eae192ebfe03
2019-10-17 13:42:34 -07:00
Colin Cross
a2bc585afe Fix compiling binaries with go 1.13
The go compile command in go 1.13 needs -p main instead of
-p $pkgPath when compiling binaries.  This also works for older
go releases.

Also turn on travis for go 1.13.

Change-Id: I45ce185aa44cfa01a7933cc6e52290a85c851641
2019-09-06 14:25:28 -07:00
Colin Cross
0e112a8b97 Revert "Put the primary builder into the console pool"
This reverts commit 8ea996f61d.

Having the primary builder hinders automatic error reporting
by separating the error message from the failure status.
2019-07-18 13:53:43 -07:00
Colin Cross
8ea996f61d Put the primary builder into the console pool
The primary builder should always be running on its own with no
other processes running in parallel, so put it in the console
pool so that it can print to stdout without buffering.

Bug: 80165685
Test: build with print in Soong
Change-Id: If34ecdb5fa18de7e47c4cd6965d551c504850176
2019-06-19 14:39:41 -07:00
Jaewoong Jung
bd0f6c3488 Skip mutated struct properties in bpdoc.
This change fixes a bug where the `blueprint:mutated` tag was ignored
when used for struct or interface properties. It also fixes a dormant
ExcludeByTag func bug.

Test: m soong_docs, properties_test.go, bpdoc_test.go
Change-Id: I926de2aa203ec552ced897174e9b78e817701a7d
2019-06-05 16:52:47 -07:00
Colin Cross
c708e1c9e3 Avoid exiting with errors in bpglob
bpglob is only used as a helper to check if the primary builder
needs to rerun due to the results of a glob changing.  A recent
change to glob support in pathtools made a glob format that was
accidentally previously accepted into an error.  If the bad
syntax was used in the most recent primary builder run, and
then an incremental build is performed after picking up the
change that made the syntax invalid, then bpglob will attempt
to rerun before the primary builder, see the now-invalid syntax,
and fail.  This will prevent the primary builder from rerunning,
which would have updated the bpglob rule with a corrected glob
syntax (or failed in the primary builder if the Blueprints file
still had the invalid glob syntax).

Avoid exiting with an error in bpglob.  Instead, write the
error to the output file along with a timestamp so that it is
always dirty, forcing the primary builder to rerun.

Bug: 129411151
Test: m nothing
Change-Id: Ib680570c33662f3c0f1f72425d60a963ed841ba6
2019-05-31 15:38:12 -07:00
Jaewoong Jung
8bc6bf1090 bpdoc preformatted text improvments
1. Apply preformatted text (<pre>...</pre>) handling logic to module
type text too. It used to be applied to property texts only.
2. Improve <pre> handling logic itself for better readability.
- Insert a blank line before <pre>.
- Prevent from ending <pre> blocks prematurely by checking if an
unindented line isn't just a blank line between indented lines.

Change-Id: Id40fa668d4c6781caf7ed140b2f40784cdeb8c35
2019-03-11 11:16:31 -07:00
Sasha Smundak
797563bec4 Export CollapseDuplicateProperties, Soong writedocs uses it 2019-02-14 10:58:48 -08:00
Jaewoong Jung
781f6b2896 bpdoc improvements
1. Extract module type documentation.
2. Support primary builder customization of factory function to use for
documentation for each module type.
3. Change the ModuleType list order so that they are grouped by package.

This is basically minor refactoring + readability improvement done on
top of https://github.com/google/blueprint/pull/232.

Change-Id: If7413e5ac23486b85f18d02fb3ba288a38730c32
2019-02-08 15:48:27 -08:00
Colin Cross
de7afaaf74 Write ninja file directly to the output file
Writing the ninja file to a byte buffer causes a significant amount
of time to be spent in memmove when growing the byte slice.  Write
the file directly to disk instead.

This also fixes some unhandled error warnings, which become more
likely when doing disk IO instead of byte buffer writes.

Change-Id: I5094e4c45cab4012713037f60c5a4fb00718f92e
2019-01-23 13:26:42 -08:00
Dan Willemsen
dac90d33ca Add --empty-ninja-file for test usecases
In cases that we want to run blueprint-based builds in many
configurations to verify all of the logic works without errors, but
don't care about running the final ninja file, writing it out only
wastes time and disk space. So add a --empty-ninja-file option that
writes out an empty ninja file instead.

Our specific use case (Soong's build_test / multiproduct_kati) runs
Soong several hundred times for different configurations, and the ninja
files are around 1GB, which leads to several hundred gigabytes of disk
writes (and persistent use during incremental generation).

Change-Id: I0198dfb2f744ce22284c05d5214dac2ab5dc9700
2018-10-26 09:57:24 -07:00
Colin Cross
1be9abc109 Add link dependency on test package archive
The test package archive was being picked up through "-L" + testRoot,
but did not have a dependency on it.  If the *_test.go file was
changed but not the implementation then the tests would not be
rerun.

Test: Add t.Fatal() to a *_test.go file and rebuild.
Change-Id: I32b8c7a0fa1d5b76b1242646887fe937e396e318
2018-10-24 13:13:13 -07:00
Colin Cross
0e58dc0176 Add missing dependencies when linking tests
Linking tests needs the same dependencies that linking binaries
got in https://github.com/google/blueprint/pull/222.

Test: m checkbuild
Change-Id: I33330f3184b8c0fd2bc20b48736c20d6edeaea68
2018-10-05 11:12:53 -07:00
Colin Cross
6c92af937e Enable restat for go binaries
Many Android build tools are written in go, and changes to Soong or
Blueprint can cause them to rebuild.  Almost everything in an
Android build is downstream of at least one of these tools, so
they all rebuild too.

Go binaries are static, so their contents will always change if any
of their dependencies change.  Only update output files of go compile
and go link if the contents change, and enable restat for the rules.

Test: m soong_zip && m soong_zip
Test: m soong_zip && touch build/soong/zip/cmd/main.go && m soong_zip
Change-Id: I9267580f644b42b44d43fb0a2674dc5234f127e5
2018-10-02 21:52:50 -07:00
Dan Willemsen
734f20c205 Fix issues found by go vet
Change-Id: If3e20a1f394d757554d6a7da5ed4c41fe194f55f
2018-07-21 13:10:25 -07:00
Dan Willemsen
ab223a512b Run globs during earlier bootstrap phases
Instead of sometimes re-running minibp/the primary builder during the
next phase, run bpglob earlier to check dependencies.

We've run into issues where the environment is slightly different
between bootstrapping phase and the main build phase. It's also a
problem because our primary builder (Soong) exports information used by
another tool (Kati) that runs in between the bootstrapping phases and
the main phase. When Soong would run in the main phase, it could get out
of sync, and would require the build to be run again.

To do this, add a "subninja" include a build-globs.ninja file to each
build.ninja file. The first time, this will be an empty file, but we'll
always run minibp / the primary builder anyway. When the builder runs,
in addition to writing a dependency file, write out the
build-globs.ninja file with the rules to run bpglob.

Since bpglob may need to be run very early, before it would normally be
built, build it with microfactory.

Change-Id: I89fcd849a8729e892f163d40060ab90b5d4dfa5d
2018-07-06 10:39:38 -07:00
Dan Willemsen
ff2c5b4cc8
Merge pull request #203 from danw/numcpu_cap
Cap the number of cpus for Go compiles
2018-03-02 16:02:06 -08:00
Dan Willemsen
20c343b89e Cap the number of cpus for Go compiles
When passing "-c" to the Go compiler, any time this value changes, we'd
force all of the Go compiles to rebuild. This could trigger a
substantial portion of the tree to rebuild (anything that transitive
depends on a Go helper tool).

We're running into issues when moving output directories between
multiple GCE machines with different core counts, but are otherwise
identical. This could also hit users moving/mounting disks between
machines, though changes to other host tools can make an impact too.

On my 48-core machine, I get a ~15% benefit from going from -c 1 to -c
48, but also ~12% benefit from going from -c 1 to -c 8. So this will
still let us scale somewhat, but prevent rebuilds when transitioning
between machines that are more likely building Android.
2018-03-02 15:32:22 -08:00
Dan Willemsen
a1e6eeeb44 Try to make GOROOT relative in Go 1.10
In Go 1.10, runtime.GOROOT() will attempt to find the current location
of the go binaries, instead of using the GOROOT_FINAL that was baked
into the binaries. This means that GOROOT() will usually return an
absolute path.

We avoid putting absolute paths into the ninja file, since any change to
the paths would then cause all of the actions including it to rebuild.
Since we've got a decent number of build tools in Android using Go now,
this causes us to rebuild a decent portion of the tree.

Instead of passing the GOROOT around manually in a side channel, just
let the Go 1.10 detection do its thing, and always try to turn the
result into a relative path.
2018-03-02 15:29:35 -08:00
Colin Cross
c1d878159c Fix format issues found by go vet
Test: m checkbuild
Change-Id: I5135e0daecd20abce9a66631f5f55230168613de
2018-02-23 14:05:03 -08:00
Dan Willemsen
2d9b59bc20 Store build.ninja deps in .ninja_deps
We're already going to load the .ninja_deps file, so we may as well use
it. These are the only two files in our build that don't.

Test: dump the .ninja_deps file, see the dependencies
2017-12-18 09:14:16 -08:00
Colin Cross
28b2843b78 Allow primary builder to stop bootstrap early
Allow the primary builder to stop bootstrap between
ResolveDependencies and PrepareBuildActions so that it can
use the build graph to write customized documentation.

Change-Id: I67ee5e6e438a0c2c3ec844c4c453d223971e60a2
2017-12-11 16:31:41 -08:00
Colin Cross
5eb116a46e Move auto-generated docs to main stage
Move the docs to the main stage and make a phony rule to build
them.  This will allow the primaray builder to choose not to
build them so that it can use its own rule instead.

Change-Id: Ib788f74717ccdcd85c2e5b6a581cc744d439341a
2017-12-11 16:31:41 -08:00
Colin Cross
92639f3fbb Refactor bpdoc
Separate the extraction of documentation-related info from writing
it out so that the primary builder can choose to write it out in
a customized format.

Change-Id: If2c91d6f13b8cea2e9e1cca75d8c2d684ad7f6df
2017-12-11 16:31:41 -08:00
Jeff Gaston
c3e2844dfe Support for a custom list of Blueprints files to parse
Bug: 64363847
Test: BLUEPRINT_LIST_FILE=out/.module_paths/Android.bp.list minibp

Change-Id: Id7f8cb1ab3a6684b3f8265d77bb32413957f1c93
2017-10-30 15:00:19 -07:00
Colin Cross
7311337d4c Support preformatted property documentation
Treat any property documentation lines that are indented as
preformatted.

Bug: 67909957
Test: manual inspection
Change-Id: Ib014bc301880eca02c60ed1692e49f07dca05fb3
2017-10-17 18:01:53 -07:00
Colin Cross
4ef23afc91 HTML escape property documentation
Property documentation that contained text that looked like HTML
tags was getting passed through.  Use html/template instead of
text/template so that it gets escaped automatically.

Bug: 67909957
Test: manual inspection
Change-Id: I2050c34c04d7be3cdacd5a76967a739f9f837e89
2017-10-17 18:01:53 -07:00
colincross
b67c1d433c Merge pull request #166 from colincross/create
Add TopDownMutatorContext.CreateModule
2017-09-20 14:09:15 -07:00
Dan Willemsen
87f71fa623 Expose the installed path for Go binaries
In case the primary builder wants to depend on a go binary by name,
expose an interface to find the real installed path of the binary.

Most of this change is changing it so that we're storing the install
path without any ninja references.

Change-Id: I873378ebdd47d8036d3cef5aab76f036846a3db1
2017-09-13 15:30:54 -07:00
Colin Cross
874a346904 Let mutators use ctx.AddNinjaFileDeps
Propagate extra ninja file deps through mutators so that they
can use ctx.AddNinjaFileDeps.

Test: blueprint tests
Change-Id: I299a0665c3f63b020ae345889fd78b91b91b215a
2017-08-01 15:12:12 -07:00
Dan Willemsen
c54c072667 Support parallel Go compiles with Go 1.9
Change-Id: I80f3ad6e563fef1b4f22e936016675262f7ce8bf
2017-07-25 11:08:20 -07:00
Dan Willemsen
feb1a225f2 Update documentation
Change-Id: I59daf7802a1b04961d0372c2417869ad9bd6d0b2
2017-07-24 14:02:51 -07:00
Dan Willemsen
cd4e0cebc9 Unify .ninja_log files
Having separate .ninja_log files doesn't work very well after
04d886b110
since the last successful timestamp is not recorded in the ninja log.
This was triggering the primary builder to be run twice on every build
since it was always considered out of date in the primary, then main
stages.

The two bootstrap stages already were unified when minibootstrap was
simplified, but this combines the main stage as well. This means that
we'll save some time running globs that are shared between minibp and
the primary builder as well (subdirs globs).

The cleanup code needed to be refactored, since the ninja_log will have
entries from the main stage as well. So instead of looking at everything
in the ninja log, take a string prefix to limit what we may clean.

Change-Id: I8d43168643aa8a0c6c3e48c6101eaa45d174bbe5
2017-07-24 14:02:51 -07:00
Dan Willemsen
af456eaae4 Define everything in both stages
Now that packages and binaries are only built when needed, they need to
be defined in both stages in case something building in the main stage
needs something that was defined, but didn't need to be built in the
earlier stage.

Change-Id: I67704fba1d670e82218eba5ce91f0c01896557a2
2017-07-24 14:02:51 -07:00
Dan Willemsen
f60525a63d Mark intermediates as optional
Instead of forcing things to be built alongside the primary builder,
mark all packages and binaries as optional, unless they're explicitly
marked as default.

Change-Id: Ie53c90a01b41ce886d651f9aa4c4900d1edb0157
2017-07-24 14:02:51 -07:00
Dan Willemsen
91e2483024 Fix and test rebuilds of bootstrap ninja when using tests
Add testing of bootstrap.bash -t to travis, and fix spacing issues
causing rebuilds.

Change-Id: Ia30367e40f5cf32de9749b2742d68b57e9201278
2017-07-24 14:02:51 -07:00
Dan Willemsen
1e72321e58 Use microfactory to build the bootstrap minibp
This duplicates building common blueprint go packages between minibp and
the primary builder, but drastically simplifies the first stage,
removing the need to check in a generated build.ninja.in.

Change-Id: I639a9637f1ed36d4210823ef276c0f7a064a83bd
2017-07-24 14:02:51 -07:00
Dan Willemsen
aee3eac6ce Fix bad ninja file if only GOOS-specific tests exist
When only linux/darwin tests were specified, we weren't setting
testArchiveFile, so we had a corrupted ninja file (build : ...).
2017-03-15 23:25:20 -07:00
Dan Willemsen
f14e096607 Support darwin/linux specific go sources
In order to use certain syscalls, some source must only be compiled on
one of darwin or linux. There are also cases where it's simplest to have
two implementations of a method (or struct) instead of choosing an
implementation at runtime.

The standard go tools use _<GOOS>.go for this, but the explicit structs
map better into the current blueprint format (and you can still use
_darwin.go / _linux.go to maintain compatiblity with those tools).

Test: Define _linux.go / _darwin.go files, ensure only one is compiled.
Change-Id: I58bffefbf839ba52dea3d8e7d3ec67eaa721c463
2017-02-18 15:12:13 -08:00
Dan Willemsen
e8a33ee623 Enable go tests for blueprint_tools
We haven't been building the tests for any Go tools that get built
during the primary stage, since we haven't been passing -t to the
primary builder. So enable them there so that we test those tools.

Change-Id: Ic7720c0679419f9e1ba61c7e151f37eb5c7bc8dc
2017-01-18 14:34:46 -08:00
Colin Cross
127d2eae8b Import globbing from Soong
Add globbing with dependency checking to blueprint.  Calling
ModuleContext.GlobWithDeps or SingletonContext.GlobWithDeps will return
a list of files that match the globs, while also adding efficient
dependencies to rerun the primary builder if a file that matches the
glob is added or removed.

Also use the globbing support for optional_subdirs=, subdirs= and build=
lines in blueprints files.  The globbing slightly changes the behavior
of subname= lines, it no longer falls back to looking for a file called
"Blueprints".  Blueprint files that need to include a subdirectory with
a different name can use build= instead of subdir= to directly include
them.  The Blueprints file is updated to reset subname="Blueprints" in
case we want to include subdirectories inside blueprint and the primary
builder has changed the subname.

Also adds a new test directory that contains a simple primary builder
tree to test regeneration for globbing, and runs the tests in travis.

Change-Id: I83ce525fd11e11579cc58ba5308d01ca8eea7bc6
2016-11-03 13:54:03 -07:00
Dan Willemsen
9bd6b38617 Refactor test running to improve parallelism
Before this change, we would ensure that tests for a package were run
before anything used that package, even just to compile. With this
change, we'll let other packages compile, but still run the tests in
dependency order, and before the binaries are finally installed.

This shrinks our initial build time from ~9 seconds to ~5 seconds on my
machine. Most of the improvements come from building the "real" package
and the test package in parallel.

Change-Id: I935670feecadacbf09b250bcaa1659a3599edc58
2016-11-02 13:10:27 -07:00
Dan Willemsen
460c80eb12 Clean up unnecessary dependencies
Other than cleaning up the code, the only change in behavior is to run
the docs build in parallel with the primary builder, saving about a
second on a Soong bootstrap (10s -> 9s).

Change-Id: Iaff51d6d1a37af842f294f12db11d5774d48d440
2016-11-02 13:10:27 -07:00
Dan Willemsen
b6d88a4f0a Allow build.ninja regeneration (fix Soong globs)
Soong (and soon, Blueprint) embeds the results of filesystem globs into
the build.ninja, and uses a helper tool to detect when the glob changes
and we need to rebuild the build.ninja file. This is more flexible than
listing the affected directories in the depfiles, since it can check to
see if a file we actually cared about was added, instead of re-running
anytime any file was added/removed (which happens on atomic file
modifications as well).

My recent bootstrap simplification broke this, since the helper rules
are in the main build.ninja, but I removed the ability to regenerate
that file from itself. So keep the current model, but add a rule into
the primary and main stages that allow themselves to re-run their
generator and write out a new build.ninja file if necessary. The actual
build rules of the generator aren't necessary, since we already built
them in the previous stage.

Change-Id: Ib51245920b2ec3ee5306c0c269361a5a8733caa8
2016-11-02 13:10:27 -07:00
Dan Willemsen
3692331afe Cleanup outfile args in bootstrap/bootstrap.go
This would have been a case for implicit outputs, but these just aren't
needed anymore.
2016-10-31 17:36:53 -07:00
Colin Cross
0b7e83e11a Ask primary builder for module names
Remove the assumed Name and Deps properties.  Instead, ask the Module
for the Name, and require them to use the existing replacements for
Deps.

Change-Id: I729045d86277686078b3aa0bba71c67d612ead2c
2016-10-12 13:33:37 -07:00
colincross
53d4357592 Merge pull request #127 from colincross/memprofile
Add support for memory profiling
2016-10-12 11:16:42 -07:00
Colin Cross
2c62844274 Improve error reporting
Report module name and variant and property names in errors.

Change-Id: I747f840402497b2c92bf8c565d7c50af33e6ab0e
2016-10-12 11:14:18 -07:00
Colin Cross
2ecec57e47 Add support for memory profiling
Add -memprofile argument to support capturing a heap profile, and -nogc
to disable the garbage collector to see all allocations.

Change-Id: I0d4c5f2bf79786aed4b2af4be112ecf0a2810a91
2016-10-12 11:12:54 -07:00
Dan Willemsen
7d0dddd84d Simplify bootstrap
tl;dr: Read if you don't use the wrapper or use SKIP_NINJA

Previously, we were relying on the ninja behavior of restarting the
build when the build.ninja file was updated to switch between different
bootstrap stages. But that means that every step that could produce a
build.ninja must pass in order to switch to a different stage. That
wasn't a big problem when we had a two stage build -- there was very
little that could fail in the second stage before we chose to go back to
the first stage. But when we had a three stage build, it was possible to
get into a state (usually during development) where you were in the
second stage, but the build was failing because the first stage needed
to be run. This was fixed in d79f1af742
by adding a wrapper that always started building at the first stage.

But this kept all of the complexity of using ninja restarts without any
of the benefits, so this change removes that complexity and just runs
each stage sequentially in the wrapper. So the wrapper is now required.

Since we're no longer going through choosestage, we can also skip the
template parsing for the later stages that don't need to be templated --
this can save a couple of seconds for large files.

In addition to all of the above, this also lets Soong reduce the number
of times the main ninja file is loaded. We had been running the wrapper
once (3 stages), then running ninja again after combining the
Soong-generated build.ninja with the Kati-generated build.ninja. This
change lets us removing the intermediate parsing of Soong's build.ninja,
so that we only execute ninja 3 times per build. It also lets us have
dependencies on pools or rules from Kati in the primary builder, since
we're never executing the main build.ninja without the Kati build.ninja.

The wrapper has a new option, NINJA to provide the path to ninja. This
used to be hardcoded to `ninja`, and will still default to that. But
we'll be running the first two bootstrap stages with $NINJA even if
SKIP_NINJA is set.

The wrapper passes "-w dupbuild=err" to ninja now -- this really should
always be turned on if you care about reliable builds.

Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
2016-08-30 17:26:56 -07:00
Colin Cross
c3d731258a Support nil pointers to structs in properties
Allow primary builders to reduce allocations of empty structures by
allowing nil pointers to concrete struct types.  Property readers will
not recurse into nil pointers, property writers will replace the nil
pointer with a pointer to the zero value of the pointer element type.

Allows a >50% primary builder time improvement with a trivial change in
Soong.

Change-Id: If6ad674bf7bf2a694c335378a074643a97d3c50b
2016-08-05 17:19:36 -07:00