This makes it possible to debug the primary builder without manually
adding the respective flags to the compilation command lines.
Test: "m nothing", "m nothing" with debugging, then "m nothing" again
Change-Id: I7d403af48528a94e5e0033d9a5c1fa9981100cfd
It's replaced with a flag in bootstrap.Args. It's a little bit ugly that
bootstrap.Main also gained a new Boolean flag in the process. The plan
is to delete that method in favor of directly calling RunBlueprint().
Test: "m nothing" (presubmits take a full workday these days)
Change-Id: Iae3421ae517a90570259b6278f18c3614bb1bb73
In addition to the command line interface, a new struct is created that
holds all the information Blueprint needs to do its job.
Test: Presubmits.
Change-Id: I0d6c924e09e069b3c1ae2000722efb52fd561e79
The end goal of this exercise is to remove all mutable global state so
that multiple Blueprint instances can be run in the same process.
This is done by plumbing the necessary data through the configuration.
Test: Presubmit.
Change-Id: I040a461ae17705258b882b3a44740a4dd5027a2c
Some of these were introduced in aosp/1610784 and some others were
present since a good while.
Test: Manual: "m nothing"
Change-Id: I7d5cb56ca2d87cb439354fd1b298c3b8d97d665d
- Add the -N -l command line arguments to Microfactory
- Pass the the Delve arguments to the primary builder on the command line
Test: Manual.
Change-Id: I4034f2d48b3e40d9863529053715183dc3dce1f5
This is so that soong_build doesn't know the current directory, thereby
avoiding one way to leak information.
Test: Manual.
Change-Id: I976a8663b11d99c5382726487102df10043c5a61
ninjaWriter repeatedly called io.WriteString() on its writer, which
does a type assertion every time. Replace its io.Writer with an
io.StringWriter and call WriteString on it directly.
Test: ninja_writer_test.go
Change-Id: Ie073d996a319190242bf6a00af07a13a60d078b5
This allows exiting bootstrap directly before writing ninja files.
This facilitates shorter runtime on integrations which do processing
which do not require ninja file output.
Test: Manually verified on Soong integration use case which involves
running bootstrap twice in a single program; stopping before ninja
output reduces runtime by ~20s, or ~11%.
Some anonymous nested properties are missing from property structs,
since setting the property to anonymous is to allow future filtering,
there is no issue if we cannot find the struct.
test: go bpdoc tests
test: m soong_docs
The primary builder may want to create variants of bootstrap
modules if they need to fit in to the primary builder's dependency
graph. Enable arbitrary variants of bootstrap modules by only
running the module's actions on the primary variant and then
copying the result to any other variants that exist.
Test: m checkbuild
Change-Id: I24b97771bb11faeacab4079ed8cf69aef59da140
* Update bpdocs to filter nested properties by tag
Prior to this change nested properties tagged with `blueprint:"mutated"`
are erroneously included in documentation, this corrects that behavior
and adds testing to verify.
Tested: go test bpdocs tests
Change-Id: I822c9a98276634d2f584d8709e83003824cdffd5
This makes a minimal difference on my AOSP tree (which is on a fast NVME
drive), but it saves ~10-15% of the time in our primary builder
(soong_build) on larger trees on slower drives.
We're always going to run the glob tool at least once outside of the
primary builder, and ninja only loads depfiles created after running the
tool, so this should be a no-op change.
We still need to write the list file (if it has changed), since that's
the file we're adding as a dependency of the primary builder, so the
effective timestamp must be earlier than the build.ninja timestamp.
Change-Id: I872b7581da50e7f2089daa7e248ca05b6703f019
This reverts commit 63085f9e7c.
It breaks the blueprint tests, and we aren't use it yet, so revert
it for now.
Change-Id: I1ac81371342285e8d57baf15fe3a223d4140c6cf
Emit the rules to build blueprint tests even if runGoTests is not
set, and add a phony rule "blueprint_tests" to run them. This will
allow Soong to stop running the tests at the beginning of every build
but still run them as part of checkbuild or with a manual
`m blueprint_tests`.
Bug: 156428456
Test: m
Test: m blueprint_tests
Change-Id: If293a0757766d3046e78bf230a1825f15adc68fd
Otherwise we'll re-run the primary builder every time this is used. This
triggered my new ninja checks that verify that all commands write their
outputs.
Change-Id: Ic92d1ea57203b953da8025a170b24e6420c5436c
The filter tag is unused, replaced with FilterPropertyStruct to
generate a new type at runtime that only contains the filtered
fields.
Test: unpack_test.go
Change-Id: Id91cf99290832094d05426f3263279836f0fea73
Sandboxing the primary builder caused the glob filelists to not
be written because they were using a relative path, causing
primary builder reruns on the second build.
Also report errors when writing the filelist files.
Test: m checkbuild
Change-Id: Id1706560d04c85f00f829cfb714967bb8600626f
Globs produce a filelist using restat rules to avoid primary builder
reruns. On the first execution of the primary builder the filelists
are written directly by the primary builder, so that the second
execution can restat them all but avoid rerunning the primary builder.
This wasn't working because the glob filelists were written after
the build.ninja file, but the build.ninja file depends on the
filelists. Switching the order avoids the extra rerun.
Test: m nothing && m nothing
Change-Id: Ia8e0924598220d4ff34235907a8de2e3e03632df
Move LoadHooks from Blueprint and run them during ParseBlueprintsFiles.
This will allow them to add scoped module types that are visible to the
rest of the Blueprints file. Requires passing the config object to
ParseBlueprintsFiles.
Test: all blueprint tests
Change-Id: Ia2a2c9a0223d5458bfd48bd22ebed0fdbd0156c6
Don't write to $buildDir/.bootstrap in the ninja file generated for
the main stage.
Test: examine main stage build.ninja file
Change-Id: I9c6183c9a434ff11d44cdd4f9844eae192ebfe03
The go compile command in go 1.13 needs -p main instead of
-p $pkgPath when compiling binaries. This also works for older
go releases.
Also turn on travis for go 1.13.
Change-Id: I45ce185aa44cfa01a7933cc6e52290a85c851641
The primary builder should always be running on its own with no
other processes running in parallel, so put it in the console
pool so that it can print to stdout without buffering.
Bug: 80165685
Test: build with print in Soong
Change-Id: If34ecdb5fa18de7e47c4cd6965d551c504850176
This change fixes a bug where the `blueprint:mutated` tag was ignored
when used for struct or interface properties. It also fixes a dormant
ExcludeByTag func bug.
Test: m soong_docs, properties_test.go, bpdoc_test.go
Change-Id: I926de2aa203ec552ced897174e9b78e817701a7d
bpglob is only used as a helper to check if the primary builder
needs to rerun due to the results of a glob changing. A recent
change to glob support in pathtools made a glob format that was
accidentally previously accepted into an error. If the bad
syntax was used in the most recent primary builder run, and
then an incremental build is performed after picking up the
change that made the syntax invalid, then bpglob will attempt
to rerun before the primary builder, see the now-invalid syntax,
and fail. This will prevent the primary builder from rerunning,
which would have updated the bpglob rule with a corrected glob
syntax (or failed in the primary builder if the Blueprints file
still had the invalid glob syntax).
Avoid exiting with an error in bpglob. Instead, write the
error to the output file along with a timestamp so that it is
always dirty, forcing the primary builder to rerun.
Bug: 129411151
Test: m nothing
Change-Id: Ib680570c33662f3c0f1f72425d60a963ed841ba6
1. Apply preformatted text (<pre>...</pre>) handling logic to module
type text too. It used to be applied to property texts only.
2. Improve <pre> handling logic itself for better readability.
- Insert a blank line before <pre>.
- Prevent from ending <pre> blocks prematurely by checking if an
unindented line isn't just a blank line between indented lines.
Change-Id: Id40fa668d4c6781caf7ed140b2f40784cdeb8c35
1. Extract module type documentation.
2. Support primary builder customization of factory function to use for
documentation for each module type.
3. Change the ModuleType list order so that they are grouped by package.
This is basically minor refactoring + readability improvement done on
top of https://github.com/google/blueprint/pull/232.
Change-Id: If7413e5ac23486b85f18d02fb3ba288a38730c32
Writing the ninja file to a byte buffer causes a significant amount
of time to be spent in memmove when growing the byte slice. Write
the file directly to disk instead.
This also fixes some unhandled error warnings, which become more
likely when doing disk IO instead of byte buffer writes.
Change-Id: I5094e4c45cab4012713037f60c5a4fb00718f92e
In cases that we want to run blueprint-based builds in many
configurations to verify all of the logic works without errors, but
don't care about running the final ninja file, writing it out only
wastes time and disk space. So add a --empty-ninja-file option that
writes out an empty ninja file instead.
Our specific use case (Soong's build_test / multiproduct_kati) runs
Soong several hundred times for different configurations, and the ninja
files are around 1GB, which leads to several hundred gigabytes of disk
writes (and persistent use during incremental generation).
Change-Id: I0198dfb2f744ce22284c05d5214dac2ab5dc9700
The test package archive was being picked up through "-L" + testRoot,
but did not have a dependency on it. If the *_test.go file was
changed but not the implementation then the tests would not be
rerun.
Test: Add t.Fatal() to a *_test.go file and rebuild.
Change-Id: I32b8c7a0fa1d5b76b1242646887fe937e396e318
Linking tests needs the same dependencies that linking binaries
got in https://github.com/google/blueprint/pull/222.
Test: m checkbuild
Change-Id: I33330f3184b8c0fd2bc20b48736c20d6edeaea68
Many Android build tools are written in go, and changes to Soong or
Blueprint can cause them to rebuild. Almost everything in an
Android build is downstream of at least one of these tools, so
they all rebuild too.
Go binaries are static, so their contents will always change if any
of their dependencies change. Only update output files of go compile
and go link if the contents change, and enable restat for the rules.
Test: m soong_zip && m soong_zip
Test: m soong_zip && touch build/soong/zip/cmd/main.go && m soong_zip
Change-Id: I9267580f644b42b44d43fb0a2674dc5234f127e5
Instead of sometimes re-running minibp/the primary builder during the
next phase, run bpglob earlier to check dependencies.
We've run into issues where the environment is slightly different
between bootstrapping phase and the main build phase. It's also a
problem because our primary builder (Soong) exports information used by
another tool (Kati) that runs in between the bootstrapping phases and
the main phase. When Soong would run in the main phase, it could get out
of sync, and would require the build to be run again.
To do this, add a "subninja" include a build-globs.ninja file to each
build.ninja file. The first time, this will be an empty file, but we'll
always run minibp / the primary builder anyway. When the builder runs,
in addition to writing a dependency file, write out the
build-globs.ninja file with the rules to run bpglob.
Since bpglob may need to be run very early, before it would normally be
built, build it with microfactory.
Change-Id: I89fcd849a8729e892f163d40060ab90b5d4dfa5d
When passing "-c" to the Go compiler, any time this value changes, we'd
force all of the Go compiles to rebuild. This could trigger a
substantial portion of the tree to rebuild (anything that transitive
depends on a Go helper tool).
We're running into issues when moving output directories between
multiple GCE machines with different core counts, but are otherwise
identical. This could also hit users moving/mounting disks between
machines, though changes to other host tools can make an impact too.
On my 48-core machine, I get a ~15% benefit from going from -c 1 to -c
48, but also ~12% benefit from going from -c 1 to -c 8. So this will
still let us scale somewhat, but prevent rebuilds when transitioning
between machines that are more likely building Android.
