This makes it possible to debug the primary builder without manually
adding the respective flags to the compilation command lines.
Test: "m nothing", "m nothing" with debugging, then "m nothing" again
Change-Id: I7d403af48528a94e5e0033d9a5c1fa9981100cfd
It's replaced with a flag in bootstrap.Args. It's a little bit ugly that
bootstrap.Main also gained a new Boolean flag in the process. The plan
is to delete that method in favor of directly calling RunBlueprint().
Test: "m nothing" (presubmits take a full workday these days)
Change-Id: Iae3421ae517a90570259b6278f18c3614bb1bb73
In addition to the command line interface, a new struct is created that
holds all the information Blueprint needs to do its job.
Test: Presubmits.
Change-Id: I0d6c924e09e069b3c1ae2000722efb52fd561e79
The end goal of this exercise is to remove all mutable global state so
that multiple Blueprint instances can be run in the same process.
This is done by plumbing the necessary data through the configuration.
Test: Presubmit.
Change-Id: I040a461ae17705258b882b3a44740a4dd5027a2c
Some of these were introduced in aosp/1610784 and some others were
present since a good while.
Test: Manual: "m nothing"
Change-Id: I7d5cb56ca2d87cb439354fd1b298c3b8d97d665d
- Add the -N -l command line arguments to Microfactory
- Pass the the Delve arguments to the primary builder on the command line
Test: Manual.
Change-Id: I4034f2d48b3e40d9863529053715183dc3dce1f5
This is so that soong_build doesn't know the current directory, thereby
avoiding one way to leak information.
Test: Manual.
Change-Id: I976a8663b11d99c5382726487102df10043c5a61
ninjaWriter repeatedly called io.WriteString() on its writer, which
does a type assertion every time. Replace its io.Writer with an
io.StringWriter and call WriteString on it directly.
Test: ninja_writer_test.go
Change-Id: Ie073d996a319190242bf6a00af07a13a60d078b5
This allows exiting bootstrap directly before writing ninja files.
This facilitates shorter runtime on integrations which do processing
which do not require ninja file output.
Test: Manually verified on Soong integration use case which involves
running bootstrap twice in a single program; stopping before ninja
output reduces runtime by ~20s, or ~11%.
Some anonymous nested properties are missing from property structs,
since setting the property to anonymous is to allow future filtering,
there is no issue if we cannot find the struct.
test: go bpdoc tests
test: m soong_docs
The primary builder may want to create variants of bootstrap
modules if they need to fit in to the primary builder's dependency
graph. Enable arbitrary variants of bootstrap modules by only
running the module's actions on the primary variant and then
copying the result to any other variants that exist.
Test: m checkbuild
Change-Id: I24b97771bb11faeacab4079ed8cf69aef59da140
* Update bpdocs to filter nested properties by tag
Prior to this change nested properties tagged with `blueprint:"mutated"`
are erroneously included in documentation, this corrects that behavior
and adds testing to verify.
Tested: go test bpdocs tests
Change-Id: I822c9a98276634d2f584d8709e83003824cdffd5
This makes a minimal difference on my AOSP tree (which is on a fast NVME
drive), but it saves ~10-15% of the time in our primary builder
(soong_build) on larger trees on slower drives.
We're always going to run the glob tool at least once outside of the
primary builder, and ninja only loads depfiles created after running the
tool, so this should be a no-op change.
We still need to write the list file (if it has changed), since that's
the file we're adding as a dependency of the primary builder, so the
effective timestamp must be earlier than the build.ninja timestamp.
Change-Id: I872b7581da50e7f2089daa7e248ca05b6703f019
This reverts commit 63085f9e7c.
It breaks the blueprint tests, and we aren't use it yet, so revert
it for now.
Change-Id: I1ac81371342285e8d57baf15fe3a223d4140c6cf
Emit the rules to build blueprint tests even if runGoTests is not
set, and add a phony rule "blueprint_tests" to run them. This will
allow Soong to stop running the tests at the beginning of every build
but still run them as part of checkbuild or with a manual
`m blueprint_tests`.
Bug: 156428456
Test: m
Test: m blueprint_tests
Change-Id: If293a0757766d3046e78bf230a1825f15adc68fd
Otherwise we'll re-run the primary builder every time this is used. This
triggered my new ninja checks that verify that all commands write their
outputs.
Change-Id: Ic92d1ea57203b953da8025a170b24e6420c5436c
The filter tag is unused, replaced with FilterPropertyStruct to
generate a new type at runtime that only contains the filtered
fields.
Test: unpack_test.go
Change-Id: Id91cf99290832094d05426f3263279836f0fea73
Sandboxing the primary builder caused the glob filelists to not
be written because they were using a relative path, causing
primary builder reruns on the second build.
Also report errors when writing the filelist files.
Test: m checkbuild
Change-Id: Id1706560d04c85f00f829cfb714967bb8600626f
Globs produce a filelist using restat rules to avoid primary builder
reruns. On the first execution of the primary builder the filelists
are written directly by the primary builder, so that the second
execution can restat them all but avoid rerunning the primary builder.
This wasn't working because the glob filelists were written after
the build.ninja file, but the build.ninja file depends on the
filelists. Switching the order avoids the extra rerun.
Test: m nothing && m nothing
Change-Id: Ia8e0924598220d4ff34235907a8de2e3e03632df
Move LoadHooks from Blueprint and run them during ParseBlueprintsFiles.
This will allow them to add scoped module types that are visible to the
rest of the Blueprints file. Requires passing the config object to
ParseBlueprintsFiles.
Test: all blueprint tests
Change-Id: Ia2a2c9a0223d5458bfd48bd22ebed0fdbd0156c6
Don't write to $buildDir/.bootstrap in the ninja file generated for
the main stage.
Test: examine main stage build.ninja file
Change-Id: I9c6183c9a434ff11d44cdd4f9844eae192ebfe03
The go compile command in go 1.13 needs -p main instead of
-p $pkgPath when compiling binaries. This also works for older
go releases.
Also turn on travis for go 1.13.
Change-Id: I45ce185aa44cfa01a7933cc6e52290a85c851641
The primary builder should always be running on its own with no
other processes running in parallel, so put it in the console
pool so that it can print to stdout without buffering.
Bug: 80165685
Test: build with print in Soong
Change-Id: If34ecdb5fa18de7e47c4cd6965d551c504850176
This change fixes a bug where the `blueprint:mutated` tag was ignored
when used for struct or interface properties. It also fixes a dormant
ExcludeByTag func bug.
Test: m soong_docs, properties_test.go, bpdoc_test.go
Change-Id: I926de2aa203ec552ced897174e9b78e817701a7d
bpglob is only used as a helper to check if the primary builder
needs to rerun due to the results of a glob changing. A recent
change to glob support in pathtools made a glob format that was
accidentally previously accepted into an error. If the bad
syntax was used in the most recent primary builder run, and
then an incremental build is performed after picking up the
change that made the syntax invalid, then bpglob will attempt
to rerun before the primary builder, see the now-invalid syntax,
and fail. This will prevent the primary builder from rerunning,
which would have updated the bpglob rule with a corrected glob
syntax (or failed in the primary builder if the Blueprints file
still had the invalid glob syntax).
Avoid exiting with an error in bpglob. Instead, write the
error to the output file along with a timestamp so that it is
always dirty, forcing the primary builder to rerun.
Bug: 129411151
Test: m nothing
Change-Id: Ib680570c33662f3c0f1f72425d60a963ed841ba6
1. Apply preformatted text (<pre>...</pre>) handling logic to module
type text too. It used to be applied to property texts only.
2. Improve <pre> handling logic itself for better readability.
- Insert a blank line before <pre>.
- Prevent from ending <pre> blocks prematurely by checking if an
unindented line isn't just a blank line between indented lines.
Change-Id: Id40fa668d4c6781caf7ed140b2f40784cdeb8c35
1. Extract module type documentation.
2. Support primary builder customization of factory function to use for
documentation for each module type.
3. Change the ModuleType list order so that they are grouped by package.
This is basically minor refactoring + readability improvement done on
top of https://github.com/google/blueprint/pull/232.
Change-Id: If7413e5ac23486b85f18d02fb3ba288a38730c32
Writing the ninja file to a byte buffer causes a significant amount
of time to be spent in memmove when growing the byte slice. Write
the file directly to disk instead.
This also fixes some unhandled error warnings, which become more
likely when doing disk IO instead of byte buffer writes.
Change-Id: I5094e4c45cab4012713037f60c5a4fb00718f92e
In cases that we want to run blueprint-based builds in many
configurations to verify all of the logic works without errors, but
don't care about running the final ninja file, writing it out only
wastes time and disk space. So add a --empty-ninja-file option that
writes out an empty ninja file instead.
Our specific use case (Soong's build_test / multiproduct_kati) runs
Soong several hundred times for different configurations, and the ninja
files are around 1GB, which leads to several hundred gigabytes of disk
writes (and persistent use during incremental generation).
Change-Id: I0198dfb2f744ce22284c05d5214dac2ab5dc9700
The test package archive was being picked up through "-L" + testRoot,
but did not have a dependency on it. If the *_test.go file was
changed but not the implementation then the tests would not be
rerun.
Test: Add t.Fatal() to a *_test.go file and rebuild.
Change-Id: I32b8c7a0fa1d5b76b1242646887fe937e396e318
Linking tests needs the same dependencies that linking binaries
got in https://github.com/google/blueprint/pull/222.
Test: m checkbuild
Change-Id: I33330f3184b8c0fd2bc20b48736c20d6edeaea68
Many Android build tools are written in go, and changes to Soong or
Blueprint can cause them to rebuild. Almost everything in an
Android build is downstream of at least one of these tools, so
they all rebuild too.
Go binaries are static, so their contents will always change if any
of their dependencies change. Only update output files of go compile
and go link if the contents change, and enable restat for the rules.
Test: m soong_zip && m soong_zip
Test: m soong_zip && touch build/soong/zip/cmd/main.go && m soong_zip
Change-Id: I9267580f644b42b44d43fb0a2674dc5234f127e5
Instead of sometimes re-running minibp/the primary builder during the
next phase, run bpglob earlier to check dependencies.
We've run into issues where the environment is slightly different
between bootstrapping phase and the main build phase. It's also a
problem because our primary builder (Soong) exports information used by
another tool (Kati) that runs in between the bootstrapping phases and
the main phase. When Soong would run in the main phase, it could get out
of sync, and would require the build to be run again.
To do this, add a "subninja" include a build-globs.ninja file to each
build.ninja file. The first time, this will be an empty file, but we'll
always run minibp / the primary builder anyway. When the builder runs,
in addition to writing a dependency file, write out the
build-globs.ninja file with the rules to run bpglob.
Since bpglob may need to be run very early, before it would normally be
built, build it with microfactory.
Change-Id: I89fcd849a8729e892f163d40060ab90b5d4dfa5d
When passing "-c" to the Go compiler, any time this value changes, we'd
force all of the Go compiles to rebuild. This could trigger a
substantial portion of the tree to rebuild (anything that transitive
depends on a Go helper tool).
We're running into issues when moving output directories between
multiple GCE machines with different core counts, but are otherwise
identical. This could also hit users moving/mounting disks between
machines, though changes to other host tools can make an impact too.
On my 48-core machine, I get a ~15% benefit from going from -c 1 to -c
48, but also ~12% benefit from going from -c 1 to -c 8. So this will
still let us scale somewhat, but prevent rebuilds when transitioning
between machines that are more likely building Android.
In Go 1.10, runtime.GOROOT() will attempt to find the current location
of the go binaries, instead of using the GOROOT_FINAL that was baked
into the binaries. This means that GOROOT() will usually return an
absolute path.
We avoid putting absolute paths into the ninja file, since any change to
the paths would then cause all of the actions including it to rebuild.
Since we've got a decent number of build tools in Android using Go now,
this causes us to rebuild a decent portion of the tree.
Instead of passing the GOROOT around manually in a side channel, just
let the Go 1.10 detection do its thing, and always try to turn the
result into a relative path.
We're already going to load the .ninja_deps file, so we may as well use
it. These are the only two files in our build that don't.
Test: dump the .ninja_deps file, see the dependencies
Allow the primary builder to stop bootstrap between
ResolveDependencies and PrepareBuildActions so that it can
use the build graph to write customized documentation.
Change-Id: I67ee5e6e438a0c2c3ec844c4c453d223971e60a2
Move the docs to the main stage and make a phony rule to build
them. This will allow the primaray builder to choose not to
build them so that it can use its own rule instead.
Change-Id: Ib788f74717ccdcd85c2e5b6a581cc744d439341a
Separate the extraction of documentation-related info from writing
it out so that the primary builder can choose to write it out in
a customized format.
Change-Id: If2c91d6f13b8cea2e9e1cca75d8c2d684ad7f6df
Treat any property documentation lines that are indented as
preformatted.
Bug: 67909957
Test: manual inspection
Change-Id: Ib014bc301880eca02c60ed1692e49f07dca05fb3
Property documentation that contained text that looked like HTML
tags was getting passed through. Use html/template instead of
text/template so that it gets escaped automatically.
Bug: 67909957
Test: manual inspection
Change-Id: I2050c34c04d7be3cdacd5a76967a739f9f837e89
In case the primary builder wants to depend on a go binary by name,
expose an interface to find the real installed path of the binary.
Most of this change is changing it so that we're storing the install
path without any ninja references.
Change-Id: I873378ebdd47d8036d3cef5aab76f036846a3db1
Propagate extra ninja file deps through mutators so that they
can use ctx.AddNinjaFileDeps.
Test: blueprint tests
Change-Id: I299a0665c3f63b020ae345889fd78b91b91b215a
Having separate .ninja_log files doesn't work very well after
04d886b110
since the last successful timestamp is not recorded in the ninja log.
This was triggering the primary builder to be run twice on every build
since it was always considered out of date in the primary, then main
stages.
The two bootstrap stages already were unified when minibootstrap was
simplified, but this combines the main stage as well. This means that
we'll save some time running globs that are shared between minibp and
the primary builder as well (subdirs globs).
The cleanup code needed to be refactored, since the ninja_log will have
entries from the main stage as well. So instead of looking at everything
in the ninja log, take a string prefix to limit what we may clean.
Change-Id: I8d43168643aa8a0c6c3e48c6101eaa45d174bbe5
Now that packages and binaries are only built when needed, they need to
be defined in both stages in case something building in the main stage
needs something that was defined, but didn't need to be built in the
earlier stage.
Change-Id: I67704fba1d670e82218eba5ce91f0c01896557a2
Instead of forcing things to be built alongside the primary builder,
mark all packages and binaries as optional, unless they're explicitly
marked as default.
Change-Id: Ie53c90a01b41ce886d651f9aa4c4900d1edb0157
This duplicates building common blueprint go packages between minibp and
the primary builder, but drastically simplifies the first stage,
removing the need to check in a generated build.ninja.in.
Change-Id: I639a9637f1ed36d4210823ef276c0f7a064a83bd
In order to use certain syscalls, some source must only be compiled on
one of darwin or linux. There are also cases where it's simplest to have
two implementations of a method (or struct) instead of choosing an
implementation at runtime.
The standard go tools use _<GOOS>.go for this, but the explicit structs
map better into the current blueprint format (and you can still use
_darwin.go / _linux.go to maintain compatiblity with those tools).
Test: Define _linux.go / _darwin.go files, ensure only one is compiled.
Change-Id: I58bffefbf839ba52dea3d8e7d3ec67eaa721c463
We haven't been building the tests for any Go tools that get built
during the primary stage, since we haven't been passing -t to the
primary builder. So enable them there so that we test those tools.
Change-Id: Ic7720c0679419f9e1ba61c7e151f37eb5c7bc8dc
Add globbing with dependency checking to blueprint. Calling
ModuleContext.GlobWithDeps or SingletonContext.GlobWithDeps will return
a list of files that match the globs, while also adding efficient
dependencies to rerun the primary builder if a file that matches the
glob is added or removed.
Also use the globbing support for optional_subdirs=, subdirs= and build=
lines in blueprints files. The globbing slightly changes the behavior
of subname= lines, it no longer falls back to looking for a file called
"Blueprints". Blueprint files that need to include a subdirectory with
a different name can use build= instead of subdir= to directly include
them. The Blueprints file is updated to reset subname="Blueprints" in
case we want to include subdirectories inside blueprint and the primary
builder has changed the subname.
Also adds a new test directory that contains a simple primary builder
tree to test regeneration for globbing, and runs the tests in travis.
Change-Id: I83ce525fd11e11579cc58ba5308d01ca8eea7bc6
Before this change, we would ensure that tests for a package were run
before anything used that package, even just to compile. With this
change, we'll let other packages compile, but still run the tests in
dependency order, and before the binaries are finally installed.
This shrinks our initial build time from ~9 seconds to ~5 seconds on my
machine. Most of the improvements come from building the "real" package
and the test package in parallel.
Change-Id: I935670feecadacbf09b250bcaa1659a3599edc58
Other than cleaning up the code, the only change in behavior is to run
the docs build in parallel with the primary builder, saving about a
second on a Soong bootstrap (10s -> 9s).
Change-Id: Iaff51d6d1a37af842f294f12db11d5774d48d440
Soong (and soon, Blueprint) embeds the results of filesystem globs into
the build.ninja, and uses a helper tool to detect when the glob changes
and we need to rebuild the build.ninja file. This is more flexible than
listing the affected directories in the depfiles, since it can check to
see if a file we actually cared about was added, instead of re-running
anytime any file was added/removed (which happens on atomic file
modifications as well).
My recent bootstrap simplification broke this, since the helper rules
are in the main build.ninja, but I removed the ability to regenerate
that file from itself. So keep the current model, but add a rule into
the primary and main stages that allow themselves to re-run their
generator and write out a new build.ninja file if necessary. The actual
build rules of the generator aren't necessary, since we already built
them in the previous stage.
Change-Id: Ib51245920b2ec3ee5306c0c269361a5a8733caa8
Remove the assumed Name and Deps properties. Instead, ask the Module
for the Name, and require them to use the existing replacements for
Deps.
Change-Id: I729045d86277686078b3aa0bba71c67d612ead2c
Add -memprofile argument to support capturing a heap profile, and -nogc
to disable the garbage collector to see all allocations.
Change-Id: I0d4c5f2bf79786aed4b2af4be112ecf0a2810a91
tl;dr: Read if you don't use the wrapper or use SKIP_NINJA
Previously, we were relying on the ninja behavior of restarting the
build when the build.ninja file was updated to switch between different
bootstrap stages. But that means that every step that could produce a
build.ninja must pass in order to switch to a different stage. That
wasn't a big problem when we had a two stage build -- there was very
little that could fail in the second stage before we chose to go back to
the first stage. But when we had a three stage build, it was possible to
get into a state (usually during development) where you were in the
second stage, but the build was failing because the first stage needed
to be run. This was fixed in d79f1af742
by adding a wrapper that always started building at the first stage.
But this kept all of the complexity of using ninja restarts without any
of the benefits, so this change removes that complexity and just runs
each stage sequentially in the wrapper. So the wrapper is now required.
Since we're no longer going through choosestage, we can also skip the
template parsing for the later stages that don't need to be templated --
this can save a couple of seconds for large files.
In addition to all of the above, this also lets Soong reduce the number
of times the main ninja file is loaded. We had been running the wrapper
once (3 stages), then running ninja again after combining the
Soong-generated build.ninja with the Kati-generated build.ninja. This
change lets us removing the intermediate parsing of Soong's build.ninja,
so that we only execute ninja 3 times per build. It also lets us have
dependencies on pools or rules from Kati in the primary builder, since
we're never executing the main build.ninja without the Kati build.ninja.
The wrapper has a new option, NINJA to provide the path to ninja. This
used to be hardcoded to `ninja`, and will still default to that. But
we'll be running the first two bootstrap stages with $NINJA even if
SKIP_NINJA is set.
The wrapper passes "-w dupbuild=err" to ninja now -- this really should
always be turned on if you care about reliable builds.
Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
Allow primary builders to reduce allocations of empty structures by
allowing nil pointers to concrete struct types. Property readers will
not recurse into nil pointers, property writers will replace the nil
pointer with a pointer to the zero value of the pointer element type.
Allows a >50% primary builder time improvement with a trivial change in
Soong.
Change-Id: If6ad674bf7bf2a694c335378a074643a97d3c50b
Passing -trace <file> on the command line produces a trace file that can
be used with go tool trace to examine concurrency.
Change-Id: I7d27bca64d0f0816cb4e518e947b70bc1c0573a3
The word "doc" spread all over bpdoc is redundant and makes it harder to
differentiate the various types and variables. Everything in bpdoc is a
doc, so remove it. Also rename Collector to Context.
Change-Id: Ia616d789ee27e7c94d9be231bdb248d45b2b7f39
Move these tools from $buildDir/.bootstrap/bin to $buildDir/bin
(configurable by the primary builder).
Also delay building them until the main stage, and give them a phony
target "blueprint_tools".
Primary builder logic is becoming complicated due to the two pass nature
of mutators that add dependencies and GenerateBuildActions that handle
the dependencies. The reason why the dependency was added is lost by
the time GenerateBuildActions is called, resulting in build logic that
has to recreate all the dependencies and try to match them up to the
modules returned by VisitDirectDeps.
Change the API of AddDependency to take a DependencyTag interface, which
is satisifed by anything that embeds BaseDependencyTag. Mutators and
GenerateBuildActions that call VisitDirectDeps can pass each Module to
ctx.OtherModuleDependencyTag to retreive the DependencyTag that was
passed when adding the dependency.
Change-Id: I0814dcd26d1670302d340b77e8dc8704ed7b60bf
A common pattern in mutators is to set a property on a module to be used
later in GenerateBuildActions. A common anti-pattern is to set a member
variable instead of a property. Setting member variables will appear to
work until a mutator calls CreateVariations after the member variable is
set. The first variant will have the member variables, but any other
variants will have zero values for all member variables.
To catch this common case early, replace all modules with clones after
running all the mutators but before GenerateBuildActions.
This catches the anti-pattern used in bootstrap, replace the buildStage
member variable with a property.
Change-Id: I6ff37580783a6227ebba2b46d577b443566a79bb
Other parts of blueprint have started allowing pointers to strings and
booleans. This code silently allowed them because of the nil check, but
it doesn't work if the module factory sets a default value.
Prefix errors with "error:" or "internal error:" colored in red using
ANSI codes. Ninja will strip the ANSI codes when build output is not
going to a terminal that can handle ANSI codes.
Embedded anonymous structs have no name, use the type as the name for
now. Eventually we should hide the name completely and put the
properties in the embedding struct.
Removing abandoned files needs to know where the .ninja_log file is
stored. Export the ninja builddir value from Context and use it to
determine the .ninja_log path in any stage.
The ninja builddir (where ninja stores its .ninja_log and .ninja_deps
files) and the bootstrap.BuildDir (where build output files are written)
are distinct, so to reduce confusion replace SetBuildDir with
SetNinjaBuildDir.
Build logic can now implement a RemoveAbandonedFiles, and the bootstrap
logic will only remove abandoned files if that method returns true.
Leaving the method unimplemented will result in the existing behavior of
always removing abandoned files.
For implicit dependencies that will be common to all users of a Rule,
add a new field 'CommandDeps' to the RuleParam. This is a list of
strings to be prepended to the implicit dependencies in each BuildParam.
This lets us have the dependencies declared next to where they are used,
instead of duplicated in areas that may be far apart.
I looked at passing this information down to ninja too, but it only
saves us a few percent of ninja file, and requires a modification to the
ninja file format.
Change-Id: Ifd910dee1506d4e32a76ed06206f853c4caec622
This removes the need to use $OLDPWD when running tests, which means
that the builddir may be an absolute or relative directory. It also
filters out the "PASS" message on successful test runs to clean up our
output.
Change-Id: I4ab937c7a87b74fe997a47cc0311e2f357f9f7e9
