There used to be two:
- The absolute source directory is now threaded through the code instead
- The command line arguments are moved to soong_build
Test: Presubmits.
Change-Id: Ie09de1a2862ec4ced2c7385198300359f805127d
Now that Blueprint has been merged with Soong, a large amount of code is
unnecessary.
Test: Presubmits.
Change-Id: I8d30d8b80f37fdc6a9db630a1792c57c7d6edb38
This is not necessary anymore now that Blueprint doesn't have a separate
existence and Soong's documentation generation uses a completely
separate code path.
Test: Presubmits.
Change-Id: Ibc9e57379c5f8ee9a06d90fbe81fac314f01ee92
This makes the distinction between what the bootstrap .ninja file and
the main .ninja files do clearer: the bootstrap .ninja file includes the
build-globs.ninja file(s) which the main .ninja file writes.
This required removing the "default" statement from said bootstrap
.ninja file: now we can build more than one thing instead of just
build.ninja and thus soong_ui explicitly tells Ninja what to do instead.
On the side, stop catering for the use case where the bootstrap
Blueprint invocation uses globs: Soong's does not and that's our only
use case.
Test: Presubmits.
Change-Id: Icffce31928242cfe9cdab56b290a37598d32a58c
This enables bp2build to write its own build-globs.ninja file.
Test: soong integration tests
Fixes: 193650250
Change-Id: Icb35f3cf3c30929dab1b2c2e9b244321be30f967
An empty glob file would be created inside the source tree if
soong_build is invoked directly (instead of via soong_ui). This logic is
now moved to soong/ui/build to prevent this behavior
Bug: 187194795
Test: Ran the following command locally for the target
art-target-arm:git_master-art
```
. ./build/envsetup.sh && lunch armv8-eng &&
art/tools/buildbot-build.sh --target
```
Before change glob file is created in $TOP/.bootstrap
After change glob file is created in $TOP/out/soong/.bootstrap
Change-Id: I6745996c940d790d1f1f387ca08aafc6a8a669df
We have `blueprint_tools` that can build all Go binaries (and run their
associates tests). But if a package is not (yet) used, it won't be
included there. This makes testing the build of new Go packages
difficult.
Test: m blueprint_go_packages
Change-Id: I7a67cf2593fedcf499d40d82c266d3b0d7ebbef2
There are a few cases that force all globs to be rerun at the beginning
of the build (changes to bpglob or dependencies, second build after a
clean build). The number of globs has gotten high enough that rerunning
them all can have significant overhead to start bpglob for each one.
Replace the per-glob bpglob invocations with sharded invocations using
1024 hash buckets.
Bug: 159845846
Test: glob_test.go
Test: m nothing && m nothing
Test: build/soong/bootstrap_test.sh
Change-Id: Ife1f7a03c8f6b25d1be01531425d8dc2c76d1ea0
Args.BuildDir was accidentally hard-wired to "." in one of my previous
changes, and since Blueprint is not really independent of Soong anymore,
it's probably better to fix it in a way that doesn't involve putting
back the almost-but-not-completely redundant "-b" command line argument
to the primary builder invocation.
Test: Presubmits.
Change-Id: I76d064a99be16cd812cb47a6c448fe44b51893dc
The only thing that was done with it was to append it to the list of
dependencies returned, which can just as well be done by the caller.
Test: Presubmits.
Change-Id: Ibedf024de411a5881f19ed0c137a91b0898f1269
bpglob is executed through the rules in build-globs.ninja to determine
whether soong_build needs to rerun. That means when the arguments
accepted by bpglob change it will be called with the old arguments,
then soong_build will rerun and update build-globs.ninja with the new
arguments.
To avoid having to maintain backwards compatibility with old arguments
across the transition, a version argument is used to detect the
transition in order to stop parsing arguments, touch the output file
and exit immediately. The version number in
pathtools.BPGlobArgumentVersion should be manually incremented when
the bpglob argument format changes.
If the version argument is not passed then a version mismatch is assumed.
Bug: 159845846
Test: build/soong/tests/bootstrap_test.sh
Change-Id: Id6742c7acc51af8d0d224d51e47bddea78e6e363
- Make the parsed command line arguments accessible from other packages
- Separate writing the depfile from writing the Ninja file
- Extract a method to compute the name of the file list file for globs
Test: Presubmits.
Change-Id: I4d0daf0f3776a9336da305be0108ebf70a314776
This is accomplished by plumbing PrimaryBuilderInvocation in from
bootstrap.Args.
The confusing removal of "primaryBuilderExtraFlags" from documentation
generation is because it used to (erroneously) refer to the func defined
in command.go and its removal apparently doesn't break anything. I
suspect that this code path is dead within Android, but we'll need to
wait until May to remove it.
Test: Presubmits.
Change-Id: I8763579b82bd79d5ee0f233edfd48327282e401c
This makes it possible to invoke the primary builder more than once in a
single Ninja file.
This required adding arguments that were hard-wired in the build.ninja
rule to extraArgs and adding a "set of primary builder invocations"
member to bootstrap.Config . The only command line arguments that remain
hard-wired are those which are expected to be relevant to every
invocation of the primary builder.
As a welcome side effect, I was able to remove a number of random
environment variables that were so far special-cased ($SOONG_DELVE,
$SOONG_DELVE_PATH and $SOONG_OUTDIR). I was also able to move writing
the empty stub ninja glob file to command.go from the bowels of the
Blueprint machinery.
In theory, $TOP and $BUILDER could be removed, too, but this would
require hard-coding the value of $TOP into build.ninja and I don't know
what would break if I did that (it's okay to hard-wired $SOONG_DELVE and
$SOONG_DELVE_PATH because those are only used for debugging and
$SOONG_OUTDIR turned out to be superfluous)
Test: Presubmits.
Change-Id: Idbfd9976c4b270bc3e5a8926c8c760a8534596cf
Replace the individual matches and deps return values with a GlobResult
struct. Use the GlobResult to create the file list file in GlobWithDepFile
(used by bpglob) and in the glob singleton, as the generated files must
match exactly so that soong_build is not rerun.
Bug: 159845846
Test: glob_test.go
Change-Id: I2159cc9d85f388073198eac7456e5bf43e813096
This makes it possible to debug the primary builder without manually
adding the respective flags to the compilation command lines.
Test: "m nothing", "m nothing" with debugging, then "m nothing" again
Change-Id: I7d403af48528a94e5e0033d9a5c1fa9981100cfd
It's replaced with a flag in bootstrap.Args. It's a little bit ugly that
bootstrap.Main also gained a new Boolean flag in the process. The plan
is to delete that method in favor of directly calling RunBlueprint().
Test: "m nothing" (presubmits take a full workday these days)
Change-Id: Iae3421ae517a90570259b6278f18c3614bb1bb73
In addition to the command line interface, a new struct is created that
holds all the information Blueprint needs to do its job.
Test: Presubmits.
Change-Id: I0d6c924e09e069b3c1ae2000722efb52fd561e79
The end goal of this exercise is to remove all mutable global state so
that multiple Blueprint instances can be run in the same process.
This is done by plumbing the necessary data through the configuration.
Test: Presubmit.
Change-Id: I040a461ae17705258b882b3a44740a4dd5027a2c
Some of these were introduced in aosp/1610784 and some others were
present since a good while.
Test: Manual: "m nothing"
Change-Id: I7d5cb56ca2d87cb439354fd1b298c3b8d97d665d
- Add the -N -l command line arguments to Microfactory
- Pass the the Delve arguments to the primary builder on the command line
Test: Manual.
Change-Id: I4034f2d48b3e40d9863529053715183dc3dce1f5
This is so that soong_build doesn't know the current directory, thereby
avoiding one way to leak information.
Test: Manual.
Change-Id: I976a8663b11d99c5382726487102df10043c5a61
ninjaWriter repeatedly called io.WriteString() on its writer, which
does a type assertion every time. Replace its io.Writer with an
io.StringWriter and call WriteString on it directly.
Test: ninja_writer_test.go
Change-Id: Ie073d996a319190242bf6a00af07a13a60d078b5
This allows exiting bootstrap directly before writing ninja files.
This facilitates shorter runtime on integrations which do processing
which do not require ninja file output.
Test: Manually verified on Soong integration use case which involves
running bootstrap twice in a single program; stopping before ninja
output reduces runtime by ~20s, or ~11%.
Some anonymous nested properties are missing from property structs,
since setting the property to anonymous is to allow future filtering,
there is no issue if we cannot find the struct.
test: go bpdoc tests
test: m soong_docs
The primary builder may want to create variants of bootstrap
modules if they need to fit in to the primary builder's dependency
graph. Enable arbitrary variants of bootstrap modules by only
running the module's actions on the primary variant and then
copying the result to any other variants that exist.
Test: m checkbuild
Change-Id: I24b97771bb11faeacab4079ed8cf69aef59da140
* Update bpdocs to filter nested properties by tag
Prior to this change nested properties tagged with `blueprint:"mutated"`
are erroneously included in documentation, this corrects that behavior
and adds testing to verify.
Tested: go test bpdocs tests
Change-Id: I822c9a98276634d2f584d8709e83003824cdffd5
This makes a minimal difference on my AOSP tree (which is on a fast NVME
drive), but it saves ~10-15% of the time in our primary builder
(soong_build) on larger trees on slower drives.
We're always going to run the glob tool at least once outside of the
primary builder, and ninja only loads depfiles created after running the
tool, so this should be a no-op change.
We still need to write the list file (if it has changed), since that's
the file we're adding as a dependency of the primary builder, so the
effective timestamp must be earlier than the build.ninja timestamp.
Change-Id: I872b7581da50e7f2089daa7e248ca05b6703f019
This reverts commit 63085f9e7c.
It breaks the blueprint tests, and we aren't use it yet, so revert
it for now.
Change-Id: I1ac81371342285e8d57baf15fe3a223d4140c6cf
Emit the rules to build blueprint tests even if runGoTests is not
set, and add a phony rule "blueprint_tests" to run them. This will
allow Soong to stop running the tests at the beginning of every build
but still run them as part of checkbuild or with a manual
`m blueprint_tests`.
Bug: 156428456
Test: m
Test: m blueprint_tests
Change-Id: If293a0757766d3046e78bf230a1825f15adc68fd
Otherwise we'll re-run the primary builder every time this is used. This
triggered my new ninja checks that verify that all commands write their
outputs.
Change-Id: Ic92d1ea57203b953da8025a170b24e6420c5436c
The filter tag is unused, replaced with FilterPropertyStruct to
generate a new type at runtime that only contains the filtered
fields.
Test: unpack_test.go
Change-Id: Id91cf99290832094d05426f3263279836f0fea73
Sandboxing the primary builder caused the glob filelists to not
be written because they were using a relative path, causing
primary builder reruns on the second build.
Also report errors when writing the filelist files.
Test: m checkbuild
Change-Id: Id1706560d04c85f00f829cfb714967bb8600626f
Globs produce a filelist using restat rules to avoid primary builder
reruns. On the first execution of the primary builder the filelists
are written directly by the primary builder, so that the second
execution can restat them all but avoid rerunning the primary builder.
This wasn't working because the glob filelists were written after
the build.ninja file, but the build.ninja file depends on the
filelists. Switching the order avoids the extra rerun.
Test: m nothing && m nothing
Change-Id: Ia8e0924598220d4ff34235907a8de2e3e03632df
Move LoadHooks from Blueprint and run them during ParseBlueprintsFiles.
This will allow them to add scoped module types that are visible to the
rest of the Blueprints file. Requires passing the config object to
ParseBlueprintsFiles.
Test: all blueprint tests
Change-Id: Ia2a2c9a0223d5458bfd48bd22ebed0fdbd0156c6
Don't write to $buildDir/.bootstrap in the ninja file generated for
the main stage.
Test: examine main stage build.ninja file
Change-Id: I9c6183c9a434ff11d44cdd4f9844eae192ebfe03
The go compile command in go 1.13 needs -p main instead of
-p $pkgPath when compiling binaries. This also works for older
go releases.
Also turn on travis for go 1.13.
Change-Id: I45ce185aa44cfa01a7933cc6e52290a85c851641
The primary builder should always be running on its own with no
other processes running in parallel, so put it in the console
pool so that it can print to stdout without buffering.
Bug: 80165685
Test: build with print in Soong
Change-Id: If34ecdb5fa18de7e47c4cd6965d551c504850176
This change fixes a bug where the `blueprint:mutated` tag was ignored
when used for struct or interface properties. It also fixes a dormant
ExcludeByTag func bug.
Test: m soong_docs, properties_test.go, bpdoc_test.go
Change-Id: I926de2aa203ec552ced897174e9b78e817701a7d
bpglob is only used as a helper to check if the primary builder
needs to rerun due to the results of a glob changing. A recent
change to glob support in pathtools made a glob format that was
accidentally previously accepted into an error. If the bad
syntax was used in the most recent primary builder run, and
then an incremental build is performed after picking up the
change that made the syntax invalid, then bpglob will attempt
to rerun before the primary builder, see the now-invalid syntax,
and fail. This will prevent the primary builder from rerunning,
which would have updated the bpglob rule with a corrected glob
syntax (or failed in the primary builder if the Blueprints file
still had the invalid glob syntax).
Avoid exiting with an error in bpglob. Instead, write the
error to the output file along with a timestamp so that it is
always dirty, forcing the primary builder to rerun.
Bug: 129411151
Test: m nothing
Change-Id: Ib680570c33662f3c0f1f72425d60a963ed841ba6
1. Apply preformatted text (<pre>...</pre>) handling logic to module
type text too. It used to be applied to property texts only.
2. Improve <pre> handling logic itself for better readability.
- Insert a blank line before <pre>.
- Prevent from ending <pre> blocks prematurely by checking if an
unindented line isn't just a blank line between indented lines.
Change-Id: Id40fa668d4c6781caf7ed140b2f40784cdeb8c35
1. Extract module type documentation.
2. Support primary builder customization of factory function to use for
documentation for each module type.
3. Change the ModuleType list order so that they are grouped by package.
This is basically minor refactoring + readability improvement done on
top of https://github.com/google/blueprint/pull/232.
Change-Id: If7413e5ac23486b85f18d02fb3ba288a38730c32
Writing the ninja file to a byte buffer causes a significant amount
of time to be spent in memmove when growing the byte slice. Write
the file directly to disk instead.
This also fixes some unhandled error warnings, which become more
likely when doing disk IO instead of byte buffer writes.
Change-Id: I5094e4c45cab4012713037f60c5a4fb00718f92e
In cases that we want to run blueprint-based builds in many
configurations to verify all of the logic works without errors, but
don't care about running the final ninja file, writing it out only
wastes time and disk space. So add a --empty-ninja-file option that
writes out an empty ninja file instead.
Our specific use case (Soong's build_test / multiproduct_kati) runs
Soong several hundred times for different configurations, and the ninja
files are around 1GB, which leads to several hundred gigabytes of disk
writes (and persistent use during incremental generation).
Change-Id: I0198dfb2f744ce22284c05d5214dac2ab5dc9700
The test package archive was being picked up through "-L" + testRoot,
but did not have a dependency on it. If the *_test.go file was
changed but not the implementation then the tests would not be
rerun.
Test: Add t.Fatal() to a *_test.go file and rebuild.
Change-Id: I32b8c7a0fa1d5b76b1242646887fe937e396e318
Linking tests needs the same dependencies that linking binaries
got in https://github.com/google/blueprint/pull/222.
Test: m checkbuild
Change-Id: I33330f3184b8c0fd2bc20b48736c20d6edeaea68
Many Android build tools are written in go, and changes to Soong or
Blueprint can cause them to rebuild. Almost everything in an
Android build is downstream of at least one of these tools, so
they all rebuild too.
Go binaries are static, so their contents will always change if any
of their dependencies change. Only update output files of go compile
and go link if the contents change, and enable restat for the rules.
Test: m soong_zip && m soong_zip
Test: m soong_zip && touch build/soong/zip/cmd/main.go && m soong_zip
Change-Id: I9267580f644b42b44d43fb0a2674dc5234f127e5
Instead of sometimes re-running minibp/the primary builder during the
next phase, run bpglob earlier to check dependencies.
We've run into issues where the environment is slightly different
between bootstrapping phase and the main build phase. It's also a
problem because our primary builder (Soong) exports information used by
another tool (Kati) that runs in between the bootstrapping phases and
the main phase. When Soong would run in the main phase, it could get out
of sync, and would require the build to be run again.
To do this, add a "subninja" include a build-globs.ninja file to each
build.ninja file. The first time, this will be an empty file, but we'll
always run minibp / the primary builder anyway. When the builder runs,
in addition to writing a dependency file, write out the
build-globs.ninja file with the rules to run bpglob.
Since bpglob may need to be run very early, before it would normally be
built, build it with microfactory.
Change-Id: I89fcd849a8729e892f163d40060ab90b5d4dfa5d
When passing "-c" to the Go compiler, any time this value changes, we'd
force all of the Go compiles to rebuild. This could trigger a
substantial portion of the tree to rebuild (anything that transitive
depends on a Go helper tool).
We're running into issues when moving output directories between
multiple GCE machines with different core counts, but are otherwise
identical. This could also hit users moving/mounting disks between
machines, though changes to other host tools can make an impact too.
On my 48-core machine, I get a ~15% benefit from going from -c 1 to -c
48, but also ~12% benefit from going from -c 1 to -c 8. So this will
still let us scale somewhat, but prevent rebuilds when transitioning
between machines that are more likely building Android.
In Go 1.10, runtime.GOROOT() will attempt to find the current location
of the go binaries, instead of using the GOROOT_FINAL that was baked
into the binaries. This means that GOROOT() will usually return an
absolute path.
We avoid putting absolute paths into the ninja file, since any change to
the paths would then cause all of the actions including it to rebuild.
Since we've got a decent number of build tools in Android using Go now,
this causes us to rebuild a decent portion of the tree.
Instead of passing the GOROOT around manually in a side channel, just
let the Go 1.10 detection do its thing, and always try to turn the
result into a relative path.
We're already going to load the .ninja_deps file, so we may as well use
it. These are the only two files in our build that don't.
Test: dump the .ninja_deps file, see the dependencies
Allow the primary builder to stop bootstrap between
ResolveDependencies and PrepareBuildActions so that it can
use the build graph to write customized documentation.
Change-Id: I67ee5e6e438a0c2c3ec844c4c453d223971e60a2
Move the docs to the main stage and make a phony rule to build
them. This will allow the primaray builder to choose not to
build them so that it can use its own rule instead.
Change-Id: Ib788f74717ccdcd85c2e5b6a581cc744d439341a
Separate the extraction of documentation-related info from writing
it out so that the primary builder can choose to write it out in
a customized format.
Change-Id: If2c91d6f13b8cea2e9e1cca75d8c2d684ad7f6df
Treat any property documentation lines that are indented as
preformatted.
Bug: 67909957
Test: manual inspection
Change-Id: Ib014bc301880eca02c60ed1692e49f07dca05fb3
Property documentation that contained text that looked like HTML
tags was getting passed through. Use html/template instead of
text/template so that it gets escaped automatically.
Bug: 67909957
Test: manual inspection
Change-Id: I2050c34c04d7be3cdacd5a76967a739f9f837e89
In case the primary builder wants to depend on a go binary by name,
expose an interface to find the real installed path of the binary.
Most of this change is changing it so that we're storing the install
path without any ninja references.
Change-Id: I873378ebdd47d8036d3cef5aab76f036846a3db1
Propagate extra ninja file deps through mutators so that they
can use ctx.AddNinjaFileDeps.
Test: blueprint tests
Change-Id: I299a0665c3f63b020ae345889fd78b91b91b215a
Having separate .ninja_log files doesn't work very well after
04d886b110
since the last successful timestamp is not recorded in the ninja log.
This was triggering the primary builder to be run twice on every build
since it was always considered out of date in the primary, then main
stages.
The two bootstrap stages already were unified when minibootstrap was
simplified, but this combines the main stage as well. This means that
we'll save some time running globs that are shared between minibp and
the primary builder as well (subdirs globs).
The cleanup code needed to be refactored, since the ninja_log will have
entries from the main stage as well. So instead of looking at everything
in the ninja log, take a string prefix to limit what we may clean.
Change-Id: I8d43168643aa8a0c6c3e48c6101eaa45d174bbe5
Now that packages and binaries are only built when needed, they need to
be defined in both stages in case something building in the main stage
needs something that was defined, but didn't need to be built in the
earlier stage.
Change-Id: I67704fba1d670e82218eba5ce91f0c01896557a2
Instead of forcing things to be built alongside the primary builder,
mark all packages and binaries as optional, unless they're explicitly
marked as default.
Change-Id: Ie53c90a01b41ce886d651f9aa4c4900d1edb0157
This duplicates building common blueprint go packages between minibp and
the primary builder, but drastically simplifies the first stage,
removing the need to check in a generated build.ninja.in.
Change-Id: I639a9637f1ed36d4210823ef276c0f7a064a83bd
In order to use certain syscalls, some source must only be compiled on
one of darwin or linux. There are also cases where it's simplest to have
two implementations of a method (or struct) instead of choosing an
implementation at runtime.
The standard go tools use _<GOOS>.go for this, but the explicit structs
map better into the current blueprint format (and you can still use
_darwin.go / _linux.go to maintain compatiblity with those tools).
Test: Define _linux.go / _darwin.go files, ensure only one is compiled.
Change-Id: I58bffefbf839ba52dea3d8e7d3ec67eaa721c463
We haven't been building the tests for any Go tools that get built
during the primary stage, since we haven't been passing -t to the
primary builder. So enable them there so that we test those tools.
Change-Id: Ic7720c0679419f9e1ba61c7e151f37eb5c7bc8dc
Add globbing with dependency checking to blueprint. Calling
ModuleContext.GlobWithDeps or SingletonContext.GlobWithDeps will return
a list of files that match the globs, while also adding efficient
dependencies to rerun the primary builder if a file that matches the
glob is added or removed.
Also use the globbing support for optional_subdirs=, subdirs= and build=
lines in blueprints files. The globbing slightly changes the behavior
of subname= lines, it no longer falls back to looking for a file called
"Blueprints". Blueprint files that need to include a subdirectory with
a different name can use build= instead of subdir= to directly include
them. The Blueprints file is updated to reset subname="Blueprints" in
case we want to include subdirectories inside blueprint and the primary
builder has changed the subname.
Also adds a new test directory that contains a simple primary builder
tree to test regeneration for globbing, and runs the tests in travis.
Change-Id: I83ce525fd11e11579cc58ba5308d01ca8eea7bc6
Before this change, we would ensure that tests for a package were run
before anything used that package, even just to compile. With this
change, we'll let other packages compile, but still run the tests in
dependency order, and before the binaries are finally installed.
This shrinks our initial build time from ~9 seconds to ~5 seconds on my
machine. Most of the improvements come from building the "real" package
and the test package in parallel.
Change-Id: I935670feecadacbf09b250bcaa1659a3599edc58
Other than cleaning up the code, the only change in behavior is to run
the docs build in parallel with the primary builder, saving about a
second on a Soong bootstrap (10s -> 9s).
Change-Id: Iaff51d6d1a37af842f294f12db11d5774d48d440
Soong (and soon, Blueprint) embeds the results of filesystem globs into
the build.ninja, and uses a helper tool to detect when the glob changes
and we need to rebuild the build.ninja file. This is more flexible than
listing the affected directories in the depfiles, since it can check to
see if a file we actually cared about was added, instead of re-running
anytime any file was added/removed (which happens on atomic file
modifications as well).
My recent bootstrap simplification broke this, since the helper rules
are in the main build.ninja, but I removed the ability to regenerate
that file from itself. So keep the current model, but add a rule into
the primary and main stages that allow themselves to re-run their
generator and write out a new build.ninja file if necessary. The actual
build rules of the generator aren't necessary, since we already built
them in the previous stage.
Change-Id: Ib51245920b2ec3ee5306c0c269361a5a8733caa8
