This is so that it doesn't need to abruptly call os.Exit(), denying
callers the opportunity to do cleanups.
Bug: 244730498
Test: Presubmits.
Change-Id: Ifd191d3bbbf2fdea2ca49e4fb552e5d1c557b80f
Many of the singletons are trivial and can be run in parallel, improving
the performance during analysis.
Bug: 281536768
Test: manual, presubmit
Change-Id: Ia63e4bc42a68e65dfa800e770982fa5826355fad
Add a VariableFuncContext argument to VariableFuncs that implements
GlobWithDeps. This will allow Soong to use optimized glob dependencies
in VariableFuncs.
Bug: 257079828
Test: no dependencies on directories in build.ninja.d
Change-Id: Iee5fc9c9ae3087662a5d1a3d7323a87462299205
It's in reality built by Microfactory quite early in the build which
takes care of incrementality in bpglob itself.
Test: Presubmits.
Change-Id: I066a6e85f48e9cfec4efd46cb1f615c68b806080
ToolDir is going to become unstable when switching between KatiEnabled
and Soong-only builds while the duplication between out/soong/host and
out/host is resolved. bpglob gets executed very early during bootstrap,
before the primary builder has run to update the paths to match the
current configuration. Move it into SoongOutDir() so that its path
is more stable.
The copy of bpglob in ToolDir is still used when bpglob is used by the
primary builder through bootstrap.GlobFile.
Bug: 204136549
Test: m nothing
Change-Id: Ida51997b6408d7c265f3ba343278e5e2968467d3
- Remove the bootstrap_go_binary module type in favor of
blueprint_go_binary
- Make it so that all the binaries used during bootstrapping are in the
host tool directory so that the bootstrap and main Ninja files are
consistent
Test: Presubmits.
Change-Id: I00744fec0474fbea89db6c0c0338856453138564
From now on, the responsibility of writing build-globs.ninja and adding
the glob list files to build.ninja.d will be the responsibility of
whoever calls Blueprint.
It's nicer this way because the bootstrap Blueprint invocation does not
need globs and Blueprint can be invoked multiple times from the
soong_build binary depending on what it does.
Regrettably, the little Blueprint invocation that generates
build-globs.ninja still requires a reference to a BootstrapConfig
because GlobRule references ${buildDir}.
Side cleanup: remove Args.DocFile, which was unused.
Test: Presubmits.
Change-Id: I37586966e3d6b7557a490ed1ae81ec596ec66ee5
There used to be two:
- The absolute source directory is now threaded through the code instead
- The command line arguments are moved to soong_build
Test: Presubmits.
Change-Id: Ie09de1a2862ec4ced2c7385198300359f805127d
Now that Blueprint has been merged with Soong, a large amount of code is
unnecessary.
Test: Presubmits.
Change-Id: I8d30d8b80f37fdc6a9db630a1792c57c7d6edb38
This makes the distinction between what the bootstrap .ninja file and
the main .ninja files do clearer: the bootstrap .ninja file includes the
build-globs.ninja file(s) which the main .ninja file writes.
This required removing the "default" statement from said bootstrap
.ninja file: now we can build more than one thing instead of just
build.ninja and thus soong_ui explicitly tells Ninja what to do instead.
On the side, stop catering for the use case where the bootstrap
Blueprint invocation uses globs: Soong's does not and that's our only
use case.
Test: Presubmits.
Change-Id: Icffce31928242cfe9cdab56b290a37598d32a58c
This enables bp2build to write its own build-globs.ninja file.
Test: soong integration tests
Fixes: 193650250
Change-Id: Icb35f3cf3c30929dab1b2c2e9b244321be30f967
There are a few cases that force all globs to be rerun at the beginning
of the build (changes to bpglob or dependencies, second build after a
clean build). The number of globs has gotten high enough that rerunning
them all can have significant overhead to start bpglob for each one.
Replace the per-glob bpglob invocations with sharded invocations using
1024 hash buckets.
Bug: 159845846
Test: glob_test.go
Test: m nothing && m nothing
Test: build/soong/bootstrap_test.sh
Change-Id: Ife1f7a03c8f6b25d1be01531425d8dc2c76d1ea0
bpglob is executed through the rules in build-globs.ninja to determine
whether soong_build needs to rerun. That means when the arguments
accepted by bpglob change it will be called with the old arguments,
then soong_build will rerun and update build-globs.ninja with the new
arguments.
To avoid having to maintain backwards compatibility with old arguments
across the transition, a version argument is used to detect the
transition in order to stop parsing arguments, touch the output file
and exit immediately. The version number in
pathtools.BPGlobArgumentVersion should be manually incremented when
the bpglob argument format changes.
If the version argument is not passed then a version mismatch is assumed.
Bug: 159845846
Test: build/soong/tests/bootstrap_test.sh
Change-Id: Id6742c7acc51af8d0d224d51e47bddea78e6e363
- Make the parsed command line arguments accessible from other packages
- Separate writing the depfile from writing the Ninja file
- Extract a method to compute the name of the file list file for globs
Test: Presubmits.
Change-Id: I4d0daf0f3776a9336da305be0108ebf70a314776
Replace the individual matches and deps return values with a GlobResult
struct. Use the GlobResult to create the file list file in GlobWithDepFile
(used by bpglob) and in the glob singleton, as the generated files must
match exactly so that soong_build is not rerun.
Bug: 159845846
Test: glob_test.go
Change-Id: I2159cc9d85f388073198eac7456e5bf43e813096
In addition to the command line interface, a new struct is created that
holds all the information Blueprint needs to do its job.
Test: Presubmits.
Change-Id: I0d6c924e09e069b3c1ae2000722efb52fd561e79
The end goal of this exercise is to remove all mutable global state so
that multiple Blueprint instances can be run in the same process.
This is done by plumbing the necessary data through the configuration.
Test: Presubmit.
Change-Id: I040a461ae17705258b882b3a44740a4dd5027a2c
This makes a minimal difference on my AOSP tree (which is on a fast NVME
drive), but it saves ~10-15% of the time in our primary builder
(soong_build) on larger trees on slower drives.
We're always going to run the glob tool at least once outside of the
primary builder, and ninja only loads depfiles created after running the
tool, so this should be a no-op change.
We still need to write the list file (if it has changed), since that's
the file we're adding as a dependency of the primary builder, so the
effective timestamp must be earlier than the build.ninja timestamp.
Change-Id: I872b7581da50e7f2089daa7e248ca05b6703f019
Sandboxing the primary builder caused the glob filelists to not
be written because they were using a relative path, causing
primary builder reruns on the second build.
Also report errors when writing the filelist files.
Test: m checkbuild
Change-Id: Id1706560d04c85f00f829cfb714967bb8600626f
Instead of sometimes re-running minibp/the primary builder during the
next phase, run bpglob earlier to check dependencies.
We've run into issues where the environment is slightly different
between bootstrapping phase and the main build phase. It's also a
problem because our primary builder (Soong) exports information used by
another tool (Kati) that runs in between the bootstrapping phases and
the main phase. When Soong would run in the main phase, it could get out
of sync, and would require the build to be run again.
To do this, add a "subninja" include a build-globs.ninja file to each
build.ninja file. The first time, this will be an empty file, but we'll
always run minibp / the primary builder anyway. When the builder runs,
in addition to writing a dependency file, write out the
build-globs.ninja file with the rules to run bpglob.
Since bpglob may need to be run very early, before it would normally be
built, build it with microfactory.
Change-Id: I89fcd849a8729e892f163d40060ab90b5d4dfa5d
This duplicates building common blueprint go packages between minibp and
the primary builder, but drastically simplifies the first stage,
removing the need to check in a generated build.ninja.in.
Change-Id: I639a9637f1ed36d4210823ef276c0f7a064a83bd
Add globbing with dependency checking to blueprint. Calling
ModuleContext.GlobWithDeps or SingletonContext.GlobWithDeps will return
a list of files that match the globs, while also adding efficient
dependencies to rerun the primary builder if a file that matches the
glob is added or removed.
Also use the globbing support for optional_subdirs=, subdirs= and build=
lines in blueprints files. The globbing slightly changes the behavior
of subname= lines, it no longer falls back to looking for a file called
"Blueprints". Blueprint files that need to include a subdirectory with
a different name can use build= instead of subdir= to directly include
them. The Blueprints file is updated to reset subname="Blueprints" in
case we want to include subdirectories inside blueprint and the primary
builder has changed the subname.
Also adds a new test directory that contains a simple primary builder
tree to test regeneration for globbing, and runs the tests in travis.
Change-Id: I83ce525fd11e11579cc58ba5308d01ca8eea7bc6
