Many of the singletons are trivial and can be run in parallel, improving
the performance during analysis.
Bug: 281536768
Test: manual, presubmit
Change-Id: Ia63e4bc42a68e65dfa800e770982fa5826355fad
With this close() missing, especially with large ninja files, soong paid
the runtime cost at arbitrary times during the rest of its execution (or
during process cleanup).
Test: Manual runs of `m nothing` and verifying timing metrics
Change-Id: Ibdfbf7371cfc4c58c485e76f96b8e9b9ad900972
This introduces a new `blueprint_package_includes` module type. If
present, other blueprint modules in that file will be analyzed
if and only if the requested include tags are met
example syntax:
```
Android.bp
blueprint_packgage_includes {
match_all: ["tag1", "tag2", ...],
}
other_module1 {...}
other_module2 {...}
```
other_module1 and other_module2 will not be analyzed unless
tag1,tag2, ... are set
This also adds a new object of type `IncludeTags` to the Context object,
which is a container for these string keys.
Test: In build/blueprint, go test ./
Test: TH
Change-Id: I79de0d7da3224a5b2025c27a5137f39d00c7382e
This is to build bazel targets that are nearing readiness for prod mode.
Bug: 254265047
Test: m nothing
Test: m nothing --bazel-mode-dev
Test: m nothing --bazel-mode-staging
Change-Id: Ie8336ec4ad70c0c9bd6d74f3b0a278c85fd5df2d
also minor refactoring to limit var scopes
Test: m nothing and ensure ninja files are generated
Bug: N/A
Change-Id: If7bdede4e54687955896b03ab11ef771855d2225
This allows for a bazel-invocation hook in mixed builds, which allows
for mixed builds to take place in only a single pass, greatly improving
its performance and complexity.
Test: Conjunction with build/soong CL
Change-Id: If89fb56830b4eb06d3263d6ca6da7b285e7ba315
In conjunction with soong/build changes, this materialized runtime
metrics for various soong_build events.
Test: Manually verified materialized protos for bp2build, mixed builds,
and legacy build.
Change-Id: Ia92403605e3063028dbf6a1ded8449c190b9e63e
PrimaryBuilderExtra flags was only used from soong_ui and has nothing to
do with Blueprint files or modules and Args contained a lot of members
that were likewise only used for plumbing withing soong_ui or for
command line processing for soong_build.
Test: Presubmits.
Change-Id: Ibf009e446b4044309a331592a88a4aa86b4801e3
Its value is a function of the call site, so it doesn't make a lot of
sense to plumb it through the configuration.
Test: Presubmits.
Change-Id: I5081650818664f283ebc5dc04daef90c26759353
It was confusing because there are two concepts with the same name:
thebootstrapping config special-cased in the bootstrap modules /
singletons and the "global" config.
This change folds the first into the second.
A further next step would be to split the Go parts and the Ninja parts
(subninjas + primary builder invocation) apart, but at some point, I
should stop gold-plating this.
Test: Presubmits.
Change-Id: Id341dee664a9d2527535f4f9980e5f0d10139eef
- Remove the bootstrap_go_binary module type in favor of
blueprint_go_binary
- Make it so that all the binaries used during bootstrapping are in the
host tool directory so that the bootstrap and main Ninja files are
consistent
Test: Presubmits.
Change-Id: I00744fec0474fbea89db6c0c0338856453138564
What's gone:
- TopFile (it's always Android.bp)
- GeneratingPrimaryBuilder (we don't make that distinction anymore)
Also rename BuildDir and NinjaBuildDir to SoongOutDir and OutDir,
respectively, to be consistent with the rest of the code.
Test: Presubmits.
Change-Id: I55ecb02a59fe68bf5674395a108181ea7557b21f
It is not used anymore; Soong assumes that the depfile is just the name
of the output file with a ".d" suffix.
Test: Presubmits.
Change-Id: Ifbf722dcdffc8420e12a032e23ab791a6be5a736
These are just out/ and out/soong/ and the old names were quite
confusing.
Test: Presubmits.
Merged-In: Idd9ce3c38a259faabcc56f0cd3fdac8b289123b1
Merged-In: I334ab40c668d2a94536f3e63d5f1fa0b401388ac
Change-Id: Ib7c568c6a97701f2240c5e3f0f2ce67397819ac0
From now on, the responsibility of writing build-globs.ninja and adding
the glob list files to build.ninja.d will be the responsibility of
whoever calls Blueprint.
It's nicer this way because the bootstrap Blueprint invocation does not
need globs and Blueprint can be invoked multiple times from the
soong_build binary depending on what it does.
Regrettably, the little Blueprint invocation that generates
build-globs.ninja still requires a reference to a BootstrapConfig
because GlobRule references ${buildDir}.
Side cleanup: remove Args.DocFile, which was unused.
Test: Presubmits.
Change-Id: I37586966e3d6b7557a490ed1ae81ec596ec66ee5
There used to be two:
- The absolute source directory is now threaded through the code instead
- The command line arguments are moved to soong_build
Test: Presubmits.
Change-Id: Ie09de1a2862ec4ced2c7385198300359f805127d
This is not necessary anymore now that Blueprint doesn't have a separate
existence and Soong's documentation generation uses a completely
separate code path.
Test: Presubmits.
Change-Id: Ibc9e57379c5f8ee9a06d90fbe81fac314f01ee92
This makes the distinction between what the bootstrap .ninja file and
the main .ninja files do clearer: the bootstrap .ninja file includes the
build-globs.ninja file(s) which the main .ninja file writes.
This required removing the "default" statement from said bootstrap
.ninja file: now we can build more than one thing instead of just
build.ninja and thus soong_ui explicitly tells Ninja what to do instead.
On the side, stop catering for the use case where the bootstrap
Blueprint invocation uses globs: Soong's does not and that's our only
use case.
Test: Presubmits.
Change-Id: Icffce31928242cfe9cdab56b290a37598d32a58c
This enables bp2build to write its own build-globs.ninja file.
Test: soong integration tests
Fixes: 193650250
Change-Id: Icb35f3cf3c30929dab1b2c2e9b244321be30f967
An empty glob file would be created inside the source tree if
soong_build is invoked directly (instead of via soong_ui). This logic is
now moved to soong/ui/build to prevent this behavior
Bug: 187194795
Test: Ran the following command locally for the target
art-target-arm:git_master-art
```
. ./build/envsetup.sh && lunch armv8-eng &&
art/tools/buildbot-build.sh --target
```
Before change glob file is created in $TOP/.bootstrap
After change glob file is created in $TOP/out/soong/.bootstrap
Change-Id: I6745996c940d790d1f1f387ca08aafc6a8a669df
There are a few cases that force all globs to be rerun at the beginning
of the build (changes to bpglob or dependencies, second build after a
clean build). The number of globs has gotten high enough that rerunning
them all can have significant overhead to start bpglob for each one.
Replace the per-glob bpglob invocations with sharded invocations using
1024 hash buckets.
Bug: 159845846
Test: glob_test.go
Test: m nothing && m nothing
Test: build/soong/bootstrap_test.sh
Change-Id: Ife1f7a03c8f6b25d1be01531425d8dc2c76d1ea0
Args.BuildDir was accidentally hard-wired to "." in one of my previous
changes, and since Blueprint is not really independent of Soong anymore,
it's probably better to fix it in a way that doesn't involve putting
back the almost-but-not-completely redundant "-b" command line argument
to the primary builder invocation.
Test: Presubmits.
Change-Id: I76d064a99be16cd812cb47a6c448fe44b51893dc
The only thing that was done with it was to append it to the list of
dependencies returned, which can just as well be done by the caller.
Test: Presubmits.
Change-Id: Ibedf024de411a5881f19ed0c137a91b0898f1269
- Make the parsed command line arguments accessible from other packages
- Separate writing the depfile from writing the Ninja file
- Extract a method to compute the name of the file list file for globs
Test: Presubmits.
Change-Id: I4d0daf0f3776a9336da305be0108ebf70a314776
This is accomplished by plumbing PrimaryBuilderInvocation in from
bootstrap.Args.
The confusing removal of "primaryBuilderExtraFlags" from documentation
generation is because it used to (erroneously) refer to the func defined
in command.go and its removal apparently doesn't break anything. I
suspect that this code path is dead within Android, but we'll need to
wait until May to remove it.
Test: Presubmits.
Change-Id: I8763579b82bd79d5ee0f233edfd48327282e401c
This makes it possible to invoke the primary builder more than once in a
single Ninja file.
This required adding arguments that were hard-wired in the build.ninja
rule to extraArgs and adding a "set of primary builder invocations"
member to bootstrap.Config . The only command line arguments that remain
hard-wired are those which are expected to be relevant to every
invocation of the primary builder.
As a welcome side effect, I was able to remove a number of random
environment variables that were so far special-cased ($SOONG_DELVE,
$SOONG_DELVE_PATH and $SOONG_OUTDIR). I was also able to move writing
the empty stub ninja glob file to command.go from the bowels of the
Blueprint machinery.
In theory, $TOP and $BUILDER could be removed, too, but this would
require hard-coding the value of $TOP into build.ninja and I don't know
what would break if I did that (it's okay to hard-wired $SOONG_DELVE and
$SOONG_DELVE_PATH because those are only used for debugging and
$SOONG_OUTDIR turned out to be superfluous)
Test: Presubmits.
Change-Id: Idbfd9976c4b270bc3e5a8926c8c760a8534596cf
It's replaced with a flag in bootstrap.Args. It's a little bit ugly that
bootstrap.Main also gained a new Boolean flag in the process. The plan
is to delete that method in favor of directly calling RunBlueprint().
Test: "m nothing" (presubmits take a full workday these days)
Change-Id: Iae3421ae517a90570259b6278f18c3614bb1bb73
In addition to the command line interface, a new struct is created that
holds all the information Blueprint needs to do its job.
Test: Presubmits.
Change-Id: I0d6c924e09e069b3c1ae2000722efb52fd561e79
The end goal of this exercise is to remove all mutable global state so
that multiple Blueprint instances can be run in the same process.
This is done by plumbing the necessary data through the configuration.
Test: Presubmit.
Change-Id: I040a461ae17705258b882b3a44740a4dd5027a2c
ninjaWriter repeatedly called io.WriteString() on its writer, which
does a type assertion every time. Replace its io.Writer with an
io.StringWriter and call WriteString on it directly.
Test: ninja_writer_test.go
Change-Id: Ie073d996a319190242bf6a00af07a13a60d078b5
This allows exiting bootstrap directly before writing ninja files.
This facilitates shorter runtime on integrations which do processing
which do not require ninja file output.
Test: Manually verified on Soong integration use case which involves
running bootstrap twice in a single program; stopping before ninja
output reduces runtime by ~20s, or ~11%.
Otherwise we'll re-run the primary builder every time this is used. This
triggered my new ninja checks that verify that all commands write their
outputs.
Change-Id: Ic92d1ea57203b953da8025a170b24e6420c5436c
Globs produce a filelist using restat rules to avoid primary builder
reruns. On the first execution of the primary builder the filelists
are written directly by the primary builder, so that the second
execution can restat them all but avoid rerunning the primary builder.
This wasn't working because the glob filelists were written after
the build.ninja file, but the build.ninja file depends on the
filelists. Switching the order avoids the extra rerun.
Test: m nothing && m nothing
Change-Id: Ia8e0924598220d4ff34235907a8de2e3e03632df
Move LoadHooks from Blueprint and run them during ParseBlueprintsFiles.
This will allow them to add scoped module types that are visible to the
rest of the Blueprints file. Requires passing the config object to
ParseBlueprintsFiles.
Test: all blueprint tests
Change-Id: Ia2a2c9a0223d5458bfd48bd22ebed0fdbd0156c6
