Storing every string without ninja variable references through
simpleNinjaString costs 24 bytes and a heap allocation. 16 bytes
is used for the ninjaString.str string, 8 bytes for the
ninjaString.variables *[]variableReference. An additional 8 bytes
is used for the resulting pointer into the heap.
The vast majority of calls to simpleNinjaString originate in
blueprint.parseBuildParams, which converts all of the parameters
passed to ctx.Build into ninjaStrings. All together this was
allocating 1.575 GB of *ninjaString objects.
Add a parseNinjaOrSimpleStrings function that converts input strings
into ninjaStrings if they have ninja variable references, but also
returns a slice of plain strings for input strings without any ninja
variable references. That still results in 1.39 GB of allocations just
for the output string slice, so also add an optimization that reuses
the input string slice as the output slice if all of the strings had
no variable references.
Plumb the resulting strings through everywhere that the []*ninjaStrings
were used.
This reduces the total memory allocations inside
blueprint.parseBuildParams in my AOSP aosp_cf_x86_64_phone-userdebug
build from 3.337 GB to 1.786 GB.
Test: ninja_strings_test.go
Change-Id: I51bc138a2a6b1cc7383c7df0a483ccb067ffa02b
Generate the rules to build and run the blueprint tests whether or
not running the tests during bootstrap is enabled, and only add them
as validation dependencies if running the tests is enabled. Export
the outputs of the tests as a phony target for checkbuild to depend on.
Bug: 269296618
Test: m nothing
Test: aninja -t path checkbuild out/host/linux-x86/bin/go/soong-java/test/test.passed
Change-Id: I09cd20d802bed5a659f3f36e87128d4281dfcfb0
Some go tests depend on testData, but do not declare it explicitly. In
preparation for the stricter sandboxing constratints imposed by Bazel,
introduce this property.
This will be a no-op in Soong for now. But bp2build will convert it into
`data` property of go_test
Test: go build ./blueprint/bootstrap
Bug: 288491147
Change-Id: I0e1e7941db1efe6f1488936d9bae7e8d295b88ba
These two module types will be special-cased in bp2build. To enable this
special-casing, export from the package alongwith getter methods to
access its properties.
Test: go build ./bootstrap
Change-Id: I9ef7ac4b6331fec99713296f8f78f614f3536847
This is so that it doesn't need to abruptly call os.Exit(), denying
callers the opportunity to do cleanups.
Bug: 244730498
Test: Presubmits.
Change-Id: Ifd191d3bbbf2fdea2ca49e4fb552e5d1c557b80f
These two module types will be special-cased in bp2build generation
logic in build/soong. To write bp2build tests, create a helper function
to register these two module types
Test: go build ./bootstrap
Change-Id: If6abd8c8911a525bf6841b199d8ce204941d7bcb
Many of the singletons are trivial and can be run in parallel, improving
the performance during analysis.
Bug: 281536768
Test: manual, presubmit
Change-Id: Ia63e4bc42a68e65dfa800e770982fa5826355fad
With this close() missing, especially with large ninja files, soong paid
the runtime cost at arbitrary times during the rest of its execution (or
during process cleanup).
Test: Manual runs of `m nothing` and verifying timing metrics
Change-Id: Ibdfbf7371cfc4c58c485e76f96b8e9b9ad900972
This introduces a new `blueprint_package_includes` module type. If
present, other blueprint modules in that file will be analyzed
if and only if the requested include tags are met
example syntax:
```
Android.bp
blueprint_packgage_includes {
match_all: ["tag1", "tag2", ...],
}
other_module1 {...}
other_module2 {...}
```
other_module1 and other_module2 will not be analyzed unless
tag1,tag2, ... are set
This also adds a new object of type `IncludeTags` to the Context object,
which is a container for these string keys.
Test: In build/blueprint, go test ./
Test: TH
Change-Id: I79de0d7da3224a5b2025c27a5137f39d00c7382e
Add a VariableFuncContext argument to VariableFuncs that implements
GlobWithDeps. This will allow Soong to use optimized glob dependencies
in VariableFuncs.
Bug: 257079828
Test: no dependencies on directories in build.ninja.d
Change-Id: Iee5fc9c9ae3087662a5d1a3d7323a87462299205
This is to build bazel targets that are nearing readiness for prod mode.
Bug: 254265047
Test: m nothing
Test: m nothing --bazel-mode-dev
Test: m nothing --bazel-mode-staging
Change-Id: Ie8336ec4ad70c0c9bd6d74f3b0a278c85fd5df2d
also minor refactoring to limit var scopes
Test: m nothing and ensure ninja files are generated
Bug: N/A
Change-Id: If7bdede4e54687955896b03ab11ef771855d2225
This allows for a bazel-invocation hook in mixed builds, which allows
for mixed builds to take place in only a single pass, greatly improving
its performance and complexity.
Test: Conjunction with build/soong CL
Change-Id: If89fb56830b4eb06d3263d6ca6da7b285e7ba315
In conjunction with soong/build changes, this materialized runtime
metrics for various soong_build events.
Test: Manually verified materialized protos for bp2build, mixed builds,
and legacy build.
Change-Id: Ia92403605e3063028dbf6a1ded8449c190b9e63e
This is used to make GODEBUG=asyncpreemptoff=1 only be set in debug mode
so that production is not affected.
Test: Presubmits.
Change-Id: Idb15ad20bb74ba3f896699715e3208c1287a5e82
This makes debugging with Delve much faster, sometimes by an order of
magnitude because Delve does not have to deal with the many SIGURG
signals that are used to stop preempted goroutines this way.
This could in theory result in a performance loss, but running
`rm -f out/soong/build.ninja; time m nothing` five times demonstrated no
change in the execution time.
Test: Presubmits + performance testing as described above.
Change-Id: I438564bcaf70d2a50147cf54cdfac6dcfe925ce5
It's in reality built by Microfactory quite early in the build which
takes care of incrementality in bpglob itself.
Test: Presubmits.
Change-Id: I066a6e85f48e9cfec4efd46cb1f615c68b806080
Presently, entries in the Soong reference docs that
share the same type are condensed into a single entry.
This is very unintuitive, so this change removes that
functionality.
Fixes: 204441523
Test: bpdoc_test.go
Test: build and compare Soong docs
Change-Id: Ic03891a8a7a29b5f7ee58c01b2fa05a0c27e0a2b
ToolDir is going to become unstable when switching between KatiEnabled
and Soong-only builds while the duplication between out/soong/host and
out/host is resolved. bpglob gets executed very early during bootstrap,
before the primary builder has run to update the paths to match the
current configuration. Move it into SoongOutDir() so that its path
is more stable.
The copy of bpglob in ToolDir is still used when bpglob is used by the
primary builder through bootstrap.GlobFile.
Bug: 204136549
Test: m nothing
Change-Id: Ida51997b6408d7c265f3ba343278e5e2968467d3
struct types defined using reflection have empty PkgPaths(), due to
which they were not added to the generated bpdocs. This CL allows
us to generate bpdocs for such struct types.
For each field in the struct, the algorithm checks the following and
creates a PropertyStruct object accordingly
1. If field is a primitive type (Base condition)
2. If field is another exported struct (Base condition)
3. If field is another struct created using reflection (Recurse)
Test: m soong_docs
Test: java_sdk_library_import before
https://ci.android.com/builds/submitted/7710820/linux/latest/view/java.html#java_sdk_library_import
and after
https://spandandas.users.x20web.corp.google.com/docs/java.html#java_sdk_library_import
Bug: 172797653
Change-Id: I0349e405fd290d427fa0b38a109b4212aace50c6
PrimaryBuilderExtra flags was only used from soong_ui and has nothing to
do with Blueprint files or modules and Args contained a lot of members
that were likewise only used for plumbing withing soong_ui or for
command line processing for soong_build.
Test: Presubmits.
Change-Id: Ibf009e446b4044309a331592a88a4aa86b4801e3
This is because docgen will be called from the bootstrap Ninja file with
cwd == /.
Side cleanup: remove an unused template.
Test: Presubmits.
Change-Id: Ia5ff10247becbabc7a76c21f4ed05d654a7b20c3
Its value is a function of the call site, so it doesn't make a lot of
sense to plumb it through the configuration.
Test: Presubmits.
Change-Id: I5081650818664f283ebc5dc04daef90c26759353
It was confusing because there are two concepts with the same name:
thebootstrapping config special-cased in the bootstrap modules /
singletons and the "global" config.
This change folds the first into the second.
A further next step would be to split the Go parts and the Ninja parts
(subninjas + primary builder invocation) apart, but at some point, I
should stop gold-plating this.
Test: Presubmits.
Change-Id: Id341dee664a9d2527535f4f9980e5f0d10139eef
- Remove the bootstrap_go_binary module type in favor of
blueprint_go_binary
- Make it so that all the binaries used during bootstrapping are in the
host tool directory so that the bootstrap and main Ninja files are
consistent
Test: Presubmits.
Change-Id: I00744fec0474fbea89db6c0c0338856453138564
What's gone:
- TopFile (it's always Android.bp)
- GeneratingPrimaryBuilder (we don't make that distinction anymore)
Also rename BuildDir and NinjaBuildDir to SoongOutDir and OutDir,
respectively, to be consistent with the rest of the code.
Test: Presubmits.
Change-Id: I55ecb02a59fe68bf5674395a108181ea7557b21f
It is not used anymore; Soong assumes that the depfile is just the name
of the output file with a ".d" suffix.
Test: Presubmits.
Change-Id: Ifbf722dcdffc8420e12a032e23ab791a6be5a736
These are just out/ and out/soong/ and the old names were quite
confusing.
Test: Presubmits.
Merged-In: Idd9ce3c38a259faabcc56f0cd3fdac8b289123b1
Merged-In: I334ab40c668d2a94536f3e63d5f1fa0b401388ac
Change-Id: Ib7c568c6a97701f2240c5e3f0f2ce67397819ac0
This makes it possible to share work between the bootstrap and the main
Ninja file.
As a side fix, remove the reference to $srcDir from moduleSrcDir() since
that is hardwired to "." anyway.
Test: Presubmits.
Change-Id: Ia4a251dd34273ec3e0436ac7a2fbafe4ef46c73f
installPath was included by androidmk.go in the Soong-generated
Android.mk file, which means that the Ninja variable reference
`$buildDir` passe through Kati, with unpredictable results.
`$buildDir` is still used in a few places, but those are not problematic
because they are not copied to Android.mk (they could probably be
removed, but it's not trivial enough to be done as a drive-by fix)
Test: Presubmits.
Change-Id: I6a5ea8b1679001ff685d263310cb061d3a6a992e
