Storing every string without ninja variable references through
simpleNinjaString costs 24 bytes and a heap allocation. 16 bytes
is used for the ninjaString.str string, 8 bytes for the
ninjaString.variables *[]variableReference. An additional 8 bytes
is used for the resulting pointer into the heap.
The vast majority of calls to simpleNinjaString originate in
blueprint.parseBuildParams, which converts all of the parameters
passed to ctx.Build into ninjaStrings. All together this was
allocating 1.575 GB of *ninjaString objects.
Add a parseNinjaOrSimpleStrings function that converts input strings
into ninjaStrings if they have ninja variable references, but also
returns a slice of plain strings for input strings without any ninja
variable references. That still results in 1.39 GB of allocations just
for the output string slice, so also add an optimization that reuses
the input string slice as the output slice if all of the strings had
no variable references.
Plumb the resulting strings through everywhere that the []*ninjaStrings
were used.
This reduces the total memory allocations inside
blueprint.parseBuildParams in my AOSP aosp_cf_x86_64_phone-userdebug
build from 3.337 GB to 1.786 GB.
Test: ninja_strings_test.go
Change-Id: I51bc138a2a6b1cc7383c7df0a483ccb067ffa02b
ninjaString is an interface, which uses 16 bytes of memory on top
of the size of the concrete type. A literalNinjaString is a string,
which is another 16 bytes for the string header for a total of 32
bytes. A varNinjaString is two slices, which are 24 bytes each
for the slice headers, for a total of 64 bytes. The slices contain
the first constant string, and then altenrating variable and string
parts of the ninjaString, resulting in 16 bytes plus 32 bytes per
variable.
This patch replaces the ninjaString interface with a *ninjaString
concrete struct type. The ninjaString struct is a string and a
pointer to a slice of variable references, for a total of 24 bytes.
ninjaStrings with no variable references (the equivalent of the old
literalNinjaString) have a nil slice, and now use 24 bytes instead
of 32 bytes.
ninjaStrings with variable references allocate a slice of variable
references that contain 32-bit start and end offsets and a Variable
interface, but reuse the original string and so avoid the extra
string headers, resulting in 24 bytes for the slice header, and
24 bytes per variable.
These savings reduce the peak memory usage averaged across 10 runs of
/bin/time -v build/soong/soong_ui.bash --make-mode nothing
on the internal master branch cf_x86_64_phone-userdebug build
from 50114842kB to 45577638kB, a savings of 4537204kB or 9%.
The new Benchmark_parseNinjaString shows savings in both time and
memory. Before:
Benchmark_parseNinjaString/constant/1-128 594251787 2.006 ns/op 0 B/op 0 allocs/op
Benchmark_parseNinjaString/constant/10-128 21191347 65.57 ns/op 16 B/op 1 allocs/op
Benchmark_parseNinjaString/constant/100-128 9983748 130.2 ns/op 112 B/op 1 allocs/op
Benchmark_parseNinjaString/constant/1000-128 2632527 445.1 ns/op 1024 B/op 1 allocs/op
Benchmark_parseNinjaString/variable/1-128 2964896 419.4 ns/op 176 B/op 4 allocs/op
Benchmark_parseNinjaString/variable/10-128 1807341 670.6 ns/op 192 B/op 7 allocs/op
Benchmark_parseNinjaString/variable/100-128 1000000 1092 ns/op 352 B/op 7 allocs/op
Benchmark_parseNinjaString/variable/1000-128 300649 3773 ns/op 1584 B/op 7 allocs/op
Benchmark_parseNinjaString/variables/1-128 2858432 441.6 ns/op 176 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/2-128 2360505 513.4 ns/op 208 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/3-128 1867136 635.6 ns/op 240 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/4-128 1584045 752.1 ns/op 272 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/5-128 1338189 885.8 ns/op 304 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/10-128 1000000 1468 ns/op 464 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/100-128 88768 12895 ns/op 3712 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/1000-128 8972 133627 ns/op 32896 B/op 4 allocs/op
After:
Benchmark_parseNinjaString/constant/1-128 584600864 2.004 ns/op 0 B/op 0 allocs/op
Benchmark_parseNinjaString/constant/10-128 19274581 64.84 ns/op 16 B/op 1 allocs/op
Benchmark_parseNinjaString/constant/100-128 9017640 127.6 ns/op 112 B/op 1 allocs/op
Benchmark_parseNinjaString/constant/1000-128 2630797 453.0 ns/op 1024 B/op 1 allocs/op
Benchmark_parseNinjaString/variable/1-128 3460422 347.0 ns/op 136 B/op 4 allocs/op
Benchmark_parseNinjaString/variable/10-128 2103404 519.9 ns/op 152 B/op 7 allocs/op
Benchmark_parseNinjaString/variable/100-128 1315778 906.5 ns/op 312 B/op 7 allocs/op
Benchmark_parseNinjaString/variable/1000-128 354812 3284 ns/op 1544 B/op 7 allocs/op
Benchmark_parseNinjaString/variables/1-128 3386868 361.5 ns/op 136 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/2-128 2675594 456.9 ns/op 160 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/3-128 2344670 520.0 ns/op 192 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/4-128 1919482 648.1 ns/op 208 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/5-128 1560556 723.9 ns/op 240 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/10-128 1000000 1169 ns/op 352 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/100-128 116738 10168 ns/op 2800 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/1000-128 10000 105646 ns/op 24688 B/op 4 allocs/op
Bug: 286423944
Test: ninja_strings_test.go
Test: out/soong/build*.ninja is the same before and after this change
Change-Id: I1ecffbaccb0d0469a41fa31255c1b17311e01687
Many of the singletons are trivial and can be run in parallel, improving
the performance during analysis.
Bug: 281536768
Test: manual, presubmit
Change-Id: Ia63e4bc42a68e65dfa800e770982fa5826355fad
Without this, we cannot correctly join action and module graph
information as we cannot distinguish variants on action graph.
Test: m json-module-graph
Change-Id: If7379845fc865d8a150f3995df6bf601456a71c3
Soong analyzes the entire source tree even though not every lunch target
needs to know about every module. For example, OEM sources can be
ignored for cuttlefish products. This functionality allows blueprint to
ignore a list of undesired directories.
Bug: 269457150
Change-Id: Icbbf8f3b66813ad639a7ebd27b1a3ec153cbf269
To build ninja hint including output path from module name
Test: m --ninja_weight_source=soong
Bug: 273282046
Change-Id: Ibb94c2c4efef4a6dedc973cbb90625231845d42e
This reduces the extract_phonys (now deduplicate_order_only_deps)
event's time from ~1.9s to ~1.5s on aosp-master, and from ~5.3s to ~4.6s
on internal master.
It does so by making keyForPhonyCandidate be based on a hash instead
of joining all the deps together. Having a hash allows us to also use
it as the name of the phony target, which simplifies the code a little.
Bug: None (original cl introducing extractPhonys also didn't have a bug)
Test: go tests
Change-Id: I2ff6e4614f19ccbfe99112ea7ae1ea33cd1df21b
1. scan if any set of order-only deps are repeated
2. if so extract them as a phony output to be shared
Test: m libc
Bug: NA
Change-Id: I0689111b97bbbd1f3b26650e8ae2e0a4ffb5085e
Collect additional metrics for individual mutators in order to
understand impact of individual mutators.
Test: m nothing
Change-Id: Ic3ecb1e79a79dd665c9f60d29f0dfd3732481c2d
This introduces a new `blueprint_package_includes` module type. If
present, other blueprint modules in that file will be analyzed
if and only if the requested include tags are met
example syntax:
```
Android.bp
blueprint_packgage_includes {
match_all: ["tag1", "tag2", ...],
}
other_module1 {...}
other_module2 {...}
```
other_module1 and other_module2 will not be analyzed unless
tag1,tag2, ... are set
This also adds a new object of type `IncludeTags` to the Context object,
which is a container for these string keys.
Test: In build/blueprint, go test ./
Test: TH
Change-Id: I79de0d7da3224a5b2025c27a5137f39d00c7382e
Add a VariableFuncContext argument to VariableFuncs that implements
GlobWithDeps. This will allow Soong to use optimized glob dependencies
in VariableFuncs.
Bug: 257079828
Test: no dependencies on directories in build.ninja.d
Change-Id: Iee5fc9c9ae3087662a5d1a3d7323a87462299205
These are more limited than bottom-up or top-down mutators but in
exchange have some pleasant properties:
- "variant not found" errors are impossible
- The logic is pleasantly split into multiple, mostly orthogonal
parts
- Theoretically, if every mutator is refactored like this, they
make it possible to partially cache the module graph
- Are quite close to a "configuration transition" in Bazel.
Bug: 231370928
Test: Presubmits.
Change-Id: Idcdb66b5ea75c0d2838f527aaa988df3b12553d8
This allows for a bazel-invocation hook in mixed builds, which allows
for mixed builds to take place in only a single pass, greatly improving
its performance and complexity.
Test: Conjunction with build/soong CL
Change-Id: If89fb56830b4eb06d3263d6ca6da7b285e7ba315
Blueprint already stores the module that called CreateModule, let's
surface it.
Test: m json-module-graph and view results
Change-Id: Ie67bf8e431d764eb23727c90200f57c9de4ab053
In conjunction with soong/build changes, this materialized runtime
metrics for various soong_build events.
Test: Manually verified materialized protos for bp2build, mixed builds,
and legacy build.
Change-Id: Ia92403605e3063028dbf6a1ded8449c190b9e63e
Variant names are part of the name of the generated Ninja rule.
toNinjaName needs to be applied to it as well.
Test: presubmit
Change-Id: I4fb908b6824179440b1c207240bad59dc85009fa
This was the only one in the source tree.
Side cleanup: remove some dead code that I assume comes from the time
where Blueprint files had to specify what subdirectories other Blueprint
files are in.
Test: Presubmits.
Change-Id: If84c4e85bc5516f30da97c1be29b56e50dddb3c4
These are just out/ and out/soong/ and the old names were quite
confusing.
Test: Presubmits.
Merged-In: Idd9ce3c38a259faabcc56f0cd3fdac8b289123b1
Merged-In: I334ab40c668d2a94536f3e63d5f1fa0b401388ac
Change-Id: Ib7c568c6a97701f2240c5e3f0f2ce67397819ac0
Propagate the ninja file deps from a LoadHook to the build.ninja.d
file.
Bug: 188547846
Test: next CL
Change-Id: If8176474b5094ee40d07df12f5da79a906ce7290
Use moduleInfo.String() to print dependency cycle errors so they
contain the variant.
Test: Test_parallelVisit
Change-Id: I2bddaa35c8abb57c42b4c424e861361a8813d588
parallelVisit supports mutating the dependency graph while it is being
visited by proceeding until there are no modules with their dependencies
satisfied, and then checking if there are modules that haven't been
visited yet. If so, it assumes there was a newly introduced dependency
cycle and tries to find it to return as an error.
Finding the dependency cycle could traverse outside of the cycle.
If the dependency cycle occurs near the bottom of the dependency graph,
that traversal could be both long and wide, leading to very long
runtimes.
Memoize traversed modules that were not found to be part of the
dependency cycle to prevent repeated traversals.
Fixes: 186572387
Test: introduce cycle into libc, m nothing
Test: Test_parallelVisit
Change-Id: I38d0749dbedffbe8a39e433d97fbe08486451321
There are a few cases that force all globs to be rerun at the beginning
of the build (changes to bpglob or dependencies, second build after a
clean build). The number of globs has gotten high enough that rerunning
them all can have significant overhead to start bpglob for each one.
Replace the per-glob bpglob invocations with sharded invocations using
1024 hash buckets.
Bug: 159845846
Test: glob_test.go
Test: m nothing && m nothing
Test: build/soong/bootstrap_test.sh
Change-Id: Ife1f7a03c8f6b25d1be01531425d8dc2c76d1ea0
Fix detecting cycles in parallelVisit when the first alphabetical
module is not part of the cycle, but depends on the cycle. Instead of
starting from the first alphabetical module, check every module in a
determinsitic order and return the first time a cycle is found.
Test: Test_parallelVisit
Change-Id: I03726f838ec42975251088ba75158103940115c2
