It's not super useful because of Go's stupid restriction on not letting
you reflect private fields.
Test: m nothing
Change-Id: I1f73f06f5e64eaf4adce58a667b98b131aede53f
blueprint.variationMap.Equal was responsible for 11.5% of allocations
and 2.2% of allocated memory. reflect.DeepEquals is an expensive way
to compare a map. variationMap.subsetOf is already iterating through
the elements of the map to compare them, replace equal with a check that
the maps are the same length and then reuse subsetOf to check that the
have the same keys and values.
Test: SOONG_PROFILE_MEM=/tmp/mem.pprof m nothing
Change-Id: Ifb7cdf612e5455fd2f412488b7f94416c4e70c54
keyForPhonyCandidate was using sha256, which is a crypto hash and
unnecessarily expensive for this use case. hash/maphash would be
much faster because it implements WriteString and so doesn't cause
an extra allocation to copy to a byte slice for every write, but it
insists on randomizing the seed, which makes it unsuitable for writing
to the build.ninja file. Use hash/fnv instead, and use unsafe to
write strings to the hash to avoid the extra allocation.
Also replace the manually rolled parallelism with the existing
parallelVisit, which will reuse goroutines and limit the parallelism
to a useful value.
The hash could collide, and using a 64-bit hash makes that more
likely, so also check the full contents to make sure they are really
equal.
Cuts 1 second off Soong analysis time.
Test: SOONG_PROFILE_MEM=/tmp/mem.pprof m nothing
Change-Id: I4d1292cb158cfc5823a0f4d8b4aeac1d0b10230e
When setProvider() is called, hash the provider and store the hash in
the module. Then after the build is done, hash all the providers again
and compare the hashes. It's an error if they don't match.
Also add a flag to control it in case this check gets slow as we convert
more things to providers. However right now it's fast (unnoticable
in terms of whole seconds) so just have the flag always enabled.
Bug: 322069292
Test: m nothing
Change-Id: Ie4e806a6a9f20542ffcc7439eef376d3fb6a98ca
The description of TransitionMutators says that "the outgoing transition
should not take the properties of the dependency into account, only those
of the module that depends on it. For this reason, the dependency is not
even passed into it as an argument." However, OutgoingTransitionContext
was returing the dependency from ctx.Module(), not the parent. This
didn't matter for the only existing TransitionMutator, as it only used
the module to get a constant value.
Test: sanitize_test.go
Change-Id: I1ce5b3144787f57be4d50e95f0c923da9b2b079f
Force a resort of the module groups before running singletons
so that two singletons running in parallel don't cause a data
race when they trigger a resort in VisitAllModules.
Test: go test -race ./...
Change-Id: Iec041cec08c33c56787aadbde6a1b2b619815142
memoizeFullName was added to variables, rules and pools as an
optimization to prevent recomputing the full name repeatedly,
but the storage of variables, rules and pools are generally global
and not tied to the Context. When running multiple tests in
parallel there will be multiple Context objects all trying to
update the memoized names on the global variables, causing a data
race.
Package names were previously memoized via a pkgNames map stored
on the Context. Expand pkgNames to a nameTracker object that
contains maps for packages, variables, rules and pools, and replace
calls to fullName with calls through nameTracker.
Test: context_test.go
Change-Id: I15040b85a6d1dab9ab3cff44f227b22985acee18
Now that nothing calls *Context.*Provider directly, make the blueprint
methods return a nil any interface instead of the zero value that was
constructed via reflection. The type-safe wrappers will return a
zero value that can be constructed without any reflection or copying.
Bug: 316410648
Test: provider_test.go
Change-Id: I0abde5bacab9964a83f03c1644b51295a6c34d0b
Using generics for the providers API allows a type to be associated
with a ProviderKey, resulting in a type-safe API without that doesn't
require runtime type assertions by every caller.
Unfortunately, Go does not allow generic types in methods, only in
functions [1]. This prevents a type-safe API on ModuleContext, and
requires moving the API to be functions that take a ModuleContext as
a parameter.
[1] https://go.googlesource.com/proposal/+/refs/heads/master/design/43651-type-parameters.md#no-parameterized-methods)
Bug: 316410648
Test: provider_test.go
Change-Id: Ide91de9f2a2a7d075b05e287c7cc86b395db0edb
Storing every string without ninja variable references through
simpleNinjaString costs 24 bytes and a heap allocation. 16 bytes
is used for the ninjaString.str string, 8 bytes for the
ninjaString.variables *[]variableReference. An additional 8 bytes
is used for the resulting pointer into the heap.
The vast majority of calls to simpleNinjaString originate in
blueprint.parseBuildParams, which converts all of the parameters
passed to ctx.Build into ninjaStrings. All together this was
allocating 1.575 GB of *ninjaString objects.
Add a parseNinjaOrSimpleStrings function that converts input strings
into ninjaStrings if they have ninja variable references, but also
returns a slice of plain strings for input strings without any ninja
variable references. That still results in 1.39 GB of allocations just
for the output string slice, so also add an optimization that reuses
the input string slice as the output slice if all of the strings had
no variable references.
Plumb the resulting strings through everywhere that the []*ninjaStrings
were used.
This reduces the total memory allocations inside
blueprint.parseBuildParams in my AOSP aosp_cf_x86_64_phone-userdebug
build from 3.337 GB to 1.786 GB.
Test: ninja_strings_test.go
Change-Id: I51bc138a2a6b1cc7383c7df0a483ccb067ffa02b
ninjaString is an interface, which uses 16 bytes of memory on top
of the size of the concrete type. A literalNinjaString is a string,
which is another 16 bytes for the string header for a total of 32
bytes. A varNinjaString is two slices, which are 24 bytes each
for the slice headers, for a total of 64 bytes. The slices contain
the first constant string, and then altenrating variable and string
parts of the ninjaString, resulting in 16 bytes plus 32 bytes per
variable.
This patch replaces the ninjaString interface with a *ninjaString
concrete struct type. The ninjaString struct is a string and a
pointer to a slice of variable references, for a total of 24 bytes.
ninjaStrings with no variable references (the equivalent of the old
literalNinjaString) have a nil slice, and now use 24 bytes instead
of 32 bytes.
ninjaStrings with variable references allocate a slice of variable
references that contain 32-bit start and end offsets and a Variable
interface, but reuse the original string and so avoid the extra
string headers, resulting in 24 bytes for the slice header, and
24 bytes per variable.
These savings reduce the peak memory usage averaged across 10 runs of
/bin/time -v build/soong/soong_ui.bash --make-mode nothing
on the internal master branch cf_x86_64_phone-userdebug build
from 50114842kB to 45577638kB, a savings of 4537204kB or 9%.
The new Benchmark_parseNinjaString shows savings in both time and
memory. Before:
Benchmark_parseNinjaString/constant/1-128 594251787 2.006 ns/op 0 B/op 0 allocs/op
Benchmark_parseNinjaString/constant/10-128 21191347 65.57 ns/op 16 B/op 1 allocs/op
Benchmark_parseNinjaString/constant/100-128 9983748 130.2 ns/op 112 B/op 1 allocs/op
Benchmark_parseNinjaString/constant/1000-128 2632527 445.1 ns/op 1024 B/op 1 allocs/op
Benchmark_parseNinjaString/variable/1-128 2964896 419.4 ns/op 176 B/op 4 allocs/op
Benchmark_parseNinjaString/variable/10-128 1807341 670.6 ns/op 192 B/op 7 allocs/op
Benchmark_parseNinjaString/variable/100-128 1000000 1092 ns/op 352 B/op 7 allocs/op
Benchmark_parseNinjaString/variable/1000-128 300649 3773 ns/op 1584 B/op 7 allocs/op
Benchmark_parseNinjaString/variables/1-128 2858432 441.6 ns/op 176 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/2-128 2360505 513.4 ns/op 208 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/3-128 1867136 635.6 ns/op 240 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/4-128 1584045 752.1 ns/op 272 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/5-128 1338189 885.8 ns/op 304 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/10-128 1000000 1468 ns/op 464 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/100-128 88768 12895 ns/op 3712 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/1000-128 8972 133627 ns/op 32896 B/op 4 allocs/op
After:
Benchmark_parseNinjaString/constant/1-128 584600864 2.004 ns/op 0 B/op 0 allocs/op
Benchmark_parseNinjaString/constant/10-128 19274581 64.84 ns/op 16 B/op 1 allocs/op
Benchmark_parseNinjaString/constant/100-128 9017640 127.6 ns/op 112 B/op 1 allocs/op
Benchmark_parseNinjaString/constant/1000-128 2630797 453.0 ns/op 1024 B/op 1 allocs/op
Benchmark_parseNinjaString/variable/1-128 3460422 347.0 ns/op 136 B/op 4 allocs/op
Benchmark_parseNinjaString/variable/10-128 2103404 519.9 ns/op 152 B/op 7 allocs/op
Benchmark_parseNinjaString/variable/100-128 1315778 906.5 ns/op 312 B/op 7 allocs/op
Benchmark_parseNinjaString/variable/1000-128 354812 3284 ns/op 1544 B/op 7 allocs/op
Benchmark_parseNinjaString/variables/1-128 3386868 361.5 ns/op 136 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/2-128 2675594 456.9 ns/op 160 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/3-128 2344670 520.0 ns/op 192 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/4-128 1919482 648.1 ns/op 208 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/5-128 1560556 723.9 ns/op 240 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/10-128 1000000 1169 ns/op 352 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/100-128 116738 10168 ns/op 2800 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/1000-128 10000 105646 ns/op 24688 B/op 4 allocs/op
Bug: 286423944
Test: ninja_strings_test.go
Test: out/soong/build*.ninja is the same before and after this change
Change-Id: I1ecffbaccb0d0469a41fa31255c1b17311e01687
Many of the singletons are trivial and can be run in parallel, improving
the performance during analysis.
Bug: 281536768
Test: manual, presubmit
Change-Id: Ia63e4bc42a68e65dfa800e770982fa5826355fad
Without this, we cannot correctly join action and module graph
information as we cannot distinguish variants on action graph.
Test: m json-module-graph
Change-Id: If7379845fc865d8a150f3995df6bf601456a71c3
Soong analyzes the entire source tree even though not every lunch target
needs to know about every module. For example, OEM sources can be
ignored for cuttlefish products. This functionality allows blueprint to
ignore a list of undesired directories.
Bug: 269457150
Change-Id: Icbbf8f3b66813ad639a7ebd27b1a3ec153cbf269
To build ninja hint including output path from module name
Test: m --ninja_weight_source=soong
Bug: 273282046
Change-Id: Ibb94c2c4efef4a6dedc973cbb90625231845d42e
This reduces the extract_phonys (now deduplicate_order_only_deps)
event's time from ~1.9s to ~1.5s on aosp-master, and from ~5.3s to ~4.6s
on internal master.
It does so by making keyForPhonyCandidate be based on a hash instead
of joining all the deps together. Having a hash allows us to also use
it as the name of the phony target, which simplifies the code a little.
Bug: None (original cl introducing extractPhonys also didn't have a bug)
Test: go tests
Change-Id: I2ff6e4614f19ccbfe99112ea7ae1ea33cd1df21b
1. scan if any set of order-only deps are repeated
2. if so extract them as a phony output to be shared
Test: m libc
Bug: NA
Change-Id: I0689111b97bbbd1f3b26650e8ae2e0a4ffb5085e
Collect additional metrics for individual mutators in order to
understand impact of individual mutators.
Test: m nothing
Change-Id: Ic3ecb1e79a79dd665c9f60d29f0dfd3732481c2d
This introduces a new `blueprint_package_includes` module type. If
present, other blueprint modules in that file will be analyzed
if and only if the requested include tags are met
example syntax:
```
Android.bp
blueprint_packgage_includes {
match_all: ["tag1", "tag2", ...],
}
other_module1 {...}
other_module2 {...}
```
other_module1 and other_module2 will not be analyzed unless
tag1,tag2, ... are set
This also adds a new object of type `IncludeTags` to the Context object,
which is a container for these string keys.
Test: In build/blueprint, go test ./
Test: TH
Change-Id: I79de0d7da3224a5b2025c27a5137f39d00c7382e
Add a VariableFuncContext argument to VariableFuncs that implements
GlobWithDeps. This will allow Soong to use optimized glob dependencies
in VariableFuncs.
Bug: 257079828
Test: no dependencies on directories in build.ninja.d
Change-Id: Iee5fc9c9ae3087662a5d1a3d7323a87462299205
These are more limited than bottom-up or top-down mutators but in
exchange have some pleasant properties:
- "variant not found" errors are impossible
- The logic is pleasantly split into multiple, mostly orthogonal
parts
- Theoretically, if every mutator is refactored like this, they
make it possible to partially cache the module graph
- Are quite close to a "configuration transition" in Bazel.
Bug: 231370928
Test: Presubmits.
Change-Id: Idcdb66b5ea75c0d2838f527aaa988df3b12553d8
This allows for a bazel-invocation hook in mixed builds, which allows
for mixed builds to take place in only a single pass, greatly improving
its performance and complexity.
Test: Conjunction with build/soong CL
Change-Id: If89fb56830b4eb06d3263d6ca6da7b285e7ba315
Blueprint already stores the module that called CreateModule, let's
surface it.
Test: m json-module-graph and view results
Change-Id: Ie67bf8e431d764eb23727c90200f57c9de4ab053
