TransitionMutators can always return "" from OutgoingVariation in
order to implement the equivalent of CreateLocalVariations - a
variation that doesn't affect which variation of dependencies is
selected. The same need applies when creating the dependency after
the TransitionMutator. Call OutgoingVariation on the parent module
when adding new dependencies unless the variation was explicitly
requested.
Bug: 319288033
Test: TestPostTransitionDeps
Flag: NONE
Change-Id: Iadb72a5a3f19579a6c1d34e3c7169ee9e2b53210
TransitionMutators sometimes apply to one variant of a module but
not to another. Deleting the variation when all variants were
untouched is insufficient if a post-mutator dependency is being added
on a variant that was untouched. Delete the variation whenever
no matching incoming variant was found.
Bug: 319288033
Test: TestPostTransitionDeps
Flag: NONE
Change-Id: Ib073f6ec3090d09e4798b6f9ca3061ec5d58d722
ReplaceDependencies[If] currently replaces dependencies on a given
module with the current module. It expects to find a variant of the
given module that has the exact same variations as the current module.
That was sometimes handled via aliases, but TransitionMutators don't
support aliases, they use IncomingTransition to rewrite the variation
instead.
In all current usages of ReplaceDependencies[If], the given module
is also a direct dependency of the current module. Instead of looking
for the exact same variations, look for the variant that is a dependency
of the current module.
Bug: 319288033
Test: all soong tests pass
Flag: NONE
Change-Id: I3e33111322040b187f6e951554366ccdcaf1bc11
When adding a dependency on a module that has already had variants
created by a TransitionMutator, ignore any variation added by the
TransitionMutator to the current module if the TransitionMutator
didn't apply to the target module. This requires passing in the
list of variations requested by the AddVariationDependencies call
so that explicitly requested variations are not ignored.
This fixes TestAndroidTest_FixTestConfig when the override mutator
is converted to a TransitionMutator, as it can't add a dependency from
the "foo_test" module with an empty override variation to the
"foo" module that has no override variation.
Bug: 319288033
Test: TestAndroidTest_FixTestConfig
Flag: NONE
Change-Id: If0fca7adc28cd702f953885aef440a0805ac017c
Adding a dependency on a module that has already had a TransitionMutator
run on it may require adjusting the variation name based on the results
of IncomingTranstion. Store the variants that existed before the
TransitionMutator ran, find one that is a subset of the requested
variant, and call TranstionMutator.IncomingTransition to update the
value.
Bug: 319288033
Test: TestPostTransitionDeps
Change-Id: I690357f9792401a3edbc5ae9fdcb666495954fbc
* changes:
Store mutator in mutatorContext instead of name
Explicitly delete old logicModule entries from Context.moduleInfo
Keep logicModule for obsolete variants
Add Provider to transition contexts
Cache outgoing transitions
Use maps and slices packages
TransitionMutators will need access to the original *mutatorInfo,
store it in mutatorContext instead of the name, and update all
accesses to the name to go through the *mutatorInfo.
Bug: 319288033
Test: go test ./...
Change-Id: I4bb1a17f44e55b23dd70708c9a5fde2ae80ce154
Creating variants has an optimization to reuse the logicModule of the
original variant as the logicModule of the first new variant. Using
the same logicModule meant that updating the Context.moduleInfo map
from logicModules to *moduleInfo would overwrite the old entry for
the original variant with the new entry for the first variant.
TransitionMutators will need to keep the original logicModule for
later use, which will require disabling the optimization that reuses
it. When the first variant is added to the map it no longer overwrites
the original variant. Excplicitly track the original logicModule and
remove it from the map if necessary.
Bug: 319288033
Test: go test ./...
Change-Id: I05ffdf6d7ce60508f6d9e9657c21032f2c4f5d9c
When splitting a module into variants the original *moduleInfo was
left in place but with logicModule set to nil as a marker that the
variant was obsolete and any incoming dependencies needed to be resolved
onto one of the newly split variants. A change to allow TransitionMutator
IncomingTransition calls when adding dependencies later will require
keeping the obsolete *moduleInfo and logicModule around so that
IncomingTransition can be called on it. so use an explicit boolean
to mark the module as obsolete instead of clearing logicModule.
Bug: 319288033
Test: go test ./...
Change-Id: I5dc201f43442aa07ac1b858240c675bab3782b55
Allow TransitionMutator OutgoingTransition and IncomingTransition
methods to access Providers from the current and dependency module
respectively.
Bug: 319288033
Test: none
Change-Id: If16ee7980ee0b8f4abaf3c112738fca3f13ab61c
TransitionMutators were calling OutgoingTransition and IncomingTransition
twice, once to determine all the necessary variations for each module,
and then again when creating the variations to resolve dependencies.
Store the final variation needed for each dependency during the first
computation, and use it to resolve the dependency variation in the second
pass.
Bug: 319288033
Test: all soong tests
Change-Id: I2e42c0d69efbfcff915267c484c18554b857e0b6
filterPropertyStructFields should treat configurable properties as
leaf nodes and not recurse into them. Configurable properties can be
made arch_variant, at least while we transition to full use of
Configurable.
Bug: 323382414
Test: soong tests when changing enabled to be configurable
Change-Id: Ib22dd3797f69e912dab0e0c3c1030d6c1f04db7e
...and pass property name to ConfigurableEvaluator.
When trying to convert Enabled to a configurable property, I enountered
a bunch of different context types that all had different error methods.
OtherModulePropertyErrorf is an error method that can be implemented
by all contexts.
Bug: 323382414
Test: m nothing --no-skip-soong-tests
Change-Id: I38b2a545c0ee8d23281cd88bc397f79f39835bc4
ctx.ModuleErrorf for the main blueprint config object doesn't
report an error, it just returns it.
If go had something like C++'s `[[nodiscard]]` or rust's `#[must_use]`
that would be useful here, but the issue has been open since 2017:
https://github.com/golang/go/issues/20803
Test: Presubmits
Change-Id: I72709e6c5466d55f5c0f3fe51a25c29df0826271
It's not super useful because of Go's stupid restriction on not letting
you reflect private fields.
Test: m nothing
Change-Id: I1f73f06f5e64eaf4adce58a667b98b131aede53f
blueprint.variationMap.Equal was responsible for 11.5% of allocations
and 2.2% of allocated memory. reflect.DeepEquals is an expensive way
to compare a map. variationMap.subsetOf is already iterating through
the elements of the map to compare them, replace equal with a check that
the maps are the same length and then reuse subsetOf to check that the
have the same keys and values.
Test: SOONG_PROFILE_MEM=/tmp/mem.pprof m nothing
Change-Id: Ifb7cdf612e5455fd2f412488b7f94416c4e70c54
keyForPhonyCandidate was using sha256, which is a crypto hash and
unnecessarily expensive for this use case. hash/maphash would be
much faster because it implements WriteString and so doesn't cause
an extra allocation to copy to a byte slice for every write, but it
insists on randomizing the seed, which makes it unsuitable for writing
to the build.ninja file. Use hash/fnv instead, and use unsafe to
write strings to the hash to avoid the extra allocation.
Also replace the manually rolled parallelism with the existing
parallelVisit, which will reuse goroutines and limit the parallelism
to a useful value.
The hash could collide, and using a 64-bit hash makes that more
likely, so also check the full contents to make sure they are really
equal.
Cuts 1 second off Soong analysis time.
Test: SOONG_PROFILE_MEM=/tmp/mem.pprof m nothing
Change-Id: I4d1292cb158cfc5823a0f4d8b4aeac1d0b10230e
When setProvider() is called, hash the provider and store the hash in
the module. Then after the build is done, hash all the providers again
and compare the hashes. It's an error if they don't match.
Also add a flag to control it in case this check gets slow as we convert
more things to providers. However right now it's fast (unnoticable
in terms of whole seconds) so just have the flag always enabled.
Bug: 322069292
Test: m nothing
Change-Id: Ie4e806a6a9f20542ffcc7439eef376d3fb6a98ca
The description of TransitionMutators says that "the outgoing transition
should not take the properties of the dependency into account, only those
of the module that depends on it. For this reason, the dependency is not
even passed into it as an argument." However, OutgoingTransitionContext
was returing the dependency from ctx.Module(), not the parent. This
didn't matter for the only existing TransitionMutator, as it only used
the module to get a constant value.
Test: sanitize_test.go
Change-Id: I1ce5b3144787f57be4d50e95f0c923da9b2b079f
Force a resort of the module groups before running singletons
so that two singletons running in parallel don't cause a data
race when they trigger a resort in VisitAllModules.
Test: go test -race ./...
Change-Id: Iec041cec08c33c56787aadbde6a1b2b619815142
memoizeFullName was added to variables, rules and pools as an
optimization to prevent recomputing the full name repeatedly,
but the storage of variables, rules and pools are generally global
and not tied to the Context. When running multiple tests in
parallel there will be multiple Context objects all trying to
update the memoized names on the global variables, causing a data
race.
Package names were previously memoized via a pkgNames map stored
on the Context. Expand pkgNames to a nameTracker object that
contains maps for packages, variables, rules and pools, and replace
calls to fullName with calls through nameTracker.
Test: context_test.go
Change-Id: I15040b85a6d1dab9ab3cff44f227b22985acee18
Now that nothing calls *Context.*Provider directly, make the blueprint
methods return a nil any interface instead of the zero value that was
constructed via reflection. The type-safe wrappers will return a
zero value that can be constructed without any reflection or copying.
Bug: 316410648
Test: provider_test.go
Change-Id: I0abde5bacab9964a83f03c1644b51295a6c34d0b
Using generics for the providers API allows a type to be associated
with a ProviderKey, resulting in a type-safe API without that doesn't
require runtime type assertions by every caller.
Unfortunately, Go does not allow generic types in methods, only in
functions [1]. This prevents a type-safe API on ModuleContext, and
requires moving the API to be functions that take a ModuleContext as
a parameter.
[1] https://go.googlesource.com/proposal/+/refs/heads/master/design/43651-type-parameters.md#no-parameterized-methods)
Bug: 316410648
Test: provider_test.go
Change-Id: Ide91de9f2a2a7d075b05e287c7cc86b395db0edb
Storing every string without ninja variable references through
simpleNinjaString costs 24 bytes and a heap allocation. 16 bytes
is used for the ninjaString.str string, 8 bytes for the
ninjaString.variables *[]variableReference. An additional 8 bytes
is used for the resulting pointer into the heap.
The vast majority of calls to simpleNinjaString originate in
blueprint.parseBuildParams, which converts all of the parameters
passed to ctx.Build into ninjaStrings. All together this was
allocating 1.575 GB of *ninjaString objects.
Add a parseNinjaOrSimpleStrings function that converts input strings
into ninjaStrings if they have ninja variable references, but also
returns a slice of plain strings for input strings without any ninja
variable references. That still results in 1.39 GB of allocations just
for the output string slice, so also add an optimization that reuses
the input string slice as the output slice if all of the strings had
no variable references.
Plumb the resulting strings through everywhere that the []*ninjaStrings
were used.
This reduces the total memory allocations inside
blueprint.parseBuildParams in my AOSP aosp_cf_x86_64_phone-userdebug
build from 3.337 GB to 1.786 GB.
Test: ninja_strings_test.go
Change-Id: I51bc138a2a6b1cc7383c7df0a483ccb067ffa02b
ninjaString is an interface, which uses 16 bytes of memory on top
of the size of the concrete type. A literalNinjaString is a string,
which is another 16 bytes for the string header for a total of 32
bytes. A varNinjaString is two slices, which are 24 bytes each
for the slice headers, for a total of 64 bytes. The slices contain
the first constant string, and then altenrating variable and string
parts of the ninjaString, resulting in 16 bytes plus 32 bytes per
variable.
This patch replaces the ninjaString interface with a *ninjaString
concrete struct type. The ninjaString struct is a string and a
pointer to a slice of variable references, for a total of 24 bytes.
ninjaStrings with no variable references (the equivalent of the old
literalNinjaString) have a nil slice, and now use 24 bytes instead
of 32 bytes.
ninjaStrings with variable references allocate a slice of variable
references that contain 32-bit start and end offsets and a Variable
interface, but reuse the original string and so avoid the extra
string headers, resulting in 24 bytes for the slice header, and
24 bytes per variable.
These savings reduce the peak memory usage averaged across 10 runs of
/bin/time -v build/soong/soong_ui.bash --make-mode nothing
on the internal master branch cf_x86_64_phone-userdebug build
from 50114842kB to 45577638kB, a savings of 4537204kB or 9%.
The new Benchmark_parseNinjaString shows savings in both time and
memory. Before:
Benchmark_parseNinjaString/constant/1-128 594251787 2.006 ns/op 0 B/op 0 allocs/op
Benchmark_parseNinjaString/constant/10-128 21191347 65.57 ns/op 16 B/op 1 allocs/op
Benchmark_parseNinjaString/constant/100-128 9983748 130.2 ns/op 112 B/op 1 allocs/op
Benchmark_parseNinjaString/constant/1000-128 2632527 445.1 ns/op 1024 B/op 1 allocs/op
Benchmark_parseNinjaString/variable/1-128 2964896 419.4 ns/op 176 B/op 4 allocs/op
Benchmark_parseNinjaString/variable/10-128 1807341 670.6 ns/op 192 B/op 7 allocs/op
Benchmark_parseNinjaString/variable/100-128 1000000 1092 ns/op 352 B/op 7 allocs/op
Benchmark_parseNinjaString/variable/1000-128 300649 3773 ns/op 1584 B/op 7 allocs/op
Benchmark_parseNinjaString/variables/1-128 2858432 441.6 ns/op 176 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/2-128 2360505 513.4 ns/op 208 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/3-128 1867136 635.6 ns/op 240 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/4-128 1584045 752.1 ns/op 272 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/5-128 1338189 885.8 ns/op 304 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/10-128 1000000 1468 ns/op 464 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/100-128 88768 12895 ns/op 3712 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/1000-128 8972 133627 ns/op 32896 B/op 4 allocs/op
After:
Benchmark_parseNinjaString/constant/1-128 584600864 2.004 ns/op 0 B/op 0 allocs/op
Benchmark_parseNinjaString/constant/10-128 19274581 64.84 ns/op 16 B/op 1 allocs/op
Benchmark_parseNinjaString/constant/100-128 9017640 127.6 ns/op 112 B/op 1 allocs/op
Benchmark_parseNinjaString/constant/1000-128 2630797 453.0 ns/op 1024 B/op 1 allocs/op
Benchmark_parseNinjaString/variable/1-128 3460422 347.0 ns/op 136 B/op 4 allocs/op
Benchmark_parseNinjaString/variable/10-128 2103404 519.9 ns/op 152 B/op 7 allocs/op
Benchmark_parseNinjaString/variable/100-128 1315778 906.5 ns/op 312 B/op 7 allocs/op
Benchmark_parseNinjaString/variable/1000-128 354812 3284 ns/op 1544 B/op 7 allocs/op
Benchmark_parseNinjaString/variables/1-128 3386868 361.5 ns/op 136 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/2-128 2675594 456.9 ns/op 160 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/3-128 2344670 520.0 ns/op 192 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/4-128 1919482 648.1 ns/op 208 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/5-128 1560556 723.9 ns/op 240 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/10-128 1000000 1169 ns/op 352 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/100-128 116738 10168 ns/op 2800 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/1000-128 10000 105646 ns/op 24688 B/op 4 allocs/op
Bug: 286423944
Test: ninja_strings_test.go
Test: out/soong/build*.ninja is the same before and after this change
Change-Id: I1ecffbaccb0d0469a41fa31255c1b17311e01687
Many of the singletons are trivial and can be run in parallel, improving
the performance during analysis.
Bug: 281536768
Test: manual, presubmit
Change-Id: Ia63e4bc42a68e65dfa800e770982fa5826355fad
Without this, we cannot correctly join action and module graph
information as we cannot distinguish variants on action graph.
Test: m json-module-graph
Change-Id: If7379845fc865d8a150f3995df6bf601456a71c3
Soong analyzes the entire source tree even though not every lunch target
needs to know about every module. For example, OEM sources can be
ignored for cuttlefish products. This functionality allows blueprint to
ignore a list of undesired directories.
Bug: 269457150
Change-Id: Icbbf8f3b66813ad639a7ebd27b1a3ec153cbf269
To build ninja hint including output path from module name
Test: m --ninja_weight_source=soong
Bug: 273282046
Change-Id: Ibb94c2c4efef4a6dedc973cbb90625231845d42e
This reduces the extract_phonys (now deduplicate_order_only_deps)
event's time from ~1.9s to ~1.5s on aosp-master, and from ~5.3s to ~4.6s
on internal master.
It does so by making keyForPhonyCandidate be based on a hash instead
of joining all the deps together. Having a hash allows us to also use
it as the name of the phony target, which simplifies the code a little.
Bug: None (original cl introducing extractPhonys also didn't have a bug)
Test: go tests
Change-Id: I2ff6e4614f19ccbfe99112ea7ae1ea33cd1df21b
