blueprint.variationMap.Equal was responsible for 11.5% of allocations
and 2.2% of allocated memory. reflect.DeepEquals is an expensive way
to compare a map. variationMap.subsetOf is already iterating through
the elements of the map to compare them, replace equal with a check that
the maps are the same length and then reuse subsetOf to check that the
have the same keys and values.
Test: SOONG_PROFILE_MEM=/tmp/mem.pprof m nothing
Change-Id: Ifb7cdf612e5455fd2f412488b7f94416c4e70c54
maphash.Hash implements WriteString, which avoids an allocation in
order to convert the string to a byte slice. Using the concrete
type instead of the io.Writer interface also allows int64Array to
be allocated on the stack.
Test: SOONG_PROFILE_MEM=/tmp/mem.pprof m nothing
Change-Id: I5894f7399c2a232f5f67d7d0724a6115ba2c278f
keyForPhonyCandidate was using sha256, which is a crypto hash and
unnecessarily expensive for this use case. hash/maphash would be
much faster because it implements WriteString and so doesn't cause
an extra allocation to copy to a byte slice for every write, but it
insists on randomizing the seed, which makes it unsuitable for writing
to the build.ninja file. Use hash/fnv instead, and use unsafe to
write strings to the hash to avoid the extra allocation.
Also replace the manually rolled parallelism with the existing
parallelVisit, which will reuse goroutines and limit the parallelism
to a useful value.
The hash could collide, and using a 64-bit hash makes that more
likely, so also check the full contents to make sure they are really
equal.
Cuts 1 second off Soong analysis time.
Test: SOONG_PROFILE_MEM=/tmp/mem.pprof m nothing
Change-Id: I4d1292cb158cfc5823a0f4d8b4aeac1d0b10230e
When setProvider() is called, hash the provider and store the hash in
the module. Then after the build is done, hash all the providers again
and compare the hashes. It's an error if they don't match.
Also add a flag to control it in case this check gets slow as we convert
more things to providers. However right now it's fast (unnoticable
in terms of whole seconds) so just have the flag always enabled.
Bug: 322069292
Test: m nothing
Change-Id: Ie4e806a6a9f20542ffcc7439eef376d3fb6a98ca
The description of TransitionMutators says that "the outgoing transition
should not take the properties of the dependency into account, only those
of the module that depends on it. For this reason, the dependency is not
even passed into it as an argument." However, OutgoingTransitionContext
was returing the dependency from ctx.Module(), not the parent. This
didn't matter for the only existing TransitionMutator, as it only used
the module to get a constant value.
Test: sanitize_test.go
Change-Id: I1ce5b3144787f57be4d50e95f0c923da9b2b079f
symlink_outputs was added so bazel could run ninja files, but we
abanoned that approach in roboleaf, and then roboleaf was cancelled
entirely. Remove this feature so we're more compatible with upstream
ninja / n2.
Bug: 160568334
Test: Presubmits
Change-Id: Ie3afd084c5574444dddac77cba1866e82ff2ca19
Force a resort of the module groups before running singletons
so that two singletons running in parallel don't cause a data
race when they trigger a resort in VisitAllModules.
Test: go test -race ./...
Change-Id: Iec041cec08c33c56787aadbde6a1b2b619815142
memoizeFullName was added to variables, rules and pools as an
optimization to prevent recomputing the full name repeatedly,
but the storage of variables, rules and pools are generally global
and not tied to the Context. When running multiple tests in
parallel there will be multiple Context objects all trying to
update the memoized names on the global variables, causing a data
race.
Package names were previously memoized via a pkgNames map stored
on the Context. Expand pkgNames to a nameTracker object that
contains maps for packages, variables, rules and pools, and replace
calls to fullName with calls through nameTracker.
Test: context_test.go
Change-Id: I15040b85a6d1dab9ab3cff44f227b22985acee18
When property a.b.c is not used, (also there is no a.* or a.b.* used)
"a", "a.b" and "a.b.c" are all in unusedNames.
removeUnnecessaryUnusedNames only keeps the last "a.b.c" as the
real unused name.
Test: TestNonExistentPropertyInSoongConfigModule, unpack_test.go and CI
Bug: 171232169
Change-Id: I861fa6933e558b07694ee5ff40ef549117d115ff
This is needed so primary builder actions can directly depend on their
glob result files.
Bug: 318434287
Test: rm -rf out && m nothing && m nothing
Change-Id: I5c67ee53c9f18f81c79c0fe13b3338eacaccdbc0
TestExternalShellEscaping and TestExternalShellEscapeIncludingSpaces
use "echo -n", which fails on darwin. These tests weren't running on
darwin because they were only run in Soong, which always limits to
only short tests. The test are now run in aosp-build-tools, which
doesn't limit to short tests.
Remove the unsupported -n argument from echo and trim the added newline
instead.
Test: TestExternalShellEscaping and TestExternalShellEscapeIncludingSpaces
Change-Id: I3d8ff1c0db0af386e1dc13cb6c2dabe561c1c89e
Now that nothing calls *Context.*Provider directly, make the blueprint
methods return a nil any interface instead of the zero value that was
constructed via reflection. The type-safe wrappers will return a
zero value that can be constructed without any reflection or copying.
Bug: 316410648
Test: provider_test.go
Change-Id: I0abde5bacab9964a83f03c1644b51295a6c34d0b
Using generics for the providers API allows a type to be associated
with a ProviderKey, resulting in a type-safe API without that doesn't
require runtime type assertions by every caller.
Unfortunately, Go does not allow generic types in methods, only in
functions [1]. This prevents a type-safe API on ModuleContext, and
requires moving the API to be functions that take a ModuleContext as
a parameter.
[1] https://go.googlesource.com/proposal/+/refs/heads/master/design/43651-type-parameters.md#no-parameterized-methods)
Bug: 316410648
Test: provider_test.go
Change-Id: Ide91de9f2a2a7d075b05e287c7cc86b395db0edb
proptools.Clear(ptr) clears a property with its zero value.
Bug: 313806237
Test: m blueprint_tests
Change-Id: Ib78f9f88a9b0a8b04e1ab6c5e545b55ba4269e5d
Go 1.21 does a better job using the same empty allocation for empty
structs, allow cloned properties to point to the original when it
is an empty struct.
Bug: 309895579
Test: TestCloneProperties
Change-Id: I064f2316a8a8017a109968671ac305dbbe3246af
NinjaEscapeList is called on every input or output of a rule, and
most of the time does not escape anything. Optimize it by returning
the input slice when nothing was escaped. This avoids 1.336 GB of
allocations in my AOSP aosp_cf_x86_64_phone-userdebug build.
Test: TestNinjaEscapeList
Change-Id: I33b9e7b77b33d10401d1ec3546caa6794c567b16
Storing every string without ninja variable references through
simpleNinjaString costs 24 bytes and a heap allocation. 16 bytes
is used for the ninjaString.str string, 8 bytes for the
ninjaString.variables *[]variableReference. An additional 8 bytes
is used for the resulting pointer into the heap.
The vast majority of calls to simpleNinjaString originate in
blueprint.parseBuildParams, which converts all of the parameters
passed to ctx.Build into ninjaStrings. All together this was
allocating 1.575 GB of *ninjaString objects.
Add a parseNinjaOrSimpleStrings function that converts input strings
into ninjaStrings if they have ninja variable references, but also
returns a slice of plain strings for input strings without any ninja
variable references. That still results in 1.39 GB of allocations just
for the output string slice, so also add an optimization that reuses
the input string slice as the output slice if all of the strings had
no variable references.
Plumb the resulting strings through everywhere that the []*ninjaStrings
were used.
This reduces the total memory allocations inside
blueprint.parseBuildParams in my AOSP aosp_cf_x86_64_phone-userdebug
build from 3.337 GB to 1.786 GB.
Test: ninja_strings_test.go
Change-Id: I51bc138a2a6b1cc7383c7df0a483ccb067ffa02b
osFs.acquire and osFs.release are surprisingly expensive, using a
combined 345.7s of runtime in an AOSP aosp_cf_x86_64_phone-userdebug
build. They are used to ensure we don't use too many simultaneous
open files, but many of the functions they are called from don't
actually open a file. Remove them from all the stat-based functions
(Exists, IsDir, IsSymlink, Lstat, Stat), and from ReadLink. After
this change the time spent in acquire and release is effectively
zero.
Test: SOONG_PROFILE_CPU=/tmp/cpu.pprof m nothing
Change-Id: Ie5e22e33c61794354821f05ab79ceb4efc3b276c
Generate the rules to build and run the blueprint tests whether or
not running the tests during bootstrap is enabled, and only add them
as validation dependencies if running the tests is enabled. Export
the outputs of the tests as a phony target for checkbuild to depend on.
Bug: 269296618
Test: m nothing
Test: aninja -t path checkbuild out/host/linux-x86/bin/go/soong-java/test/test.passed
Change-Id: I09cd20d802bed5a659f3f36e87128d4281dfcfb0
Currently when a directory path is specified bpfmt only processes files
named "Blueprints" so change this to also process files with a `.bp`
suffix.
Test: Manual + bpfmt -d frameworks/base/services shows differences
Change-Id: I5a6356f387892934ee8e83362db13cda6156ed51
Signed-off-by: Rashid Zaman <rashidz@meta.com>
