keyForPhonyCandidate was using sha256, which is a crypto hash and
unnecessarily expensive for this use case. hash/maphash would be
much faster because it implements WriteString and so doesn't cause
an extra allocation to copy to a byte slice for every write, but it
insists on randomizing the seed, which makes it unsuitable for writing
to the build.ninja file. Use hash/fnv instead, and use unsafe to
write strings to the hash to avoid the extra allocation.
Also replace the manually rolled parallelism with the existing
parallelVisit, which will reuse goroutines and limit the parallelism
to a useful value.
The hash could collide, and using a 64-bit hash makes that more
likely, so also check the full contents to make sure they are really
equal.
Cuts 1 second off Soong analysis time.
Test: SOONG_PROFILE_MEM=/tmp/mem.pprof m nothing
Change-Id: I4d1292cb158cfc5823a0f4d8b4aeac1d0b10230e
Storing every string without ninja variable references through
simpleNinjaString costs 24 bytes and a heap allocation. 16 bytes
is used for the ninjaString.str string, 8 bytes for the
ninjaString.variables *[]variableReference. An additional 8 bytes
is used for the resulting pointer into the heap.
The vast majority of calls to simpleNinjaString originate in
blueprint.parseBuildParams, which converts all of the parameters
passed to ctx.Build into ninjaStrings. All together this was
allocating 1.575 GB of *ninjaString objects.
Add a parseNinjaOrSimpleStrings function that converts input strings
into ninjaStrings if they have ninja variable references, but also
returns a slice of plain strings for input strings without any ninja
variable references. That still results in 1.39 GB of allocations just
for the output string slice, so also add an optimization that reuses
the input string slice as the output slice if all of the strings had
no variable references.
Plumb the resulting strings through everywhere that the []*ninjaStrings
were used.
This reduces the total memory allocations inside
blueprint.parseBuildParams in my AOSP aosp_cf_x86_64_phone-userdebug
build from 3.337 GB to 1.786 GB.
Test: ninja_strings_test.go
Change-Id: I51bc138a2a6b1cc7383c7df0a483ccb067ffa02b
ninjaString is an interface, which uses 16 bytes of memory on top
of the size of the concrete type. A literalNinjaString is a string,
which is another 16 bytes for the string header for a total of 32
bytes. A varNinjaString is two slices, which are 24 bytes each
for the slice headers, for a total of 64 bytes. The slices contain
the first constant string, and then altenrating variable and string
parts of the ninjaString, resulting in 16 bytes plus 32 bytes per
variable.
This patch replaces the ninjaString interface with a *ninjaString
concrete struct type. The ninjaString struct is a string and a
pointer to a slice of variable references, for a total of 24 bytes.
ninjaStrings with no variable references (the equivalent of the old
literalNinjaString) have a nil slice, and now use 24 bytes instead
of 32 bytes.
ninjaStrings with variable references allocate a slice of variable
references that contain 32-bit start and end offsets and a Variable
interface, but reuse the original string and so avoid the extra
string headers, resulting in 24 bytes for the slice header, and
24 bytes per variable.
These savings reduce the peak memory usage averaged across 10 runs of
/bin/time -v build/soong/soong_ui.bash --make-mode nothing
on the internal master branch cf_x86_64_phone-userdebug build
from 50114842kB to 45577638kB, a savings of 4537204kB or 9%.
The new Benchmark_parseNinjaString shows savings in both time and
memory. Before:
Benchmark_parseNinjaString/constant/1-128 594251787 2.006 ns/op 0 B/op 0 allocs/op
Benchmark_parseNinjaString/constant/10-128 21191347 65.57 ns/op 16 B/op 1 allocs/op
Benchmark_parseNinjaString/constant/100-128 9983748 130.2 ns/op 112 B/op 1 allocs/op
Benchmark_parseNinjaString/constant/1000-128 2632527 445.1 ns/op 1024 B/op 1 allocs/op
Benchmark_parseNinjaString/variable/1-128 2964896 419.4 ns/op 176 B/op 4 allocs/op
Benchmark_parseNinjaString/variable/10-128 1807341 670.6 ns/op 192 B/op 7 allocs/op
Benchmark_parseNinjaString/variable/100-128 1000000 1092 ns/op 352 B/op 7 allocs/op
Benchmark_parseNinjaString/variable/1000-128 300649 3773 ns/op 1584 B/op 7 allocs/op
Benchmark_parseNinjaString/variables/1-128 2858432 441.6 ns/op 176 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/2-128 2360505 513.4 ns/op 208 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/3-128 1867136 635.6 ns/op 240 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/4-128 1584045 752.1 ns/op 272 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/5-128 1338189 885.8 ns/op 304 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/10-128 1000000 1468 ns/op 464 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/100-128 88768 12895 ns/op 3712 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/1000-128 8972 133627 ns/op 32896 B/op 4 allocs/op
After:
Benchmark_parseNinjaString/constant/1-128 584600864 2.004 ns/op 0 B/op 0 allocs/op
Benchmark_parseNinjaString/constant/10-128 19274581 64.84 ns/op 16 B/op 1 allocs/op
Benchmark_parseNinjaString/constant/100-128 9017640 127.6 ns/op 112 B/op 1 allocs/op
Benchmark_parseNinjaString/constant/1000-128 2630797 453.0 ns/op 1024 B/op 1 allocs/op
Benchmark_parseNinjaString/variable/1-128 3460422 347.0 ns/op 136 B/op 4 allocs/op
Benchmark_parseNinjaString/variable/10-128 2103404 519.9 ns/op 152 B/op 7 allocs/op
Benchmark_parseNinjaString/variable/100-128 1315778 906.5 ns/op 312 B/op 7 allocs/op
Benchmark_parseNinjaString/variable/1000-128 354812 3284 ns/op 1544 B/op 7 allocs/op
Benchmark_parseNinjaString/variables/1-128 3386868 361.5 ns/op 136 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/2-128 2675594 456.9 ns/op 160 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/3-128 2344670 520.0 ns/op 192 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/4-128 1919482 648.1 ns/op 208 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/5-128 1560556 723.9 ns/op 240 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/10-128 1000000 1169 ns/op 352 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/100-128 116738 10168 ns/op 2800 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/1000-128 10000 105646 ns/op 24688 B/op 4 allocs/op
Bug: 286423944
Test: ninja_strings_test.go
Test: out/soong/build*.ninja is the same before and after this change
Change-Id: I1ecffbaccb0d0469a41fa31255c1b17311e01687
Previously, warnings about missing modules were printed directly to
stderr. Instead we can pass these messages along as errors using the
existing pathways.
Bug: 269457150
Test: m nothing
Test: add -external to PRODUCT_SOURCE_ROOT_DIRS and observe missing
module errors
Change-Id: Ib6624b9edbd103247f7f6e6d4c3030f1959aa56c
Soong analyzes the entire source tree even though not every lunch target
needs to know about every module. For example, OEM sources can be
ignored for cuttlefish products. This functionality allows blueprint to
ignore a list of undesired directories.
Bug: 269457150
Change-Id: Icbbf8f3b66813ad639a7ebd27b1a3ec153cbf269
This reduces the extract_phonys (now deduplicate_order_only_deps)
event's time from ~1.9s to ~1.5s on aosp-master, and from ~5.3s to ~4.6s
on internal master.
It does so by making keyForPhonyCandidate be based on a hash instead
of joining all the deps together. Having a hash allows us to also use
it as the name of the phony target, which simplifies the code a little.
Bug: None (original cl introducing extractPhonys also didn't have a bug)
Test: go tests
Change-Id: I2ff6e4614f19ccbfe99112ea7ae1ea33cd1df21b
1. scan if any set of order-only deps are repeated
2. if so extract them as a phony output to be shared
Test: m libc
Bug: NA
Change-Id: I0689111b97bbbd1f3b26650e8ae2e0a4ffb5085e
This introduces a new `blueprint_package_includes` module type. If
present, other blueprint modules in that file will be analyzed
if and only if the requested include tags are met
example syntax:
```
Android.bp
blueprint_packgage_includes {
match_all: ["tag1", "tag2", ...],
}
other_module1 {...}
other_module2 {...}
```
other_module1 and other_module2 will not be analyzed unless
tag1,tag2, ... are set
This also adds a new object of type `IncludeTags` to the Context object,
which is a container for these string keys.
Test: In build/blueprint, go test ./
Test: TH
Change-Id: I79de0d7da3224a5b2025c27a5137f39d00c7382e
This was the only one in the source tree.
Side cleanup: remove some dead code that I assume comes from the time
where Blueprint files had to specify what subdirectories other Blueprint
files are in.
Test: Presubmits.
Change-Id: If84c4e85bc5516f30da97c1be29b56e50dddb3c4
Use moduleInfo.String() to print dependency cycle errors so they
contain the variant.
Test: Test_parallelVisit
Change-Id: I2bddaa35c8abb57c42b4c424e861361a8813d588
Fix detecting cycles in parallelVisit when the first alphabetical
module is not part of the cycle, but depends on the cycle. Instead of
starting from the first alphabetical module, check every module in a
determinsitic order and return the first time a cycle is found.
Test: Test_parallelVisit
Change-Id: I03726f838ec42975251088ba75158103940115c2
Pass a channel to visitor functions called by parallelVisit that
allows them to pause the current visitor until a given visitor
has finished. This allows parallelVisit to work on a dependency
graph while it is being mutated.
Test: Test_parallelVisit
Change-Id: Id8b1542c22ac9914439310e31d992ae0d7318d69
AddFarVariationDependencies claims to take the first variant that
matches all the requested variations, but it did nothing of the sort.
It took the first variant that either matched or did not contain each
of the requested variations. A module with no variations was always
matched, and requesting variations that didn't apply to a module
still matched (for example, requesting an image variation for a
host module that was ignored by the image mutator).
Fix AddFarVariationDependencies by making subset actually check
for subsets.
Test: Test_findVariant
Change-Id: I10063fec342db2a1c0685a7db08e4a650d14bd4e
AddFarVariationDependencies takes the first matching variant. To
maintain sensible behavior on a module with aliases, the ordering
of aliases and module variants needs to be maintained so that
AddFarVariationDependencies can find an earlier matching alias
instead of a more specific variant.
Test: go test .
Change-Id: I78f4e96edd98159f3a62d94e240e5d652667bec4
Add tests for findVariant behavior that provides the matching
behaviors of AddVariationDependencies, AddFarVariationDependencies,
etc.
Test: Test_findVariant
Change-Id: I3494d57179c8b3d62f7d32e5a1b43c9b9672c2df
Previously, WalkDeps() would record that a module was visited after the
first time it encountered the module irrespective of whether it recursed
into or not. This change moves the recording so it happens only after it
has been recursed into.
Added TestWalkDepsDuplicates_IgnoreFirstPath to test the change. Without
the change the test fails because it does not visit E.
Test refactoring:
* A depsMutator was added instead of relying on blueprintDepsMutator to
allow different tags to be used for different dependency types.
* Modified barModule and fooModule to support the new depsMutator and
add support for another type of dependency that is ignored by the
walking code.
* Extracted walkDependencyGraph() function to reuse common code.
Move LoadHooks from Blueprint and run them during ParseBlueprintsFiles.
This will allow them to add scoped module types that are visible to the
rest of the Blueprints file. Requires passing the config object to
ParseBlueprintsFiles.
Test: all blueprint tests
Change-Id: Ia2a2c9a0223d5458bfd48bd22ebed0fdbd0156c6
Use module groups instead of passing around the list of modules
extracted from a module group.
Test: blueprint tests
Change-Id: I02a79950c6377441c49129ebeb5f12be257df668
Throw an error if run into a module without a name when generating
contexts.
Test: context_test.go
Change-Id: I3976d86d1f15b8ac10a7a38aa42ae277740a8f3b
WalkDeps was following every possible path through the dependency
tree, which can be enormous. Modify it to only call visit for
any particular (child, parent) pair once for each direct dependency
by not recursing into child if visitDown returns true but child
has already been visited.
Test: TestWalkDeps, TestWalkDepsDuplicates
Change-Id: Ieef28399bd10e744417cdeb661dfa04fbeb4ec60
Allow adding multiple dependencies on the same module. Adds a flag
to Context.walkDeps to determine whether dependencies should be
visited multiple times, and clarifies the behavior of
VisitDepsDepthFirst (continues to visit each module once, even if
there are multiple dependencies on it) and VisitDirectDeps (may
visit a module multiple times).
Test: TestWalkDepsDuplicates, TestVisit
Change-Id: I58afbe90490aca102d242d63e185386e1fe55d73
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
'build/blueprint:work^^^' 'build/blueprint:work'
Test: put a syntax error in a file and see that the
reported error reports the location of the violation
Change-Id: Iaeedb91ea8e816cb8e9ee954f21cd6c6bc4afa48
in facilitate moving name resolution to Soong
Bug: 65683273
Test: build/soong/scripts/diff_build_graphs.sh \
--products=aosp_arm \
'build/blueprint:work^' 'build/blueprint:work'
# and see that the only changes were:
# 1. adding the name_interface.go file
# 2. changing some line numbers
Change-Id: Ifa7603cca59b3b3d592f2f146fdafe57012bd4b9
Propagate extra ninja file deps through mutators so that they
can use ctx.AddNinjaFileDeps.
Test: blueprint tests
Change-Id: I299a0665c3f63b020ae345889fd78b91b91b215a
Allow a module to create other modules as if they were specified
in a blueprint file. CreateModule takes the name of a module type,
calls the factory function the module type, and then inserts
properties into the property structs returned by the factory.
Bug: 64161749
Test: TestCreateModule
Change-Id: Ic1903305d6b08eae1edc7f0fb4137e85544aac0f
Add globbing with dependency checking to blueprint. Calling
ModuleContext.GlobWithDeps or SingletonContext.GlobWithDeps will return
a list of files that match the globs, while also adding efficient
dependencies to rerun the primary builder if a file that matches the
glob is added or removed.
Also use the globbing support for optional_subdirs=, subdirs= and build=
lines in blueprints files. The globbing slightly changes the behavior
of subname= lines, it no longer falls back to looking for a file called
"Blueprints". Blueprint files that need to include a subdirectory with
a different name can use build= instead of subdir= to directly include
them. The Blueprints file is updated to reset subname="Blueprints" in
case we want to include subdirectories inside blueprint and the primary
builder has changed the subname.
Also adds a new test directory that contains a simple primary builder
tree to test regeneration for globbing, and runs the tests in travis.
Change-Id: I83ce525fd11e11579cc58ba5308d01ca8eea7bc6
Remove the assumed Name and Deps properties. Instead, ask the Module
for the Name, and require them to use the existing replacements for
Deps.
Change-Id: I729045d86277686078b3aa0bba71c67d612ead2c
VisitDirectDeps[If] need to know the parent in order to support
OtherModuleDependencyTag(), remove Context.visitDirectDeps[If] and
reimplement [Module]Context.VisitDirectDeps[If] using Context.walkdeps.
Change-Id: I2faac2d100d2a0ad2a4f1d8250997eed1a7ef6f0
Test: TestVisit
DynamicDependencies can be implemented more flexibly by a
BottomUpMutator. If there are no DynamicDependencies, then
EarlyMutators are identical to BottomUpMutators. Deperecate both, and
reimplement DynamicDependencies inside a BottomUpMutator that is
guaranteed to be registered first.
build is a magic variable that acts similarly to subdirs, but specifies
files to build instead of subdirs. Using files provides a few advantages:
files can be named something more appropriate for the project than
"Blueprints", whatever you want instead of Blueprints, and multiple
Blueprints files can exist in the same directory.
A new variable is used instead of putting filenames into subdirs to avoid
unexpected behavior when a glob matches both files and directories.
subdirs= and build= entries that don't match any directories or files
are now reported as errors.
Change-Id: Id329504ace251eab4ccea1081a3c8665a4c52f5a
Make integrating with go tools easier by putting the blueprint package
files in the top level directory of the git project instead of in a
subdirectory called blueprint.
Change-Id: I35c144c5fe7ddf34e478d0c47c50b2f6c92c2a03
