memoizeFullName was added to variables, rules and pools as an
optimization to prevent recomputing the full name repeatedly,
but the storage of variables, rules and pools are generally global
and not tied to the Context. When running multiple tests in
parallel there will be multiple Context objects all trying to
update the memoized names on the global variables, causing a data
race.
Package names were previously memoized via a pkgNames map stored
on the Context. Expand pkgNames to a nameTracker object that
contains maps for packages, variables, rules and pools, and replace
calls to fullName with calls through nameTracker.
Test: context_test.go
Change-Id: I15040b85a6d1dab9ab3cff44f227b22985acee18
Storing every string without ninja variable references through
simpleNinjaString costs 24 bytes and a heap allocation. 16 bytes
is used for the ninjaString.str string, 8 bytes for the
ninjaString.variables *[]variableReference. An additional 8 bytes
is used for the resulting pointer into the heap.
The vast majority of calls to simpleNinjaString originate in
blueprint.parseBuildParams, which converts all of the parameters
passed to ctx.Build into ninjaStrings. All together this was
allocating 1.575 GB of *ninjaString objects.
Add a parseNinjaOrSimpleStrings function that converts input strings
into ninjaStrings if they have ninja variable references, but also
returns a slice of plain strings for input strings without any ninja
variable references. That still results in 1.39 GB of allocations just
for the output string slice, so also add an optimization that reuses
the input string slice as the output slice if all of the strings had
no variable references.
Plumb the resulting strings through everywhere that the []*ninjaStrings
were used.
This reduces the total memory allocations inside
blueprint.parseBuildParams in my AOSP aosp_cf_x86_64_phone-userdebug
build from 3.337 GB to 1.786 GB.
Test: ninja_strings_test.go
Change-Id: I51bc138a2a6b1cc7383c7df0a483ccb067ffa02b
ninjaString is an interface, which uses 16 bytes of memory on top
of the size of the concrete type. A literalNinjaString is a string,
which is another 16 bytes for the string header for a total of 32
bytes. A varNinjaString is two slices, which are 24 bytes each
for the slice headers, for a total of 64 bytes. The slices contain
the first constant string, and then altenrating variable and string
parts of the ninjaString, resulting in 16 bytes plus 32 bytes per
variable.
This patch replaces the ninjaString interface with a *ninjaString
concrete struct type. The ninjaString struct is a string and a
pointer to a slice of variable references, for a total of 24 bytes.
ninjaStrings with no variable references (the equivalent of the old
literalNinjaString) have a nil slice, and now use 24 bytes instead
of 32 bytes.
ninjaStrings with variable references allocate a slice of variable
references that contain 32-bit start and end offsets and a Variable
interface, but reuse the original string and so avoid the extra
string headers, resulting in 24 bytes for the slice header, and
24 bytes per variable.
These savings reduce the peak memory usage averaged across 10 runs of
/bin/time -v build/soong/soong_ui.bash --make-mode nothing
on the internal master branch cf_x86_64_phone-userdebug build
from 50114842kB to 45577638kB, a savings of 4537204kB or 9%.
The new Benchmark_parseNinjaString shows savings in both time and
memory. Before:
Benchmark_parseNinjaString/constant/1-128 594251787 2.006 ns/op 0 B/op 0 allocs/op
Benchmark_parseNinjaString/constant/10-128 21191347 65.57 ns/op 16 B/op 1 allocs/op
Benchmark_parseNinjaString/constant/100-128 9983748 130.2 ns/op 112 B/op 1 allocs/op
Benchmark_parseNinjaString/constant/1000-128 2632527 445.1 ns/op 1024 B/op 1 allocs/op
Benchmark_parseNinjaString/variable/1-128 2964896 419.4 ns/op 176 B/op 4 allocs/op
Benchmark_parseNinjaString/variable/10-128 1807341 670.6 ns/op 192 B/op 7 allocs/op
Benchmark_parseNinjaString/variable/100-128 1000000 1092 ns/op 352 B/op 7 allocs/op
Benchmark_parseNinjaString/variable/1000-128 300649 3773 ns/op 1584 B/op 7 allocs/op
Benchmark_parseNinjaString/variables/1-128 2858432 441.6 ns/op 176 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/2-128 2360505 513.4 ns/op 208 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/3-128 1867136 635.6 ns/op 240 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/4-128 1584045 752.1 ns/op 272 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/5-128 1338189 885.8 ns/op 304 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/10-128 1000000 1468 ns/op 464 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/100-128 88768 12895 ns/op 3712 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/1000-128 8972 133627 ns/op 32896 B/op 4 allocs/op
After:
Benchmark_parseNinjaString/constant/1-128 584600864 2.004 ns/op 0 B/op 0 allocs/op
Benchmark_parseNinjaString/constant/10-128 19274581 64.84 ns/op 16 B/op 1 allocs/op
Benchmark_parseNinjaString/constant/100-128 9017640 127.6 ns/op 112 B/op 1 allocs/op
Benchmark_parseNinjaString/constant/1000-128 2630797 453.0 ns/op 1024 B/op 1 allocs/op
Benchmark_parseNinjaString/variable/1-128 3460422 347.0 ns/op 136 B/op 4 allocs/op
Benchmark_parseNinjaString/variable/10-128 2103404 519.9 ns/op 152 B/op 7 allocs/op
Benchmark_parseNinjaString/variable/100-128 1315778 906.5 ns/op 312 B/op 7 allocs/op
Benchmark_parseNinjaString/variable/1000-128 354812 3284 ns/op 1544 B/op 7 allocs/op
Benchmark_parseNinjaString/variables/1-128 3386868 361.5 ns/op 136 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/2-128 2675594 456.9 ns/op 160 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/3-128 2344670 520.0 ns/op 192 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/4-128 1919482 648.1 ns/op 208 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/5-128 1560556 723.9 ns/op 240 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/10-128 1000000 1169 ns/op 352 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/100-128 116738 10168 ns/op 2800 B/op 4 allocs/op
Benchmark_parseNinjaString/variables/1000-128 10000 105646 ns/op 24688 B/op 4 allocs/op
Bug: 286423944
Test: ninja_strings_test.go
Test: out/soong/build*.ninja is the same before and after this change
Change-Id: I1ecffbaccb0d0469a41fa31255c1b17311e01687
buildDef.WriteTo was calling valueList to convert all the build
parameter ninjaStrings into strings, which uses ValueWithEscaper
to build a strings.Builder. This results in building a string
only to immediately copy it into the output writer's buffer.
Instead, pass an io.StringWriter to ValueWithEscaper so it can
build the string directly into the output writer's buffer. This
requires converting ninjaWriterWithWrap into an io.StringWriter.
Test: ninja_writer_test.go
Change-Id: I02e1cf8259306267b9d2d0ebe8c81e13dd443725
ninjaWriter repeatedly called io.WriteString() on its writer, which
does a type assertion every time. Replace its io.Writer with an
io.StringWriter and call WriteString on it directly.
Test: ninja_writer_test.go
Change-Id: Ie073d996a319190242bf6a00af07a13a60d078b5
The Android fork of Ninja supports "validations", a node that is
added to the top level of the build graph whenever another node
is in the build graph. Add support in Blueprint to specify them
on build statements and write them to the ninja.
Test: ninja_writer_test.go
Change-Id: I87cd1281dbd2ed113cc26a265c50d20c65118c91
Writing the ninja file to a byte buffer causes a significant amount
of time to be spent in memmove when growing the byte slice. Write
the file directly to disk instead.
This also fixes some unhandled error warnings, which become more
likely when doing disk IO instead of byte buffer writes.
Change-Id: I5094e4c45cab4012713037f60c5a4fb00718f92e
Instead of sometimes re-running minibp/the primary builder during the
next phase, run bpglob earlier to check dependencies.
We've run into issues where the environment is slightly different
between bootstrapping phase and the main build phase. It's also a
problem because our primary builder (Soong) exports information used by
another tool (Kati) that runs in between the bootstrapping phases and
the main phase. When Soong would run in the main phase, it could get out
of sync, and would require the build to be run again.
To do this, add a "subninja" include a build-globs.ninja file to each
build.ninja file. The first time, this will be an empty file, but we'll
always run minibp / the primary builder anyway. When the builder runs,
in addition to writing a dependency file, write out the
build-globs.ninja file with the rules to run bpglob.
Since bpglob may need to be run very early, before it would normally be
built, build it with microfactory.
Change-Id: I89fcd849a8729e892f163d40060ab90b5d4dfa5d
Prebuild a string of all spaces to avoid calling strings.Repeat for
every indented line. Replace appendWithWrap function with a class
that handles counting bytes on a line as they are written instead of
buffering lines, and avoids extra string concatenations by providing
a function that implicitly adds a space. Saves 10% wall time on one
workload.
Change-Id: I571d70366ee5db21c40c1c98d0fe92d640efefab
Make integrating with go tools easier by putting the blueprint package
files in the top level directory of the git project instead of in a
subdirectory called blueprint.
Change-Id: I35c144c5fe7ddf34e478d0c47c50b2f6c92c2a03
