Add -memprofile argument to support capturing a heap profile, and -nogc
to disable the garbage collector to see all allocations.
Change-Id: I0d4c5f2bf79786aed4b2af4be112ecf0a2810a91
Blueprint files included with build= statements were not adding a
dependency to cause regenerating when they were changed.
Bug: 32085516
Test: touch external/boringssl/sources.bp && mmma -j external/boringssl
Change-Id: Id4fdf3b6788ae5c1e94547dc63ec6b55424a66a0
Blueprint properties that end up as command line arguments need to be
both ninja and shell escaped. Provide helpers that primary builders can
use to appropriately escape them.
Change-Id: Ifd697d87edb1c6f0a910377835c391bbe8f95b42
In Go 1.7, all generated types (including StructOf) are marked as
unexported types. This is not a problem with regular named fields, since
the whether the field is exported is based on the name, not the type.
But for unnamed (anonymous) fields, there is no name, and it falls back
to whether the type is exported or not. When the field isn't exported,
we're unable to use reflection to set the value.
Special case fields named "BlueprintEmbed" to act like exported unnamed
fields during unpacking. There is no functionality lost here, since code
can not directly access these runtime generated fields.
Structures embedded in the source shouldn't be affected by this change,
unless they happen to use the "BlueprintEmbed" field, which is not a
normal field name due to the capitalization.
Change-Id: I53f3b32b2286c47b2bb1918d6008d0180ad0605e
tl;dr: Read if you don't use the wrapper or use SKIP_NINJA
Previously, we were relying on the ninja behavior of restarting the
build when the build.ninja file was updated to switch between different
bootstrap stages. But that means that every step that could produce a
build.ninja must pass in order to switch to a different stage. That
wasn't a big problem when we had a two stage build -- there was very
little that could fail in the second stage before we chose to go back to
the first stage. But when we had a three stage build, it was possible to
get into a state (usually during development) where you were in the
second stage, but the build was failing because the first stage needed
to be run. This was fixed in d79f1af742
by adding a wrapper that always started building at the first stage.
But this kept all of the complexity of using ninja restarts without any
of the benefits, so this change removes that complexity and just runs
each stage sequentially in the wrapper. So the wrapper is now required.
Since we're no longer going through choosestage, we can also skip the
template parsing for the later stages that don't need to be templated --
this can save a couple of seconds for large files.
In addition to all of the above, this also lets Soong reduce the number
of times the main ninja file is loaded. We had been running the wrapper
once (3 stages), then running ninja again after combining the
Soong-generated build.ninja with the Kati-generated build.ninja. This
change lets us removing the intermediate parsing of Soong's build.ninja,
so that we only execute ninja 3 times per build. It also lets us have
dependencies on pools or rules from Kati in the primary builder, since
we're never executing the main build.ninja without the Kati build.ninja.
The wrapper has a new option, NINJA to provide the path to ninja. This
used to be hardcoded to `ninja`, and will still default to that. But
we'll be running the first two bootstrap stages with $NINJA even if
SKIP_NINJA is set.
The wrapper passes "-w dupbuild=err" to ninja now -- this really should
always be turned on if you care about reliable builds.
Change-Id: I6f656b74eb3d064b8b9e69d1d6dac1129d72b747
Allow Blueprint build statements and rules to reference rules and pools
defined outside of Blueprint.
Change-Id: I48644497d16a4c2bab88db400dbe7f7fe169d933
Allow using ExtendMatchingProperties to extend pointer to a struct or an
interface containing a pointer to a struct using a struct, and
vice-versa.
Also fixes a pre-existing bug where extending a nested structure could
fail if there were multiple possible destnations and some of them did
not have a matching nested property.
Change-Id: I6e69d78eb6595ba7dd2603e3aa7dd8de3f292744
Don't add a "_" to the variant name if the variation name is empty. For
example, when splitting a variant with name "foo" into variations "" and
"bar", the new variants would be named "foo" and "foo_bar" instead of
"foo_" and "foo_bar".
Change-Id: I82342d57e2a8e9f2d65a7d8d2872dcb7b3512899
runMutators was sending the wrong value into errsCh, hiding all errors.
parallelVisit had a data race on error, send the cancel signal over a
channel instead of writing to a bool from a goroutine.
Change-Id: I18919d71464c094db5846badbfac80f99366d70c
Convert parallelVisitAllBottomUp into a generic parallelVisit that takes
a visitOrderer to select top down or bottom up. Combine
runTopDownMutator and runBottomUpMutator into runMutators that takes a
mutatorDirection to select the visitOrderer for parallelVisit and which
function pointer in the mutatorInfo to run. Optimize out the
updateDependencies to only run if dependencies have been modified to
avoid running it after TopDownMutators that cannot modify dependencies.
Change-Id: Ib00302db1108ebab2ce8e01b20aa026140d382a4
cloneModules duplicates every module, which can be slow. Spawn a new
goroutine for each module to spread the factory and CopyProperties calls
over multiple cpus.
Change-Id: I2597f921dadfd321ccb705aec9a904c71b75f9ef
Allow BottomUpMutators to run in parallel by calling Parallel() on the
return value of RegisterBottomUpMutator. To avoid locking, moves
updates of global state into a separate goroutine that receives updates
over channels from the mutator goroutines.
Change-Id: Ic59b612da9b406cf59ec44940f0c1dee0c051a51
Get rid of runEarlyMutators, and implement EarlyMutators as a shim
around BottomUpMutators. Removes mostly-duplicate code between
runEarlyMutators and runBottomUpMutator to allow parallelizing
runBottomUpMutator.
Change-Id: I66a051cb8710770b47da21d35e55aa49c84e9be8
unpackProperties was instantiating all nil struct pointers in any struct
that had a property set. Check if the property is set first, and only
instantiate the struct if it was set.
This has a slight behavioral change, as now structures that only exist
through nil pointers and are never set into will not be type-checked.
This should be fixed in a later patch set that moves the type checking
to be done across all property structs once per factory before blueprint
files are loaded.
Change-Id: I0dea34d7fff76bb4fc907516a2d996e4ea2408d6
VisitDirectDeps[If] need to know the parent in order to support
OtherModuleDependencyTag(), remove Context.visitDirectDeps[If] and
reimplement [Module]Context.VisitDirectDeps[If] using Context.walkdeps.
Change-Id: I2faac2d100d2a0ad2a4f1d8250997eed1a7ef6f0
Test: TestVisit
Allow primary builders to reduce allocations of empty structures by
allowing nil pointers to concrete struct types. Property readers will
not recurse into nil pointers, property writers will replace the nil
pointer with a pointer to the zero value of the pointer element type.
Allows a >50% primary builder time improvement with a trivial change in
Soong.
Change-Id: If6ad674bf7bf2a694c335378a074643a97d3c50b
Passing -trace <file> on the command line produces a trace file that can
be used with go tool trace to examine concurrency.
Change-Id: I7d27bca64d0f0816cb4e518e947b70bc1c0573a3
proptools cloning and extending are a significant portion of the run
time for Soong. Optimize out calls to reflect.Type.Field(), which must
allocate a []int to store the index, by caching all the fields of each
type as it is seen, and by iterating over a slice of cached fields
instead of calling Field(i) for each one. Also avoid calling
reflect.Value.Interface() twice on the same Value.
Change-Id: I4e13fc85f30d8614a5586283e928c0a6d7f24809
It can be useful for a BottomUpMutator to visit its dependencies. Embed
all of TopDownMutatorContext into BottomUpMutatorContext.
Change-Id: I7c7262fbe3d8fa9cc0d26899eee704cc862835df
In order to implement some build statistics in Soong, expose the module
type (the string used to define the module) to singletons.
Change-Id: I441d12c7782bcf338b3654cfe907b8d2a7253594
value.Type and value.Pos changed to methods, change the users that were
not caught by error checking to use Type() or Pos().
Change-Id: I295a658c007fa2de68c89fb85ee367fbea5ed1aa
There may be multiple skipped comments in a row, convert
p.skippedComments to a list and add a test.
Change-Id: I30dcff269bee56fd51ef9513dab7c7885c44b7d7
Determining which comments are contiguous is difficult once they have
been parsed into an out-of-band comment list, as any intervening nodes
are in a separate structure. Group the comments into CommentGroups
during the parsing stage instead.
Change-Id: I9444c58e75333b7521b58dbfbd36ff29d139b6e3
