Commit graph

2954 commits

Author SHA1 Message Date
Cole Faust
4f968700d1 Add OtherModulePropertyErrorf
...and pass property name to ConfigurableEvaluator.

When trying to convert Enabled to a configurable property, I enountered
a bunch of different context types that all had different error methods.
OtherModulePropertyErrorf is an error method that can be implemented
by all contexts.

Bug: 323382414
Test: m nothing --no-skip-soong-tests
Change-Id: I38b2a545c0ee8d23281cd88bc397f79f39835bc4
2024-03-22 13:02:51 -07:00
Cole Faust
894b318528 Fix error reporting in MatchesIncludeTags
ctx.ModuleErrorf for the main blueprint config object doesn't
report an error, it just returns it.

If go had something like C++'s `[[nodiscard]]` or rust's `#[must_use]`
that would be useful here, but the issue has been open since 2017:
https://github.com/golang/go/issues/20803

Test: Presubmits
Change-Id: I72709e6c5466d55f5c0f3fe51a25c29df0826271
2024-03-21 16:16:49 -07:00
Cole Faust
d2f1141871 Get rid of some extra newlines in bpfmt
Bug: 323382414
Test: go tests, and I also ran both the old/new bpfmts on the whole aosp source tree, and the only difference between the results was one extra removed line in external/uwb
Change-Id: I4942c9247a66f1de5028de39caa5cd34b66093c3
2024-03-19 17:07:00 -07:00
Cole Faust
a962edf763 Add proptools.Slice
To make it easier to handle the result of
Configurable[[]string].Evaluate()

Bug: 329711542
Test: go tests
Change-Id: I7364170564b1049a33873424c6da4fc26aff305b
2024-03-14 14:28:52 -07:00
Cole Faust
6437d4e737 Select statements
Select statements are a new blueprint feature inspired by bazel's select
statements. They are essentially alternative syntax for soong config
variables that require less boilerplate. In addition, they support
making decisions based on a module's variant, which will eliminate
the need for manual property struct manipulation, such as the arch
mutator's arch: and target: properties.

In order to support decisions based on the variant, select statements
cannot be evaluated as soon as they're parsed. Instead, they must be
stored in the property struct unevaluated. This means that individual
properties need to change their type from say, string, to
Configurable[string]. Currently, only configurable strings, bools, and
string slices are supported, but more types can be added later.
The module implementation must call my_property.Evaluate(ctx) in order
to get the final, resolved value of the select statement.

Bug: 323382414
Test: go tests
Change-Id: I62f8721d7f0ac3d1df4a06d7eaa260a5aa7fcba3
2024-03-06 15:00:39 -08:00
LaMont Jones
34847c828d debugValue: handle Interfaces better. am: 89e90b4c60
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2975295

Change-Id: Iba23845605d8cdeb400f65cc3da34e4e2239b1d6
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-02-26 20:35:19 +00:00
LaMont Jones
89e90b4c60 debugValue: handle Interfaces better.
Bug: none
Test: manual
Change-Id: Ic8ee668abf4387fc2f00e1f520efea360eebd66c
2024-02-23 15:04:04 -08:00
Joe Onorato
2e39a573ce Add maps to soong-debug-info.json am: 5920e5507e
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2968297

Change-Id: I3506b8e2ab0928381d11d0ff14e07f207ac28a43
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-02-20 18:49:14 +00:00
Joe Onorato
5920e5507e Add maps to soong-debug-info.json
Test: m nothing
Change-Id: Iffa308e1b93a5f7e1cea7587bef3540d1eb37de5
2024-02-18 14:47:44 -08:00
Joe Onorato
6dacd69f65 Add tag data to soong-debug-info.json. am: c7ec117556
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2956143

Change-Id: Id701b9172a56db6780c1e1e899569e6bec689cc2
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-02-12 19:55:21 +00:00
Joe Onorato
b4058b4182 Add contents of properties and providers to soong-debug-info.json am: e66903065b
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2956892

Change-Id: I54953f828ebbedf9d8d65f54cc755a6489132bdd
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-02-10 00:50:12 +00:00
Joe Onorato
c7ec117556 Add tag data to soong-debug-info.json.
It's not super useful because of Go's stupid restriction on not letting
you reflect private fields.

Test: m nothing
Change-Id: I1f73f06f5e64eaf4adce58a667b98b131aede53f
2024-02-09 16:26:16 -08:00
Joe Onorato
e66903065b Add contents of properties and providers to soong-debug-info.json
Test: whatever
Change-Id: I9e602c690ecbbf2150a65174d334ae2cbacc62b5
2024-02-08 23:06:19 -08:00
Colin Cross
d15eef65e1 Merge "Optimize (Extend|Append|Prepend)[Matching]Properties" into main am: 51d8322241
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2947018

Change-Id: I49a3461e3008fd67afc978f62ccdc3aaac643365
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-02-05 21:23:33 +00:00
Colin Cross
51d8322241 Merge "Optimize (Extend|Append|Prepend)[Matching]Properties" into main 2024-02-05 21:02:02 +00:00
Colin Cross
409c25a85a Merge "Optimize blueprint.variationMap.equal" into main am: 4a918d4e91
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2947017

Change-Id: I9adb50dd5f8d9549fa7b738b680a4610e2295670
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-02-05 20:41:37 +00:00
Colin Cross
1daaad4500 Merge changes I5894f739,I4d1292cb into main am: 9beaa476d6
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2947016

Change-Id: I2fdb25ebda8e2679fc9a1a0ad8acefbce5cab070
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-02-05 20:41:03 +00:00
Colin Cross
4a918d4e91 Merge "Optimize blueprint.variationMap.equal" into main 2024-02-05 20:29:19 +00:00
Colin Cross
9beaa476d6 Merge changes I5894f739,I4d1292cb into main
* changes:
  Use WriteString in hashProviderInternal
  Optimize deduplicateOrderOnlyDeps
2024-02-05 20:00:38 +00:00
Joe Onorato
2f019ce883 Add new soong module debug info json. am: 8ad2b88c8d
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2949327

Change-Id: Id900142900cc5c6b302e69e4d88770cdb5654db5
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-02-04 00:51:59 +00:00
Joe Onorato
8ad2b88c8d Add new soong module debug info json.
Enabled by setting GENERATE_SOONG_DEBUG=true

Test: GENERATE_SOONG_DEBUG=true m nothing ; soongdbg ...
Change-Id: If70c7d91cc59a5a7642e453c85278e8022693ac7
2024-02-03 14:37:02 -08:00
Colin Cross
c4fc4b447f Optimize (Extend|Append|Prepend)[Matching]Properties
The property squashing functions were performing 19% of all allocations
and allocating 3.43 GB, most of which was used to track the name of the
property, but the name of the property is only used to print errors.
Keep the elements of the name in a preallocated slice instead, and only
compute the name when an error occurs.

Calling reflect.Value.Interface() was also expensive, and only passed
to filter and order functions, none of which use the values.  Modify
the signature of the filter and order functions to remove the interfaces
and the property name.

Test: extend_test.go
Change-Id: I517f89daf251bb43f7cfefa6f1e83951c0e271b7
2024-02-02 15:57:26 -08:00
Colin Cross
1d82de2aab Optimize blueprint.variationMap.equal
blueprint.variationMap.Equal was responsible for 11.5% of allocations
and 2.2% of allocated memory.  reflect.DeepEquals is an expensive way
to compare a map.  variationMap.subsetOf is already iterating through
the elements of the map to compare them, replace equal with a check that
the maps are the same length and then reuse subsetOf to check that the
have the same keys and values.

Test: SOONG_PROFILE_MEM=/tmp/mem.pprof m nothing
Change-Id: Ifb7cdf612e5455fd2f412488b7f94416c4e70c54
2024-02-02 15:57:26 -08:00
Colin Cross
2ef2c35664 Use WriteString in hashProviderInternal
maphash.Hash implements WriteString, which avoids an allocation in
order to convert the string to a byte slice.  Using the concrete
type instead of the io.Writer interface also allows int64Array to
be allocated on the stack.

Test: SOONG_PROFILE_MEM=/tmp/mem.pprof m nothing
Change-Id: I5894f7399c2a232f5f67d7d0724a6115ba2c278f
2024-02-02 15:57:26 -08:00
Colin Cross
dbf18bec98 Optimize deduplicateOrderOnlyDeps
keyForPhonyCandidate was using sha256, which is a crypto hash and
unnecessarily expensive for this use case.  hash/maphash would be
much faster because it implements WriteString and so doesn't cause
an extra allocation to copy to a byte slice for every write, but it
insists on randomizing the seed, which makes it unsuitable for writing
to the build.ninja file.  Use hash/fnv instead, and use unsafe to
write strings to the hash to avoid the extra allocation.

Also replace the manually rolled parallelism with the existing
parallelVisit, which will reuse goroutines and limit the parallelism
to a useful value.

The hash could collide, and using a 64-bit hash makes that more
likely, so also check the full contents to make sure they are really
equal.

Cuts 1 second off Soong analysis time.

Test: SOONG_PROFILE_MEM=/tmp/mem.pprof m nothing
Change-Id: I4d1292cb158cfc5823a0f4d8b4aeac1d0b10230e
2024-02-02 15:57:26 -08:00
Cole Faust
689b3e5ce4 Enforce that providers are not changed am: 7add62142d
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2928138

Change-Id: I9509cd10e21d24b9792f35a1a208b755adae08c2
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-01-31 01:31:41 +00:00
Cole Faust
7add62142d Enforce that providers are not changed
When setProvider() is called, hash the provider and store the hash in
the module. Then after the build is done, hash all the providers again
and compare the hashes. It's an error if they don't match.

Also add a flag to control it in case this check gets slow as we convert
more things to providers. However right now it's fast (unnoticable
in terms of whole seconds) so just have the flag always enabled.

Bug: 322069292
Test: m nothing
Change-Id: Ie4e806a6a9f20542ffcc7439eef376d3fb6a98ca
2024-01-30 15:18:24 -08:00
Colin Cross
1236187824 Merge changes from topic "transition-mutator-config" into main am: 1b3fe6bd2c
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2922240

Change-Id: I881619ef13ed58d5fd65fe03d6923b76f8d7d7c7
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-01-24 20:36:28 +00:00
Colin Cross
8530c625b9 Fix OutgoingTransitionContext to match the description of TransitionMutators am: 33bec91aec
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2923530

Change-Id: I8f76a7cdc0694e0f8ea34fb1b17ade729a6cf412
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-01-24 20:36:25 +00:00
Colin Cross
1b3fe6bd2c Merge changes from topic "transition-mutator-config" into main
* changes:
  Add Config() to TransitionMutator contexts
  Fix OutgoingTransitionContext to match the description of TransitionMutators
2024-01-24 20:03:46 +00:00
Colin Cross
9c2f78f6c4 Add Config() to TransitionMutator contexts
Test: builds
Change-Id: Iadcea3081d8537220d54b4eeb1ba3691105e1d82
2024-01-22 17:18:45 -08:00
Colin Cross
33bec91aec Fix OutgoingTransitionContext to match the description of TransitionMutators
The description of TransitionMutators says that "the outgoing transition
should not take the properties of the dependency into account, only those
of the module that depends on it. For this reason, the dependency is not
even passed into it as an argument."  However, OutgoingTransitionContext
was returing the dependency from ctx.Module(), not the parent.  This
didn't matter for the only existing TransitionMutator, as it only used
the module to get a constant value.

Test: sanitize_test.go
Change-Id: I1ce5b3144787f57be4d50e95f0c923da9b2b079f
2024-01-22 17:18:40 -08:00
Cole Faust
01b4981fe2 Merge "Remove SymlinkOutputs" into main am: 079c92fb04
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2917003

Change-Id: Icc339be8a89a9ad1bb186f6706c5e6402c7ed755
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-01-19 19:59:24 +00:00
Cole Faust
079c92fb04 Merge "Remove SymlinkOutputs" into main 2024-01-19 19:38:55 +00:00
Colin Cross
21ed16ee0b Fix data race in a parallel singletons calling VisitAllModules am: e70495ba9a
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2917002

Change-Id: I1ed3c25b4e82337337f311a0eb1b9624d7454daf
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-01-19 00:48:43 +00:00
Colin Cross
c0b0a19454 Move name memoization out of variables am: 6bc984abca
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2917001

Change-Id: I240b9a3db493a0114c53195e99721644586184da
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-01-19 00:48:29 +00:00
Cole Faust
828e6b4f20 Remove SymlinkOutputs
symlink_outputs was added so bazel could run ninja files, but we
abanoned that approach in roboleaf, and then roboleaf was cancelled
entirely. Remove this feature so we're more compatible with upstream
ninja / n2.

Bug: 160568334
Test: Presubmits
Change-Id: Ie3afd084c5574444dddac77cba1866e82ff2ca19
2024-01-18 12:41:16 -08:00
Colin Cross
e70495ba9a Fix data race in a parallel singletons calling VisitAllModules
Force a resort of the module groups before running singletons
so that two singletons running in parallel don't cause a data
race when they trigger a resort in VisitAllModules.

Test: go test -race ./...
Change-Id: Iec041cec08c33c56787aadbde6a1b2b619815142
2024-01-18 12:28:49 -08:00
Colin Cross
6bc984abca Move name memoization out of variables
memoizeFullName was added to variables, rules and pools as an
optimization to prevent recomputing the full name repeatedly,
but the storage of variables, rules and pools are generally global
and not tied to the Context.  When running multiple tests in
parallel there will be multiple Context objects all trying to
update the memoized names on the global variables, causing a data
race.

Package names were previously memoized via a pkgNames map stored
on the Context.  Expand pkgNames to a nameTracker object that
contains maps for packages, variables, rules and pools, and replace
calls to fullName with calls through nameTracker.

Test: context_test.go
Change-Id: I15040b85a6d1dab9ab3cff44f227b22985acee18
2024-01-18 12:28:49 -08:00
Zi Wang
510abfab4f Merge "Remove unnecessary used names before reporting" into main am: 2f0f395cc9 am: 79e4baab88 am: b0cce0bde8
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2810185

Change-Id: Ibb04d7649d3b78d5216034cef3c570e9cf89ae15
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-01-05 01:25:47 +00:00
Zi Wang
b0cce0bde8 Merge "Remove unnecessary used names before reporting" into main am: 2f0f395cc9 am: 79e4baab88
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2810185

Change-Id: I2357012eec65f3eba246a1aeb4dde17b5728cea8
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-01-05 00:36:23 +00:00
Zi Wang
79e4baab88 Merge "Remove unnecessary used names before reporting" into main am: 2f0f395cc9
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2810185

Change-Id: I2bb7aad8bf50e16c77f26839c9ce55d1baa65459
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-01-05 00:07:04 +00:00
Zi Wang
2f0f395cc9 Merge "Remove unnecessary used names before reporting" into main 2024-01-04 23:35:05 +00:00
Zi Wang
e5913c17e9 Remove unnecessary used names before reporting
When property a.b.c is not used, (also there is no a.* or a.b.* used)
"a", "a.b" and "a.b.c" are all in unusedNames.
removeUnnecessaryUnusedNames only keeps the last "a.b.c" as the
real unused name.

Test: TestNonExistentPropertyInSoongConfigModule, unpack_test.go and CI

Bug: 171232169
Change-Id: I861fa6933e558b07694ee5ff40ef549117d115ff
2024-01-04 12:24:39 -08:00
Cole Faust
1ad5037f43 Merge "Add implicit inputs to primary builder actions" into main am: 6733423618 am: fa7e35cab5 am: 7a0266716d
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2893438

Change-Id: Icad92b0b11c1aabf20c7924c0ba628d081fa54a7
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-01-03 07:52:35 +00:00
Cole Faust
7a0266716d Merge "Add implicit inputs to primary builder actions" into main am: 6733423618 am: fa7e35cab5
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2893438

Change-Id: Iee2656fb666f0e938c9d2617a511d092726ddf17
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-01-03 07:19:25 +00:00
Cole Faust
fa7e35cab5 Merge "Add implicit inputs to primary builder actions" into main am: 6733423618
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2893438

Change-Id: If3cfb7ffe612b2c577f554561edb788ddcf01abd
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-01-03 06:45:56 +00:00
Cole Faust
6733423618 Merge "Add implicit inputs to primary builder actions" into main 2024-01-03 06:14:23 +00:00
Cole Faust
ef75beb068 Add implicit inputs to primary builder actions
This is needed so primary builder actions can directly depend on their
glob result files.

Bug: 318434287
Test: rm -rf out && m nothing && m nothing
Change-Id: I5c67ee53c9f18f81c79c0fe13b3338eacaccdbc0
2024-01-02 16:58:49 -08:00
Treehugger Robot
9593d9cd8c Merge "Optimize returning the zero value from provider APIs" into main am: 321aad0a8c am: 162690bb2a am: fb95058b94
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/2876680

Change-Id: I6221dadb2f0e563f15b3210fb87a6c9148c9bfba
Signed-off-by: Automerger Merge Worker <android-build-automerger-merge-worker@system.gserviceaccount.com>
2024-01-02 21:47:04 +00:00