Commit graph

1688 commits

Author SHA1 Message Date
Lukacs T. Berki
6ec924c12b Remove Srcdir() from BootstrapConfig.
It was always ".".

Test: Presubmits.
Change-Id: I0612b9b07233659adc708b9710db9aec6e24f224
2021-08-16 17:04:50 +02:00
Lukacs T. Berki
980c1f0c72 Remove global variables in command.go .
There used to be two:
- The absolute source directory is now threaded through the code instead
- The command line arguments are moved to soong_build

Test: Presubmits.

Change-Id: Ie09de1a2862ec4ced2c7385198300359f805127d
2021-08-16 15:28:04 +02:00
Lukacs T. Berki
016a7dd6ef Delete support for standalone Blueprint.
Now that Blueprint has been merged with Soong, a large amount of code is
unnecessary.

Test: Presubmits.
Change-Id: I8d30d8b80f37fdc6a9db630a1792c57c7d6edb38
2021-08-16 14:09:47 +02:00
Lukacs T. Berki
64ee4262a0 Remove the ability to emit Blueprint docs.
This is not necessary anymore now that Blueprint doesn't have a separate
existence and Soong's documentation generation uses a completely
separate code path.

Test: Presubmits.
Change-Id: Ibc9e57379c5f8ee9a06d90fbe81fac314f01ee92
2021-08-16 13:42:42 +02:00
Lukacs T. Berki
0eb24a7041 Explain why build.ninja is a non-default output.
Test: Only comment change.
Change-Id: I335670b7f3bed0fe34fdb1d625b6e459146cc192
2021-08-16 11:48:51 +02:00
Lukacs T. Berki
d3a0637702 Separate the concept of subninjas and glob files.
This makes the distinction between what the bootstrap .ninja file and
the main .ninja files do clearer: the bootstrap .ninja file includes the
build-globs.ninja file(s) which the main .ninja file writes.

This required removing the "default" statement from said bootstrap
.ninja file: now we can build more than one thing instead of just
build.ninja and thus soong_ui explicitly tells Ninja what to do instead.

On the side, stop catering for the use case where the bootstrap
Blueprint invocation uses globs: Soong's does not and that's our only
use case.

Test: Presubmits.

Change-Id: Icffce31928242cfe9cdab56b290a37598d32a58c
2021-08-16 09:16:09 +02:00
Jingwen Chen
8979d4c54c Expose build-globs.ninja write function to android/soong.
This enables bp2build to write its own build-globs.ninja file.

Test: soong integration tests
Fixes: 193650250
Change-Id: Icb35f3cf3c30929dab1b2c2e9b244321be30f967
2021-08-12 10:13:44 +00:00
Chris Parsons
030150d8f9 Add function to retrieve module by name.
This function should only be used in corner-cases where it is not
possible to retrieve the module by traversing a dependency graph; there
are no guarantees about which variant is returned, and the function will
eagerly panic if something is amiss (the name doesn't belong to a module
but to an alias instead).

This function is particularly useful for bp2build, which does not use
real variants when evaluating the blueprint graph.

Test: With soong change, USE_BAZEL_ANALYSIS=1 m libc
Change-Id: I72b8335b642ed2d05e0a38e448cd380acc8d65b0
2021-07-21 16:16:40 -04:00
Colin Cross
1c3530ab58 Support AppendMatchingProperties on an embedded anonymous struct
Recurse into embedded anonymous structs and the BlueprintEmbed
workaround structs when looking for properties in
extendPropertiesRecursive.

Test: proptools/extend_test.go
Change-Id: I975651a64e5173747403629a09263562761f1495
2021-06-28 17:08:52 -07:00
Lukacs T. Berki
1602226f23 Module/providers can now emit extra JSON data.
Test: Presubmits.
Change-Id: I448e85f9144b9b35e7822ab7629329bae7a2fb8e
2021-06-25 10:31:10 +02:00
Spandan Das
78f4755aaa Delete creation of empty glob file am: 3a495910ad
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1736533

Change-Id: I9f3365344ef37afa9f389b604c4a68a2d7f8c995
2021-06-15 19:46:19 +00:00
Spandan Das
3a495910ad Delete creation of empty glob file
An empty glob file would be created inside the source tree if
soong_build is invoked directly (instead of via soong_ui). This logic is
now moved to soong/ui/build to prevent this behavior

Bug: 187194795
Test: Ran the following command locally for the target
art-target-arm:git_master-art
```
. ./build/envsetup.sh && lunch armv8-eng &&
art/tools/buildbot-build.sh --target
```
Before change glob file is created in $TOP/.bootstrap
After change glob file is created in $TOP/out/soong/.bootstrap

Change-Id: I6745996c940d790d1f1f387ca08aafc6a8a669df
2021-06-15 17:57:03 +00:00
Treehugger Robot
2b3a6a50d5 Merge "Add support for maps as properties" am: 16caaa4817
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1715913

Change-Id: I134c53bd884308243b4c0c2d026736bec92288c8
2021-05-27 20:06:35 +00:00
Treehugger Robot
16caaa4817 Merge "Add support for maps as properties" 2021-05-27 19:32:37 +00:00
Colin Cross
bec4769941 Add a test for AddNinjaFileDeps am: 7a2a352825
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1715723

Change-Id: Ieb32a2a0a399fc3fe7f354cf0bd20621ea3d5342
2021-05-26 17:21:44 +00:00
Liz Kammer
c1ccfee2bd Add support for maps as properties
This support enables specifying properties of the type "map" within a
Soong module, but explicitly does not allow them to be used within a bp
file.

This means that rather than specifying each arch/os/target within a
struct to support arch-variant properties/attributes, we can use a map.
This allows us to simplify the implementation of LabelAttribute,
StringListAttribute, and LabelListAttribute as the number of select
statements supported becoming large and hard results in a lot of
duplication.

Test: go test blueprint tests
Test: m nothing
Change-Id: I88cc5952a6bdb60a2344fa0737216f016086cea5
2021-05-26 09:54:22 -04:00
Colin Cross
7a2a352825 Add a test for AddNinjaFileDeps
Add a test for all the contexts that call AddNinjaFileDeps to ensure
they are propagated back to the main method.

Bug: 188547846
Test: TestAddNinjaFileDeps
Change-Id: Iaab32b8237eecee07fb930f2f9fe337c7d4e047c
2021-05-24 17:20:55 -07:00
Dan Willemsen
5832a98338 Add blueprint_go_packages phony to build all go packages am: 0f0e304d08
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1713431

Change-Id: I6fa5e01c5e68cd933eb8abe29d808ff4941bcf0d
2021-05-24 21:38:44 +00:00
Dan Willemsen
0f0e304d08 Add blueprint_go_packages phony to build all go packages
We have `blueprint_tools` that can build all Go binaries (and run their
associates tests). But if a package is not (yet) used, it won't be
included there. This makes testing the build of new Go packages
difficult.

Test: m blueprint_go_packages
Change-Id: I7a67cf2593fedcf499d40d82c266d3b0d7ebbef2
2021-05-24 11:36:37 -07:00
Treehugger Robot
2548a93c36 Merge "Fix AddNinjaFileDeps in a LoadHook" am: 69e2bbe2df
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1712169

Change-Id: Ic8a91706868263a18bee9b51af172a13884ddce4
2021-05-21 23:18:15 +00:00
Treehugger Robot
69e2bbe2df Merge "Fix AddNinjaFileDeps in a LoadHook" 2021-05-21 23:02:18 +00:00
Bill Peckham
4c3f2a4a4b Merge "Add OtherModuleFarDependencyVariantExists." am: 1f716db99f
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1697063

Change-Id: I0ee202a7a8003d0226c263cb999d973121e55dc9
2021-05-21 02:01:14 +00:00
Bill Peckham
1f716db99f Merge "Add OtherModuleFarDependencyVariantExists." 2021-05-21 01:37:21 +00:00
Martin Stjernholm
53bf14cda7 Add OtherModuleFarDependencyVariantExists.
This is the logical extension corresponding to
AddFarVariationDependencies in the same way
OtherModuleDependencyVariantExists corresponds to
AddVariationDependencies.

Test: m nothing
Bug: 188398129
Change-Id: I5517bfd6be5e2c58432c2902dfd1ca7668c76598
2021-05-20 16:30:02 +00:00
Colin Cross
13b5befc5c Fix AddNinjaFileDeps in a LoadHook
Propagate the ninja file deps from a LoadHook to the build.ninja.d
file.

Bug: 188547846
Test: next CL
Change-Id: If8176474b5094ee40d07df12f5da79a906ce7290
2021-05-19 10:30:58 -07:00
Colin Cross
419285d831 Merge "Support go 1.16 in gotestmain" am: d7d1e39aaf
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1696685

Change-Id: I400af9c08bbc2f29324035cb8a27523013edeabf
2021-05-14 22:31:30 +00:00
Colin Cross
d7d1e39aaf Merge "Support go 1.16 in gotestmain" 2021-05-14 22:13:32 +00:00
Colin Cross
47331e6630 Add support for setting string property in bpmodify am: b1abbada6a
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1692188

Change-Id: Icb4ba0eed829c4fd415e5b445996259c1c3499f0
2021-05-07 15:30:29 +00:00
Colin Cross
459fe182ff Improve bpmodify_test.go am: 5ef7b6608c
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1692187

Change-Id: Ib85af9584a93acdac73dd4d4cd657c9d81b55747
2021-05-07 15:30:27 +00:00
Colin Cross
ce447fc83e Support go 1.16 in gotestmain
go 1.16 adds a new method on the testDeps interface passed to
testing.MainStart.

Test: m nothing
Change-Id: I5c42a4294a775337696dc2d64791dee301eac390
2021-05-04 19:29:23 -07:00
Colin Cross
b1abbada6a Add support for setting string property in bpmodify
Add a -str argument to set a property to a string value.

Bug: 186723288
Test: bpmodify_test.go
Change-Id: I490e3be182693c8720020814da579e6e788b3d9f
2021-05-04 08:02:13 -07:00
Colin Cross
5ef7b6608c Improve bpmodify_test.go
Make TestProcessModule use subtests, give the tests names, and
use named fields in the testcase struct.

Test: bpmodify_test.go
Change-Id: I85345bd419e4c3ccf0050b7efae0479dd03d8bea
2021-04-29 19:11:39 -07:00
Colin Cross
c930745caa Make AddVariationDependencies return list of nil for non-parallel mutators am: 2f7d3dd6fb
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1689707

Change-Id: I0f09501f6e5e884fed48a0b67adf23ccae9dbb90
2021-04-29 00:13:53 +00:00
Colin Cross
54d8f87981 Add variant to dependency cycle errors am: e5ff770c95
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1689306

Change-Id: I5146b753d84b0df9125c14a0dc617198f7c51852
2021-04-29 00:13:52 +00:00
Colin Cross
010f5bbd8d Speed up finding dependency cycles am: 9793b0a5e0
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1689305

Change-Id: I28b57ccfe40e04e446d8ff4c721752ff1a69aae9
2021-04-29 00:13:51 +00:00
Colin Cross
2f7d3dd6fb Make AddVariationDependencies return list of nil for non-parallel mutators
According to the documentation, Add[Variation]Dependencies should be
returning a list the same length as the input list.  A non-parallel
mutator doesn't support pausing to wait for new dependencies to finish
the current mutator pass, so it can't return the new dependencies.  Make
it return a list of nil instead to match the contract.

Bug: 186539038
Test: TestAddVariationDependencies
Change-Id: I07d0ac1f3024dcb7c94a2518910a68b61cd731e2
2021-04-28 11:50:37 -07:00
Colin Cross
e5ff770c95 Add variant to dependency cycle errors
Use moduleInfo.String() to print dependency cycle errors so they
contain the variant.

Test: Test_parallelVisit
Change-Id: I2bddaa35c8abb57c42b4c424e861361a8813d588
2021-04-28 09:40:47 -07:00
Colin Cross
9793b0a5e0 Speed up finding dependency cycles
parallelVisit supports mutating the dependency graph while it is being
visited by proceeding until there are no modules with their dependencies
satisfied, and then checking if there are modules that haven't been
visited yet.  If so, it assumes there was a newly introduced dependency
cycle and tries to find it to return as an error.

Finding the dependency cycle could traverse outside of the cycle.
If the dependency cycle occurs near the bottom of the dependency graph,
that traversal could be both long and wide, leading to very long
runtimes.

Memoize traversed modules that were not found to be part of the
dependency cycle to prevent repeated traversals.

Fixes: 186572387
Test: introduce cycle into libc, m nothing
Test: Test_parallelVisit
Change-Id: I38d0749dbedffbe8a39e433d97fbe08486451321
2021-04-28 09:40:47 -07:00
Colin Cross
c12ed83e19 Merge "Speed up globs with sharding" am: 9021eef07b
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1668808

Change-Id: Ic86c4034860e607bfce3523a07e94fba69bcccc3
2021-04-20 16:27:29 +00:00
Colin Cross
9021eef07b Merge "Speed up globs with sharding" 2021-04-20 16:16:40 +00:00
Lukács T. Berki
412934164e Merge "Remove the extraNinjaFileDeps arg of RunBlueprint." am: d04dcc89f4
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1676293

Change-Id: I957eb9766c62a83075c882ec2bb9e2d083c4472e
2021-04-19 07:04:21 +00:00
Lukács T. Berki
9e791184e5 Merge "Use the correct build dir." am: 8308aab95e
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1676296

Change-Id: I83c05f0ef7e9f0bdb5985546005a953064ee874a
2021-04-19 07:04:05 +00:00
Lukács T. Berki
d04dcc89f4 Merge "Remove the extraNinjaFileDeps arg of RunBlueprint." 2021-04-19 06:46:52 +00:00
Lukács T. Berki
8308aab95e Merge "Use the correct build dir." 2021-04-19 06:46:16 +00:00
Colin Cross
2523698c12 Speed up globs with sharding
There are a few cases that force all globs to be rerun at the beginning
of the build (changes to bpglob or dependencies, second build after a
clean build).  The number of globs has gotten high enough that rerunning
them all can have significant overhead to start bpglob for each one.

Replace the per-glob bpglob invocations with sharded invocations using
1024 hash buckets.

Bug: 159845846
Test: glob_test.go
Test: m nothing && m nothing
Test: build/soong/bootstrap_test.sh
Change-Id: Ife1f7a03c8f6b25d1be01531425d8dc2c76d1ea0
2021-04-15 11:04:11 -07:00
Colin Cross
a8444548f3 Merge "Version bpglob command line arguments" am: 45222ec3ca
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1673987

Change-Id: I207265e94a6ea9b2dcd3456cbfa54773c8e1d04d
2021-04-15 17:09:12 +00:00
Colin Cross
45222ec3ca Merge "Version bpglob command line arguments" 2021-04-15 16:42:08 +00:00
Lukacs T. Berki
af5d832934 Make Blueprint better callable from Go: am: 98e0efb9a4
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1675531

Change-Id: Ic1d17a40736c9a7934745db0217cb301c5785487
2021-04-15 13:59:45 +00:00
Lukacs T. Berki
f646afb52f Use the correct build dir.
Args.BuildDir was accidentally hard-wired to "." in one of my previous
changes, and since Blueprint is not really independent of Soong anymore,
it's probably better to fix it in a way that doesn't involve putting
back the almost-but-not-completely redundant "-b" command line argument
to the primary builder invocation.

Test: Presubmits.
Change-Id: I76d064a99be16cd812cb47a6c448fe44b51893dc
2021-04-15 15:46:35 +02:00
Lukacs T. Berki
f4d43ac0d7 Remove the extraNinjaFileDeps arg of RunBlueprint.
The only thing that was done with it was to append it to the list of
dependencies returned, which can just as well be done by the caller.

Test: Presubmits.
Change-Id: Ibedf024de411a5881f19ed0c137a91b0898f1269
2021-04-15 15:05:26 +02:00