Commit graph

1979 commits

Author SHA1 Message Date
Colin Cross
ce447fc83e Support go 1.16 in gotestmain
go 1.16 adds a new method on the testDeps interface passed to
testing.MainStart.

Test: m nothing
Change-Id: I5c42a4294a775337696dc2d64791dee301eac390
2021-05-04 19:29:23 -07:00
Colin Cross
b1abbada6a Add support for setting string property in bpmodify
Add a -str argument to set a property to a string value.

Bug: 186723288
Test: bpmodify_test.go
Change-Id: I490e3be182693c8720020814da579e6e788b3d9f
2021-05-04 08:02:13 -07:00
Colin Cross
5ef7b6608c Improve bpmodify_test.go
Make TestProcessModule use subtests, give the tests names, and
use named fields in the testcase struct.

Test: bpmodify_test.go
Change-Id: I85345bd419e4c3ccf0050b7efae0479dd03d8bea
2021-04-29 19:11:39 -07:00
Colin Cross
f22bd2471a Make AddVariationDependencies return list of nil for non-parallel mutators am: 2f7d3dd6fb am: c930745caa am: 40ee8f7236 am: 2d803d6102
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1689707

Change-Id: I03c91fc176509a8a383fcfaf478808d01bbb229c
2021-04-29 04:07:02 +00:00
Colin Cross
c1765fe8c2 Add variant to dependency cycle errors am: e5ff770c95 am: 54d8f87981 am: 2ef3c79d58 am: f1c5c636b5
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1689306

Change-Id: I11ae75de0eefea9d796c0298b309377caaf896cc
2021-04-29 04:07:02 +00:00
Colin Cross
3b3965db39 Speed up finding dependency cycles am: 9793b0a5e0 am: 010f5bbd8d am: 92f64550cd am: 08d29011da
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1689305

Change-Id: I143cceaa1af1d74f1b2f3ad0e87186b3de0bb922
2021-04-29 04:07:01 +00:00
Colin Cross
2d803d6102 Make AddVariationDependencies return list of nil for non-parallel mutators am: 2f7d3dd6fb am: c930745caa am: 40ee8f7236
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1689707

Change-Id: Ic902f494d368c6ffec8150a2b310e4830d2f9819
2021-04-29 03:37:37 +00:00
Colin Cross
f1c5c636b5 Add variant to dependency cycle errors am: e5ff770c95 am: 54d8f87981 am: 2ef3c79d58
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1689306

Change-Id: Ifbffea250fb729d65aa9848edb491e894db68b58
2021-04-29 03:37:36 +00:00
Colin Cross
08d29011da Speed up finding dependency cycles am: 9793b0a5e0 am: 010f5bbd8d am: 92f64550cd
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1689305

Change-Id: I7fce8059f965a5d4db187e8165c518040ccff10b
2021-04-29 03:37:35 +00:00
Colin Cross
40ee8f7236 Make AddVariationDependencies return list of nil for non-parallel mutators am: 2f7d3dd6fb am: c930745caa
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1689707

Change-Id: If389553339f8e6998e27beee1a859577fad49543
2021-04-29 03:00:27 +00:00
Colin Cross
2ef3c79d58 Add variant to dependency cycle errors am: e5ff770c95 am: 54d8f87981
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1689306

Change-Id: I896d8c4d6dfb1b8fedf0fcee78d0f6cfdee41a67
2021-04-29 03:00:26 +00:00
Colin Cross
92f64550cd Speed up finding dependency cycles am: 9793b0a5e0 am: 010f5bbd8d
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1689305

Change-Id: I65385bbe06dc0877fb4d8d01cb8e636804722ce2
2021-04-29 03:00:25 +00:00
Colin Cross
c930745caa Make AddVariationDependencies return list of nil for non-parallel mutators am: 2f7d3dd6fb
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1689707

Change-Id: I0f09501f6e5e884fed48a0b67adf23ccae9dbb90
2021-04-29 00:13:53 +00:00
Colin Cross
54d8f87981 Add variant to dependency cycle errors am: e5ff770c95
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1689306

Change-Id: I5146b753d84b0df9125c14a0dc617198f7c51852
2021-04-29 00:13:52 +00:00
Colin Cross
010f5bbd8d Speed up finding dependency cycles am: 9793b0a5e0
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1689305

Change-Id: I28b57ccfe40e04e446d8ff4c721752ff1a69aae9
2021-04-29 00:13:51 +00:00
Colin Cross
2f7d3dd6fb Make AddVariationDependencies return list of nil for non-parallel mutators
According to the documentation, Add[Variation]Dependencies should be
returning a list the same length as the input list.  A non-parallel
mutator doesn't support pausing to wait for new dependencies to finish
the current mutator pass, so it can't return the new dependencies.  Make
it return a list of nil instead to match the contract.

Bug: 186539038
Test: TestAddVariationDependencies
Change-Id: I07d0ac1f3024dcb7c94a2518910a68b61cd731e2
2021-04-28 11:50:37 -07:00
Colin Cross
e5ff770c95 Add variant to dependency cycle errors
Use moduleInfo.String() to print dependency cycle errors so they
contain the variant.

Test: Test_parallelVisit
Change-Id: I2bddaa35c8abb57c42b4c424e861361a8813d588
2021-04-28 09:40:47 -07:00
Colin Cross
9793b0a5e0 Speed up finding dependency cycles
parallelVisit supports mutating the dependency graph while it is being
visited by proceeding until there are no modules with their dependencies
satisfied, and then checking if there are modules that haven't been
visited yet.  If so, it assumes there was a newly introduced dependency
cycle and tries to find it to return as an error.

Finding the dependency cycle could traverse outside of the cycle.
If the dependency cycle occurs near the bottom of the dependency graph,
that traversal could be both long and wide, leading to very long
runtimes.

Memoize traversed modules that were not found to be part of the
dependency cycle to prevent repeated traversals.

Fixes: 186572387
Test: introduce cycle into libc, m nothing
Test: Test_parallelVisit
Change-Id: I38d0749dbedffbe8a39e433d97fbe08486451321
2021-04-28 09:40:47 -07:00
Colin Cross
7690bc67a6 Merge "Speed up globs with sharding" am: 9021eef07b am: c12ed83e19 am: 00e3b274ab
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1668808

Change-Id: I49d1ff1c5802006dbb0cb72dab42198d3d35fc37
2021-04-20 17:15:29 +00:00
Colin Cross
00e3b274ab Merge "Speed up globs with sharding" am: 9021eef07b am: c12ed83e19
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1668808

Change-Id: I19ac2ee09c0f5219b618bd08580ab15bc20083e0
2021-04-20 16:51:28 +00:00
Colin Cross
c12ed83e19 Merge "Speed up globs with sharding" am: 9021eef07b
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1668808

Change-Id: Ic86c4034860e607bfce3523a07e94fba69bcccc3
2021-04-20 16:27:29 +00:00
Colin Cross
9021eef07b Merge "Speed up globs with sharding" 2021-04-20 16:16:40 +00:00
Lukács T. Berki
8a20c8eae7 Merge "Remove the extraNinjaFileDeps arg of RunBlueprint." am: d04dcc89f4 am: 412934164e am: 308afcf580
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1676293

Change-Id: I59a5dbdeed6d9c28e9595864efdd99ba02d3e108
2021-04-19 07:58:31 +00:00
Lukács T. Berki
4e3fda237c Merge "Use the correct build dir." am: 8308aab95e am: 9e791184e5 am: a74c3fc65e
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1676296

Change-Id: Id63b5a46a3e351c6d227be51b38555c696c45bd9
2021-04-19 07:58:20 +00:00
Lukács T. Berki
308afcf580 Merge "Remove the extraNinjaFileDeps arg of RunBlueprint." am: d04dcc89f4 am: 412934164e
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1676293

Change-Id: Icc0b38209fb2007a89a9fe3ec67baf843e102f27
2021-04-19 07:32:26 +00:00
Lukács T. Berki
a74c3fc65e Merge "Use the correct build dir." am: 8308aab95e am: 9e791184e5
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1676296

Change-Id: I978937cdc9581255d6e5f14f83d431e5513df377
2021-04-19 07:32:16 +00:00
Lukács T. Berki
412934164e Merge "Remove the extraNinjaFileDeps arg of RunBlueprint." am: d04dcc89f4
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1676293

Change-Id: I957eb9766c62a83075c882ec2bb9e2d083c4472e
2021-04-19 07:04:21 +00:00
Lukács T. Berki
9e791184e5 Merge "Use the correct build dir." am: 8308aab95e
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1676296

Change-Id: I83c05f0ef7e9f0bdb5985546005a953064ee874a
2021-04-19 07:04:05 +00:00
Lukács T. Berki
d04dcc89f4 Merge "Remove the extraNinjaFileDeps arg of RunBlueprint." 2021-04-19 06:46:52 +00:00
Lukács T. Berki
8308aab95e Merge "Use the correct build dir." 2021-04-19 06:46:16 +00:00
Colin Cross
b05765bfd7 Merge "Version bpglob command line arguments" am: 45222ec3ca am: a8444548f3 am: f995bb2235
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1673987

Change-Id: I63ea818cc1c0e36fa2a7a5e082a88114ac605455
2021-04-15 18:42:19 +00:00
Colin Cross
2523698c12 Speed up globs with sharding
There are a few cases that force all globs to be rerun at the beginning
of the build (changes to bpglob or dependencies, second build after a
clean build).  The number of globs has gotten high enough that rerunning
them all can have significant overhead to start bpglob for each one.

Replace the per-glob bpglob invocations with sharded invocations using
1024 hash buckets.

Bug: 159845846
Test: glob_test.go
Test: m nothing && m nothing
Test: build/soong/bootstrap_test.sh
Change-Id: Ife1f7a03c8f6b25d1be01531425d8dc2c76d1ea0
2021-04-15 11:04:11 -07:00
Colin Cross
f995bb2235 Merge "Version bpglob command line arguments" am: 45222ec3ca am: a8444548f3
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1673987

Change-Id: I136354e7217590babf7a6d0b6ed73e1721a9b73c
2021-04-15 17:53:11 +00:00
Colin Cross
a8444548f3 Merge "Version bpglob command line arguments" am: 45222ec3ca
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1673987

Change-Id: I207265e94a6ea9b2dcd3456cbfa54773c8e1d04d
2021-04-15 17:09:12 +00:00
Colin Cross
45222ec3ca Merge "Version bpglob command line arguments" 2021-04-15 16:42:08 +00:00
Lukacs T. Berki
0f811944c9 Make Blueprint better callable from Go: am: 98e0efb9a4 am: af5d832934 am: 20d39d68df
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1675531

Change-Id: If3cb4784017a0b079f7821654fcbc5d67148cd58
2021-04-15 15:19:31 +00:00
Lukacs T. Berki
20d39d68df Make Blueprint better callable from Go: am: 98e0efb9a4 am: af5d832934
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1675531

Change-Id: I01df353a3beca3a8af8dc27f9906124d8be49b82
2021-04-15 14:51:12 +00:00
Lukacs T. Berki
af5d832934 Make Blueprint better callable from Go: am: 98e0efb9a4
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1675531

Change-Id: Ic1d17a40736c9a7934745db0217cb301c5785487
2021-04-15 13:59:45 +00:00
Lukacs T. Berki
f646afb52f Use the correct build dir.
Args.BuildDir was accidentally hard-wired to "." in one of my previous
changes, and since Blueprint is not really independent of Soong anymore,
it's probably better to fix it in a way that doesn't involve putting
back the almost-but-not-completely redundant "-b" command line argument
to the primary builder invocation.

Test: Presubmits.
Change-Id: I76d064a99be16cd812cb47a6c448fe44b51893dc
2021-04-15 15:46:35 +02:00
Lukacs T. Berki
f4d43ac0d7 Remove the extraNinjaFileDeps arg of RunBlueprint.
The only thing that was done with it was to append it to the list of
dependencies returned, which can just as well be done by the caller.

Test: Presubmits.
Change-Id: Ibedf024de411a5881f19ed0c137a91b0898f1269
2021-04-15 15:05:26 +02:00
Lukacs T. Berki
98d28c63e3 Make it possible to call the primary builder more than once. am: 78df853e07 am: a098ae38ab am: 59186d0207
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1675525

Change-Id: Ia4c00a35c7a14a18d466e0349742af50424ebae1
2021-04-15 12:51:33 +00:00
Lukacs T. Berki
59186d0207 Make it possible to call the primary builder more than once. am: 78df853e07 am: a098ae38ab
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1675525

Change-Id: I4fdff24ac1bcba1a6318d2b5754fe394b1e92dfd
2021-04-15 12:22:22 +00:00
Lukacs T. Berki
a098ae38ab Make it possible to call the primary builder more than once. am: 78df853e07
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1675525

Change-Id: I9d1649cb218a1f02ffa7b40d855e2d93e2de4170
2021-04-15 11:44:51 +00:00
Colin Cross
7e6f6b7932 Version bpglob command line arguments
bpglob is executed through the rules in build-globs.ninja to determine
whether soong_build needs to rerun.  That means when the arguments
accepted by bpglob change it will be called with the old arguments,
then soong_build will rerun and update build-globs.ninja with the new
arguments.

To avoid having to maintain backwards compatibility with old arguments
across the transition, a version argument is used to detect the
transition in order to stop parsing arguments, touch the output file
and exit immediately.  The version number in
pathtools.BPGlobArgumentVersion should be manually incremented when
the bpglob argument format changes.

If the version argument is not passed then a version mismatch is assumed.

Bug: 159845846
Test: build/soong/tests/bootstrap_test.sh
Change-Id: Id6742c7acc51af8d0d224d51e47bddea78e6e363
2021-04-14 09:28:22 -07:00
Lukacs T. Berki
98e0efb9a4 Make Blueprint better callable from Go:
- Make the parsed command line arguments accessible from other packages
- Separate writing the depfile from writing the Ninja file
- Extract a method to compute the name of the file list file for globs

Test: Presubmits.
Change-Id: I4d0daf0f3776a9336da305be0108ebf70a314776
2021-04-14 13:47:52 +02:00
Lukacs T. Berki
78df853e07 Make it possible to call the primary builder more than once.
This is accomplished by plumbing PrimaryBuilderInvocation in from
bootstrap.Args.

The confusing removal of "primaryBuilderExtraFlags" from documentation
generation is because it used to (erroneously) refer to the func defined
in command.go and its removal apparently doesn't break anything. I
suspect that this code path is dead within Android, but we'll need to
wait until May to remove it.

Test: Presubmits.
Change-Id: I8763579b82bd79d5ee0f233edfd48327282e401c
2021-04-14 10:28:54 +02:00
Colin Cross
f6417d0bc3 Merge "Make Glob return a GlobResult" am: 5ce0d78363 am: 646545f2e8 am: 8b5ccb279e
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1668807

Change-Id: I35f86c97ebb4655d2a7cd331d240ca2ae8548b9d
2021-04-13 19:12:06 +00:00
Colin Cross
8b5ccb279e Merge "Make Glob return a GlobResult" am: 5ce0d78363 am: 646545f2e8
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1668807

Change-Id: I4101f77d97dfd3e18e0064f2a57e5993696bf88e
2021-04-13 18:24:45 +00:00
Colin Cross
646545f2e8 Merge "Make Glob return a GlobResult" am: 5ce0d78363
Original change: https://android-review.googlesource.com/c/platform/build/blueprint/+/1668807

Change-Id: I19d371687f18c9e011f4c1ce29c2574a09d6282a
2021-04-13 17:41:01 +00:00
Colin Cross
5ce0d78363 Merge "Make Glob return a GlobResult" 2021-04-13 17:01:39 +00:00