Commit graph

523 commits

Author SHA1 Message Date
Treehugger Robot
e1b9e95ddd Merge "Discard PT_INTERP in embedded linker linker script" 2022-02-17 04:08:58 +00:00
Colin Cross
4d9b1f1d87 Discard PT_INTERP in embedded linker linker script
Discarding PT_INTERP in the linker script generated for the embedded
linker allows linking without the --no-dynamic-linker argument.

Bug: 216192129
Test: manual
Change-Id: I0e293944c5c804b4512fb834f8f18e1d6ecd053c
2022-02-16 15:36:29 -08:00
Liz Kammer
124967f597 Show failing products in multiproduct_kati
Test: cause products to fail and verify they appear in error message
Change-Id: I4111b919aebc0b8d80ecbd17be3c2745c75b76a5
2022-02-14 15:42:53 -05:00
Liz Kammer
4ae119c14c In Soong, set max files soft limit to hard limit
Test: m nothing
Change-Id: I4f45ebcf8c7b74315c371012603aeb7c541ae336
2022-02-09 13:16:23 -05:00
Liz Kammer
a754178d4f Add logging of file resource limits in Soong
Test: m nothing and verify in logs
Change-Id: I4b51089ce1d222927a180b260fb9bc0e3b77c468
2022-02-07 13:38:52 -05:00
Kousik Kumar
5ddefa3df4 Merge "Move env var config loading to be within the config package." 2022-01-31 04:40:50 +00:00
Usta Shrestha
b3af19caa2 Merge "list all valid flags (commands) to soong_ui upon unknown command" 2022-01-29 06:34:19 +00:00
kgui
67007248a5 Add new file name to output inputs/outputs of actions of modules into a new file when m json-module-graph is executed.
And also delete the logic in droidstubs to output JSONDataActions.

Test: local.

Change-Id: Ib5d6f1f69c16c41f128e481131775c7699f96464
2022-01-28 13:52:50 +08:00
Kousik Kumar
3ff037e3d9 Move env var config loading to be within the config package.
When the environment variable config file is loaded after config object
is initialized, we  end up loading the USE_RBE variable in the config
file after the code to start reproxy process is run. This causes a
problem when USE_RBE variable itself is set in the config files. To
avoid this, I've moved the config file loading code to be within the
config package, this makes the main.go file cleaner too.

Test:
Ran a build with USE_RBE:true set in the config json file. The build
fails without this change and succeeds with this change.

Bug: b/209486170
Change-Id: Iab3957b64f4b5456a861057d16ad318f4f78e0cb
2022-01-26 18:40:43 -05:00
Usta
6feae38aa3 list all valid flags (commands) to soong_ui upon unknown command
Test: m nothing
Bug: N/A
Change-Id: I0dd94c18e945a0ef4311b076d49086281ef288af
2022-01-15 01:42:19 -05:00
Kousik Kumar
7b7dca4316 Add additional directories from which env config can be loaded
This is useful for external users to be able to specify their own config files that can be loaded by soong during startup. In addition, we need this for upcoming changes to incorporate an experiments framework in Soong since the config file will be fetched from CDPush and put into the OUT_DIR folder by the config file fetcher binary.

Note: Once this is merged into internal branch, I'll fully get rid of
the vendor/google/ path from Soong in aosp.

Test:
1. Ran a build in aosp with these changes and no config file was loaded.
2. Ran a build in internal master with these changes and the current
config file inside vendor/google was loaded as expected.

Bug: b/214035335
Change-Id: I9af83687d4eaeee1ffb0f88a750cfeb7c6d2bafb
2022-01-14 00:22:36 -05:00
Alex Márquez Pérez Muñíz Díaz Púras Thaureaux
947fdbfdee Log bp2build_metrics .pb
Also share `Save(pb proto.Message, filepath string)`

Bug: 201539536
Test: bp2build_metrics.pb has expected content & path
Test: m nothing
Test: {bp2build,mixed_{libc,droid}}.sh
Test: CI
Change-Id: I7d8ad87fca6a4b0355010090a527f5ae67b27c88
2021-12-13 13:03:44 -05:00
Liz Kammer
1aefc9d2d2 Support a quieter bp2build
Tie verbosity of bp2build to env var BP2BUILD_VERBOSE

Test: m bp2build
Test: BP2BUILD_VERBOSE=true m bp2build
Change-Id: Ib8f10ad73ed06c935126a77d18c1d9febc25348f
2021-12-02 10:00:29 -05:00
Treehugger Robot
ec014011ea Merge "sbox: run commands using script for large commands" 2021-11-19 22:06:02 +00:00
Colin Cross
ffbcd1d8a0 Extract primary apk from apk set zip
Extract and install the primary apk normally, and then unzip the rest
of them as a post install command.

Bug: 204136549
Test: app_set_test.go
Change-Id: I17437ff27f49df6bc91bdbbea6173b46c7d3ec4e
2021-11-17 19:55:33 -08:00
Kevin DuBois
0a5da7025e sbox: run commands using script for large commands
Linux has limit of 32 pages for a single argument. Sbox's method of
handling commands as a single large string can run into this limitation.

Write the large command to a script file and then execute that file.
This better accommodates the large commands, and leaves a script in the
temporary directory useful for manual inspection if necessary.

Bug: 177070955
Bug: 174232579
Fixes: 202297224
Test: m
Test: make particular target with very long command exceeding limit.
Change-Id: Ia298fdfd7a759821c37f540deaf800026041e511
2021-11-12 00:48:34 +00:00
Jingwen Chen
a47f28d28e bp2build: add support for soong_config_module_type.
Test: CI, go unit test
Bug: 198556411
Change-Id: Idf862904d51d822f92af0c072341c31b7a02fc64
2021-11-08 13:38:28 +00:00
Lukacs T. Berki
eb0454b2a4 Generate actions when building the module graph.
This is necessary because GenerateAndroidBuildActions() is effectively a
mutator and therefore can change the data stored in each module.
Sometimes one wants to dump that data in the JSON module graph, and in
order to do so, actions need to be generated.

I briefly entertained removing StopBeforePrepareBuildActions(), but it
incurs a 10-second penalty on "m soong_docs" and "m bp2build" so I
decided to keep it.

Test: Presubmits.
Change-Id: I438aaf981e989dc39a09e8a2ba73f7a08ee55386
2021-10-26 13:32:59 +02:00
Dan Willemsen
2a109db10b Merge "Reduce modules exposed to Make in Mac builds" 2021-10-19 00:32:54 +00:00
Dan Willemsen
def7b5d198 Reduce modules exposed to Make in Mac builds
Device builds are no longer supported on Mac, but we do support building
various host tools, including the SDK build-tools and platform-tools
packages. These have dependencies on [java] device modules, so we don't
completely disable device modules, only hide them from Make (which makes
them more difficult to trigger from the command line).

Also fix the mac build of multiproduct_kati, so that `m blueprint_tools`
works on Mac.

Bug: 187222815
Test: `m`, `m dist`, etc on Mac
Change-Id: I92f16605d5cd173d431cbcb79081234d45cc6e2e
2021-10-18 12:44:45 -07:00
Colin Cross
5cb7366c58 Up per-product memory usage to 30 GB
Reduce multiproduct_kati parallelism by upping required memory per product
to 30 GB.

Bug: 201671935
Test: builds
Change-Id: Ia756f309459d28e22a49f190a5227a139b356a1f
2021-10-18 10:18:51 -07:00
Romain Jobredeaux
6a779a4b50 Merge "remove redundant 'deps' section from template with 'exports' section." 2021-09-27 20:54:51 +00:00
Dan Willemsen
66213a64b2 Disable protobuf deterministic randomness in soong_build
Soong writes out text protos for the sbox manifests, and if those change
every time soong changes, we trigger rebuilds of significant portions of
the tree. This is very annoying when attempting to iterate on Soong, so
disable this functionality at the soong_build level.

Bug: 200727888
Change-Id: Icf21e1775a4c4200e983c1b8ac66ddd780d7c90c
2021-09-22 15:07:53 -07:00
Romain Jobredeaux
b669398c45 remove redundant 'deps' section from template with 'exports' section.
Test: pom2bp -regen prebuilts/sdk/current/support/BUILD.bazel
Bug: 198418951
Change-Id: I4d0e3d100d0e319473723a75077e2520fc1535e1
2021-09-22 16:50:33 -04:00
Colin Cross
fac6fd0927 Merge changes I0b78ceeb,Ic57e1efd
* changes:
  Add environment variable to force keeping ANSI codes
  Disable table mode in smart status if window size is not available
2021-09-22 18:19:15 +00:00
Romain Jobredeaux
5ff942285c Merge "Add a flag-protected (-pom2build) feature for pom2bp to produce Bazel BUILD files." 2021-09-22 14:47:02 +00:00
Romain Jobredeaux
89cb22416d Add a flag-protected (-pom2build) feature for pom2bp to produce Bazel BUILD files.
Note that this is an initial, rather crude attempt at this,
that unblocks generating BUILD files needed to build AOSP android apps.
In particular, note that
- Jetifier step currently ignored
- Visibility set to public for all output targets
- All dependencies are gathered in 'deps', though they are split under various categories in bp files
- Host, host-only notions are ignored for now

Test: Copy first 2 lines of, e.g. prebuilts/sdk/current/support/Android.bp to a BUILD.bazel file and run pom2bp -pom2build -regen BUILD.bazel
Bug: 198433754
Change-Id: I5efa93517e6c319fd828da5b16ba7efd2d90bca8
2021-09-21 20:36:07 +00:00
Spandan Das
235a9443b7 Merge "Add Property.Type to bpdocs template" 2021-09-21 20:00:35 +00:00
Colin Cross
3c0fe0edc0 Add environment variable to force keeping ANSI codes
If SOONG_UI_ANSI_OUTPUT is set to a true value force the simple status
output to keep ANSI codes.  This will allow buildbots to collect logs
with ANSI codes in them.  Smart status output is not affected as it
always keeps ANSI codes.

Bug: 147310922
Test: manual
Change-Id: I0b78ceebb65125b8e8dafb4787816fb679d3eb3e
2021-09-21 12:53:30 -07:00
Colin Cross
2f08bc1a5c Increase timeouts in Test_runWithTimeout
Under load sometimes the 1 millisecond timeouts are not run before
a 1 second sleep finishes, or `echo foo` is not run before a 1
second timeoutt.  Increase the long sleeps and timeouts to 10 seconds.
This doesn't affect normal test runtime as the command should timeout
after 1 millisecond.

Bug: 199207577
Test: Test_runWithTimeout
Change-Id: I31166850a473b9cde37906a62d026580e7a7bd3f
2021-09-17 21:36:29 +00:00
Liz Kammer
83d07a4c41 Merge "Expand handling of unconverted deps in bp2build" 2021-09-15 13:49:48 +00:00
Liz Kammer
6eff323206 Expand handling of unconverted deps in bp2build
Support three options for converting modules with unconverted
dependencies
1. (default) Warn when converting a module if it has unconverted deps.
2. Error when encountering a module with unconverted deps. (not hooked
up yet)

Test: build/bazel/ci/bp2build.sh
Test: build/bazel/ci/mixed_libc.sh
Test: BP2BUILD_ERROR_UNCONVERTED=1 build/bazel/ci/bp2build.sh with
      unconverted deps -- get appropriate error
Bug: 181155349
Change-Id: Ifaabf0cd2e43e963366dc137159c705294165c3d
2021-09-14 19:55:55 +00:00
Alan Viverette
b89663b7d9 Merge "Patch "Add support for --optional-uses-libs to pom2bp" to match master" 2021-09-14 15:04:28 +00:00
Alan Viverette
bcbfc5f53c Patch "Add support for --optional-uses-libs to pom2bp" to match master
Cherry pick that was originally submitted was not up-to-date.

Change-Id: Ia599ea294926666cddfd84f8c2a3e22e7e7c0434
2021-09-13 19:28:35 +00:00
Spandan Das
eabda9708a Add Property.Type to bpdocs template
This field was missing for Property objects that have non-empty inner
Property objects

Bug: 172797653
Test: m soong_docs
Change-Id: Iee9c66f8b85d68a6b5bf18fd9787143191c4f002
2021-09-12 18:40:37 +00:00
Treehugger Robot
c521d5717d Merge "Add support for --optional-uses-libs to pom2bp" 2021-09-10 20:15:35 +00:00
Lukacs T. Berki
89fcdcb788 Refactor the creation of soong_build calls.
They were a bit repetitive.

In addition, make the environment checks more correct; they could
probably use an integration test or two to make sure that when the
environment changes, exactly those outputs are rebuilt that need to be,
but for now, this is an improvement already.

Test: Presubmits.
Change-Id: Idd79b81ca6975d57d00e5bf4699d266152505ff8
2021-09-08 09:36:22 +02:00
Lukacs T. Berki
c6012f36e1 Invoke soong_docs from the bootstrap Ninja file.
This makes soong_ui the only place where soong_build is invoked, thus
greatly simplifying the conceptual model of the build.

It comes with the slight limitation that now soong_docs (and queryview
and the JSON module graph) are not Make targets anymore, but I suppose
that's an acceptable loss.

The only place where someone depended on soong_docs from a Makefile is
removed in a separate change.

Test: Presubmits.
Change-Id: I3f9ac327725c15d84de725d05e3cdde1da3dcbe2
2021-09-08 09:31:52 +02:00
Lukacs T. Berki
ffc9e8d812 Pass StopBefore as an argument to RunBlueprint.
Its value is a function of the call site, so it doesn't make a lot of
sense to plumb it through the configuration.

Test: Presubmits.
Change-Id: If928b34de075969fd42932212ce9187808cbdf86
2021-09-07 18:14:07 +02:00
Lukacs T. Berki
7edd91c0f4 Make incremental queryview builds work.
The breakage was because:
- aosp/1818236 removed the "rm -fr" invocation from the Ninja file that
removed the old queryview tree
- queryview files were created with permissions 440 to discourage manual
editing

These two together made incremental queryview invocations not be able to
overwrite its output files.

(Of course, it also made them incorrect since old files were lft in place)

Test: Presubmits.
Change-Id: Idaa3e1db82c9eb8325c48e37aca5146e69af7333
2021-09-07 08:47:21 +02:00
Lukacs T. Berki
3a82169a55 Invoke queryview from the bootstrap Ninja file.
It used to be invoked from out/soong/build.ninja, which required two
soong_build invocations one after the other (ne to generate
out/soong/build.ninja, one to generate the queryview workspace). This
was slower and required some shell-quoted-in-ninja-quoted-in-Go .

Test: Presubmits.
Change-Id: Idda79c067606663b66e9f94626fa24f3b5af4114
2021-09-06 17:10:49 +02:00
Lukacs T. Berki
d6cee7e374 Make OutDir() and SoongOutDir() be consistent.
They both used to return out/soong. Unfortunately, the tests seem to
assume the old behavior, so I had to keep tests inconsistent and I was
scared of dexpreopt, too.

Test: Presubmits.
Change-Id: Ib00c7956fd2cdb28cc11d481c17e5a3ff5ad563a
2021-09-02 13:35:49 +02:00
Lukacs T. Berki
ea1a31c07f Remove the bootstrap.Config class.
It was confusing because bootstrapping uses two configurations: the
"global" config and the special-cased bootstrap one.

This change merges them.

Test: Presubmits.
Change-Id: I82b482cbe28a343ab6991374b2a28667e1a06b48
2021-09-02 09:59:35 +02:00
Lukacs T. Berki
a806e4176b Remove bootstrap_go_binary .
blueprint_go_binary is enough.

Test: Presubmits.
Change-Id: Ic95ed8533603dff396740bc1d2a7c60ff2cebf20
2021-09-01 11:55:22 +02:00
Alan Viverette
24658d07f8 Add support for --optional-uses-libs to pom2bp
Fixes: 198303986
Test: ./update_prebuilts.py -x <build-id>
Change-Id: I4eb322e905f636eb18d1940f05682f38e448a344
2021-08-31 20:30:41 +00:00
Lukacs T. Berki
bf5bdb2f3f Write logs.zip + logs to the correct place.
I erroneously assumed that the continuous builds don't set DIST_DIR,
whereas they do.

The variable naming is quite questionable, but for now, the important
thing is that the code works and I wanted to do that with the least
amount of change possible.

Also add some more logging:
- Shell environment
- Command line arguments
- Location of logs.zip
- The logs directory

Test: Presubmits.
Change-Id: I651b1045eeef568aa860fd2457fa7799ad68ba38
2021-08-31 19:58:51 +02:00
Lukacs T. Berki
b078ade28d Remove some unused args from Blueprint.
These are: TopFile and GeneratingPrimaryBuilder.

Also re-shuffle the list of flags to make a bit more sense and finish
the rename of BuildDir and NinjaBuildDir to SoongOutDir and OutDir,
respectively.

Test: Presubmits.
Change-Id: I103ff5f09f1c0d16f695a7da5dea13b55028e33e
2021-08-31 11:42:48 +02:00
Lukacs T. Berki
e571dc3bd0 Print the JSON module graph correctly.
Before, we piggybacked on the implementation of regular soong_build and
wrote a fake build.ninja file to satisfy Ninja.

Now, instead, the JSON module graph is a a separate action in the Ninja
output file. This has the pleasant side effect that one can flip back
and forth between generating the JSON file and regular Soong without
loss of incrementality.

Side cleanup: write .d files in a slightly cleaner way.

Test: Presubmits.
Change-Id: Ia853383567b9dd31c53f3bdf56cfc8d517b498ec
2021-08-27 15:55:51 +02:00
Lukacs T. Berki
9f6c24a887 Rename BuildDir and NinjaBuildDir.
These are just out/ and out/soong/ and the old names were quite
confusing.

Test: Presubmits.
Merged-In: I999ca62162062f27e449f83ffb00fa580d4065b8
Merged-In: I2fcd80a9e376a6842a06aa104a9f6f5c5e16f8c7
Change-Id: Ib481d2aac40df8da1b74174dc8ec576c1cb48be0
2021-08-27 10:08:49 +02:00
Lukacs T. Berki
809d2ed548 Do globbing-related paperwork explicitly.
In particular, the writing of build-globs.ninja and the collection of
glob list files for build.ninja.d is now done outside of the Blueprint
invocation.

This is useful because RunBlueprint() is sometimes called multiple times
in a single soong_build invocation and because now glob processing is
not hidden by the machinery of Blueprint.

Test: Presubmits.
Change-Id: I6eb19b357901e452e6d1e7ce055f8a3bab4a74f4
2021-08-18 17:11:56 +02:00