Commit graph

184 commits

Author SHA1 Message Date
Chris Parsons
21f80277c3 Remove bazel dev mode
This mode is no longer in use, and is dead code.

Fixes: 282809863
Test: Presubmits
Change-Id: I45ab7d7c9dee153f1cbc66262c2cf9f1e9487a28
2023-06-15 14:14:19 +00:00
Cole Faust
319abae1c7 Remove --noexperimental_platform_in_output_dir
This is a followup to aosp/2606989.

This flag is not necessary now that we're using one platform name
for all of mixed builds.

Also rename current_product to mixed_builds_product so that it's clear
that that this platform should only be used for mixed builds.

In addition, make the bazelrc files point to the named products again
instead of the mixed build product so that b builds will still have
qualified outputs, but mixed builds won't.

Test: Presubmit and kernel build tools abtd run
Change-Id: I7f764cf42cd1323f4b495d1320931f59a076ac63
2023-06-08 17:58:35 -07:00
Treehugger Robot
ca69a6fc21 Merge "Revert "Remove --noexperimental_platform_in_output_dir"" 2023-06-08 12:10:14 +00:00
Matthias Männich
08ccec7c11 Revert "Remove --noexperimental_platform_in_output_dir"
This reverts commit 24af24e0f2.

Reason for revert: Breaks kernel build tools build https://ci.android.com/builds/branches/aosp_kernel-build-tools/grid?head=10282004&tail=10282004

Change-Id: I486d7f305d44cfecc51a0e039cc41c0afa6495f3
2023-06-08 08:19:14 +00:00
Treehugger Robot
731c4525ca Merge "Remove --noexperimental_platform_in_output_dir" 2023-06-08 06:08:14 +00:00
Spandan Das
bd1568178b Create GetPrebuiltFileInfo to Mixed builds cquery api
prebuilt_file Bazel targets do not have any build actions per se, but
return a PrebuiltFileInfo provider that determines installation rules.
To integrate this info into mixed builds, add a `GetPrebuiltFileInfo` method to
the API.

Bug: 280094612
Test: unit tests for prebuilt_etc in the next CL of this stack
Change-Id: I79da6b75b6a6cbd30869860d3620aeda731ea36a
2023-06-07 22:08:30 +00:00
Cole Faust
24af24e0f2 Remove --noexperimental_platform_in_output_dir
This is a followup to aosp/2606989.

This flag is not necessary now that we're using one platform name
for all of mixed builds.

Also rename current_product to mixed_builds_product so that it's clear
that that this platform should only be used for mixed builds.

In addition, make the bazelrc files point to the named products again
instead of the mixed build product so that b builds will still have
qualified outputs, but mixed builds won't.

Test: Presubmit
Change-Id: I5acbd18869589f67efb0ad3ebfda2c4d5a6a3850
2023-06-07 11:08:44 -07:00
Jingwen Chen
583ab219ea Create a "current_product" named platform.
This enables output paths of product agnostic targets (e.g. mainline
apex) to be identical when switching between target products, since the
product name is no longer embedded in:

1) the output path prefix in bazel-out
2) the PlatformOptions contribution to the configuration hash as part of a transition

Test: treehugger
Change-Id: I3fd0f7134209aa7eede3ed98412a1c5482689b57
2023-05-30 15:40:18 +00:00
Treehugger Robot
0d149f4a4f Merge "Remove GetPythonBinary request type" 2023-05-23 21:42:31 +00:00
Vinh Tran
c1910c9e07 Merge "Support sandboxing genrule" 2023-05-23 21:29:27 +00:00
Chris Parsons
6b49888a17 Remove GetPythonBinary request type
This has been dead code for over a year.

Fixes: 232976601
Test: m nothing
Change-Id: Ida305fc8fa61d8ac83d74ea2cf007073a0019cec
2023-05-23 17:57:38 +00:00
Treehugger Robot
04a1809d2a Merge "Handle missing dependencies in mixed builds" 2023-05-23 16:07:01 +00:00
Liz Kammer
c13f785061 Handle missing dependencies in mixed builds
Bazel will fail on queries with missing dependencies. Instead, we check
for missing dependencies in mixed builds and we fall back to Soong when
we are aware of missing dependencies in a module.

Test: go test soong tests
Change-Id: I4f83752704970d8b43650d5b55ff35799c7bc625
2023-05-23 12:14:02 +00:00
Yu Liu
6a7940cc99 Support sandboxing genrule
Bug: 281067655
Test: local build
Change-Id: I123b5e51eaf1476d1215e5208331b6540d45fd98
2023-05-22 15:02:36 -07:00
Jingwen Chen
379221f086 Move disable downloads to bazelrc from mixed build
Keeps mixed builds and b consistent.

Bug: 275724420
Test: presubmits, especially bp2build-incremental
Change-Id: I89942111f259bf9a88c27e124adbdb73936c514c
2023-05-11 13:30:59 -04:00
Sam Delmerico
316b0158fe Merge "add LLVM_*_VERSION envvars to allowlist" 2023-05-01 20:23:16 +00:00
Mark Dacek
dc4617aa66 Merge "Refactor MixedBuildsEnabled and add --ensure-allowlist-integrity." 2023-04-26 15:23:11 +00:00
MarkDacek
f47e142ffd Refactor MixedBuildsEnabled and add --ensure-allowlist-integrity.
Currently, there is little verification around allowlisted modules
actually being mixed-built. This flag would allow us to verify
that a module allowlisted is mixed-built for at least one variant.

Bug: 278910100
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
--bazel-force-enabled-modules=com.google.android.neuralnetworks (This
fails, as expected)
Test: build/soong/test/mixed_mode_test.sh

Change-Id: Icd5976f4f44f1a8caca1e5247d986642f7995f97
2023-04-25 20:55:07 +00:00
Chris Parsons
c9089dcc7b Refactor and cleanup of bazel handler
- Creation of a bazel command is independent of test/real implementation
- Use a custom struct instead of cmd.Exec
- Move mock bazel runner to the test

Bug: 270989498
Test: m nothing
Test: USE_PERSISTENT_BAZEL=0 m nothing
Test: Treehugger
Change-Id: Ieec35dad5e21aac644d5b8dc79a92397d42db861
2023-04-25 15:53:58 +00:00
Sam Delmerico
0ffa1f369d add LLVM_*_VERSION envvars to allowlist
Bug: 273563660
Change-Id: I7c1e6b244e137b27126abcba22684db66fae0327
2023-04-19 14:18:20 -04:00
Zi Wang
a4f7dae7b9 Fix api fingerprinting with apex.
Test: b build adbd and tzdata and verify target sdk version is correct

Bug: 277921995
Change-Id: I05a98fcb628f2a1d83358fe79f19c58010f05430
2023-04-18 21:12:02 +00:00
MarkDacek
9c094ca153 Fix issue where bazel-force-enabled-modules aren't actually analyzed.
Bug: 273910287
Test: run mixed_mode_test.sh

Change-Id: I309c423b6e7f505ad1db0d9327e7de6b8298dfe9
2023-04-14 02:17:54 +00:00
Treehugger Robot
7a9626c655 Merge "Make buildroot.cquery determinisitc" 2023-04-12 21:04:52 +00:00
Cole Faust
8181f4b916 Replace json_encode with json.encode
Now that the json module is available in the cquery starlark execution
enviornment.

Bug: 245315355
Test: m nothing
Change-Id: Ie7c314e2f4e2ef4257bd84abc1add01ddb668f0b
2023-03-19 14:24:23 -07:00
Vinh Tran
cbee4e3f2e Remove --extra_toolchains flag from aquery runner
When specifying --extra_toolchains=//prebuilts/clang/host/linux-x86:all, there is no control over the sort order of toolchains, which can result in a more generic toolchain being used rather than the most specific (and correct) toolchain.

Apparently, this flag is causing Bazel to drop some flags from CppCompile actions. This causes mixed-build's outputs different from Soong build. The mixed-build also generates different from Bazel build because we don't use the flag when using `b` to build the targets.

Test: Inspect differences in CppCompile actions from Soong's Bazel aquery handler and b aquery
Bug: 273995121
Change-Id: Id9e32c0cd12ab8577cd5b223ca9e19c982f3ae1f
2023-03-17 15:34:37 -04:00
Jingwen Chen
2d37b641c5 Populate apexBundle#filesInfo using bazel info.
This CL adds a few things:

1) Populate the filesInfo struct with cquery'd information from an
apex's ApexMkInfo provider. This filesInfo is then used in
apex/androidmk.go to generate Make modules (soong_cc_rust_prebuilt.mk),
which are then used in packaging to generate zip files of symbols in $PRODUCT_OUT.
2) Make a list of dicts of primitives JSON-encodable.
3) Tests.

Bug: 271423316
Bug: 271423062
Test: presubmits
Change-Id: Iaa34f51044de310510e580d9cf1fe60bbef801c1
2023-03-16 18:09:27 +00:00
Chris Parsons
38851d86aa Make buildroot.cquery determinisitc
Test: determinism_test.sh
Change-Id: Ia80dc425a781247f518ace8a5d11b49cd1ae60ce
2023-03-15 00:19:55 -04:00
Chris Parsons
9402ca881c Add a persistent bazel server between builds
This feature is toggled on with USE_PERSISTENT_BAZEL, which is off by
default. Those that opt-in will have a bazel server running between
builds (with a 3hr default TTL) which will greatly improve analysis on
subsequent builds. (As Bazel maintains a cache of analysis results).

Bug: 266983462
Test: Manual `m nothing` runs (timing with and without the feature)
Test: New integration test
Test: Presubmits
Change-Id: I3af4948baa0c490e9b87c48ffdbe9f67732586c7
2023-03-01 23:30:19 -05:00
Yu Liu
bfb2362cde Only apply apex transition to DCLA libs for now.
Bug: 270339610
Test: mixed_droid.sh
Change-Id: I6a8b7fdefeb504a25256e92647de9b487d5b65a3
2023-02-22 10:43:46 -08:00
Yu Liu
e43124023c Support DCLA
Bug: 240424572
Test: Manual tests:
1. m --dev-mode-staging com.android.adbd com.android.media.swcodec.
2. verify the DCLA libs from the two apexes have the same size and
   sha1sum, and also match the libs in bazel-out.
3. empty the DCLA libs list in allowlist.go and repeat step 1
4. repeat step 2 and verify the opposite result
5. build git_master: mainline_modules_bundles-userdebug in ABTD
   with the cl, then follow go/build-sideload-dcla-locally to
   download the adbd and swcodec aab files, run the DCLA trimming
   workflow locally, and verify the symlinks in the two trimmed
   apexes are identical and also match the lib path in the DCLA
   apex that was created by the workflow.

Change-Id: Ib2f8a29126a54829c0e10eba17b256a79930fd70
2023-02-21 10:12:47 -08:00
Liz Kammer
a4655a96c0 Handle aquery build statements in a goroutine
Creation of build statements is largely parallelizable because each
action is independent apart from updates/reads to
depsetHashToArtifactPathsCache. Locally resulted in build statements
taking ~.45 seconds on staging mode to ~.02 seconds

Test: CI
Change-Id: Iab00c8394a9eab17353f71230885ff0870e17f24
2023-02-15 11:29:49 -05:00
Treehugger Robot
80936bd15f Merge "Collect more metrics for aquery handling" 2023-02-14 19:25:53 +00:00
Liz Kammer
690fbac200 Collect more metrics for aquery handling
Test: go test soong tests
Test: m nothing & verify metrics
Change-Id: Id19e004d90dfbaa1b1706c607d5250bf845c0f3b
2023-02-14 11:10:19 -05:00
Sam Delmerico
e56fd9467d Merge "pass TIDY_EXTERNAL_VENDOR envvar to Bazel" 2023-02-13 19:02:29 +00:00
Sam Delmerico
700b4d3e1a pass TIDY_EXTERNAL_VENDOR envvar to Bazel
Bug: 268668248
Test: verify that Bazel tidy actions don't have a large diff from Soong tidy actions:
  (add -v to Ninja invocation in ui/build/ninja.go [and disable fifo file])
  WITH_TIDY=1 DISABLE_ARTIFACT_PATH_REQUIREMENTS=true m com.android.adbd 2>&1 | tee soong_build-tidy.adbd &&
  WITH_TIDY=1 DISABLE_ARTIFACT_PATH_REQUIREMENTS=true m com.android.adbd --bazel-mode-staging 2>&1 | tee mixed_build-tidy.adbd &&
  grep -Po "prebuilts/clang/host/linux-x86/clang-r475365b/bin/clang-tidy.sh \K\S+" soong_build-tidy.adbd | sort -u > soong-clang-tidy.adbd &&
  grep -Po "prebuilts/clang/host/linux-x86/clang-r475365b/bin/clang-tidy.sh \K\S+" mixed_build-tidy.adbd | sort -u > mixed-clang-tidy.adbd &&
  vimdiff soong_build-tidy.adbd mixed_build-tidy.adbd
Change-Id: I22804d6d92ed56d9727eaf430318df0b041712a8
2023-02-10 16:46:33 -05:00
Sam Delmerico
db23e22835 Merge "export allowlist of environment variables to Bazel" 2023-02-07 19:25:39 +00:00
Christopher Parsons
a9242e32d5 Merge "Only write soong_injection files if changed" 2023-02-06 22:23:54 +00:00
Sam Delmerico
cb3c52c766 export allowlist of environment variables to Bazel
Previously, Bazel was invoked during mixed builds with a stripped down
environment. This CL adds an allowlist of environment variables that are
passed to Bazel.

Test: WITH_TIDY=1 DISABLE_ARTIFACT_PATH_REQUIREMENTS=true mss tidy-packages-modules-NeuralNetworks --bazel-mode-dev
Change-Id: I23147bec59f6522953cf623e7bcaa0f1f99a75a3
2023-02-06 14:34:43 -05:00
Chris Parsons
3a8d0fbede Only write soong_injection files if changed
This also fixes determinism of these files (by ensuring that bazel
requests and their configurations are sorted, as these are used in the
soong_injection file output)

Bug: 266983462
Test: Manually verified soong_injection files are the same among
multiple runs
Test: Unit test
Test: m nothing

Change-Id: I1568930549cff0bc5676825434cc448d16ebdd4e
2023-02-03 18:29:02 -05:00
Usta Shrestha
13fd5aead7 prune out empty bazel depsets
Instead of using a sentinel file that was created under bazel_workspace
prune them out. The sentinel file was being created under bazel_workspace,
which would be deleted and thus rendered useless from the perspective of
not retriggering a build.

Test: `touch Android.bp` then `NINJA_ARGS="-d explain" m --bazel-mode-staging adbd_test`
Bug: 265155778
Change-Id: If5d8f0ea7f3b8828fda0646faafd1a621f0cb27c
2023-01-30 11:37:04 -05:00
Sam Delmerico
ce39f83538 add error handling to cquery in MockBazelContext
Change-Id: Ia01072b6528caff2f3ceaca67cdba8b900924fe5
2023-01-24 10:39:07 -05:00
Sasha Smundak
39a301c963 Clean up mixed builds module names filtering
* Remove `Bp2buildCcLibraryStaticOnlyList` it is never set
* Remove the rest of the sutff related to it
* Streamline MixedBuildsEnabled
* Rename BazelContext.BazelAllowlisted to BazelContext.NameFilter
  to reflect its action
* Rename bazelContext to mixedBuildBazelContext
* Fix minor warnings from Go static analyzer

Test: treehugger
Change-Id: Ie245c992f94bbfd5c7c23dd06917026200a28263
2023-01-12 00:32:45 +00:00
Cole Faust
b85d1a15cc Bp2build support for multiple product configs
Create a
build/bazel/product_config/generated/products/<product_name>/BUILD
file that contains the platform definitions needed for
a particular product. Currently we just create it for the
current lunch target, but the idea is that eventually when
all product config is in starlark, all the products will
have their platform definitions in the tree at once.

Bug: 249685973
Test: Presubmits
Change-Id: I08c82ff28dcf62f09d3b1d2e3186a6b961e12f6e
2023-01-05 12:49:14 -08:00
Paul Duffin
58f7784e95 Merge "Stop changes to BAZEL_METRICS_DIR regenerating ninja file" 2023-01-05 18:02:10 +00:00
Paul Duffin
184366a18f Stop changes to BAZEL_METRICS_DIR regenerating ninja file
Bug: 262738679
Test: BAZEL_METRICS_DIR=dir1 m nothing
      BAZEL_METRICS_DIR=dir2 m nothing
      # Make sure that the second does not regenerate the ninja file.
Change-Id: If30f13eee5d27d902c894af04af38078f437b6df
2023-01-04 10:55:20 +00:00
Cole Faust
a734749148 Use WriteFileRule instead of custom echo commands
These instances could use WriteFileRule instead of
making their own shell code to write a file.

Test: Presubmits
Change-Id: I9c809b2164a68b4ce1c22fbbd0d7497240110b39
2023-01-03 14:42:23 -08:00
Colin Cross
62e4fe15dc Merge changes I47b8fdd9,Ifbd121ab
* changes:
  Return linux as host OS from getConfigString when using musl
  Use //build/bazel/platforms/os:linux_glibc for osLinux
2022-12-21 19:33:14 +00:00
Colin Cross
046fcea830 Return linux as host OS from getConfigString when using musl
linux_musl is an alternative host OS, return "linux" from
getConfigString when it is being used.

Bug: 259266326
Test: build/bazel/ci/mixed_libc.sh
Test: USE_HOST_MUSL=true build/bazel/ci/mixed_libc.sh
Change-Id: I47b8fdd93b36345f82dd4e8455edb4c579f805e9
2022-12-20 15:32:18 -08:00
Treehugger Robot
597894279d Merge "mixed builds: add cc_test bazel handler, and add adbd_test to staging allowlist." 2022-12-20 03:12:53 +00:00
Cole Faust
705968d043 Allow exporting bazel mixed build allowlists to simple text files
So that they the allowlists can be read from build_mainline_modules_bundles.sh.

Bug: 262302253
Test: m out/soong/bazel_prod_mixed_builds_enabled_list.txt out/soong/bazel_staging_mixed_builds_enabled_list.txt && cat out/soong/bazel_prod_mixed_builds_enabled_list.txt && echo && cat out/soong/bazel_staging_mixed_builds_enabled_list.txt
Change-Id: Id5f89ebb651a52dc5368681a20d6183703e05bd4
2022-12-15 15:40:36 -08:00