Commit graph

230 commits

Author SHA1 Message Date
Trevor Radcliffe
56b1a2b575 LTO Bp2build
Bug: 261733821
Test: Unit Tests
Change-Id: I8c3721d35c464e296012145b2e95a7f0866aac37
2023-03-02 17:45:01 +00:00
Chris Parsons
9402ca881c Add a persistent bazel server between builds
This feature is toggled on with USE_PERSISTENT_BAZEL, which is off by
default. Those that opt-in will have a bazel server running between
builds (with a 3hr default TTL) which will greatly improve analysis on
subsequent builds. (As Bazel maintains a cache of analysis results).

Bug: 266983462
Test: Manual `m nothing` runs (timing with and without the feature)
Test: New integration test
Test: Presubmits
Change-Id: I3af4948baa0c490e9b87c48ffdbe9f67732586c7
2023-03-01 23:30:19 -05:00
Liz Kammer
a4655a96c0 Handle aquery build statements in a goroutine
Creation of build statements is largely parallelizable because each
action is independent apart from updates/reads to
depsetHashToArtifactPathsCache. Locally resulted in build statements
taking ~.45 seconds on staging mode to ~.02 seconds

Test: CI
Change-Id: Iab00c8394a9eab17353f71230885ff0870e17f24
2023-02-15 11:29:49 -05:00
Liz Kammer
00629db646 Use aquery proto directly
This prevents unnecessary copies of data from the query by using a
pointer to the proto or its contents

Test: go test soong tests
Test: m nothing
Change-Id: I5962b27fe31f7103d2af6cef159f11747cff3ed9
2023-02-15 11:29:47 -05:00
Liz Kammer
f15a0797c0 Only create a command line replacer once
Test: CI
Change-Id: Ibb9b822aea808c936fedc4c8fea7b7d787b101d5
2023-02-14 11:10:20 -05:00
Liz Kammer
690fbac200 Collect more metrics for aquery handling
Test: go test soong tests
Test: m nothing & verify metrics
Change-Id: Id19e004d90dfbaa1b1706c607d5250bf845c0f3b
2023-02-14 11:10:19 -05:00
Sam Delmerico
4ed95e263f mixed build targets need transitive tidy files
Previously in mixed builds, only the tidy files for the boundary module
were built, whereas all of its transitive dependencies' tidy files were
not being built. Instead we should export the list of transitive tidy
files for a module so that we can run clang-tidy for the boundary module
as well as its dependencies.

Bug: 195029134
Test: WITH_TIDY=1 DISABLE_ARTIFACT_PATH_REQUIREMENTS=true mss tidy-packages-modules-NeuralNetworks --bazel-mode-dev
Change-Id: I463646d2ae1fc4aa075a54c264e1c34571c3fd5c
2023-02-07 16:50:30 -05:00
Christopher Parsons
94f8a60f5d Merge "Fix nondeterminism in bp2build" 2023-01-30 22:58:17 +00:00
Chris Parsons
7b3289b471 Fix nondeterminism in bp2build
This fixes two main sources of nondeterminism:

1. Fix a bug in the ConfigurationAxis comparator (which caused
   ConfigurationAxis sorting to be nondeterministic)
2. Process C++ dependencies using the sorted ConfigurationAxis order. In
   theory, the order in which dependencies are processed shouldn't
   matter (as they should end up in different select stanzas). However,
   in the case of InApex stubs, this is not the case; we now ensure
   that lists are concatenated in a predictable order.

Added bonus: Some cleanup with SortConfigurationAxes which
makes use of go generics (this made it easier to debug this issue).

Will follow-up with regression tests.

Test: Manually verified that build.ninja checksum and BUILD.bazel checksums do not change after running `m nothing` 6 times in AOSP (with comment-only Android.bp changes in between each run)

Change-Id: I81168e45bdbbcd61ea95ff665cf6c4bc180aa4e0
2023-01-30 21:26:57 +00:00
Usta (Tsering) Shrestha
e136efdcfa Merge "prune out empty bazel depsets" 2023-01-30 21:04:08 +00:00
Usta Shrestha
13fd5aead7 prune out empty bazel depsets
Instead of using a sentinel file that was created under bazel_workspace
prune them out. The sentinel file was being created under bazel_workspace,
which would be deleted and thus rendered useless from the perspective of
not retriggering a build.

Test: `touch Android.bp` then `NINJA_ARGS="-d explain" m --bazel-mode-staging adbd_test`
Bug: 265155778
Change-Id: If5d8f0ea7f3b8828fda0646faafd1a621f0cb27c
2023-01-30 11:37:04 -05:00
Sam Delmerico
78aca80710 Merge "add androidmk cc-related variables to androidmk" 2023-01-27 21:04:29 +00:00
Treehugger Robot
6e7441edde Merge "Read ApexMkInfo for modules to be installed." 2023-01-27 15:45:08 +00:00
Jingwen Chen
29743c8423 Read ApexMkInfo for modules to be installed.
This piggybacks onto the ApexInfo cquery handler, so we're issuing a
single bazel query call that reads two providers in the starlark expr.

Also rename requiredDeps to makeModulesToInstall to differentiate it from
APEX's required/provided libs in the apex manifest.

Test: unit test
Test: mkdiff
Fixes: 263123189
Change-Id: Ib7e43f1586f29864eee8627dba3631bfaff27afa
2023-01-27 04:02:45 +00:00
Liz Kammer
748d70786c Correct global excludes not always being excluded
Test: bp2build go tests
Fixes: 266617441
Change-Id: I82418ef8da4cca880d5adac98853805a18a9780e
2023-01-26 11:16:39 -05:00
Sam Delmerico
4e115cc90d add androidmk cc-related variables to androidmk
The adbd_test androidmk definition is missing some cc-related variables
for cc_test in mixed builds. These variables should be populated from
information from Bazel.

Bug: 265758350
Change-Id: I59d017e2eb2f139188ba3383c457cc0055372b61
2023-01-25 15:14:03 -05:00
Trevor Radcliffe
3a79caa3b9 Merge "Add musl defaults in bp2build" 2023-01-19 16:01:04 +00:00
Liz Kammer
9d2d41065c Handle soong config vars for string attrs
Previously if there was a value set outside of product variables and one
inside a conditions default soong config var, we would ignore the
conditions default. Now, we maintain the conditions default and use
the root value in cases where there is not a value for a product
variable.

Test: go test soong tests
Change-Id: Ic991e3aebe5bb6039353f4e3d25625e7c5190f96
2023-01-11 22:01:01 -08:00
Liz Kammer
f3963f8b9c bp2build correctly handle empty soong_config_var
Previously in cases where a variable value was set but empty in a bp
file we ignored the value in bp2build; however, the correct behavior is
to take the empty value for each case. The tests for soong config vars
with empty_prop_for_string_var and unused_string_var:
https://cs.android.com/android/platform/superproject/+/master:build/soong/android/soong_config_modules_test.go;l=243-283;drc=72beb34609217f3c98623fb6795b6417c7f0fc65
highlight the difference in when conditions default should be used or
not.

Test: go test bp2build tests
Change-Id: I9768241f013a1a1bc8fa23657a066b5b3f52f09b
2023-01-11 22:01:01 -08:00
Trevor Radcliffe
0d1b4029b8 Add musl defaults in bp2build
Bug: 261657184
Test: Unit Tests
Test: Manual Inspection of generated files
Change-Id: I69864abb410fddecd530fc75f2531aff33b33c97
2023-01-11 18:43:20 +00:00
Jingwen Chen
94098e880e apex mixed builds: use the .capex when required.
The .capex is the compressed apex. Depending on various conditions, the
.capex should be installed, and thus the
apex.go#ProcessBazelQueryResponse should set the .capex as the main apex
output file.

Fixes: 264873511
Test: new unit test with install rules
Test: m com.android.adbd --bazel-mode-staging gives:

$ ls -lah
out/target/product/generic_arm64/system/apex/com.android.adbd.capex
-rw-r----- 1 jingwen primarygroup 3.6M Jan 10 14:44 out/target/product/generic_arm64/system/apex/com.android.adbd.capex

Change-Id: Ic3143cc6999f111428c6adb76760be8ddc4b6344
2023-01-10 14:50:42 +00:00
Zi Wang
9f609db46a Add variant_prepend support for all the properties in bp2build
This is a follow up of aosp/2336916. All the properties that have
variant_prepend tag are supported in bp2build.

Bug: 261644490
Test: TH and modified tests with updated expected behaviors
Change-Id: I13bb5d4d8fcd38bf153fa9083c9c945aec2dfa19
2023-01-04 11:19:50 -08:00
Zi Wang
0f82844b75 Add variant_prepend test for cc_library
The comments in aosp/2336916 are also addressed here

Test: TH and TestCcLibraryVariantPrependPropOrder
Change-Id: If1472658fcd4b5544dec2e2691049a180520c84e
2022-12-28 11:18:11 -08:00
Colin Cross
62e4fe15dc Merge changes I47b8fdd9,Ifbd121ab
* changes:
  Return linux as host OS from getConfigString when using musl
  Use //build/bazel/platforms/os:linux_glibc for osLinux
2022-12-21 19:33:14 +00:00
Colin Cross
133782e4bb Use //build/bazel/platforms/os:linux_glibc for osLinux
osLinux is implicitly glibc, use //build/bazel/platforms/os:linux_glibc
for the configurability select key for it.

Bug: 259266326
Test: build/bazel/ci/mixed_libc.sh
Test: USE_HOST_MUSL=true build/bazel/ci/mixed_libc.sh
Change-Id: Ifbd121ab3d943f3fdfbd3f6c3e982d1eaaaa9ec1
2022-12-20 15:29:31 -08:00
Jingwen Chen
3952a9039e mixed builds: add cc_test bazel handler, and add adbd_test to staging allowlist.
This will allow the test_suites_{arm64, x86_64} CI targets on
git_master-bazel to build adbd_test with bazel in `m`.

This is currently implemented similar to the cc_binary handler to
hydrate the output path field.

The tradefed config is still currently handled by Soong in
testBinary#install, which is called from both ProcessBazelQueryResponse
and GenerateAndroidBuildActions. We'll move the tradefed config path to
a mixed builds handler when it's ready on the cc_test bazel rule side.

Tested with `atest` with cuttlefish. Since `atest` doesn't have a way to pass
--bazel-mode-staging to its internal call to Soong, so I manually tested
this by adding adbt_test to the prod allowlist, and confirmed that the
installed adbd_test is the same binary as the one in bazel-out.

Test: $ showcommands adbd_test | grep bazel-out | tail -n1

```
/bin/bash -c "rm -f
out/target/product/vsoc_x86_64/data/nativetest64/adbd_test/adbd_test &&
cp -f -d
out/bazel/output/execroot/__main__/bazel-out/android_x86_64_silvermont-opt-ST-59d4124ca041/bin/packages/modules/adb/adbd_test
out/target/product/vsoc_x86_64/data/nativetest64/adbd_test/adbd_test"
```

Test: Add adbd_test to prod allowlist and run 'lunch aosp_cf_x86_64_phone-userdebug; atest adbd_test', all tests pass.
Bug: 258426425
Bug: 261692372
Change-Id: I6be329f39d607af8daa1dc772b23e8dd0247418c
2022-12-15 08:01:44 +00:00
Sam Delmerico
eddd3c0ba4 add APEX transitive dependency validation
Bug: 218419109
Test: b build //packages/modules/adb/apex:com.android.adbd
Change-Id: I93b0c99d6521e419e52c63271646448f6d708c22
2022-12-13 14:32:19 -05:00
Zi Wang
1cb11800f7 Add support for variant_prepend properties in bp2build for cc library
Some properties in soong are marked "variant_prepend", which means
in bp2build output, variant properties(select ...) should come before
general properties.
Prepend property is added in StringListAttribute
stuct so that bp2build can handlle variant_prepend properly, if
Prepend is properly set in different module logics.
This change also coordinates cc library with Prepend.
Coordination with other module types will be in separate CLs.

Bug: 261644490
Test: TestCcLibraryHeadersSimple,
TestCcLibraryHeadersArchAndTargetExportSystemIncludes and TH

Change-Id: I3b61c491c54bbe91ba9aa2af9b4c84193c91ae7f
2022-12-09 16:41:23 -08:00
Yu Liu
24214aacc2 Support abi check in mixed build.
Bug: 253498204
Test: Manual tested.
Change-Id: I451a6109a2a7efa24bafc860b7b757e23fde09e1
2022-12-01 10:59:06 -08:00
Treehugger Robot
efc1bc7f2f Merge "Improve mixed builds error messages" 2022-11-28 22:06:21 +00:00
Liz Kammer
ffc17e4edb Use excludes from OS axis for OsAndInApex
The axis for OS and in APEX was previously ignoring the excludes from
OS, which would result in excludes for non-Android OSes being
incorrectly ignored.

Test: go test Soong tests
Test: b build //hardware/libhardware:all
Bug: 260131489
Change-Id: Ie77f763bea0f473ac69a0c6b6bc3216e7359ad12
2022-11-23 10:01:30 -05:00
Cole Faust
97d1527dd2 Improve mixed builds error messages
When a target does not have a certain provider we need,
currently we just fail at the dictionary lookup. Change
it so that we fail with a message that tells us what
target had the issue.

Bug: 260131489
Test: Presubmits
Change-Id: I99fd018835c14eba0d4db54ba2769b2da1b278f6
2022-11-22 14:38:24 -08:00
Wei Li
78c07de1dc Add support of <APEX>-installed-files.txt in mixed build.
Bug: 239084750
Test: CIs
Change-Id: Ie3823d1a620c3924111e71aa97df727565261c98
2022-11-11 22:46:33 +00:00
Liz Kammer
dbb9fe8e92 Merge "Increase testing of request_type" 2022-11-10 13:30:58 +00:00
Treehugger Robot
d5fd30d94a Merge "Add reference to bundle_file in request_type_test" 2022-11-10 01:04:12 +00:00
Treehugger Robot
01dd619766 Merge changes from topic "clang-tidy-mixed-builds"
* changes:
  add tidy files to mixed builds
  add --norun_validations to bazel build action
2022-11-10 01:00:15 +00:00
Liz Kammer
1b7ed9bcbc Increase testing of request_type
Test: go test soong tests
Change-Id: Iacb773a91a4cc097320f06ea1d1669c5923f8076
2022-11-09 11:22:11 -05:00
Sam Delmerico
9a9f652bdd add tidy files to mixed builds
Bug: 195029134
Test: m --bazel-mode-dev
Change-Id: I6fa370334c6ff2b6ae4313a7fcbaa34b16b0fa92
2022-11-09 10:12:00 -05:00
Liz Kammer
08626ec9d7 Add reference to bundle_file in request_type_test
Test: go test soong tests
Change-Id: I60e8930e4565a9b076e1f487300e1e2e1d3e66d0
2022-11-09 09:03:52 -05:00
Liz Kammer
303978dcd2 Connect base.zip to mixed builds
Test: go test soong tests
Test: mixed_droid
Change-Id: Iaf1fbd1e7cde63892b01a9c9f4cd29ae69e2d6e3
2022-11-09 08:16:54 -05:00
Jingwen Chen
1ec7785ec5 bazel apex: Add mixed builds support for <module>_using.xml file.
This file contains the java APIs parsed by the 'dexdeps' tool within the gen_java_usedby_apex.sh.

Bug: 239084755
Fixes: 239084755
Test: presubmits
Change-Id: Ia271783a6be3ea3a343481306cde1aaba2166e88
2022-11-09 06:30:03 +00:00
Wei Li
cc73a05670 Add support of <APEX>_backing.txt in mixed build.
Bug: 239081456
Test: CIs
Change-Id: I8a8a5e606d5b1bc19294fc92cd469e7aa01cdb43
2022-11-08 12:00:42 -08:00
Jingwen Chen
0c9a276ded bazel apex: Add mixed builds support for <module>_using.txt file.
Test: presubmits
Change-Id: I7203fcccb09f6c93e702550ffa47a4029c3351dd
2022-11-07 23:16:12 +00:00
Liz Kammer
be6a712767 Rename ApexCqueryInfo to ApexInfo
cquery is redundant given this is in the cquery package.

Test: go test soong tests
Change-Id: Ib38794b712f548129eec964b8fd85bf8401a5f86
2022-11-04 16:24:37 -04:00
Wei Li
aff9a46ca5 Merge "Support new attributes added in ApexInfo in mixed build." 2022-10-29 01:16:39 +00:00
Wei Li
32dcdf9038 Support new attributes added in ApexInfo in mixed build.
Bug: 243748589
Test: CI
Change-Id: I6c9869464fe91ebf62811d92cf4403f2c7c9e8a9
2022-10-28 14:06:57 -07:00
Jason Wu
118fd2b443 Rollforward "Use proto output instead of jsonproto output for aquery""
Test: use m --bazel-mode-dev and mixed_droid.sh
Bug: 233798334

This reverts commit d27e28c2a7.

Change-Id: Ifd37ac449979330529dd25c92a5eca2f5a80827e
2022-10-27 18:41:50 +00:00
Jason Wu
d27e28c2a7 Revert "Use proto output instead of jsonproto output for aquery"
This reverts commit 6fe8721e12.

Reason for revert: Broke aosp_kernel-build-tools/linux

Change-Id: I502ec4c275cbc56e1024a42fadf68b308750939d
2022-10-27 05:18:40 +00:00
Jason Wu
6fe8721e12 Use proto output instead of jsonproto output for aquery
Test: use m --bazel-mode-dev and mixed_droid.sh
Bug: 233798334
Change-Id: Ib945359d3d05e6f51afe70432ec380e4e584175b
2022-10-26 17:07:48 -04:00
Sasha Smundak
33407b86c1 Refactor parsing JSON-encoded cquery results.
Test: treehugger
Change-Id: I8f1157ad4c3f0e563c9722bb6a1f0fc03e795700
2022-10-14 15:02:56 -07:00