Commit graph

160 commits

Author SHA1 Message Date
Chris Parsons
8152a94816 Read BUILD files in bp2build
The parsed BUILD files will be scanned for obvious definitions of BUILD
targets which have Android.bp counterparts. In such cases, bp2build will
automatically omit conversion of these defined modules (to prevent
collisions). With this change, we no longer need one-off denylisting of
modules which have BUILD file definitions.

This has a 0.03s to 0.2s slowdown for bp2build with current state. This
impact is identical on a heavier test branch, as well. I also ran an
experiment that applied BUILD scanning to all source BUILD files
(regardless of allowlisting), and this had a 2 second slowdown.

We may want to look into parallelizing or improving the performance of
this evaluation, but it's probably not worth the effort at this time,
since the current performance hit is small.

Test: New integration test
Test: Removed libprotobuf-python from denylist and tested building the
package
Test: Treehugger

Change-Id: Ibde3bab12cd4a8fed642ad46e5344a56953bec91
2023-06-09 16:02:40 +00:00
Treehugger Robot
ca69a6fc21 Merge "Revert "Remove --noexperimental_platform_in_output_dir"" 2023-06-08 12:10:14 +00:00
Matthias Männich
08ccec7c11 Revert "Remove --noexperimental_platform_in_output_dir"
This reverts commit 24af24e0f2.

Reason for revert: Breaks kernel build tools build https://ci.android.com/builds/branches/aosp_kernel-build-tools/grid?head=10282004&tail=10282004

Change-Id: I486d7f305d44cfecc51a0e039cc41c0afa6495f3
2023-06-08 08:19:14 +00:00
Treehugger Robot
731c4525ca Merge "Remove --noexperimental_platform_in_output_dir" 2023-06-08 06:08:14 +00:00
Cole Faust
24af24e0f2 Remove --noexperimental_platform_in_output_dir
This is a followup to aosp/2606989.

This flag is not necessary now that we're using one platform name
for all of mixed builds.

Also rename current_product to mixed_builds_product so that it's clear
that that this platform should only be used for mixed builds.

In addition, make the bazelrc files point to the named products again
instead of the mixed build product so that b builds will still have
qualified outputs, but mixed builds won't.

Test: Presubmit
Change-Id: I5acbd18869589f67efb0ad3ebfda2c4d5a6a3850
2023-06-07 11:08:44 -07:00
MarkDacek
bf7a5f1c71 Add b_test to run_integration_tests.sh
Bug: 281547018
Test: run the script
Change-Id: I4cb0205ba47c6c97015146fa0782d7a30fb80bac
2023-06-07 00:52:09 +00:00
Yu Liu
7301fc8aeb Add the DCLA libs used by swcodec to the staging list
Bug: 269342245
Test: run dcla_apex_comparison_test
Change-Id: I26bcbf6447b9216258c18c42e80f729e5651658a
2023-06-05 16:47:34 -07:00
Trevor Radcliffe
59b3729825 Update apex_comparison_test.sh to support CFI
The CFI transition being applied creates a situation where Bazel
clears its convenience symlinks. This change points the script
directly to the location of the deapexer artifact instead of the
symlink.

Bug: 261733820
Test: Ran the tests
Change-Id: I23951f0fe75ce1fbc3c0dd969244a797ac1ff983
2023-06-05 18:07:42 +00:00
Yu Liu
62461ea576 Merge "Tool to detect modules that generate the incorrect output in sandbox" 2023-05-31 21:34:49 +00:00
Yu Liu
fb458e9255 Tool to detect modules that generate the incorrect output in sandbox
Bug: 281067655
Test: Test locally
Change-Id: Ie38e3658eee4d577d1f871c9ed8129b4ac7d7d07
2023-05-26 14:45:12 -07:00
Wei Li
704146094a Exclude diffs caught by sbom_test.
Bug: 284279034
Test: build/soong/tests/sbom_test.sh
Change-Id: I152c2e2c55b6547b2dddf7b1b2524681c9af952e
2023-05-25 16:29:20 -07:00
Treehugger Robot
873691ae99 Merge "simplify m clean" 2023-05-24 18:36:39 +00:00
Usta Shrestha
6e1aa7830c simplify m clean
`rm -rf out` used to fail because bazel would not set write permission on some files. Now it's been fixed and thus there is little reason to prefer `m clean` (also users are accustomed to `rm -rf out`)

Bug: NA
Test: verified `m clean` works;interestingly soong_metrics file is created at the end
Change-Id: I000d16508613045811fc7792e5798f7c150dcc05
2023-05-23 17:34:15 -04:00
Treehugger Robot
1befe407a1 Merge "Revert "Revert "Restrict plugins to an existing allowlist""" 2023-05-23 17:08:46 +00:00
Jihoon Kang
f530696762 Add missing dependency in create_mock_soong
This change adds missing dependency of Soong modules as symlink
directory in create_mock_soong so that a correct ninja graph can be
created when invoking androidmk_test.sh.

Test: build/soong/test/run_integration_tests.sh
Bug: 283905797
Change-Id: I7dff0094fdc23b7c438c917dbc00a5143a4dc5ae
2023-05-23 06:57:38 +00:00
Liz Kammer
33cc80eb1f Revert "Revert "Restrict plugins to an existing allowlist""
This reverts commit b20a12553e.

Reason for revert: Rollforward with fix

soong-ca-certificates-apex was missing from allowlist

Test: verify fixes CI breakages
(cherry picked from https://googleplex-android-review.googlesource.com/q/commit:dfe4a9e4dd088f42ba5e0a3c4f7777c253d378d3)
Merged-In: I00687bd2962dbdaaf55189ca4aa82183264eb8d7
Change-Id: I00687bd2962dbdaaf55189ca4aa82183264eb8d7
2023-05-18 18:20:28 +00:00
Yu Liu
09a8d459d5 Test dcla libs for b build.
Bug: 273862399
Test: local
Change-Id: Ic72239779d63d64b68a902de6f5b3c6567385878
2023-05-17 20:01:28 -07:00
Chris Parsons
a3ae007641 Touch soong_build output at the end of main
This fixes an incrementality bug that resulted in superfluous
build.ninja regeneration after a new glob definition is added.

Fixes: 279674820
Test: Treehugger
Test; New integration test in this CL

Change-Id: Ifefe66a0eb1c125e9ad5373d60437a1cb1e6fdec
2023-05-12 13:40:12 +00:00
Cole Faust
c9508aac4c Load starlark files from soong
There are a number of instances where we are exporting information
from soong to bazel via soong_injection. This could be more bazel-centric
if the information was instead held in bzl files, and both bazel and
soong read it from there.

Add a starlark package that will run
//build/bazel/constants_exported_to_soong.bzl at initialization time,
and then results can be retreived with GetStarlarkValue.

Since changes to the starlark files mean that soong has to rerun,
add them as ninja deps.

Unfortunately, the starlark code has to be run at runtime rather than
pregenerating their results, because tests run from intellij wouldn't
go through any pregeneration steps. This means that starlark is run
multiple times during the build, once per test package and once per
primary builder invocation. (currently 3, could be reduced to 2 if we
made the symlink forest generation into its own standalone tool) The
starlark code we have so far in this cl is very fast, roughly half a
millisecond, so it's not a big deal for now, but something to keep an
eye on as we add more starlark constants.

Bug: 279095899
Test: go test
Change-Id: I1e7ca1df1d8d67333cbfc46e8396e229820e4476
2023-04-26 17:18:19 -07:00
MarkDacek
f47e142ffd Refactor MixedBuildsEnabled and add --ensure-allowlist-integrity.
Currently, there is little verification around allowlisted modules
actually being mixed-built. This flag would allow us to verify
that a module allowlisted is mixed-built for at least one variant.

Bug: 278910100
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
--bazel-force-enabled-modules=com.google.android.neuralnetworks (This
fails, as expected)
Test: build/soong/test/mixed_mode_test.sh

Change-Id: Icd5976f4f44f1a8caca1e5247d986642f7995f97
2023-04-25 20:55:07 +00:00
MarkDacek
53506361da Squelch output for a failure case in mixed_mode_test
Currently, the expected output shows as the main text for
unrelated presubmit failures.
Bug: 278748024
Test: build/soong/tests/mixed_mode_test.sh

Change-Id: I553a82597c7c7791a822e330eb9b2663f35567a1
2023-04-21 17:55:42 +00:00
MarkDacek
9c094ca153 Fix issue where bazel-force-enabled-modules aren't actually analyzed.
Bug: 273910287
Test: run mixed_mode_test.sh

Change-Id: I309c423b6e7f505ad1db0d9327e7de6b8298dfe9
2023-04-14 02:17:54 +00:00
Wei Li
7c1e7f0563 Remove diff excludes for partition compatibility symlink files in SBOM integration test.
Bug: 272358980
Test: CIs
Change-Id: I5436c77d6a63942771422a17a8b4876b6c81150a
2023-04-06 08:36:01 -07:00
Wei Li
0766f4c66c Merge "Update the SBOM integration test to include ramdisk partition and resolve some diffs." 2023-04-04 00:55:38 +00:00
Ibrahim Kanouche
7c1de51d9e Add spdx-tools and licenseclassifier to soong mocks
Change-Id: I8d394fe1f0633f433b76f66929294e6537e1f809
2023-03-31 17:19:29 +00:00
Wei Li
8a8d5a9c7a Update the SBOM integration test to include ramdisk partition and resolve some diffs.
Bug: 272360068
Test: CIs
Change-Id: Ieff900fd083f36daf77243be9159fcee6c94d1d3
2023-03-29 15:04:33 -07:00
Cole Faust
d3cc7c9bef Merge "Add testing android products" 2023-03-29 00:19:51 +00:00
Cole Faust
ebc01a3225 Add testing android products
These must be added in the soong_injection code as opposed to just
defined loosely in checked-in bzl files because the product_vars
select statement must be updated to support the new platforms.

Bug: 269577299
Test: b test --config=android //build/bazel/...
Change-Id: I7bba9af214896dd3b5938bae70b7c0cea4f75e41
2023-03-28 11:27:30 -07:00
Trevor Radcliffe
391a25d7fa Supporting changes for CFI toolchain features
Mostly exporting variables to Bazel, but also allowlisting a BUILD
file.

Bug: 251217226
Test: Unit tests
Change-Id: Id87015a3cd5d970700c4058ec989bb0c14c36bcb
2023-03-27 18:07:40 +00:00
Usta (Tsering) Shrestha
1cb8b5b57a Merge "cosmetic: log test completion" 2023-03-22 06:08:14 +00:00
usta
94e89a4091 cosmetic: log test completion
Test: run_intergration_tests.sh
Bug: NA
Change-Id: Iaacc708d05f0bfdc5fe6a0c60834efcb9b4eb98b
2023-03-21 17:01:52 +00:00
Yu Liu
6b7c316cfc Treat libcrypto in com.android.tethering differently
This module uses librypto from cronet which is different from the
default one, so the test would always fail.

Bug: 274473434
Test: run the script locally.
Change-Id: I9aac4da0990953db312e2163ee8af05e692f6f3f
2023-03-20 14:46:29 -07:00
Treehugger Robot
94d107020a Merge "Fix CI failures, comment out dcla cases temporarily." 2023-03-20 21:35:10 +00:00
Wei Li
4a77c55b81 Fix CI failures, comment out dcla cases temporarily.
Bug: 274473434
Test: CIs
Change-Id: Ia1b8860ac46dad6b9670880613b6559561e37dbf
2023-03-20 18:07:40 +00:00
Wei Li
898918f7c8 Build partition images and SBOM seperately in case the build of SBOM introduces extra dependencies that included in partition images.
Bug: 272358980
Test: build/soong/tests/sbom_test.sh
Test: CIs
Change-Id: I99bd2357344f9a3d5827cf052b69e3e8545241c0
2023-03-16 22:52:35 +00:00
Wei Li
44ac6dad27 Merge "Revert "Revert "Revert "Revert "Add integration test for testing generat...""" 2023-03-16 18:20:48 +00:00
Wei Li
73032c8fa3 Revert "Revert "Revert "Revert "Add integration test for testing generat...""
This reverts commit 1e75eccb18.

Reason for revert: The SBOM generation change has been submitted in aosp/2485540, so its integration test in this change should be ok to submit now.

Change-Id: Idb8c84a871b2a5633bed998eb7483fc91d30e97a
2023-03-15 18:31:38 +00:00
Kevin Han
a2e8ae1771 Merge "Revert "Revert "Revert "Add integration test for testing generat..."" 2023-03-13 21:03:23 +00:00
Kevin Han
1e75eccb18 Revert "Revert "Revert "Add integration test for testing generat..."
Revert submission 2481062-sbom

Reason for revert: b/273332655

Reverted changes: /q/submissionid:2481062-sbom

Change-Id: I9297f585ab426d0b5586a7a77e983e8bafb19f20
2023-03-13 20:57:32 +00:00
Wei Li
601fc0137e Merge "Revert "Revert "Add integration test for testing generated SBOM with .im...""" 2023-03-13 17:43:59 +00:00
Wei Li
cba64e4cf1 Revert "Revert "Add integration test for testing generated SBOM with .im...""
This reverts commit 68101fad40.

Reason for revert: fixed the build error and try to commit again.

Change-Id: Iec8afa95a59dc1c7ce9b48911a85667f2956fbce
2023-03-10 16:21:31 +00:00
Android Culprit Assistant
68101fad40 Revert "Add integration test for testing generated SBOM with .im..."
This revert was created by Android Culprit Assistant. The culprit was identified in the following culprit search session (http://go/aca-get/5b65d203-d364-4ade-aabb-1330fe45236a).

Change-Id: I10231bc624a15a2ba477712b3a5950f5fc9113e8
2023-03-10 06:18:01 +00:00
Spandan Das
68bcbb588a Add an integration test for API export from another bazel package
This test ensures that API export works ok if the api file exists in a
different directory (precisely, package) than the *_api_contribution
target.

Test: tests/run_integration_tests.sh
Change-Id: I1ff171b93773b514a9a081f962606f4c28abe42e
2023-03-10 02:32:18 +00:00
Wei Li
eecee5a695 Add integration test for testing generated SBOM with .img files.
Bug: 266726655
Test: CIs
Change-Id: Iacaaae50125c7f310828c79b7881ac1d83a1f9ed
2023-03-08 13:47:51 -08:00
Yu Liu
525bae7ac4 Merge "Run the dcla comparison test for both pure soong build and mixed build." 2023-03-08 06:20:58 +00:00
Yu Liu
fa3a152de7 Run the dcla comparison test for both pure soong build and mixed build.
Bug: 268337606
Test: Manually run the script.
Change-Id: I7afaf46de2a4575b36be0cf348de7a1b7f90719f
2023-03-02 15:43:36 -08:00
Christopher Parsons
7720f5704c Merge "Add a persistent bazel server between builds" 2023-03-02 14:22:34 +00:00
Christopher Parsons
647b8518ec Merge "Avoid deleting bp2build directory" 2023-03-02 14:22:13 +00:00
Chris Parsons
9402ca881c Add a persistent bazel server between builds
This feature is toggled on with USE_PERSISTENT_BAZEL, which is off by
default. Those that opt-in will have a bazel server running between
builds (with a 3hr default TTL) which will greatly improve analysis on
subsequent builds. (As Bazel maintains a cache of analysis results).

Bug: 266983462
Test: Manual `m nothing` runs (timing with and without the feature)
Test: New integration test
Test: Presubmits
Change-Id: I3af4948baa0c490e9b87c48ffdbe9f67732586c7
2023-03-01 23:30:19 -05:00
Chris Parsons
520e88b740 Avoid deleting bp2build directory
With this CL, we avoid deleting the bp2build directory and regenerating
all BUILD files. Instead, we regenerate BUILD files which have changes
on the filesystem, and delete old BUILD files which should no longer
exist.

This improves incremental Bazel server performance by about ~5 seconds.
Previously, Bazel would have cache misses for regenerated packages, and
would thus need to analyze them even if no meaningful changes took
place.

For ease of implementation, we avoid removing stale (empty) directories
from the bp2build workspace, but this should have no effect on the
build. (These will be removed on next clean)

Test: New integration tests
Test: Manual benchmarking in conjunction with persistent bazel server
Change-Id: I3e489ff403be34040122876012329060a2506366
2023-03-01 23:15:30 -05:00