Commit graph

150 commits

Author SHA1 Message Date
Wei Li
704146094a Exclude diffs caught by sbom_test.
Bug: 284279034
Test: build/soong/tests/sbom_test.sh
Change-Id: I152c2e2c55b6547b2dddf7b1b2524681c9af952e
2023-05-25 16:29:20 -07:00
Treehugger Robot
873691ae99 Merge "simplify m clean" 2023-05-24 18:36:39 +00:00
Usta Shrestha
6e1aa7830c simplify m clean
`rm -rf out` used to fail because bazel would not set write permission on some files. Now it's been fixed and thus there is little reason to prefer `m clean` (also users are accustomed to `rm -rf out`)

Bug: NA
Test: verified `m clean` works;interestingly soong_metrics file is created at the end
Change-Id: I000d16508613045811fc7792e5798f7c150dcc05
2023-05-23 17:34:15 -04:00
Treehugger Robot
1befe407a1 Merge "Revert "Revert "Restrict plugins to an existing allowlist""" 2023-05-23 17:08:46 +00:00
Jihoon Kang
f530696762 Add missing dependency in create_mock_soong
This change adds missing dependency of Soong modules as symlink
directory in create_mock_soong so that a correct ninja graph can be
created when invoking androidmk_test.sh.

Test: build/soong/test/run_integration_tests.sh
Bug: 283905797
Change-Id: I7dff0094fdc23b7c438c917dbc00a5143a4dc5ae
2023-05-23 06:57:38 +00:00
Liz Kammer
33cc80eb1f Revert "Revert "Restrict plugins to an existing allowlist""
This reverts commit b20a12553e.

Reason for revert: Rollforward with fix

soong-ca-certificates-apex was missing from allowlist

Test: verify fixes CI breakages
(cherry picked from https://googleplex-android-review.googlesource.com/q/commit:dfe4a9e4dd088f42ba5e0a3c4f7777c253d378d3)
Merged-In: I00687bd2962dbdaaf55189ca4aa82183264eb8d7
Change-Id: I00687bd2962dbdaaf55189ca4aa82183264eb8d7
2023-05-18 18:20:28 +00:00
Yu Liu
09a8d459d5 Test dcla libs for b build.
Bug: 273862399
Test: local
Change-Id: Ic72239779d63d64b68a902de6f5b3c6567385878
2023-05-17 20:01:28 -07:00
Chris Parsons
a3ae007641 Touch soong_build output at the end of main
This fixes an incrementality bug that resulted in superfluous
build.ninja regeneration after a new glob definition is added.

Fixes: 279674820
Test: Treehugger
Test; New integration test in this CL

Change-Id: Ifefe66a0eb1c125e9ad5373d60437a1cb1e6fdec
2023-05-12 13:40:12 +00:00
Cole Faust
c9508aac4c Load starlark files from soong
There are a number of instances where we are exporting information
from soong to bazel via soong_injection. This could be more bazel-centric
if the information was instead held in bzl files, and both bazel and
soong read it from there.

Add a starlark package that will run
//build/bazel/constants_exported_to_soong.bzl at initialization time,
and then results can be retreived with GetStarlarkValue.

Since changes to the starlark files mean that soong has to rerun,
add them as ninja deps.

Unfortunately, the starlark code has to be run at runtime rather than
pregenerating their results, because tests run from intellij wouldn't
go through any pregeneration steps. This means that starlark is run
multiple times during the build, once per test package and once per
primary builder invocation. (currently 3, could be reduced to 2 if we
made the symlink forest generation into its own standalone tool) The
starlark code we have so far in this cl is very fast, roughly half a
millisecond, so it's not a big deal for now, but something to keep an
eye on as we add more starlark constants.

Bug: 279095899
Test: go test
Change-Id: I1e7ca1df1d8d67333cbfc46e8396e229820e4476
2023-04-26 17:18:19 -07:00
MarkDacek
f47e142ffd Refactor MixedBuildsEnabled and add --ensure-allowlist-integrity.
Currently, there is little verification around allowlisted modules
actually being mixed-built. This flag would allow us to verify
that a module allowlisted is mixed-built for at least one variant.

Bug: 278910100
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
--bazel-force-enabled-modules=com.google.android.neuralnetworks (This
fails, as expected)
Test: build/soong/test/mixed_mode_test.sh

Change-Id: Icd5976f4f44f1a8caca1e5247d986642f7995f97
2023-04-25 20:55:07 +00:00
MarkDacek
53506361da Squelch output for a failure case in mixed_mode_test
Currently, the expected output shows as the main text for
unrelated presubmit failures.
Bug: 278748024
Test: build/soong/tests/mixed_mode_test.sh

Change-Id: I553a82597c7c7791a822e330eb9b2663f35567a1
2023-04-21 17:55:42 +00:00
MarkDacek
9c094ca153 Fix issue where bazel-force-enabled-modules aren't actually analyzed.
Bug: 273910287
Test: run mixed_mode_test.sh

Change-Id: I309c423b6e7f505ad1db0d9327e7de6b8298dfe9
2023-04-14 02:17:54 +00:00
Wei Li
7c1e7f0563 Remove diff excludes for partition compatibility symlink files in SBOM integration test.
Bug: 272358980
Test: CIs
Change-Id: I5436c77d6a63942771422a17a8b4876b6c81150a
2023-04-06 08:36:01 -07:00
Wei Li
0766f4c66c Merge "Update the SBOM integration test to include ramdisk partition and resolve some diffs." 2023-04-04 00:55:38 +00:00
Ibrahim Kanouche
7c1de51d9e Add spdx-tools and licenseclassifier to soong mocks
Change-Id: I8d394fe1f0633f433b76f66929294e6537e1f809
2023-03-31 17:19:29 +00:00
Wei Li
8a8d5a9c7a Update the SBOM integration test to include ramdisk partition and resolve some diffs.
Bug: 272360068
Test: CIs
Change-Id: Ieff900fd083f36daf77243be9159fcee6c94d1d3
2023-03-29 15:04:33 -07:00
Cole Faust
d3cc7c9bef Merge "Add testing android products" 2023-03-29 00:19:51 +00:00
Cole Faust
ebc01a3225 Add testing android products
These must be added in the soong_injection code as opposed to just
defined loosely in checked-in bzl files because the product_vars
select statement must be updated to support the new platforms.

Bug: 269577299
Test: b test --config=android //build/bazel/...
Change-Id: I7bba9af214896dd3b5938bae70b7c0cea4f75e41
2023-03-28 11:27:30 -07:00
Trevor Radcliffe
391a25d7fa Supporting changes for CFI toolchain features
Mostly exporting variables to Bazel, but also allowlisting a BUILD
file.

Bug: 251217226
Test: Unit tests
Change-Id: Id87015a3cd5d970700c4058ec989bb0c14c36bcb
2023-03-27 18:07:40 +00:00
Usta (Tsering) Shrestha
1cb8b5b57a Merge "cosmetic: log test completion" 2023-03-22 06:08:14 +00:00
usta
94e89a4091 cosmetic: log test completion
Test: run_intergration_tests.sh
Bug: NA
Change-Id: Iaacc708d05f0bfdc5fe6a0c60834efcb9b4eb98b
2023-03-21 17:01:52 +00:00
Yu Liu
6b7c316cfc Treat libcrypto in com.android.tethering differently
This module uses librypto from cronet which is different from the
default one, so the test would always fail.

Bug: 274473434
Test: run the script locally.
Change-Id: I9aac4da0990953db312e2163ee8af05e692f6f3f
2023-03-20 14:46:29 -07:00
Treehugger Robot
94d107020a Merge "Fix CI failures, comment out dcla cases temporarily." 2023-03-20 21:35:10 +00:00
Wei Li
4a77c55b81 Fix CI failures, comment out dcla cases temporarily.
Bug: 274473434
Test: CIs
Change-Id: Ia1b8860ac46dad6b9670880613b6559561e37dbf
2023-03-20 18:07:40 +00:00
Wei Li
898918f7c8 Build partition images and SBOM seperately in case the build of SBOM introduces extra dependencies that included in partition images.
Bug: 272358980
Test: build/soong/tests/sbom_test.sh
Test: CIs
Change-Id: I99bd2357344f9a3d5827cf052b69e3e8545241c0
2023-03-16 22:52:35 +00:00
Wei Li
44ac6dad27 Merge "Revert "Revert "Revert "Revert "Add integration test for testing generat...""" 2023-03-16 18:20:48 +00:00
Wei Li
73032c8fa3 Revert "Revert "Revert "Revert "Add integration test for testing generat...""
This reverts commit 1e75eccb18.

Reason for revert: The SBOM generation change has been submitted in aosp/2485540, so its integration test in this change should be ok to submit now.

Change-Id: Idb8c84a871b2a5633bed998eb7483fc91d30e97a
2023-03-15 18:31:38 +00:00
Kevin Han
a2e8ae1771 Merge "Revert "Revert "Revert "Add integration test for testing generat..."" 2023-03-13 21:03:23 +00:00
Kevin Han
1e75eccb18 Revert "Revert "Revert "Add integration test for testing generat..."
Revert submission 2481062-sbom

Reason for revert: b/273332655

Reverted changes: /q/submissionid:2481062-sbom

Change-Id: I9297f585ab426d0b5586a7a77e983e8bafb19f20
2023-03-13 20:57:32 +00:00
Wei Li
601fc0137e Merge "Revert "Revert "Add integration test for testing generated SBOM with .im...""" 2023-03-13 17:43:59 +00:00
Wei Li
cba64e4cf1 Revert "Revert "Add integration test for testing generated SBOM with .im...""
This reverts commit 68101fad40.

Reason for revert: fixed the build error and try to commit again.

Change-Id: Iec8afa95a59dc1c7ce9b48911a85667f2956fbce
2023-03-10 16:21:31 +00:00
Android Culprit Assistant
68101fad40 Revert "Add integration test for testing generated SBOM with .im..."
This revert was created by Android Culprit Assistant. The culprit was identified in the following culprit search session (http://go/aca-get/5b65d203-d364-4ade-aabb-1330fe45236a).

Change-Id: I10231bc624a15a2ba477712b3a5950f5fc9113e8
2023-03-10 06:18:01 +00:00
Spandan Das
68bcbb588a Add an integration test for API export from another bazel package
This test ensures that API export works ok if the api file exists in a
different directory (precisely, package) than the *_api_contribution
target.

Test: tests/run_integration_tests.sh
Change-Id: I1ff171b93773b514a9a081f962606f4c28abe42e
2023-03-10 02:32:18 +00:00
Wei Li
eecee5a695 Add integration test for testing generated SBOM with .img files.
Bug: 266726655
Test: CIs
Change-Id: Iacaaae50125c7f310828c79b7881ac1d83a1f9ed
2023-03-08 13:47:51 -08:00
Yu Liu
525bae7ac4 Merge "Run the dcla comparison test for both pure soong build and mixed build." 2023-03-08 06:20:58 +00:00
Yu Liu
fa3a152de7 Run the dcla comparison test for both pure soong build and mixed build.
Bug: 268337606
Test: Manually run the script.
Change-Id: I7afaf46de2a4575b36be0cf348de7a1b7f90719f
2023-03-02 15:43:36 -08:00
Christopher Parsons
7720f5704c Merge "Add a persistent bazel server between builds" 2023-03-02 14:22:34 +00:00
Christopher Parsons
647b8518ec Merge "Avoid deleting bp2build directory" 2023-03-02 14:22:13 +00:00
Chris Parsons
9402ca881c Add a persistent bazel server between builds
This feature is toggled on with USE_PERSISTENT_BAZEL, which is off by
default. Those that opt-in will have a bazel server running between
builds (with a 3hr default TTL) which will greatly improve analysis on
subsequent builds. (As Bazel maintains a cache of analysis results).

Bug: 266983462
Test: Manual `m nothing` runs (timing with and without the feature)
Test: New integration test
Test: Presubmits
Change-Id: I3af4948baa0c490e9b87c48ffdbe9f67732586c7
2023-03-01 23:30:19 -05:00
Chris Parsons
520e88b740 Avoid deleting bp2build directory
With this CL, we avoid deleting the bp2build directory and regenerating
all BUILD files. Instead, we regenerate BUILD files which have changes
on the filesystem, and delete old BUILD files which should no longer
exist.

This improves incremental Bazel server performance by about ~5 seconds.
Previously, Bazel would have cache misses for regenerated packages, and
would thus need to analyze them even if no meaningful changes took
place.

For ease of implementation, we avoid removing stale (empty) directories
from the bp2build workspace, but this should have no effect on the
build. (These will be removed on next clean)

Test: New integration tests
Test: Manual benchmarking in conjunction with persistent bazel server
Change-Id: I3e489ff403be34040122876012329060a2506366
2023-03-01 23:15:30 -05:00
Yu Liu
e0c4d35cc5 Add a test to compare the DCLA libs across the apexes.
Bug: 268337606
Test: manual run it.
Change-Id: I48525ed4724736594080af757800f0e224852abd
2023-02-25 14:02:36 -08:00
Usta (Tsering) Shrestha
9393264eff fix flakey test
since bp2build runs concurrently, errors can occur in any order

Bug: 265845583
Change-Id: Iea6d26bc73112dc89c963545b3e0fc7e1b7fa349
Test: run test localy multiple times
2023-02-13 20:25:12 +00:00
Usta (Tsering) Shrestha
c4c07b12b6 reduce forest generation to be incremental
Previously, symlink forest generation involved removing the entire
symlink forest and recreating it from scratch. With this change,
a) symlinks which need not change are untouched,
b) symlinks pointing to the wrong location are fixed, and
c) symlinks which should no longer exist are removed.

On AOSP on my local machine, this reduces the symlink forest generation
step from 2.5s to 1.1s clean, and 0.6s when a single file is added to
a source directory.

Bug: 257528847
Test: m bp2build, touch `fakefile` under the forest, remove a file
from the source tree, rerun m bp2build. Manually verify the new forest
does not retain the link to the deleted source file, and that fakefile
no longer exists in the forest.

Change-Id: I481371ae487e9419af6a3a4370c552578b07d650
2023-02-07 06:23:43 +00:00
Cole Faust
56080ddc08 Merge "Precompile python sources" 2023-01-30 20:02:27 +00:00
Cole Faust
5c503d1c43 Precompile python sources
This signifigantly improves the startup time of soong-built
python binaries. For example, running
`m apexer && time out/host/linux-x86/bin/apexer` gives
0.734s before this cl, and 0.094s after.

Fixes: 259718110
Test: Presubmits
Change-Id: Ib19e83e2c60c39a849525be117279c318de3afa7
2023-01-27 15:43:38 -08:00
Wei Li
80f1ef15cc Change grep pattern so it can report number of CppCompile actions correctly without being impacted by setting -march on command line twice.
Bug: 266930443
Test: build/soong/tests/apex_cc_module_arch_variant_tests.sh
Change-Id: Ifd74e8cfc86dd94fce632beac049cc81430bfc46
2023-01-27 22:52:20 +00:00
Usta Shrestha
071f6c29a3 for consistency symlink_tree failure = os.Exit(1)
Brings down time taken to fail from 18s to 13s (on aosp branch)

Test: temporarily coded random failure
Bug: N/A
Change-Id: Ib694eec977293f4dd7054e779d1b82b8cace93f6
2023-01-13 14:07:10 -05:00
Treehugger Robot
0ddd60fc1d Merge "Reference deapexer directly in bazel-bin" 2023-01-12 01:01:24 +00:00
Spandan Das
df01030ab5 Merge changes I78639a2d,Iddb0aa1a
* changes:
  Create a smoke test for api_bp2build workspace
  Add a new CreateSoongInjectionDirFiles function
2023-01-11 22:37:27 +00:00
Chris Parsons
41eff2e4b6 Reference deapexer directly in bazel-bin
This keeps this test compatible with a Bazel release change which
affects which files are present in `bazel cquery --output=files` for
py_binary targets. Since the executable will be present in a predictable
location under bazel-bin, we can hardcode that, instead.

We could have alternatively used --output_groups=python_zip_file for
'hermetic' python, but the bp2build-built deapexer target is missing
some dependencies on protobuf.

Bug: 265176532
Test: Manual runs of apex_comparison_tests.sh with the new Bazel release
Change-Id: Ib7021e7e81d1a7bf5ff4d222e2a7947a06d16791
2023-01-11 17:30:55 -05:00