Commit graph

188 commits

Author SHA1 Message Date
Wei Li
b82c236cbb Merge "Add integration test for verifying package verification code in SBOM." into main 2023-08-02 19:10:52 +00:00
Wei Li
8ea2ffc825 Add integration test for verifying package verification code in SBOM.
The test case calculates package verification code of package product according to the SPDX spec and compare it to the one in SBOM file which should have the same SHA1 hash value. This helps verify the python logic of generating it in SBOM files.

Bug: 293304694
Test: build/soong/tests/sbom_test.sh
Change-Id: I37c96d90a1990fbeb786f1bd4e8dc87102e0f0cd
2023-08-02 19:10:23 +00:00
Wei Li
9d1ff41f3d Remove sbom_test from soong integration test.
Bug: 294129889
Test: CIs
Change-Id: Ic2925b214bd1e9299b7fa39f7ad0540bca282b53
2023-08-01 16:40:15 -07:00
Treehugger Robot
e4d58523b6 Merge "Ignore some new files added to partition images in sbom_test.sh." into main 2023-07-21 14:34:44 +00:00
Romain Jobredeaux
ca4706b474 Symlink to rules_java in soong integration test.
Bug: 292023656
Test: CI + ABTD
Change-Id: I3ac87154c2d77cb18c7ba60f638b5787cef57606
2023-07-20 09:41:29 -04:00
Cole Faust
b51a05844c Use rbc_dashboard.py instead of rbc_regression_test.sh
rbc_dashboard.py is an alternative implementation of
rbc_regression_test.sh. These tests both run soong 2 times with
starlark product config enabled/disabled and diff the resulting ninja
files. However, the shell script test runs the soong invications in
series and the python one runs them in parallel, meaning the python
one is faster. On my computer the shell one takes ~7 minutes and the
python one takes ~4 minutes.

Bug: 291548757
Test: Presubmits
Change-Id: I20fd6ca09f47dd0547f2681fbd78ba70991493b0
2023-07-17 23:20:57 +00:00
Wei Li
08246b92a6 Ignore some new files added to partition images in sbom_test.sh.
Test: build/soong/tests/sbom_test.sh
Change-Id: I883ba741a47120f56f9f4d0594caebe78ce5242d
2023-07-17 11:33:33 -07:00
Treehugger Robot
b35cc91ba0 Merge "Cleanup genrule_sandbox_test.py" into main 2023-07-14 19:50:23 +00:00
Wei Li
9a8df6669b Some new libraries are added to system partition and are not captured in
product SBOM of cuttlefish devices, ignore these diffs for now in the
SBOM integration test to fix the CI breakage.

Bug: 291073898
Test: build/soong/tests/sbom_test.sh
Change-Id: Idf5569f67642784ee422847301c59dddc8ff0907
2023-07-13 10:29:55 -07:00
Cole Faust
bbe2cc6191 Cleanup genrule_sandbox_test.py
In preparation for adding the ability to check all genrules in the tree.

Bug: 290816499
Test: ./build/soong/tests/genrule_sandbox_test.py extensions_db.pb
Change-Id: I6454be7e88a876a2ab1105c04f5ecda03ecd7771
2023-07-12 18:01:32 -07:00
Wei Li
59df0ee442 Add integration test for m build SBOM of APEXs/APKs.
Bug: 266726655
Test: build/soong/tests/sbom_test.sh
Change-Id: Id8002042f607f8e158f561ca47e0a84a8b8927e2
2023-07-10 15:50:38 -07:00
Cole Faust
837c3b9a55 Revert "Commend out dcla_apex_comparison_test.sh"
This reverts commit a56663f695.

Reason for revert: Original breaking change was reverted

Change-Id: I8e0c97f0b511f686f5efdb8a9e8c9f862fc41010
2023-06-30 22:53:10 +00:00
Cole Faust
a56663f695 Commend out dcla_apex_comparison_test.sh
Is currently failing on ci, uncomment when fix is submitted.

Bug: 289141798
Test: Presubmits
Change-Id: Ic7b220f917cb0b9f0914d8e9575081004610cf63
2023-06-29 12:36:00 -07:00
Treehugger Robot
3da9587eda Merge changes from topic "revert-2637031-revert-2626204-rules_go_repo-OSFOZKENFO-KSSEOFEKLG"
* changes:
  Revert^2 "Symlink external/bazelbuild-rules_go in mock tests"
  Revert^2 "Always use the BUILD files from bazelbuild-rules_go"
2023-06-26 08:01:19 +00:00
Spandan Das
f7373e62dc Revert^2 "Symlink external/bazelbuild-rules_go in mock tests"
5479e0f202

Change-Id: Idc4baa3d8afd1841ee3620e2f2e29a30c85a5473
2023-06-26 03:44:04 +00:00
Matthias Männich
5479e0f202 Revert "Symlink external/bazelbuild-rules_go in mock tests"
Revert submission 2626204-rules_go_repo

Reason for revert: breaks the aosp_kernel-build-tools (https://ci.android.com/builds/branches/aosp_kernel-build-tools/grid)
Reverted changes: /q/submissionid:2626204-rules_go_repo

Change-Id: I1c816bdb739e08794da4affb6fca3b3a052ed355
2023-06-23 07:45:26 +00:00
Wei Li
cd9948f66e Cleanup SBOM test script for later adding tests of generating SBOM of
unbundled APKs/APEXs.

Bug: 266726655
Test: build/soong/tests/sbom_test.sh
Change-Id: I2bb3e8405dad716837d3004d4c6fd54ae80527c1
2023-06-22 17:52:31 -07:00
Spandan Das
60133b3070 Symlink external/bazelbuild-rules_go in mock tests
Test: TH
Change-Id: I067bda143252c66bfac63699d51ca3b80f5a5f63
2023-06-21 00:48:05 +00:00
Jason Macnak
394f1b7b86 Fix local var error, warn if no outputs, handle "}generate{"
1. UnboundLocalError: cannot access local variable 'output' where
   it is not associated with a value

2. `build/soong/tests/genrule_sandbox_test.py WmediumdServerProto_cc`
   would try to build the entire world due to 3)

3. WmediumdServerProto_cc 's module action desc had '}generate{'
   without a space which would cause 2)

Bug: b/287470077
Test: build/soong/tests/genrule_sandbox_test.py WmediumdServerProto_cc
Change-Id: Ie9d5db44195a6f700cc876ec889bd01c11e66cff
2023-06-15 10:59:20 -07:00
Mark Dacek
ff0f42a3ac Merge "Add integration test for ensuring that stale metrics files are deleted." 2023-06-14 19:14:31 +00:00
MarkDacek
5040c36649 Add integration test for ensuring that stale metrics files are deleted.
Bug: 286083726
Test: run the script
Change-Id: I2dd382b3a0ee1b226f4931fed3bc4b5df0a3528c
2023-06-14 15:05:36 +00:00
Trevor Radcliffe
ded095ce45 Bp2build Sanitizer Blocklist
Bug: 286894426
Test: Unit tests
Change-Id: I382e028410a185a5017dba1fc47c83ad5b2432cf
2023-06-13 20:46:27 +00:00
Treehugger Robot
01f8784246 Merge "Read BUILD files in bp2build" 2023-06-09 22:42:39 +00:00
Treehugger Robot
cbe72299ce Merge "Add feature to print output paths for module" 2023-06-09 19:34:00 +00:00
Treehugger Robot
7cb703fb99 Merge "Make genrule sandbox script a python script" 2023-06-09 19:32:11 +00:00
Chris Parsons
8152a94816 Read BUILD files in bp2build
The parsed BUILD files will be scanned for obvious definitions of BUILD
targets which have Android.bp counterparts. In such cases, bp2build will
automatically omit conversion of these defined modules (to prevent
collisions). With this change, we no longer need one-off denylisting of
modules which have BUILD file definitions.

This has a 0.03s to 0.2s slowdown for bp2build with current state. This
impact is identical on a heavier test branch, as well. I also ran an
experiment that applied BUILD scanning to all source BUILD files
(regardless of allowlisting), and this had a 2 second slowdown.

We may want to look into parallelizing or improving the performance of
this evaluation, but it's probably not worth the effort at this time,
since the current performance hit is small.

Test: New integration test
Test: Removed libprotobuf-python from denylist and tested building the
package
Test: Treehugger

Change-Id: Ibde3bab12cd4a8fed642ad46e5344a56953bec91
2023-06-09 16:02:40 +00:00
Liz Kammer
2fb361cfb8 Add feature to print output paths for module
Test: ./genrule_sandbox_test.py -o gen_fstab.gs201 \
    libbt_topshim_bridge_header \
    android-support-multidex-instrumentation-version
Change-Id: Ied74d727054a1bc3b447c1f10a4ee41b3fdfdf1d
2023-06-09 11:31:28 -04:00
Liz Kammer
767fad4b05 Make genrule sandbox script a python script
Unfortunately, genrules are not always available with `m`, instead we
need to know their output paths in order to build them and diff them.
Rewriting in Python lets us store module:output path maps more easily.

Test: ./genrule_sandbox_test.py gen_fstab.gs201 \
      libbt_topshim_bridge_header \
      android-support-multidex-instrumentation-version
Change-Id: If74130e5a4381cc0e1fab396ebb90dfd5a595a1c
2023-06-09 11:31:08 -04:00
Cole Faust
319abae1c7 Remove --noexperimental_platform_in_output_dir
This is a followup to aosp/2606989.

This flag is not necessary now that we're using one platform name
for all of mixed builds.

Also rename current_product to mixed_builds_product so that it's clear
that that this platform should only be used for mixed builds.

In addition, make the bazelrc files point to the named products again
instead of the mixed build product so that b builds will still have
qualified outputs, but mixed builds won't.

Test: Presubmit and kernel build tools abtd run
Change-Id: I7f764cf42cd1323f4b495d1320931f59a076ac63
2023-06-08 17:58:35 -07:00
Treehugger Robot
ca69a6fc21 Merge "Revert "Remove --noexperimental_platform_in_output_dir"" 2023-06-08 12:10:14 +00:00
Matthias Männich
08ccec7c11 Revert "Remove --noexperimental_platform_in_output_dir"
This reverts commit 24af24e0f2.

Reason for revert: Breaks kernel build tools build https://ci.android.com/builds/branches/aosp_kernel-build-tools/grid?head=10282004&tail=10282004

Change-Id: I486d7f305d44cfecc51a0e039cc41c0afa6495f3
2023-06-08 08:19:14 +00:00
Treehugger Robot
731c4525ca Merge "Remove --noexperimental_platform_in_output_dir" 2023-06-08 06:08:14 +00:00
Cole Faust
24af24e0f2 Remove --noexperimental_platform_in_output_dir
This is a followup to aosp/2606989.

This flag is not necessary now that we're using one platform name
for all of mixed builds.

Also rename current_product to mixed_builds_product so that it's clear
that that this platform should only be used for mixed builds.

In addition, make the bazelrc files point to the named products again
instead of the mixed build product so that b builds will still have
qualified outputs, but mixed builds won't.

Test: Presubmit
Change-Id: I5acbd18869589f67efb0ad3ebfda2c4d5a6a3850
2023-06-07 11:08:44 -07:00
MarkDacek
bf7a5f1c71 Add b_test to run_integration_tests.sh
Bug: 281547018
Test: run the script
Change-Id: I4cb0205ba47c6c97015146fa0782d7a30fb80bac
2023-06-07 00:52:09 +00:00
Yu Liu
7301fc8aeb Add the DCLA libs used by swcodec to the staging list
Bug: 269342245
Test: run dcla_apex_comparison_test
Change-Id: I26bcbf6447b9216258c18c42e80f729e5651658a
2023-06-05 16:47:34 -07:00
Trevor Radcliffe
59b3729825 Update apex_comparison_test.sh to support CFI
The CFI transition being applied creates a situation where Bazel
clears its convenience symlinks. This change points the script
directly to the location of the deapexer artifact instead of the
symlink.

Bug: 261733820
Test: Ran the tests
Change-Id: I23951f0fe75ce1fbc3c0dd969244a797ac1ff983
2023-06-05 18:07:42 +00:00
Yu Liu
62461ea576 Merge "Tool to detect modules that generate the incorrect output in sandbox" 2023-05-31 21:34:49 +00:00
Yu Liu
fb458e9255 Tool to detect modules that generate the incorrect output in sandbox
Bug: 281067655
Test: Test locally
Change-Id: Ie38e3658eee4d577d1f871c9ed8129b4ac7d7d07
2023-05-26 14:45:12 -07:00
Wei Li
704146094a Exclude diffs caught by sbom_test.
Bug: 284279034
Test: build/soong/tests/sbom_test.sh
Change-Id: I152c2e2c55b6547b2dddf7b1b2524681c9af952e
2023-05-25 16:29:20 -07:00
Treehugger Robot
873691ae99 Merge "simplify m clean" 2023-05-24 18:36:39 +00:00
Usta Shrestha
6e1aa7830c simplify m clean
`rm -rf out` used to fail because bazel would not set write permission on some files. Now it's been fixed and thus there is little reason to prefer `m clean` (also users are accustomed to `rm -rf out`)

Bug: NA
Test: verified `m clean` works;interestingly soong_metrics file is created at the end
Change-Id: I000d16508613045811fc7792e5798f7c150dcc05
2023-05-23 17:34:15 -04:00
Treehugger Robot
1befe407a1 Merge "Revert "Revert "Restrict plugins to an existing allowlist""" 2023-05-23 17:08:46 +00:00
Jihoon Kang
f530696762 Add missing dependency in create_mock_soong
This change adds missing dependency of Soong modules as symlink
directory in create_mock_soong so that a correct ninja graph can be
created when invoking androidmk_test.sh.

Test: build/soong/test/run_integration_tests.sh
Bug: 283905797
Change-Id: I7dff0094fdc23b7c438c917dbc00a5143a4dc5ae
2023-05-23 06:57:38 +00:00
Liz Kammer
33cc80eb1f Revert "Revert "Restrict plugins to an existing allowlist""
This reverts commit b20a12553e.

Reason for revert: Rollforward with fix

soong-ca-certificates-apex was missing from allowlist

Test: verify fixes CI breakages
(cherry picked from https://googleplex-android-review.googlesource.com/q/commit:dfe4a9e4dd088f42ba5e0a3c4f7777c253d378d3)
Merged-In: I00687bd2962dbdaaf55189ca4aa82183264eb8d7
Change-Id: I00687bd2962dbdaaf55189ca4aa82183264eb8d7
2023-05-18 18:20:28 +00:00
Yu Liu
09a8d459d5 Test dcla libs for b build.
Bug: 273862399
Test: local
Change-Id: Ic72239779d63d64b68a902de6f5b3c6567385878
2023-05-17 20:01:28 -07:00
Chris Parsons
a3ae007641 Touch soong_build output at the end of main
This fixes an incrementality bug that resulted in superfluous
build.ninja regeneration after a new glob definition is added.

Fixes: 279674820
Test: Treehugger
Test; New integration test in this CL

Change-Id: Ifefe66a0eb1c125e9ad5373d60437a1cb1e6fdec
2023-05-12 13:40:12 +00:00
Cole Faust
c9508aac4c Load starlark files from soong
There are a number of instances where we are exporting information
from soong to bazel via soong_injection. This could be more bazel-centric
if the information was instead held in bzl files, and both bazel and
soong read it from there.

Add a starlark package that will run
//build/bazel/constants_exported_to_soong.bzl at initialization time,
and then results can be retreived with GetStarlarkValue.

Since changes to the starlark files mean that soong has to rerun,
add them as ninja deps.

Unfortunately, the starlark code has to be run at runtime rather than
pregenerating their results, because tests run from intellij wouldn't
go through any pregeneration steps. This means that starlark is run
multiple times during the build, once per test package and once per
primary builder invocation. (currently 3, could be reduced to 2 if we
made the symlink forest generation into its own standalone tool) The
starlark code we have so far in this cl is very fast, roughly half a
millisecond, so it's not a big deal for now, but something to keep an
eye on as we add more starlark constants.

Bug: 279095899
Test: go test
Change-Id: I1e7ca1df1d8d67333cbfc46e8396e229820e4476
2023-04-26 17:18:19 -07:00
MarkDacek
f47e142ffd Refactor MixedBuildsEnabled and add --ensure-allowlist-integrity.
Currently, there is little verification around allowlisted modules
actually being mixed-built. This flag would allow us to verify
that a module allowlisted is mixed-built for at least one variant.

Bug: 278910100
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
Test: m nothing --bazel-mode-staging --ensure-allowlist-integrity
--bazel-force-enabled-modules=com.google.android.neuralnetworks (This
fails, as expected)
Test: build/soong/test/mixed_mode_test.sh

Change-Id: Icd5976f4f44f1a8caca1e5247d986642f7995f97
2023-04-25 20:55:07 +00:00
MarkDacek
53506361da Squelch output for a failure case in mixed_mode_test
Currently, the expected output shows as the main text for
unrelated presubmit failures.
Bug: 278748024
Test: build/soong/tests/mixed_mode_test.sh

Change-Id: I553a82597c7c7791a822e330eb9b2663f35567a1
2023-04-21 17:55:42 +00:00
MarkDacek
9c094ca153 Fix issue where bazel-force-enabled-modules aren't actually analyzed.
Bug: 273910287
Test: run mixed_mode_test.sh

Change-Id: I309c423b6e7f505ad1db0d9327e7de6b8298dfe9
2023-04-14 02:17:54 +00:00