Commit graph

493 commits

Author SHA1 Message Date
Sebastian Pickl
90355f79bf Revert "Change symlink_forest to use relative symlinks."
Revert submission 2673616-relativesymlinks-fix

Reason for revert: this breaks tests verified with go/abtd https://android-build.googleplex.com/builds/abtd/run/L36000000963001181

Bug: 300122962

Reverted changes: /q/submissionid:2673616-relativesymlinks-fix

Change-Id: I5a97c4fbe4df5727c0604a07137093d0f00c7776
2023-09-12 18:51:39 +00:00
Mark Dacek
4eabebec69 Merge "Change symlink_forest to use relative symlinks." into main 2023-09-12 15:47:42 +00:00
MarkDacek
b98b3a429f Change symlink_forest to use relative symlinks.
Also add script to remove the Bazel output base.
This will assist with supporting movable checkouts alongside
mixed builds.

Bug: 259191764
Test: m && (move topic and prepare_moved_top.sh) && m
Test: m && prepare_moved_top.sh && m
Test: build/soong/tests/relative_symlinks_test.sh

Change-Id: I0f53da8d99f752fad496cf3ac61b01f001b7296d
2023-09-12 04:02:23 +00:00
Jooyung Han
4bc102672a apex: invoke conv_linker_config validate as validation
`conv_linker_config validate` command is used to validate the linker
configuration embedded in APEX to detect common mistakes.

For example, when used in APEX, linker configuration can't set
provideLibs/requireLibs. For APEX, there are
provideSharedLibs/requireSharedLibs in APEX manifest for that purpose.

One might make mistake by setting provideLibs in linker config.
Now, when these unsupported properties are set, there'll be build-time
error like:

 // set provideLibs key in com.android.art's linker config.
 $ m com.android.art
   ...image.apex/etc/linker.config.pb: provideLibs is set. Use provideSharedLibs in apex_manifest

Bug: 264341796
Test: m com.android.art (see above)
Change-Id: Ibaf7322616ad333569e6d721680f3d72243402a2
2023-09-08 15:40:12 +09:00
Cole Faust
51d7bfd9a1 Revert^2 "Only allow setting presigned without preprocessed on targetSdk < 30"
This reverts commit d293e28f52.

Reason for revert: The underlying issue was fixed in ag/24685010

Change-Id: I06810d37dba37aa12f9a1e14b0749f1e1eb41136
2023-09-07 05:31:32 +00:00
Karl Shaffer
d293e28f52 Revert "Only allow setting presigned without preprocessed on targetSdk < 30"
This reverts commit 6158528e15.

Reason for revert: DroidMonitor-triggered revert due to breakage https://android-build.corp.google.com/builds/quarterdeck?branch=git_udc-d1-dev-plus-aosp&target=aosp_bramble-trunk_staging-userdebug&lkgb=10771573&lkbb=10771600&fkbb=10771587, bug https://buganizer.corp.google.com/issues/299369971

BUG: 299369971

Change-Id: I6bf6eb5c0fb9e30197e145121adc7ed58871526f
2023-09-07 00:51:09 +00:00
Cole Faust
9c5c09f0ad Error out if skip_preprocessed_apk_checks is set when it's not necessary
So that we don't erroneously add skip_preprocessed_apk_checks
everywhere.

Bug: 185811447
Test: Presubmits
Change-Id: Icb1c6163d170ca4181c5c6a814b51fda4777746f
2023-09-06 16:11:44 -07:00
Cole Faust
6158528e15 Only allow setting presigned without preprocessed on targetSdk < 30
When targetSdk is >= 30, the system verifies that you use a valid
signature V2+ certificate. Uncompressing ndk/dex files or aligning
the zip file will break a signature V2, so these apks should really
just set preprocessed: true.

Fixes: 185811447
Test: Presubmits
Change-Id: Id89c42bcd5b5daa6eda1716bff4023423298036b
2023-08-29 11:18:44 -07:00
A. Cody Schuffelen
f29ca58e88 Darwin/Mac OS host rust compilation fixes
- Don't pass `--as-needed` to the linker on Mac OS which is unsupported
  there
- Use `--force_load` rather than `--Wl,--whole-archive` on Mac OS
- Scan `rustc`'s linker arguments for `-dylib` and `-dynamiclib`, which
  it can use instead of `-shared` on Mac OS:
  7bd81ee190/compiler/rustc_codegen_ssa/src/back/linker.rs (L319)

Test: m libhalf serde_derive
Bug: 291164566
Change-Id: Iecd6c2532fa31c9476834f49b109de98cbd2dccf
2023-08-03 16:26:44 -07:00
Alyssa Ketpreechasawat
ac6c853ee6 Fix the number of required parameters in deapexer script.
Regarding aosp/2624311, `blkid_path` was removed from the command, so the number of parameters should be reduced by 1.

Test: MODULE_BUILD_FROM_SOURCE=false m (in internal-main branch)
Bug: 279858383
Change-Id: Iad494b757e2a4152a6867098c3eb7419c3bba71b
2023-07-14 17:26:13 +00:00
Jooyung Han
6c762092ee Remove --blkid_path argument
deapexer doesn't need it.

Bug: 279858383
Test: presubmit
Test: m MODULE_BUILD_FROM_SOURCE=false droid
Change-Id: Ie8311c74682f366cf6aeebb669500d01675cd0bf
2023-06-14 15:23:03 +09:00
Jiakai Zhang
a449678996 Move CLC construction to Ninja phase.
Before this change, dexpreopt was often broken with optional libraries.
This was because the CLC construction was done in Soong at an early
stage, where we don't have sufficient information to determine whether
an optional library is installed or not.

For example, the "Settings" package uses an optional library called
"androidx.window.extensions". On some devices, the library is installed,
but on some other devices, it's not. Soong always adds the library to
the CLC, meaning the CLC is wrong for devices which don't have the
library. This change fixes the problem. See the tests below.

After this change, the CLC construction is done by a Python script
invoked at a very late stage. It uses product_packages.txt, which is
generated by Make, to determine whether an optional library is
installed or not, and filter out libraries that are not installed.

Note that optional libraries are still added as dependencies by Soong.
This is because dependencies have to be added at an early stage. This
means what dex2oat eventually uses will be a subset of the dependencies,
which is fine.

Bug: 282877248
Test: m
Test: atest construct_context_test
Test: -
  1. lunch aosp_cf_x86_64_phone-userdebug && m
  2. Check the .invocation file of the "Settings" package (defined in
     .bp file)
  3. See androidx.window.extensions
Test: -
  1. lunch aosp_redfin-userdebug && m
  2. Check the .invocation file of the "Settings" package (defined in
     .bp file)
  3. Don't see androidx.window.extensions
Test: Check the .invocation file of the "Dialer" package (defined in
  .mk file)
Test: -
  1. Build a Pixel 5 system image and flash it to a Pixel 5 device.
  2. adb shell pm art dump
  3. See "reason=prebuilt" instead of "reason=vdex".
     (https://diff.googleplex.com/#key=fB6Ls9q2QGSN, before: left,
     after: right)

Change-Id: Ia112bd7c2328373e68db6bffb74bf34030f683d8
2023-05-30 15:46:38 +01:00
Cole Faust
8f065e6c5d Delete rbc-run script
It was inlined into the usage in the other cl in this topic.

Bug: 280685526
Test: ./build/bazel/ci/rbc_dashboard.py --quick aosp_arm64-userdebug
Change-Id: I68af9186e17a996baf4676d97cdc9895abcb7e06
2023-05-04 18:02:22 -07:00
Almaz Mingaleev
cb97a76ad4 Merge "Add jdk.internal.access to the allowed package list." 2023-05-04 18:01:13 +00:00
Almaz Mingaleev
2429776555 Add jdk.internal.access to the allowed package list.
This is part of OpenJDK17 update.

Bug: 270674727
Test: m
Change-Id: I0708ee286cd7c3c6eef593784affaeffbdf0a76c
2023-05-03 07:21:27 +00:00
Andrei Onea
580f7b0094 Add script for modifying privapp permission allowlists
This script will be used to change the package that is being allowed to
use privileged permissions.

Test: python scripts/modify_permissions_allowlist.py
Bug: 242509786
Change-Id: I1d7cb3c906ad39aef42ad4394953251db40d1bf1
2023-05-02 15:10:16 -04:00
Treehugger Robot
f5cd477841 Merge "Split Rust crate builds into compile and link phases." 2023-04-13 22:45:02 +00:00
Peter Collingbourne
e7c71c344d Split Rust crate builds into compile and link phases.
Teach rustc to generate object files instead of a linked executable
for binary and shared library crates. This lets us skip the lengthy
Rust compilation phase when only the C++ dependencies of these crates
have changed.

This works using a Python script that replaces the linker invoked by
rustc and converts the linker command line into a rspfile that can be
used during the link phase. The script makes certain assumptions about
how rustc invokes the linker, but since we control the version of Rust
we use, we can update the script if those assumptions ever break. I've
asked the Rust developers about adding an official way to do this so
hopefully we won't need to rely on this hack in the future.

The rustc rule for Kythe has been altered to pass linker=true, somewhat
consistent with the main rustc rule, which now doesn't invoke the actual
linker either. `m xref_rust` still succeeds, but hopefully we can find
someone who knows more about the rest of the Kythe pipeline who can let
us know whether it should still work. Otherwise, let's use this CL as
an opportunity to find out whether the rest of the pipeline is happy
with linker=true, since otherwise we will find it hard to maintain the
Kythe rule in the future.

Change-Id: Ifb9c8689c1bd78c0afcf6fdfd111140d20377536
2023-04-12 15:51:41 -07:00
Jooyung Han
4b4f03fd82 Make conv_linker_config visible to linkerconfig
linkerconfig_diff_test uses conv_linker_config.

Bug: 276890767
Test: atest --host linkerconfig_diff_test
Change-Id: I2f669b83f24b9dd563fc034738fe4ba83f95fd49
2023-04-12 17:20:47 +09:00
Elliott Hughes
7d8f39e127 build-ndk-prebuilts.sh: disable bazel.
```
ERROR: /buildbot/src/android/master/out/soong/workspace/external/libcxx/BUILD.bazel:30:18: While resolving toolchains for target //external/libcxx:libc++: com.google.devtools.build.lib.packages.NoSuchTargetException: no such target '@soong_injection//product_config_platforms/products/ndk-eng:ndk-eng_android_riscv64': target 'ndk-eng_android_riscv64' not declared in package 'product_config_platforms/products/ndk-eng' defined by /buildbot/src/android/master/out/bazel/output/external/soong_injection/product_config_platforms/products/ndk-eng/BUILD (Tip: use `query "@soong_injection//product_config_platforms/products/ndk-eng:*"` to see all the targets in that package)
```

Bug: https://github.com/google/android-riscv64/issues/10
Bug: http://b/274509603
Bug: http://b/262192655
Test: build/soong/scripts/build-ndk-prebuilts.sh
Change-Id: I4b6633ab5aab3f01953b628d78ac1d79f2a4ee3c
2023-03-21 00:42:35 +00:00
Elliott Hughes
65a97364ba Merge "Generate NDK stubs for riscv64 too." 2023-03-20 21:10:53 +00:00
Elliott Hughes
f7d3109972 Generate NDK stubs for riscv64 too.
These will only be provisional, but there are enough different groups
that want *something* to make a start on porting that it's probably
time...

Ignore missing dependencies in build-ndk-prebuilts.sh for now, because
various pieces (that the NDK doesn't care about) are still missing.

Bug: http://b/273792258
Test: build/soong/scripts/build-ndk-prebuilts.sh
Change-Id: I39202a953a73dc449a10a887d497d09079c43402
2023-03-17 18:49:18 +00:00
Romain Jobredeaux
6d8716c176 Create a sh_binary target for jars-to-module-info
This will enable its bp2build conversion, and is needed for java_system_modules support in Bazel

Change-Id: I4f3ff5e36c8cd7f78efbb42f641efb2f76a8b71d
Bug: 215230098
2023-03-17 14:58:08 +00:00
Jooyung Han
fba49bbf56 Add tests for conv_linker_config
to cover the recent features.
- conv_linker_config proto with empty input
- conv_linker_config proto with existing output
- conv_linker_config proto with --append
- conv_linker_config proto with --force

Bug: n/a
Test: conv_linker_config_test
Change-Id: I0de79b6e05c2608e0e2f30dfbf04d8289672f362
2023-03-09 07:46:25 +00:00
Jooyung Han
f6fd4c2823 Fix conv_linker_config with multiple input json files
`conv_linker_config proto -s` should work with multiple json input
files, but ParseDict() overwrites list fields (e.g. provideLibs), not
appending elements.

Also added a test.

Bug: 264330513
Test: conv_linker_config_test
Change-Id: Idc482f941201f15e5fc276c0ffc0dfeaa09d0cc2
2023-03-09 07:46:00 +00:00
Jooyung Han
3397b6a2aa conv_linker_config handling existing output file
`proto` sub command now fails when the output file exists.
This is to avoid accidental overwrite.

To handle the case properly, it support --force and --append.

This is to support amending /{partition}/etc/linker.config.pb in the
build process.

Bug: 244531518
Test: manual testing
Change-Id: I0af8c83015e485f2c7533221cae8caf6143403c8
2023-03-07 21:18:17 +00:00
Treehugger Robot
6bc5e2cefa Merge "conv_linker_config proto works with empty input" 2023-03-06 18:36:10 +00:00
Jooyung Han
b531beecd5 conv_linker_config proto works with empty input
PRODUCT_VENDOR_LINKER_CONFIG_FRAGMENTS lists json files for vendor
linker config. It's annoying to handle the case of empty list.

`proto` subcommand now works for empty input. This is useful to generate
the empty linker config.

Bug: 244531518
Test: conv_linker_config proto --source --output output.pb
Change-Id: Iec6de67a979814a818730c393d9a4a7ca5d2eebe
2023-03-04 08:28:40 +09:00
Sorin Basca
93bca1585e Merge changes from topic "java17-record-support"
* changes:
  Allow java.lang.runtime in boot image
  Emit record annotation
2023-03-03 15:12:28 +00:00
Joe Onorato
442b6fc701 Remove OWNERS access for non-build team people to core build system
Test: treehugger
Change-Id: I0c318ddd7d78110327bbd849ed9aa3973c86de5d
2023-02-28 19:29:47 -08:00
Victor Chang
7c05d2d185 Allowlist java.lang.constant in libcore
Bug: 270028670
Test: m droid
Change-Id: I7612998fd518442defd5f9f00115e6a16e261b18
2023-02-20 17:27:18 +00:00
Sorin Basca
4025390468 Allow java.lang.runtime in boot image
Test: m
Change-Id: Icb208bb235f9c6ac97492c54c32a3174982756ae
2023-02-10 07:31:51 +00:00
Seth Moore
c6f4b532f3 Allow android tests to specify a mainline package name
Tests that run against mainline modules generally need to be built
twice: once for aosp (unsigned) and once for internal (signed).
The override_android_test rule is really useful for generating a
signed version of the unsigned test.

However, mainline tests need to use a MainlineTestModuleController
to only run tests on targets with the expected module. Since unsigned
and signed mainline modules have different package names, this means
the test config needs to be updated when overriding a test.

Add a new "mainline_package_name" argument so that android test rules
can update the targeted mainline package. Without this, the only way
to create a mainline package test is to instead use android_test, which
results in a lot of copy-pasta.

Test: added tests to soong
Test: manual, using a locally-modified RkpdAppUnitTest
Test: test_config_fixer_test.py
Change-Id: Idaffd63f225719a2bfda41018fda630b17db0080
2023-02-08 12:38:50 -08:00
Almaz Mingaleev
8b04d82ff1 Add jdk.internal.util.jar in the allowed package list.
This is part of OpenJDK17 update. sun.misc.JarIndex was moved to
jdk.internal.util.jar.

Bug: 260718199
Test: m
Change-Id: I7107f980d6afe500acff88164b1e6cb04cded896
2023-02-03 11:42:46 +00:00
Paul Duffin
0f386bc12b Allow obfuscated classes on bootclasspath
Previously, the signature_trie python library would reject classes
which started with a lower case character which in turn caused the
verify_overlaps tool to fail. That meant that it was impossible to
add obfuscated classes (which commonly used lower case characters
for their name) from being used on the bootclasspath.

This change removes that restriction and the accompanying test.

Bug: 265833521
Test: TH and partner testing
Change-Id: I70710484e427f64d79fb30301f3413f3b67b27e7
2023-01-18 12:59:23 +00:00
Jooyung Han
014ccd4ef0 conv_linker_config proto supports multiple input json files
conv_linker_config proto -s input1.json:input2.json generates a merged
proto output file.

Bug: 264330513
Test: conv_linker_config proto -s input1.json:input2.json
Change-Id: I4806fc606d115c42a1c745fba67960ce6db8c8d7
2023-01-09 16:26:53 +09:00
Sasha Smundak
648e32c341 Run ckati step of the Android build tracing given variables
Test: `m nothing && build/soong/scripts/run-ckati.sh FOO BAR`, see FOO
and BAR assignments in out/ckati.trace

Change-Id: Ife8bb6d716bfde5ee9f75b85b1e5738ab2238dce
2023-01-04 10:08:43 -08:00
Paul Duffin
e17c31611a Fix deapexer usage in prebuilt_apex/apex_set
A previous change to the deapexer script to check that blkid_path and
fsckerofs_path were set broke the scripts/unpack-prebuilt-apex.sh
scripts use of deapexer. This change fixes it by passing the paths
through to the script which then passes it on to deapexer.

Bug: 255963179
Bug: 240288941
Bug: 259958590
Test: m MODULE_BUILD_FROM_SOURCE=false droid
Change-Id: Ief7f36219b7fe1cf7104c08163e10d8e39f181d0
2022-12-12 17:40:26 +00:00
Jingwen Chen
2d61a7d005 bazel apex: clean up gen_ndk_usedby_apex.sh to work in bazel sandbox.
This script is responsible for generating <module>_using.txt, which is a
file containing all NDK symbols used by the shared libraries in an APEX.

This script doesn't work as-is in the bazel sandbox; namely it tries to
create tmp directories in read only input dirs, for example:

"mkdir: cannot create directory 'bazel-out/android_target-fastbuild-ST-7fe6d3cd3905/bin/packages/modules/adb/apex/com.android.adbd_staging_dir/tmpUnzipped': Permission denied"

The staging dir inputs are also symlinks, so the `find` call needs a new
-L flag to follow symlinks.

This CL fixes that by using a tmpdir, and also refactors the script to
be less fragile.

I'm not a fan of the camelCase, but I'll save that for another change.

Test: presubmits
Bug: 239081455
Bug: 257226023
Fixes: 257226023
Change-Id: I1d0fe93c47c92d046e72c0ab32014a5dc8bbd597
2022-11-07 08:38:01 +00:00
Chris Parsons
4d0b4df833 Add prebuilts/ to microfactory includes
This allows go sources under subdirectories of prebuilts/ to be
referencable by soong_ui and soong_build. Without this change,
microfactory does not correctly include go libraries under prebuilts.

Test: Conjunction with aosp/2260126
Change-Id: I4cba6fb04d1e274af759fa92f9ae4b2e3ea18447
2022-10-19 11:11:45 -04:00
Treehugger Robot
f5410382ab Merge "Remove obsolete script for building unbundled Mainline targets." 2022-09-21 18:11:20 +00:00
Martin Stjernholm
4fc6f4ee62 Remove obsolete script for building unbundled Mainline targets.
The supported scripts for doing that now live in
packages/modules/common/build.

Test: Check in CodeSearch that the script isn't actively used nor
  mentioned in docs.
Bug: 180394948
Change-Id: I787cbe469bec2121f89de7b74695484310b00b5e
2022-09-21 11:50:08 +01:00
Jooyung Han
7113b19be8 Add a host tool to create an APEX bundle
Even though the soong build system generates a bundle module for an
apex, the bundle module itself should be further processed to be an
AppBundle (.aab) file which can be uploaded to Play.

This script fills the gap by invoking bundletool to create an
AppBundle (.aab) file out of soong-built bundle module for an apex.
(Note: uploading APEX bundle (.aab) to Play is not supported yet.)

You can create an .aab file by:
- TARGET_BUILD_APPS={apex name} m dist
- m build-apex-bundle
- build-apex-bundle --output out.aab out/dist/{apex name}-base.zip

For now it creates a single-ABI APEX bundle. In the future it can be
extended to support multiple-ABI APEX bundles.

Bug: 236673372
Test: m build-apex-bundle
Test: TARGET_BUILD_APPS=com.google.cf.bt m dist
Test: build-apex-bundle --output bt.aab out/dist/com.google.cf.bt-base.zip
Change-Id: Id321efcd42c0fe60294a8348047c9ebbf7acf391
2022-09-21 13:27:49 +09:00
Krzysztof Kosiński
ef8d6a68f4 Remove Python enables identical to global defaults.
Bug: 245854393
Test: presubmit
Change-Id: Ia99d3fc9f1626891d45c55d679f4d8a049f9e1e8
2022-09-18 01:30:19 +00:00
Cole Faust
bcc3d05aba Fix module-file name collisions
Bazel doesn't allow a module and file with the same name.

Bug: 198619163
Test: Presubmits
Change-Id: Ie9731b627945d2ff221d3a5d08a32a5a172f9dcd
2022-09-01 15:20:00 -07:00
Colin Cross
d079e0b270 Reformat build/soong for go 1.19
Test: none
Change-Id: I132368f0fcbdb5ea088b5b84dbe4ccfdd9e94cad
2022-08-17 10:43:13 -07:00
Alexei Nicoara
69cf0f3756 Overriding placeholder version in updatable apks
Test: presubmit, checked the app version after build locally
Bug: 231691162
Change-Id: Icd242432540ea424235b226a45aac839dbc995be
2022-08-03 12:05:14 +01:00
William Loh
50dbe0a074 Merge "Propagate max_sdk_version to manifest_fixer" 2022-07-14 00:20:35 +00:00
Alexei Nicoara
1e3d27821b Merge changes Ibcf908d9,I786ab152
* changes:
  Cleaning up the version sed rule
  Adding support to conditionally replace a value
2022-07-12 13:07:28 +00:00
Alexei Nicoara
7d69b1d8e6 Adding support to conditionally replace a value
Bug: 231691643
Test: presubmit
Change-Id: I786ab152a94126ebc8a9d7dea0bb68e07d789a0b
2022-07-12 11:53:07 +01:00
Treehugger Robot
6b773a97b8 Merge "Make ‘rustfmt.toml’ available for other modules" 2022-07-11 18:42:42 +00:00
Wei Li
18b7a2e8a2 JSON format doesn't support comments, so a JSONWithCommentsDecoder is added to support line comments start with // that used in apex_manifest.json files. This fixes the CI errors reported in b/238399517.
Bug: 238399517
Test: build/bazel/ci/bp2build.sh
Change-Id: Iaa3c2ab319eb7a52cbedaddd057646fc089d745a
2022-07-08 19:14:01 -07:00
Martin Geisler
4e86b5e07a Make ‘rustfmt.toml’ available for other modules
We need the config file to ensure consistent output when running
tests.

Bug: 237977382
Test: manual
Change-Id: Ib19d4b739305a5cbed94a40908d76dbbeec67fe5
2022-07-04 08:35:44 +00:00
William Loh
5a082f9a33 Propagate max_sdk_version to manifest_fixer
If max_sdk_version is included in Android.bp that value will now be
propagated to manifest_fixer.py. This value will then be used to
override any maxSdkVersion attribute set on permission or
uses-permission tags in the android manifest if maxSdkVersion="-1".

Bug: 223902327
Test: add max_sdk_version to Android.bp for test app
Test: create permission in test app manifest with maxSdkVersion="-1"
Test: run test to check maxSdkVersion=max_sdk_version
Change-Id: Ic533ef2a41b9ecc9ee68c69399026df47ee945b7
2022-06-28 22:07:23 +00:00
Treehugger Robot
802b059417 Merge "Revert "manifest_check.py: add uses-libraries propagaged via dex..."" 2022-06-13 22:21:40 +00:00
Ulya Trofimovich
c68b289125 Revert "manifest_check.py: add uses-libraries propagaged via dex..."
Reason for revert: performance regression on AppStartup.
Reverted Changes:
I6522319a8:Dexpreopt: prepare to merge class loader context f...
Ic8528dffd:manifest_check.py: add uses-libraries propagaged v...

Bug: 235304939
Bug: 214255490
Change-Id: I054df19dbddb4897d07d74e51d7b651dfae85fbe
2022-06-13 09:06:44 +00:00
Ulya Trofimovich
e0862867d9 Merge "manifest_check.py: add uses-libraries propagaged via dexpreopt.config." 2022-06-06 12:50:34 +00:00
Ulya Trafimovich
24abbe85d1 manifest_check.py: add uses-libraries propagaged via dexpreopt.config.
Some of the uses-libraries in the manifest are propagated from
dependencies (which are not necessarily uses-libraries themselves).
In makefiles this has to be done via dexpreopt.config files to ensure
topological order. This patch makes manifest_check.py add such
propagated entries before doing the check (as it might otherwise fail).

Also, the patch adds deduplication of added library entries.

Bug: 214255490
Test: lunch aosp_cf_x86_64_phone-userdebug && m && launch_cvd \
    && adb wait-for-device && adb root \
    && adb logcat | grep -E 'ClassLoaderContext [a-z ]+ mismatch'
    # a few unrelated errors caused by missing optional uses-libraries
    #   android.net.ipsec.ike
    #   androidx.window.extensions
    #   androidx.window.sidecar
    # that were present before this patch
Change-Id: Ic8528dffd47fa5818b38b912cd6b606855480e33
2022-05-24 16:19:13 +01:00
Artur Satayev
74cead891e Add support to max-target-s to generate_hiddenapi_lists
Test: make
Bug: 169748350
Change-Id: I2662a643a190ed531d85755d01e1be9c49c02e25
Merged-In: I2662a643a190ed531d85755d01e1be9c49c02e25
2022-05-06 08:59:39 +00:00
Ge Bian
93b542459a Revert "Add support to max-target-s to generate_hiddenapi_lists"
Revert submission 1931011-max-target-s

Reason for revert: Seems to cause build breakage
Reverted Changes:
I2b513f93a:Add support to max-target-s to generate_hiddenapi_...
Ifdd7b7a71:Add support for max-target-s
I06d05840e:Add support for max-target-s

Change-Id: I67413a3deda42c503b876727377e932f7e4652e9
2022-05-03 20:41:53 +00:00
Narayan Kamath
ad0ec2ae1f Add support to max-target-s to generate_hiddenapi_lists
Test: make
Bug: 169748350
Change-Id: I2b513f93a509bc1163b114d399895cbe9486817e
2022-04-28 18:34:58 +01:00
Treehugger Robot
7436a7121f Merge "Updates test_config_fixer to also work for SuiteApkInstaller" 2022-04-22 07:12:45 +00:00
Sasha Smundak
fe11851029 Recursively list Android image contents.
Relies on debugfs utility to inspect image contents.
The output looks something like this:
```
./
./app/
./app/Calendar/
./app/Calendar/Calendar.apk
```

Test: manual
Change-Id: Ia05f4e857dfd28ac76ac2b450b08f5cf5577d21f
2022-04-20 20:02:11 -07:00
Rahul Sabnis
48a8f0d10b Updates test_config_fixer to also work for SuiteApkInstaller
Bug: 228385580
Test: atest BluetoothInstrumentationTests
Merged-In: I878309354c4aa5a492fb3d2bcecf8b1410e011ee
Change-Id: I878309354c4aa5a492fb3d2bcecf8b1410e011ee
2022-04-20 10:36:58 -07:00
Paul Duffin
bd88c882f6 Remove implementation details from stub flags in sdk snapshot
Previously, the build applied the same filtering to remove
implementation details from the sdk snapshot's stub-flags.csv file as
it did for its all-flags.csv, i.e. removing the signatures that only
had a "blocked" flag. Unfortunately, that had no effect on the stub
flags as the implementation signatures had no flags, not a single
blocked flag. That meant that the sdk snapshot's
filtered-stub-flags.csv file contained a lot of implementation details.

This change removes signatures from stub-flags.csv that have no flags
which removes all implementation details from the sdk snapshot.

Bug: 194063708
Test: atest --host verify_overlaps_test
      m out/soong/hiddenapi/hiddenapi-flags.csv
      m art-module-sdk
      # Check contents of its filtered-stub-flags.csv file
Change-Id: I30edc77348fad118ea732e787ae8e206c8841f84
2022-04-12 17:44:15 +01:00
Paul Duffin
0c12b78ae8 Use named options for verify_overlaps
Previously, verify_overlaps used positional arguments, the first was
the monolithi flags and the rest were the module flag pairs (filtered
flags file and signature patterns file). This change makes them use
named options to make the purpose of the arguments clearer on the
command line.

Bug: 194063708
Test: atest --host verify_overlaps_test
      m out/soong/hiddenapi/hiddenapi-flags.csv
Change-Id: Ife0af0016eb0f91416e8330d5d98cb53c97d68a4
2022-04-08 10:56:21 +01:00
Paul Duffin
181b56cf4d Cleanup verify_overlaps*.py
Fix issues reported by pylint, Intellij Python checks and also try and
adhere to the Google Python Style Guide.

Bug: 194063708
Test: atest --host verify_overlaps_test
      m out/soong/hiddenapi/hiddenapi-flags.csv
      /usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/verify_overlaps*.py
      pyformat -s 4 --force_quote_type single -i scripts/hiddenapi/verify_overlaps*.py
Change-Id: I6273456abc3a4da4af780ae1a0473ca63521dff5
2022-04-08 10:49:24 +01:00
Paul Duffin
b99d480e1d analyze_bcpf: Explain why a package is split/single
Previously, the script would determine whether a package was split,
single or could be used as a prefix but did not explain why. This
change provides additional information to explain why they are split.

Bug: 202154151
Test: m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment
      m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment --fix
      atest --host analyze_bcpf_test
Change-Id: I3e2d5c0b54b5cc028013ce5ea979ebd9b9bf2c0d
2022-04-04 18:24:40 +01:00
Paul Duffin
ea836c2baf analyze_bcpf: Indent multi-line text to improve formatting
Addresses a comment on the review of the initial changes to add
analyze_bcpf script.

Bug: 202154151
Test: m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment
      m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment --fix
      atest --host analyze_bcpf_test
Change-Id: I8c030dcc1a29f106ca57dca8c97a5cb4425e7674
2022-04-04 16:59:36 +01:00
Paul Duffin
dbbb8374a2 signature_trie: Avoid unnecessary wrapping and unwrapping of values
Previously, Leaf.values() and Leaf.append_values() would wrap the
Leaf's value inside a list before appending it to the list of values.
So, the values list was actually a list of lists of values. The
get_matching_rows method would then use chain.from_iterable() to
flatten that list of list of values into a list of values.

This change removes the initial wrapping in a list and so removes the
need to flatten them into a single list. It also adds a test for the
values() method. Prior to this change the expected value would have
been [[1], ["A"], [{}]].

Bug: 202154151
Test: atest --host analyze_bcpf_test signature_trie_test verify_overlaps_test
Change-Id: Ida78500c9ab4466de127b2c36501b3606d0f3fe5
2022-04-01 15:04:23 +01:00
Bob Badour
de6a087d2d Revert "Revert "Build notice files from license metadata.""
This reverts commit 77807b3c27.

Reason for revert: fix and roll forward again

Change-Id: I80c796ca81ecf29df9522babe54d684a0d11b9e6
2022-04-01 18:00:00 +00:00
Wei Sheng Shih
77807b3c27 Revert "Build notice files from license metadata."
This reverts commit 43c2dcaef6.

Reason for revert: suspect build break

Bug: 227682036

Test: TARGET_BUILD_VARIANT=userdebug UNBUNDLED_BUILD_SDKS_FROM_SOURCE=true vendor/google/build/mainline_modules_bundles.sh -j97


Change-Id: Ibfb8b4fefc264f52f32ba661c269a9cd625d800a
2022-04-01 14:10:53 +00:00
Bob Badour
43c2dcaef6 Build notice files from license metadata.
Switch from generate-notice-files.py and mergenotice.py to htmlnotice.

Bug: 151177513
Bug: 213388645
Bug: 210912771

Test: m droid dist reportmissinglicenses
Change-Id: I6cac049d24f35ec358c6f341a04f4ba6161703bf
2022-03-31 17:29:01 +00:00
Treehugger Robot
91dac89447 Merge "analyze_bcpf: Compute hidden_api package properties" 2022-03-30 20:00:43 +00:00
Cole Faust
71350eaff2 Merge "Always build mk2rbc/rbcrun" 2022-03-30 19:43:06 +00:00
Paul Duffin
dd97fd2520 analyze_bcpf: Compute hidden_api package properties
Analyzes the signatures of the class members contained within the
bootclasspath_fragment to construct the split_packages,
single_packages and package_prefixes hidden_api properties that are
necessary to remove the internal implementation details from the
hidden API flags.

Bug: 202154151
Test: m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment
      m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment --fix
      m analyze_bcpf && analyze_bcpf --bcpf com.android.mediaprovider-bootclasspath-fragment --fix
      atest --host analyze_bcpf_test
Change-Id: I4a8e5a8bfee2a44775e714b9226cd4a7382e0123
2022-03-30 16:00:08 +01:00
Paul Duffin
26f19919ea analyze_bcpf: Add --fix option
Add a --fix option that will cause the script to automatically fix the
issues that it finds. It uses the bpmodify tool to add values to the
bootclasspath_fragment's hidden_api properties.

This adds analyze_bcpf to bp2buildModuleDoNotConvertList as
analyze_bcpf depends on bpmodify which is a blueprint_go_binary which
is not yet supported by bazel.

Bug: 202154151
Test: m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment
      m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment --fix
      atest --host analyze_bcpf_test

Change-Id: I5ee52419b4829474f6dbeb47f86ab2aeb22b1382
2022-03-30 16:00:08 +01:00
Paul Duffin
4dcf65951b Add analyze_bcpf.py script to help manage bootclasspath_fragments
The analyze_bcpf.py script can be used by bootclasspath_fragment owners
to help setup their fragment and also to debug issues that arise during
use.

It is run by specifying the name of the bootclasspath_fragment module
in the --bcpf option. If specified the --apex and --sdk options will
allow the tool to produce more helpful messages.

The script currently:
* Builds out/soong/hiddenapi/hiddenapi-stub-flags.txt and checks any
  inconsistencies between those flags and the flags generated for the
  bootclasspath_fragment.

* Builds out/soong/hiddenapi/hiddenapi-flags.csv and checks any
  inconsistencies between those flags and the flags generated for the
  bootclasspath_fragment. It will describe the changes that need to
  be made in order to fix any inconsistencies in the flags.

Bug: 202154151
Test: m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment
      atest --host analyze_bcpf_test
Change-Id: I824fd9d527d246dfe56725c073d89a0b0e72d25a
2022-03-30 12:18:56 +01:00
Cole Faust
d9932ad053 Always build mk2rbc/rbcrun
Currently, mk2rbc/rbcrun aren't built in certain phases
of the build, which causes the Starlark product configuration
to fail when using get_build_var if a regular build hasn't
been run before.

Bug: 226677850
Test: Manually
Change-Id: If85608eef63be25fddc4d916c82247c13eb0195f
2022-03-29 14:55:17 -07:00
Jooyung Han
1df464a90b Revert "Add a tool to list contents of .img file"
Revert submission 2041990-microdroid-contents

Reason for revert: b/227144320 (broken build)
Reverted Changes:
Ie2f471d39:Add the golden list of microdroid contents
I7c4fca184:Add a tool to list contents of .img file

Change-Id: If2b5d6cb1325cb17ef2bdc4f13ded003ab2dd6d7
2022-03-28 09:44:33 +00:00
Jooyung Han
b3a3a2893b Add a tool to list contents of .img file
list_image <img> lists the contents of the file.

For example:
  $ m microdroid
  $ list_image <path to microdroid.img>
  ./
  ./apex/
  ./bin
  ..

Bug: 195425111
Bug: 225121718
Test: list_image .img
Change-Id: I7c4fca184751ba20066fd25fa4c366a955dbabf6
2022-03-25 02:44:36 +09:00
Paul Duffin
846beb7f54 Make bootclasspath_fragment hidden API package checks exhaustive
Previously, the bootclasspath_fragment's hidden_api.split_packages and
hidden_api.package_prefixes properties did not specify an exhaustive
set of packages that were provided by the fragment. They excluded
packages which were either not split or which could not be used as a
package prefix because it would match sub-packages provided by other
bootclasspath modules.

This change adds the hidden_api.single_packages list to specify those
additional packages and then uses that information to verify that any
bootclasspath_fragment that specifies at least one of split_packages,
single_packages or package_prefixes properties only contains classes
from a package that matches one of those properties. That will
prevent a module from accidentally including unexpected classes, such
as might happen when statically including a common utility library.

It also adds coverage specific versions of the properties as additional
packages are added to the art-bootclasspath-fragment when building
coverage builds.

Bug: 194063708
Test: atest signature_patterns_test
      m out/soong/hiddenapi/hiddenapi-flags.csv
      m EMMA_INSTRUMENT=true EMMA_INSTRUMENT_FRAMEWORK=true out/soong/hiddenapi/hiddenapi-flags.csv
      # Breaks without corresponding change to add android.system to
      # the art-bootclasspath-fragment.
      /usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_patterns*.py
      pyformat -s 4 --force_quote_type single -i scripts/hiddenapi/signature_patterns*.py
Change-Id: Iddf6c59cd4dc8c36dde7943a9840ccef5794b320
2022-03-18 10:54:51 +00:00
Paul Duffin
1024f1d72b Cleanup signature_patterns*.py
Fix issues reported by pylint, Intellij Python checks and also try and
adhere to the Google Python Style Guide.

Bug: 194063708
Test: atest signature_patterns_test
      m out/soong/hiddenapi/hiddenapi-flags.csv
      /usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_patterns*.py
      pyformat -s 4 --force_quote_type single -i scripts/hiddenapi/signature_patterns*.py
Change-Id: I64d64e9cb269f58d65f4e10ec2f0e874154919e6
2022-03-17 10:36:29 +00:00
Paul Duffin
92532e72a1 Allow traversal over the trie structure
Previously, there was no way to traverse the trie structure and no way
to identify specific nodes in the trie. That made it impossible to
analyze the trie structure resulting from loading a set of flags. This
change adds type and selector properties to nodes as well as access to
the child nodes of a node to allow for the structure to be analyzed.

Bug: 202154151
Test: m out/soong/hiddenapi/hiddenapi-flags.csv
      atest --host signature_trie_test verify_overlaps_test
      pyformat -s 4 --force_quote_type double -i scripts/hiddenapi/signature_trie*
      /usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_trie*
Change-Id: Ia4714dbf59f6fd143aa3bf3ad1a59cd073d2175b
2022-03-15 15:33:51 +00:00
Paul Duffin
ea93542e90 Switch signature_to_elements to use tuple
Previously, it used a structured string of format <type>:<value>. A
tuple is more efficient and less prone to edge cases, such as when the
value is a field signature which itself contains a ":".

Bug: 202154151
Test: m out/soong/hiddenapi/hiddenapi-flags.csv
      atest --host signature_trie_test verify_overlaps_test
      pyformat -s 4 --force_quote_type double -i scripts/hiddenapi/signature_trie*
      /usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_trie*
Change-Id: I80abaff243d98aad325df1b5a655bba7f9d04e2c
2022-03-15 15:16:04 +00:00
Paul Duffin
19255f1d91 Make signature_to_elements stricter and more consistent
Previously, signature_to_elements would return a string array where
non-wildcard strings were of the form <type>:<value> but wildcard
strings were just * or **. This change makes it handle wildcards
consistently with the other element types and adds some extra
checking for edge cases.

Bug: 202154151
Test: m out/soong/hiddenapi/hiddenapi-flags.csv
      atest --host signature_trie_test verify_overlaps_test
      pyformat -s 4 --force_quote_type double -i scripts/hiddenapi/signature_trie*
      /usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_trie*
Change-Id: I5bfaf5e75c7da54b6241f68e03231939c9d65501
2022-03-15 15:16:04 +00:00
Paul Duffin
1f8a6b2f27 Extract Node superclass of Leaf and InteriorNode
Adds a Node superclass to declare and documented the methods that need
to be implemented by Leaf and InteriorNode. Also uses @dataclasses to
make it easy to declare and document the properties of each class.

This refactoring is in preparation for future changes that will need to
add common behavior to both Leaf and InteriorNode so having a
superclass will make that easier.

Bug: 202154151
Test: m out/soong/hiddenapi/hiddenapi-flags.csv
      atest --host signature_trie_test verify_overlaps_test
      pyformat -s 4 --force_quote_type double -i scripts/hiddenapi/signature_trie*
      /usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_trie*
Change-Id: If660855f6b1f45a8ea5b90b3cc0236da9e07d090
2022-03-09 15:43:57 +00:00
Paul Duffin
b5cd522053 Extract signature_trie.py from verify_overlaps.py
Makes the efficient pattern matching of hidden API flags that is used
by verify_overlaps.py available for use in other scripts.

As part of the move this cleans up the python to use consistent quotes,
and fix pylint issues.

Bug: 202154151
Test: m out/soong/hiddenapi/hiddenapi-flags.csv
      atest --host signature_trie_test verify_overlaps_test
      pyformat -s 4 --force_quote_type double -i scripts/hiddenapi/signature_trie*
      /usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_trie*
Change-Id: I758ca70bb5b7e6806f14b72fd04f821a069f188f

Change-Id: I73fdb7e02127a8c0171a285221d9e6024310953d
2022-03-09 14:04:23 +00:00
Anton Hansson
b850f84cb9 Generate soong.variables for ndk build
Update the ndk build script to have Make generate the soong.variables
via the config step, using a new "ndk" product.

Bug: 174315599
Test: build/soong/scripts/build-ndk-prebuilts.sh
Change-Id: I0f6fbf523cf9209ea17b7752dd7470012e0baf8c
2022-02-23 12:16:48 +00:00
Anton Hansson
3a3f169b56 Add support for sdk extensions in prebuilt_apis
This makes it possible to pass an extensions_dir containing finalized
module APIs to prebuilt_apis. The extension versions are compared to the
api level versions to figure out what the "latest" finalized API is for
each module. This is done using the base_sdk_extension_version, such
that any extension higher than than base_sdk_extension_version is
assumed to be finalized after any of the existing api level versions.

Bug: 220086085
Test: prebuilt_apis_test.go
Test: existing module in prebuilts/sdk
Change-Id: Ib792f84202d436f594ba5e8716c6a187f9cd60dc
2022-02-21 16:43:43 +00:00
Thiébaud Weksteen
8fae31138a Update rustfmt.toml to 2021 edition
Bug: 210475268
Test: $ cd system/security; find . -name \*.rs -exec rustfmt --check {} \;
Change-Id: I49044e19a17d3806570d04c3215ccd3554ce7903
2022-02-10 16:55:01 +11:00
Cole Faust
8d47c48082 Use soong's finder to find included makefiles in mk2rbc
Having soong generate a list of makefiles for mk2rbc
to look through is much faster than having mk2rbc search
itself. Profiling the readLinesFromeFile() function that
reads the list of makefiles shows it takes on the order
of 200 microseconds, much faster than the ~5 seconds
it takes for mk2rbc to search the tree itself.

This CL also allows include statements that are prefixed
with a variable. The concern with this was that there
would be a lot of load statemnts emitted for generic
include statements, causing the generated code to look
ugly, and converting and loading all those files could
cause performance issues. On the performance issues
front, there's already a check that it doesn't result
in over 150 potentially included files. We can lower
that number if necessary, but it's probably good for now.
On the generated code front, while it's true that it's
ugly, it's better to have working but ugly generated
code than refusing to generate anything working at all.

To ensure the soong finder step isn't slowed down due
to having to find a bunch of new makefiles, I profiled
the combination of newSourceFinder and FindSources in main.go:

Baseline incremental:
338.011634ms
340.853335ms
348.541762ms
333.229644ms
349.124824ms

Baseline clean:
1.003836419s
1.006203912s
996.193648ms
1.031005604s
1.03691152s

Modified incremental:
349.029285ms
349.264496ms
351.774948ms
337.63187ms
359.425306ms

Modified clean:
1.028238704s
1.053103506s
1.032757506s
1.016631201s
1.04288108s

So we can see the times are barely affected by this change.

Fixes: 213508006
Test: go test
Change-Id: Iab18bfb127ba3b7e63f2c01f69064805a8398764
2022-02-07 11:41:28 -08:00
Gurpreet Singh
75d65f360c Add testOnly attribute to AndroidManifest file of apex_test
If the build file contains the apex_test module, add the
testOnly attribute to the application element of the
corresponding AndroidManifest file and set its value to true.
If the testOnly attribute is already present and has value
false, then do nothing.

Tests added in manifest_fixer_test.py to check if the updated
AndroidManifest file has the testOnly attribute set to true or not.

Bug: 213310150
Test: atest --host manifest_fixer_test
Test: m nothing
Test: manually checked the AndroidManifest file generated
Change-Id: I36247dbe0261c342d451a4422c314fd8fe0c2369
2022-02-01 12:03:48 +00:00
satayev
83bda94c12 Remove OWNERS for deleted build-mainline-modules.sh
Bug: 180394948
Test: presubmit
Change-Id: I76a157f9292d28155a7f62acc93cf7fa7fb39cb9
2021-12-15 18:45:22 +00:00
satayev
035fba700a Remove unused build-mainline-modules.sh
This script is no longer used in any CI or in any other scripts.

Bug: 180394948
Test: presubmit
Change-Id: Ia1b9f699ae3b9263b6e1d6c28ebec449ee31ef84
2021-12-15 16:21:02 +00:00
sophiez
bc82ba55c5 Java used-by API list generation
Update the java used by script to fix the xml format when dexdeps processing resource only APKs

Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only
Change-Id: I61a58d7c739e74a571e97166a7c9b4091e6f8da0
2021-12-14 14:52:58 -08:00
sophiez
65a981569f Java used-by API list generation
Wrap the dexdeps output content with "<externals></externals>" to
simplify backend xml parsing.

Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only
Change-Id: I1457eb9558763e50911dc9c689b9bc3fb833a52c
2021-12-10 13:43:10 -08:00
Nikita Iashchenko
84bec80023 Merge "Add jdk.internal.ref to package_allowed_list.txt (OpenJDK 11)" 2021-12-08 13:40:44 +00:00
Cole Faust
864028a71b Support passing input variables to the product configuration
Since rblf_env / rblf_cli are not typed properly, accept
input variables via a file so that they can be converted
with the correct types.

Bug: 201700692
Test: go test
Change-Id: I9b56067cfe396d1bcd8d62c353ff222dd61a6c9f
2021-12-02 16:19:19 -08:00
Cole Faust
c903f00e3f Remove python 2 support from manifest_utils
Bug: 203436762
Test: Presubmits
Change-Id: If93033a428aa1aa0dd2db973666d4790af8f0912
2021-11-30 12:24:02 -08:00