Commit graph

113 commits

Author SHA1 Message Date
Spandan Das
5ae65ee6e0 Allow dexpreopt of source sdklib in prebuilt apex builds
aosp/2984037 disabled dexperopt of the source apex system server jars
when prebuilts are selected. This was done to prevent duplicate
installation and dex2oat deps.

AOSP art has some additional variants like com.android.art.debug. In
source builds, this apex should contain service-art.jar and
service-art.jar.prof (a dexpeopt artifact). We have a test to check this
(`art_apex_test.py`). If we disable dexpreopt of source sdlib when
prebuilts are selected, this test needs to be disabled. This is the
behavior at ToT.

This CL changes the behavior to enable running this test even when
com.google.android.art prebuilt is active in a specific release
configuraiton. To prevent collisions that prompted aosp/2984037, this CL
special-cases the installation and dex2oat rules instead of disabling
dexpreopt of the source sdklib altogether.

b/331665856 tracks the principled solution to prevent duplicate
dexpreopt rules.

Implementation:
Add a new copyApexSystemServerJarDex arg to GenerateDexpreoptRule API.
If true, the dexjar file will be copied to
out/soong/system_server_jars/. For soong modules, the value of this will
be the inverse of disableSourceApexVariant. Since none of the apex
system server jars are in make, this will be a noop in dexpreopt_gen

Bug: 331665856
Test: modified trunk_staging.scl locally to select
art.google.contributions.prebuilt; mmma art; (with the sibling CL in
topic)

Change-Id: Idb59e424f83d126cdc8b1671dde358745979bd8d
2024-04-17 23:15:20 +00:00
Jiakai Zhang
7d292228c1 Determine GC type based on BUILT_KERNEL_VERSION_FILE.
How it works:
1. build/make/core/Makefile generates a txt file with the kernel
   version, which is taken from an explicit BOARD_KERNEL_VERSION value,
   or extracted from the kernel image on the source tree, or extracted
   from the kernel image extracted from the prebuilt boot.img.
   The file is saved at
   $ANDROID_PRODUCT_OUT/obj/PACKAGING/check_vintf_all_intermediates/kernel_version.txt.
2. If PRODUCT_ENABLE_UFFD_GC is "default", meaning the GC type needs to
   be determined by the kernel version, build/make/core/Makefile copies
   kernel_version.txt to
   out/soong/dexpreopt/kernel_version_for_uffd_gc.txt.
3. build/soong/dexpreopt/config.go writes the the UFFD GC flag to
   out/soong/dexpreopt/uffd_gc_flag.txt. The flag is determined by an
   explicit PRODUCT_ENABLE_UFFD_GC value or by contruct_uffd_gc_flag.py,
   which reads kernel_version_for_uffd_gc.txt and determines the flag
   accordingly.
4. dex2oat takes the UFFD GC flag from uffd_gc_flag.txt.
5. post_process_props.py mangles ro.dalvik.vm.enable_uffd_gc based on
   the same logic.

Bug: 321751629
Bug: 319554951
Bug: 318763448
Bug: 319648491
Test: m --no-skip-soong-tests nothing
Test: atest uffd_gc_utils_test
Test: Build with `OVERRIDE_ENABLE_UFFD_GC=default m` for device with no
  UFFD support -
  1. Check the existence of `-Xgc:CMC` in
     out/soong/dexpreopt_arm64/dex_bootjars/android/system/framework/arm64/boot.invocation
     (dex2oat invocation for a boot image)
  2. Check the existence of `-Xgc:CMC` in
     out/soong/.intermediates/packages/apps/Settings/Settings/android_common/dexpreopt/Settings/oat/arm64/package.invocation
     (dex2oat invocation for an app defined in .bp)
  3. Check the existence of `-Xgc:CMC` in
     $ANDROID_PRODUCT_OUT/obj/APPS/Dialer_intermediates/oat/arm64/package.invocation
     (dex2oat invocation for an app defined in .mk)
  4. Check the value of ro.dalvik.vm.enable_uffd_gc in
     $ANDROID_PRODUCT_OUT/product/etc/build.prop
Test: Build with `OVERRIDE_ENABLE_UFFD_GC=default m` for device with
  UFFD support, and do the steps above.
Test: Build with `OVERRIDE_ENABLE_UFFD_GC=true m`, and do the steps
  above.
Test: Build with `OVERRIDE_ENABLE_UFFD_GC=false m`, and do the steps
  above.

Change-Id: I035ad32233e49e2a30ce11f6c7c318a648511ef8
2024-02-02 09:23:37 +08:00
Spandan Das
b2fd4ff43f Revert "Ensure sscp jars get copied to $OUT/soong/system_server_dexjars"
This reverts commit 0b7089f134.

Reason for revert: Breaks auto builds https://groups.google.com/a/google.com/g/android-build-police/c/DfXeyxkyfYk/m/4SvZa-jkAAAJ

Change-Id: Ia6ebdf64d03284d2318978c35cb87fd4f0b79531
2024-01-25 04:25:38 +00:00
Spandan Das
0b7089f134 Ensure sscp jars get copied to $OUT/soong/system_server_dexjars
(This was missed in aosp/2876756, which copied only the bcp jars)

This hardcoded location is used by dex2oat to compile the dexpreopt
artifacts. The copy rules are currently generated by java_(sdk)_library
for source builds, and their prebuilt counterparts in prebuilt builds.

After this change, the copy rule will be bifurcated between source and
prebuilt builds
1. For source builds, it will come from java_(sdk)_library
2. For prebuilt builds, it will come from the top-level prebuilt apex.
   Since there can be multiple prebuilt apexes in trunk stable,
   HideFromMake will be used to determine which deapexed jar to copy.

The bifurfaction is expected to be temporary. It is needed for now since
the `apex_contributions` which will be used for source vs prebuilt
selection have not been populated completely.

Test: Added a unit test
Test: Presubmits
Test: git_master-art-host:
art-gtest (https://android-build.corp.google.com/builds/abtd/run/L40800030001459791)
Test: git_main:art_standalone_tests
(https://android-build.corp.google.com/builds/abtd/run/L09000030001463855)

Bug: 308790457

Change-Id: I3105d3b3a7e5c41cb601d07806f4ea483a61b50a
2024-01-24 05:14:40 +00:00
Jiakai Zhang
db93553b02 Remove PreoptExtractedApk.
This was added by r.android.com/513843 to force dexpreopt some GMS core
modules even if dexpreopt is disabled, to avoid some memory usage
regression. We no longer need it because dexpreopt is never disabled on
production builds.

Bug: 313505540
Test: m
Change-Id: I605b7569c17ee715cd4df167768e25aaf51bcd37
2023-11-28 13:40:58 +00:00
Jiakai Zhang
2398442ec0 Change OnlyPreoptBootImageAndSystemServer to OnlyPreoptArtBootImage.
OnlyPreoptBootImageAndSystemServer was for making the eng build faster.
This change makes it even faster.

Bug: 309011617
Test: Presubmit
Change-Id: If9001e99593b57deafa7b56f0ca6b3d4d80f56d3
2023-11-09 16:50:58 +00:00
Treehugger Robot
da169a1ee3 Merge "Fix FirstUniqueStrings after conversion to generics" 2023-06-29 18:46:20 +00:00
Jiakai Zhang
51b2a8b5eb Use per-app package list to avoid unnecessary dexpreopt.
Starting from aosp/2594905, dexpreopt depends on
`$PRODUCT_OUT/product_packages.txt`. When PRODUCT_PACKAGES changes,
dexpreopt has to rerun for all apps. This is not ideal.

After this change, dexpreopt uses a per-app product_packages.txt that is
filtered by the app's dependencies, and it uses `rsync --checksum` to
prevent the file's mtime from being changed if the contents don't change.
This avoids unnecessary dexpreopt reruns.

Bug: 288218403
Test: m
Test: Change PRODUCT_PACKAGES and see no dexpreopt reruns.
Change-Id: I5788a9ee987dfd0abfd7d91cbcef748452290004
2023-06-28 17:59:56 +01:00
Colin Cross
48016d5a2f Fix FirstUniqueStrings after conversion to generics
The conversion of FirstUniqueStrings to be implemented on top of the
generic firstUnique accidentally used a different threshold to switch
from the list implementation to the map implementation.  Modify the
threshold of firstUnique to match the old value from FirstUniqueStrings
now that it doesn't have the reflection overhead.  While we're at it,
also make firstUnique make a copy of the list, and make FirstUniqueStrings
a pure wrapper around firstUnique.

Test: BenchmarkFirstUniqueStrings
Change-Id: Icc2febea663142c508ff2e4be65a8a68121631d5
2023-06-27 11:07:55 -07:00
Jiakai Zhang
a449678996 Move CLC construction to Ninja phase.
Before this change, dexpreopt was often broken with optional libraries.
This was because the CLC construction was done in Soong at an early
stage, where we don't have sufficient information to determine whether
an optional library is installed or not.

For example, the "Settings" package uses an optional library called
"androidx.window.extensions". On some devices, the library is installed,
but on some other devices, it's not. Soong always adds the library to
the CLC, meaning the CLC is wrong for devices which don't have the
library. This change fixes the problem. See the tests below.

After this change, the CLC construction is done by a Python script
invoked at a very late stage. It uses product_packages.txt, which is
generated by Make, to determine whether an optional library is
installed or not, and filter out libraries that are not installed.

Note that optional libraries are still added as dependencies by Soong.
This is because dependencies have to be added at an early stage. This
means what dex2oat eventually uses will be a subset of the dependencies,
which is fine.

Bug: 282877248
Test: m
Test: atest construct_context_test
Test: -
  1. lunch aosp_cf_x86_64_phone-userdebug && m
  2. Check the .invocation file of the "Settings" package (defined in
     .bp file)
  3. See androidx.window.extensions
Test: -
  1. lunch aosp_redfin-userdebug && m
  2. Check the .invocation file of the "Settings" package (defined in
     .bp file)
  3. Don't see androidx.window.extensions
Test: Check the .invocation file of the "Dialer" package (defined in
  .mk file)
Test: -
  1. Build a Pixel 5 system image and flash it to a Pixel 5 device.
  2. adb shell pm art dump
  3. See "reason=prebuilt" instead of "reason=vdex".
     (https://diff.googleplex.com/#key=fB6Ls9q2QGSN, before: left,
     after: right)

Change-Id: Ia112bd7c2328373e68db6bffb74bf34030f683d8
2023-05-30 15:46:38 +01:00
Jiakai Zhang
cf61e3c591 Revert^2 "Generate app profiles even if dexpreopt is disabled."
Revert submission 2580631-revert-2574032-XXTWCJDTDQ

Reason for revert: Fixed build breakages

Reverted changes: /q/submissionid:2580631-revert-2574032-XXTWCJDTDQ

Bug: 280440941
Test: lunch aosp_cf_riscv64_minidroid-userdebug && m UNSAFE_DISABLE_HIDDENAPI_FLAGS=true dist
Test: Disable dex2oat on host (to simulate macOS) and build
Change-Id: I6090b4b74cedb6d129fcbeef58d075c8ccdcc4e2
2023-05-08 21:29:23 +01:00
Qiao Yang
3d08c388b9 Revert "Generate app profiles even if dexpreopt is disabled."
Revert submission 2574032

Reason for revert: DroidMonitor-triggered revert due to breakage <https://android-build.googleplex.com/builds/quarterdeck?branch=aosp-master&target=aosp_cf_riscv64_minidroid-userdebug&lkgb=10069333&lkbb=10075041&fkbb=10071083>, bug <b/280902279>

Reverted changes: /q/submissionid:2574032

Change-Id: Ia9d05f3b7439604eb4a4b4100f46879fe11f5820
BUG: <280902279>
2023-05-05 15:03:24 +00:00
Jiakai Zhang
7b845e808f Generate app profiles even if dexpreopt is disabled.
Bug: 280440941
Test: -
  1. Patch ag/22302622 to disable dexpreopt.
  2. lunch aosp_cf_x86_64_phone-userdebug && m
  3. See app profiles still generated.
Test: -
  1. Patch ag/20592051 to enable profile for service-art.
  2. banchan com.android.art x86_64 && m
  3. See the profile for service-art generated.
Change-Id: I4e721b475b84a2f667bbccc030a8947078f26bb0
2023-05-04 13:53:22 +01:00
Jiakai Zhang
3317ce725d Install system server jar profiles into the APEX.
After this change, if profile-guided compilation is enabled for an APEX
system server jar, the profile will be installed next to the jar with
the ".prof" suffix, ("javalib/<name>.jar.prof"). This file will later be
used by odrefresh and dexpreopt from prebuilt APEX.

Bug: 241823638
Test: m nothing
Test: -
  1. Patch ag/20581649 PS2.
  2. banchan com.android.btservices x86_64 && m
  3. Check that "javalib/service-bluetooth.jar.prof" exists in the APEX.
Change-Id: Ibcfc6257dade92bd40a6d4b7368148717d0863b9
2023-02-08 19:25:37 +08:00
Jiakai Zhang
616be0695e Pass "-Xgc:CMC" to dex2oat when userfaultfd GC is enabled.
This change is a no-op change. It doesn't enable userfaultfd GC by
default. ENABLE_UFFD_GC=true can be passed to the build system to enable
userfaultfd GC for testing purposes.

Bug: 242553398
Test: -
  1. lunch aosp_oriole-userdebug
  2. ENABLE_UFFD_GC=true m
  3. See "--runtime-arg -Xgc:CMC" in .invocation files.
Change-Id: I789b49a71b9604fd41bf1ef77d0ac5bac4cbdf25
2022-11-16 12:04:53 +00:00
liulvping
76d56ee9f8 support system server jars install in system_ext
Bug: 245383870
Bug: 155630745
Test:m nothing
Test:-
1.Add a jar installed in the system_ext partition toSYSTEMSERVERJARS
(e.g.b/245383870 PRODUCT_SYSTEM_SERVER_JARS_EXTRA += system_ext:oem-services)
2.Build a ROM
3.See the odex and vdex files generated in $ANDROID_PRODUCT_OUT/system_ext/framework/oat/
4.Flash the ROM to a device
5.Check the maps information of the system_server , the newly added odex has the r-xp flags
6.Run `adb shell am restart;adb root;adb shell setprop dalvik.vm.extra-opts
-verbose:deopt,heap,jit,oat,image,dex,verifier;`
7.Check ClassLoaderContext classpath size matching information
8.Run `atest art_standalone_odrefresh_tests`
`atest art_standalone_dexpreopt_tests`
`atest CtsSdkExtensionsTestCases`

Change-Id: Ib1083ee130b5fef9c23d56523868fa1e29be879a
Signed-off-by: liulvping <liulvping@xiaomi.com>
2022-09-16 20:36:42 +08:00
Ulya Trafimovich
6fad1d0dba Dexpreopt: use "speed-profile" if a system server jar has a profile.
The motivation is to reduce file size and in-process memory footprint
for "services" (which is currently the only system server jar that has a
profile). For other system server jars that have no profile keep using
"speed", as enabling "speed-profile" for them would completely disable
AOT-compiled code.

Compare .odex file size:
  - before: 44360 services.odex
  - after:  24968 services.odex

Bug: b/237399630
Test: lunch aosp_sargo-userdebug && m
Change-Id: I844b9607c496d3d6e7048dc6bb8cd958ecbaa441
2022-07-07 12:12:50 +01:00
Jiakai Zhang
389a647320 Dexpreopt standalone system server jars.
Standalone system server jars are dynamically loaded by system server
using a `PathClassLoader` whose parent is `SYSTEMSERVERCLASSPATH`. They
are listed in `PRODUCT_STANDALONE_SYSTEM_SERVER_JARS` and
`PRODUCT_APEX_STANDALONE_SYSTEM_SERVER_JARS` in Makefile. We need to
dexpreopt them to achieve better performance.

Bug: 203198541
Test: m nothing
Test: -
  1. Add a standalone system server jar (e.g., by patching
     aosp/1906158)
  2. Build a system image.
  3. See the odex and vdex files generated in
     $ANDROID_PRODUCT_OUT/system/framework/oat/
  4. Flash the image to a device.
  5. Run `atest art_standalone_dexpreopt_tests`.
Change-Id: I358a62d34989c5c8eba12e18fe6167e0b72ff69d
2021-12-20 15:07:26 +00:00
Jiakai Zhang
0a0a2fbea9 Add a build-time check for dexpreopting system server jars.
This change adds a Soong module that stores the paths to the compilation
artifacts of system server jars in a variable, which will then be
consumed by Make to check if the artifacts are installed. When the check
fails, it means that dexpreopting is not working for some system server
jars and needs to be fixed.

Bug: 201371822
Test: m nothing
Test: manual -
  1. Add "service-permission" to DEXPREOPT_DISABLED_MODULES (https://cs.android.com/android/platform/superproject/+/master:build/make/core/product_config.mk?q=DEXPREOPT_DISABLED_MODULES)
  2. m nothing
  3. See the error:
    Missing compilation artifacts. Dexpreopting is not working for some system server jars
    Offending entries:
    system/framework/oat/x86_64/apex@com.android.permission@javalib@service-permission.jar@classes.odex
    system/framework/oat/x86_64/apex@com.android.permission@javalib@service-permission.jar@classes.vdex
Change-Id: I4816f19668f1dae180a34adafdbfa448c97aa0db
2021-10-29 11:55:20 +00:00
Ulya Trafimovich
29e95d94bf Move copy rule for system server jars from dexpreopt_gen to makefiles.
This is necessary in order to expose the copy rule to Ninja. Otherwise
Ninja has a build rule that depends on the copied jar (namely, dexpreopt
commands for all subsequent system server jars that have the current one
in class loader context), but no rule that creates it, so Ninja fails
with a "no rule to make ..." error.

The change only affects system server ed in Android.mk

Previously the problem existed, but was hidden by the fact that the only
system server jar defined in Android.mk was the last one on the list, so
no other jar depended on it. Now that updatable apex jars are also
dexpreopted (https://r.android.com/1828115) the problem was uncovered.

Bug: 200297762
Test: lunch bertha_x86_64-userdebug && m
Change-Id: Idbf2eacdf7ae89847aa9d755ef688eda3720ce89
2021-09-21 14:08:32 +01:00
Jiakai Zhang
519c5c82e5 Revert^2 "Preopt APEX system server jars."
This reverts commit 92346c4832.

Reason for revert: Fixed build error.

The build error is fixed by ag/15841934. This CL remains unchanged. This
CL will be submitted AFTER ag/15841934 is submitted.

Bug: 200024131
Test: 1. Patch this CL and ag/15841934 into internal master.
  2. sudo vendor/google/build/build_test.bash

Change-Id: I5f2b8357846fc7dda56e25ebe6ffb095e8047ec8
2021-09-16 06:25:26 +00:00
Adrian Roos
92346c4832 Revert "Preopt APEX system server jars."
This reverts commit ca9bc98e0c.

Reason for revert: breaks build
Bug: 200024131

Change-Id: Ide07b4c4d267370ae31107b1598b2f878c701282
2021-09-15 14:11:07 +00:00
Jiakai Zhang
ca9bc98e0c Preopt APEX system server jars.
The path to the artifacts will in the form of
/system/framework/oat/<arch>/<encoded-jar-path>@classes.{odex,vdex,art},
where <encoded-jar-path> is the path to the jar file with "/" replaced
by "@". For example,
/system/framework/oat/x86_64/apex@com.android.art@javalib@service-art.jar@classes.odex

There will be a follow-up CL to update ART runtime to recognize
artifacts in that path.

Test: m com.android.art
Bug: 194150908
Change-Id: Ic89fd63c4b1cd565684cead83fc91dae3bc97a4c
2021-09-15 09:01:07 +00:00
Ryan Savitski
aa3c355820 eng builds: dexpreopt: stop explicitly removing minidebuginfo
This special-casing is actively unhelpful nowadays, as it breaks
callstack unwinding and symbolisation by tools that work using the
on-device debug info (e.g. perfetto's profilers). It's also highly
surprising that eng builds have less debug info than userdebug/user.

As far as I can tell, the history behind this case is (with some
internal-only references as the context is in the review comments): the
minidebuginfo was originally rolled out in device-specific makefiles
(e.g. ag/2433691). Eng builds were excluded, presumably due to time/size
concerns for rebuild+resync workflow, as well as the assumption that
engineers using eng builds already have the necessary debug info
off-device. Later, as some more makefile variables were introduced, the
conversation in ag/990059 shows that the overheads were no longer a
concern, and the last patchset generalised the option to include eng
builds. However at a later point, during device-specific makefile
cleanup, the eng-exclusion got restored: ag/2455645. This then ended up
in the makefile->soong transition, leaving us with the current special
casing that is undesireable.

Checking things with an internal master build on my workstation:
dexpreopting system services is 10s regardless of this minidebuginfo
flag, and the resulting odex size is 37MB vs 38MB. Overall, I only see 4
dexpreopt invocations as part of a local build, so the effect of this
patch should be negligible.

Bug: 194835631
Change-Id: I80de75a84259c056cdae0696fe611afe7e0162f5
2021-08-04 19:34:55 +01:00
satayev
492b17d533 Rename UpdatableSystemServerJars to ApexSystemServerJars.
This is consistent with naming for boot jars, e.g. ApexBootJars.

Bug: 191127295
Test: presubmit
Change-Id: I6e8828d55ac86b7f5260ed7f8d4eca1c0b3dced8
2021-07-28 14:04:49 +01:00
Jeongik Cha
4dda75e73e Add usesTargetFiles option in dexpreopt_gen
For running dex2oat on the target_files, the paths should be use the
device install path instead of the path starting with $(OUT).
So add usesTargetFiles option and basePath option which indicates
extracted path. With those options, the path is replaced with
$(basePath)/$(device path)

And also, add DexPreoptImageDeviceLocations in the config which refers
to the boot image path(without arch) on the device. Because
DexPreoptImage related device path was missing.

Bug: 158843648
Test: dexpreopt_gen -usesTargetFiles -basePath (extract path) and then
check if paths in the generated shell script are based on on-device
path.

Change-Id: I9667fadbf3b7c6f770e0d1bcbee5d67c1ecd8a3d
2021-05-15 00:29:49 +09:00
Treehugger Robot
564fce4578 Merge "Rename fields in dexpreopt config" 2021-05-08 01:47:39 +00:00
Jeongik Cha
a596909342 Rename fields in dexpreopt config
Add 'host' into the name of fields regarding path on the host side to
distinguish between paths on the device(which will be added in the
following commit), and paths on the host.

Bug: 158843648
Test: build and flash, and then
  adb wait-for-device \
    && adb -s $S root \
    && adb -s $S logcat \
    | grep -E 'ClassLoaderContext [a-z ]+ mismatch' -C 1
Change-Id: Ib2645ed51591ba2f4b726c115b401ad2bd6675da
2021-05-07 21:40:23 +09:00
satayev
9a6f87ebd6 Make SystemServerJars ConfiguredJarList.
Consistent with plumbing of boot jars. SystemServerJars only support /system/framework/ jars.

Bug: 180105615, 155630745
Test: m && launch_cvd
Change-Id: I58b005b7c4103c8e250090e995b1d9b2f9ed4a76
2021-05-05 14:24:02 +01:00
Vladimir Marko
982e384a63 Fix profileCommand to remove the profile first.
This fixes a discrepancy between clean and incremental
builds exposed by
    https://android-review.googlesource.com/1671710 .

Test: m
Bug: 148067697
Change-Id: I0de078045075417a4123f7c3821c723f1e8e6b68
2021-04-29 09:10:30 +01:00
Vladimir Marko
230bd421a8 profman: Clean up output profile type options.
Test: m  # Check output boot profiles with hexdump.
Bug: 148067697
Change-Id: Ic20f415ad4eee1fd6396f320b757420884764d2c
2021-04-23 15:19:16 +01:00
Jeongik Cha
33a3a8182f Made ManifestPath OptionalPath instead of Path
because it is actually an optional value.

Bug: 158843648
Test: m nothing
Change-Id: If323eacf6a7aa83dcf5ce4825ad8cabeeaa97ac9
2021-04-15 09:12:49 +09:00
Ulya Trafimovich
dd622951a4 Fix AAPT path in dexpreopt.
Dexpreopt uses AAPT to parse `targetSdkVersion` from the manifest, so it
the generated dexpreopt.sh script that calls AAPT must depend on it,
otherwise AAPT might not be built by the time it is used. Tool
dependencies are handled via the global Soong config and the
DEXPREOPT_GEN_DEPS variable that Soong generates for Make. This config
always uses Soong tool paths, like out/soong/host/linux-x86/bin/aapt
rather than out/host/linux-x86/bin/aapt.

This CL fixes a mistake in dexpreopt rule that used context-dependent
AAPT path (so, when called from Make, the dependency was on Soong tool,
but a Make tool was actually used, so it failed sometimes).

Bug: 132357300
Test: lunch aosp_cf_x86_64_phone-userdebug && m
Change-Id: I1f0ab4afac98e6239f324e7f3571d670fd7a36cd
2021-03-25 12:52:49 +00:00
Ulya Trafimovich
bd6b076f80 Allow modules that don't run verify_uses_libraries to have nonempty CLC.
There are modules that would have passed verify_uses_library check, but
don't run it for some reason (the check gets enabled either with an
explicit setting, or if the module has nonempty <ues-library> lists in
the build properties). Previously all such modules were assumed to have
empty CLC, which is not always true. In particular, compatibility
libraries are ignored, which affected e.g. Calendar and messaging apps.

This CL gives such apps a chance to have correct CLC. The goal for the
future is to enforce verify_library_check by default.

Bug: 132357300
Test: lunch aosp_cf_x86_64_phone-userdebug && m
Change-Id: Iea3be0fc9d7775c52950848b5a3fd3b7fcd36c53
2021-03-22 10:49:31 +00:00
Ulya Trafimovich
0aba252e15 Reimplement verify_uses_libraries.sh in manifest_check.py.
Previously there were two different scripts that did similar things:
1) build/soong/scripts/manifest_check.py
2) build/make/core/verify_uses_libraries.sh

Both scripts extracted <uses-library> tags and `targetSdkVersion` from
the manifests of Java modules, but 1) worked for XML manifests, and 2)
worked for APKs. This CL reimplements the functionality from 2) in 1),
so that one script can handle both XML manifests and APKs.

Bug: 132357300

Test: lunch cf_x86_64_phone-userdebug && m && launch_cvd \
      adb wait-for-device && adb root && adb logcat \
      | grep -E 'ClassLoaderContext [a-z ]+ mismatch'
      # empty grep output, no errors

Change-Id: Id1b66e4f3f30f307dba70cb111c7571762cb546a
2021-03-15 13:50:49 +00:00
Nicolas Geoffray
b4f9d5c837 Merge "Stop using '&' for class loader context." 2021-03-08 11:16:09 +00:00
Nicolas Geoffray
863f3b9fb4 Stop using '&' for class loader context.
Test: m
Bug: 132357300
Change-Id: I01a5008d5b84820f793c1c23fe2e4c4687cf4acf
2021-03-05 18:14:10 +00:00
Ulya Trafimovich
4a13acb07d Use compiler filter "verify" for dexpreopt w/o class loader context.
In cases when class loader context cannot be computed at build time and
verify_uses_libraries check is relaxed (RELAX_USES_LIBRARY_CHECK=true)
dexpreopt uses a special compiler filter that suppresses AOT-compilation
to native code and applies only those optimizations that do not require
class loader context. Previously the "extract" filter was used. Now ART
supports using "verify" filter in this configuration.

Bug: 132357300
Test: lunch cf_x86_64_phone-userdebug && m && launch_cvd \
      adb wait-for-device && adb root && adb logcat \
      | grep -E 'ClassLoaderContext [a-z ]+ mismatch'
      # empty grep output, no errors
Change-Id: Id4933aa94ea96894278355283383c16d103e98c9
2021-03-02 12:32:23 +00:00
Ulya Trafimovich
8c35fcf4d5 Add non-fatal mode for verify_uses_libraries check.
The new mode is enabled with environment variable
RELAX_USES_LIBRARY_CHECK. If the variable is set to true, then a
verify_uses_libraries check failure does not fail the build, instead it
sets a special compiler filter "extract" for dexpreopt, which means that
the DEX file will be extracted, but it won't be compiled to native code.
Class loader context will be set to empty in this case (not &, as it is
going to be deprecated soon).

If the variable RELAX_USES_LIBRARY_CHECK is unset or set to something
other than "true", then the old behaviour of the verify_uses_libraries
check is preserved.

The intended use case for this flag is to have a smoother migration path
for the Java modules that need to add <uses-library> information in
the build files. The flag allows to quickly silence build errors. This
flag should be used with caution and only as a temporary measure, as it
masks real errors and affects performance.

verify_uses_libraries check is reworked so that it writes the error
message to a status file (which is used instead of the former timestamp
file). Currently the stored error message is not used, but it may be
used later to produce a warning. Dexpreopt command checks if the status
file exists and is nonempty; if that is the case, then compiler filter
is set to "extract".
Bug: 132357300

Test: Manually add some mismatch between the libraries in the Android.bp
      and Android.mk files for dexpreopted apps, build with
      RELAX_USES_LIBRARY_CHECK=true and obsserve that the build doesn't
      fail and they are compiled with compiler-filter "extract".
      Unset RELAX_USES_LIBRARY_CHECK and observe that the build fails.

Change-Id: Ibb5d993a25b1df1d2e70b7d5aafc6997f9d64e67
2021-02-22 15:41:03 +00:00
Colin Cross
f1a035e6be Pass pctx and ctx to NewRuleBuilder
Enable the RuleBuilder and RuleBuilderCommand methods to access
the BuilderContext by passing it to NewRuleBuilder instead of
RuleBuilder.Build.

Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I63e6597e19167393876dc2259d6f521363b7dabc
2020-12-01 16:22:16 -08:00
Ulya Trafimovich
8cbc5d269b Rework class loader context implementation.
The old representation consisted of a list of libraries (UsesLibraries),
a list of optional libraries (OptionalUsesLibraries) and a mapping from
library name to its build/install paths (LibraryPaths). The separation
into lists and map was necessary because of special handling of
compatibility libraries, which is now unified with normal libraries.

The new representation is a mapping from target SDK version to a tree
structure ClassLoaderContext. Each node of the tree represents a library
and contains library name, build/install paths and a slice of
subcontexts for dependencies. The same library may occur in the tree
multiple times in case it is a dependency of multiple libraries. The
order in which libraries are added matters (the resulting tree shape may
be different).

Test results have to be updated, as the resulting <uses-library> list is
reodered (previously it was a sorted list of map keys, and now it is
formed by a depth-first preorder traversal of the class loader tree).

Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Bug: 168686456
Change-Id: I11be8cd2967f004fd58753d7c5fb99fed179cd63
2020-11-03 15:15:46 +00:00
Ulya Trafimovich
6961267298 Add unit tests for class loader context.
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: I96a1e6cef86652e429b1678a655fc0b02f40d00c
2020-10-29 17:47:34 +00:00
Ulya Trafimovich
eb26886c85 Move class loader context definitions to a separate file.
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: I1e7e9db1654d0b835276be1cfa6a8eeffc5e96ee
2020-10-29 17:47:34 +00:00
Ulya Trafimovich
8130c482ab Move part of logic from construct_context.py to Soong.
construct_context.py is a script that runs at Ninja stage and constructs
class loader context arguments for dex2oat. Previously it accepted lists
of library paths and concatenated them into one class loader context
string. The script also handled the dependency of "android.hidl.manager"
on "android.hidl.base" that is not tracked by the build system and must
be handled in a special way.

Now that class loader context representation is going to change from
flat lists to trees, passing paths to individual libraries as script
arguments is no longer possible, because a list of paths cannot
represent a class loader context tree. Passing the trees in a serialized
form is also inconvenient, because the script would have to parse them,
which would complicate it a lot.

Therefore this patch ports all the concatenation and "android.hidl.base"
handling to Soong. It is not possible to port the remaining script to
Soong because Soong has no information about the targetSdkVersiion of
the dexpreopted apps (it is in the manifest and sometimes in an APK, and
has to be extracted at Ninja time).

Test: construct_context_test.py
Test: new subtests in TestUsesLibs
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: Icdb03cf00d1e27e4cff3844b89bfaec4de502dd7
2020-10-27 17:05:30 +00:00
Ulya Trafimovich
46b3d5bd05 Fix handling of "android.test.mock" in class loader context.
Mimick the way PackageManager handles it at runtime: do add it to class
loader context for apps with targetSdkVersion < 30, but only if
"android.test.runner" is used. Previously it was not added at all.

Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: I4c06635277ab13e21069b9fa0b46eb6a2547dfdd
2020-10-27 17:05:30 +00:00
Ulya Trafimovich
24813e1d80 Unify handling of compat and normal libs in class loader contexts.
Also, add tests for compatibility libraries in class loader context.

This CL separates special-case handling of compatibility libraries into
a "fixup" step that is done after class loader context is constructed by
Soong. This allows to handle compatibility libraries and normal
libraries uniformly, which should enable further simplification of class
loader context representation (in subsequent CLs).

Currently the only "fixup" step is removal of libraries from conditional
class loader context if they already are in unconditional context. This
check cannot be done at the time when the libraries are added to
conditional context, because the full uncoditional context is not yet
known at that time. Previously construction of unconditional context was
delayed, now it is no longer delayed and handled in the same way as
unconditional context, and the "fixup" does the filtering.

Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: Ie71e9fb2d3d529b5317cd84e09ab3c853017c349
2020-10-27 17:05:30 +00:00
Ulya Trafimovich
a54d33be77 Fail the build if dexpreopt cannot find path to a <uses-library>.
This doesn't apply to compatibility libraries (those added to
conditional class loader context base on the targetSdkVersion in the
manifest), because Soong doesn't know if they are needed or not (the
build rule that parses the manifest runs at Ninja stage).

In the future we may need to relax this and instead of failing the build
to disable dexpreopt in such cases. But currently there are no use cases
that fail.

Also, this patch replaces hardcoded install path with a constant.

Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: Ic894bc17e8ffd8ff173111ea09366252001ef3e9
2020-10-06 14:04:49 +01:00
Anton Hansson
da4d9d967d Revert "Re-land "Move odexes of non-system apps into system_other""
This reverts commit 75480154e2.

Bug: 141707536
Test: Clean revert
Reason for revert: breaks some builds
Change-Id: If0cf736099ec12e35bcb190d0fc33daf7f1da98e
2020-09-15 09:29:11 +00:00
Anton Hansson
75480154e2 Re-land "Move odexes of non-system apps into system_other"
Previously, the patterns in SYSTEM_OTHER_ODEX_FILTER only matched
subpaths of /system. Broaden this to match /<p> as well as /system/<p>.

Because SYSTEM_OTHER_ODEX_FILTER includes system_ext and product apps,
the immediate effect of this is that the odex and vdex files of
apps on those partitions move to system_other for A/B devices.

This makes the product and system_ext partitions smaller, which
in turn makes OTAs smaller.

Bug: 141707536
Test: soong unit tests
Test: (adb shell dumpsys package com.android.stk &&
       adb shell dumpsys package com.android.quicksearchbox) | grep -i dexopt -A3;
      verify dexpreopt state is 'prebuilt' for both
Test: adb shell find /data/dalvik-cache -name '*QuickSearch*.vdex' | wc -l == 1
Test: adb shell find /data/dalvik-cache -name '*Stk*.vdex' | wc -l == 1
Change-Id: I0cf141014b9ddf32159cff47b0325d9c78b549f2
2020-09-11 14:45:51 +01:00
Ulya Trafimovich
fc24ad3d4e Propagate transitive SDK Java library dependencies to dexpreopt.
For some dependencies, like stubs, the SDK library may not be found at
build time (either because the implementation library is not among the
dependencies of the dexpreopted module, or because it's part of a
prebuilt, or because it's missing from the build altogether). In such
cases dexpreopt is useless, because dex2oat does not have access to the
full classpath (unless the &-classpath is used). Therefore do not
dexpreopt in such cases.

Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: If289088cfd103011ccb16165e95a97b30fd31b81
2020-08-27 11:47:15 +01:00