platform_build_soong/dexpreopt/dexpreopt.go

625 lines
24 KiB
Go
Raw Normal View History

// Copyright 2018 Google Inc. All rights reserved.
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
// The dexpreopt package converts a global dexpreopt config and a module dexpreopt config into rules to perform
// dexpreopting.
//
// It is used in two places; in the dexpeopt_gen binary for modules defined in Make, and directly linked into Soong.
//
// For Make modules it is built into the dexpreopt_gen binary, which is executed as a Make rule using global config and
// module config specified in JSON files. The binary writes out two shell scripts, only updating them if they have
// changed. One script takes an APK or JAR as an input and produces a zip file containing any outputs of preopting,
// in the location they should be on the device. The Make build rules will unzip the zip file into $(PRODUCT_OUT) when
// installing the APK, which will install the preopt outputs into $(PRODUCT_OUT)/system or $(PRODUCT_OUT)/system_other
// as necessary. The zip file may be empty if preopting was disabled for any reason.
//
// The intermediate shell scripts allow changes to this package or to the global config to regenerate the shell scripts
// but only require re-executing preopting if the script has changed.
//
// For Soong modules this package is linked directly into Soong and run from the java package. It generates the same
// commands as for make, using athe same global config JSON file used by make, but using a module config structure
// provided by Soong. The generated commands are then converted into Soong rule and written directly to the ninja file,
// with no extra shell scripts involved.
package dexpreopt
import (
"fmt"
"path/filepath"
"runtime"
"strings"
"android/soong/android"
"github.com/google/blueprint/pathtools"
)
const SystemPartition = "/system/"
const SystemOtherPartition = "/system_other/"
var DexpreoptRunningInSoong = false
// GenerateDexpreoptRule generates a set of commands that will preopt a module based on a GlobalConfig and a
// ModuleConfig. The produced files and their install locations will be available through rule.Installs().
func GenerateDexpreoptRule(ctx android.BuilderContext, globalSoong *GlobalSoongConfig,
Allow dexpreopt of source sdklib in prebuilt apex builds aosp/2984037 disabled dexperopt of the source apex system server jars when prebuilts are selected. This was done to prevent duplicate installation and dex2oat deps. AOSP art has some additional variants like com.android.art.debug. In source builds, this apex should contain service-art.jar and service-art.jar.prof (a dexpeopt artifact). We have a test to check this (`art_apex_test.py`). If we disable dexpreopt of source sdlib when prebuilts are selected, this test needs to be disabled. This is the behavior at ToT. This CL changes the behavior to enable running this test even when com.google.android.art prebuilt is active in a specific release configuraiton. To prevent collisions that prompted aosp/2984037, this CL special-cases the installation and dex2oat rules instead of disabling dexpreopt of the source sdklib altogether. b/331665856 tracks the principled solution to prevent duplicate dexpreopt rules. Implementation: Add a new copyApexSystemServerJarDex arg to GenerateDexpreoptRule API. If true, the dexjar file will be copied to out/soong/system_server_jars/. For soong modules, the value of this will be the inverse of disableSourceApexVariant. Since none of the apex system server jars are in make, this will be a noop in dexpreopt_gen Bug: 331665856 Test: modified trunk_staging.scl locally to select art.google.contributions.prebuilt; mmma art; (with the sibling CL in topic) Change-Id: Idb59e424f83d126cdc8b1671dde358745979bd8d
2024-04-17 00:03:26 +02:00
global *GlobalConfig, module *ModuleConfig, productPackages android.Path, copyApexSystemServerJarDex bool) (
Move CLC construction to Ninja phase. Before this change, dexpreopt was often broken with optional libraries. This was because the CLC construction was done in Soong at an early stage, where we don't have sufficient information to determine whether an optional library is installed or not. For example, the "Settings" package uses an optional library called "androidx.window.extensions". On some devices, the library is installed, but on some other devices, it's not. Soong always adds the library to the CLC, meaning the CLC is wrong for devices which don't have the library. This change fixes the problem. See the tests below. After this change, the CLC construction is done by a Python script invoked at a very late stage. It uses product_packages.txt, which is generated by Make, to determine whether an optional library is installed or not, and filter out libraries that are not installed. Note that optional libraries are still added as dependencies by Soong. This is because dependencies have to be added at an early stage. This means what dex2oat eventually uses will be a subset of the dependencies, which is fine. Bug: 282877248 Test: m Test: atest construct_context_test Test: - 1. lunch aosp_cf_x86_64_phone-userdebug && m 2. Check the .invocation file of the "Settings" package (defined in .bp file) 3. See androidx.window.extensions Test: - 1. lunch aosp_redfin-userdebug && m 2. Check the .invocation file of the "Settings" package (defined in .bp file) 3. Don't see androidx.window.extensions Test: Check the .invocation file of the "Dialer" package (defined in .mk file) Test: - 1. Build a Pixel 5 system image and flash it to a Pixel 5 device. 2. adb shell pm art dump 3. See "reason=prebuilt" instead of "reason=vdex". (https://diff.googleplex.com/#key=fB6Ls9q2QGSN, before: left, after: right) Change-Id: Ia112bd7c2328373e68db6bffb74bf34030f683d8
2023-05-17 17:57:30 +02:00
rule *android.RuleBuilder, err error) {
defer func() {
if r := recover(); r != nil {
if _, ok := r.(runtime.Error); ok {
panic(r)
} else if e, ok := r.(error); ok {
err = e
rule = nil
} else {
panic(r)
}
}
}()
rule = android.NewRuleBuilder(pctx, ctx)
generateProfile := module.ProfileClassListing.Valid() && !global.DisableGenerateProfile
generateBootProfile := module.ProfileBootListing.Valid() && !global.DisableGenerateProfile
var profile android.WritablePath
if generateProfile {
profile = profileCommand(ctx, globalSoong, global, module, rule)
}
if generateBootProfile {
bootProfileCommand(ctx, globalSoong, global, module, rule)
}
Use precise class loader context for system server jars. This patch excludes non-Soong system server jars from dexpreopting. System server jars should be dexpreopted together for better performance: each jar should have all preceding system server jars in its class loader context (that is passed to dex2oat and recorded in the resulting .oat file to be used by the loader). This means that we need a total order on system server jars. The default order provided by PRODUCT_SYSTEM_SERVER_JARS is not good, as it does not always respect genuine dependencies between jars (counter- examples are rare, but non-trivial to fix: b/148219586). This patch adds a post deps mutator pass that creates additional dependencies and enforces global order. These are later used to generate precise class loader contexts and system server classpath. Test: Class loader contexts in the oat files for system server jars match expectations: $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/com.android.location.provider.odex | grep '^classpath' classpath = PCL[] $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/services.odex | grep '^classpath' classpath = PCL[/system/framework/com.android.location.provider.jar*1919890654] $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/wifi-service.odex | grep '^classpath' classpath = PCL[/system/framework/com.android.location.provider.jar*1919890654:/system/framework/services.jar*4269704903:/system/framework/services.jar!classes2.dex*134345935] ... Test: The phone boots and logcat has no scary messages related to class loader contexts: $ lunch aosp_walleye-userdebug && m $ adb reboot bootloader && fastboot flashall -w && adb wait-for-device $ adb root $ adb shell stop $ adb logcat -c $ adb shell setprop dalvik.vm.extra-opts -verbose:oat $ adb shell start $ adb logcat | egrep -i 'system_server: .*load.*/system/framework' 02-03 14:14:26.912 5016 5016 I system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1 02-03 14:14:26.914 5016 5016 I system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1 02-03 14:14:26.914 5016 5016 I system_server: Loading /system/framework/oat/arm64/services.odex with executable: 1 02-03 14:14:26.916 5016 5016 I system_server: Successfully loaded /system/framework/oat/arm64/services.odex with executable: 1 02-03 14:14:26.927 5016 5016 I system_server: Loading /system/framework/oat/arm64/wifi-service.odex with executable: 1 02-03 14:14:26.933 5016 5016 I system_server: Successfully loaded /system/framework/oat/arm64/wifi-service.odex with executable: 1 02-03 14:14:26.933 5016 5016 I system_server: Loading /system/framework/oat/arm64/ethernet-service.odex with executable: 1 02-03 14:14:26.934 5016 5016 I system_server: Successfully loaded /system/framework/oat/arm64/ethernet-service.odex with executable: 1 02-03 14:14:26.946 5016 5016 I system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 0 02-03 14:14:26.947 5016 5016 I system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 0 02-03 14:14:26.947 5016 5016 I system_server: Loading /system/framework/oat/arm64/services.odex with executable: 0 02-03 14:14:26.948 5016 5016 I system_server: Successfully loaded /system/framework/oat/arm64/services.odex with executable: 0 02-03 14:14:26.948 5016 5016 I system_server: Loading /system/framework/oat/arm64/wifi-service.odex with executable: 0 02-03 14:14:26.948 5016 5016 I system_server: Successfully loaded /system/framework/oat/arm64/wifi-service.odex with executable: 0 02-03 14:14:26.949 5016 5016 I system_server: Loading /system/framework/oat/arm64/ethernet-service.odex with executable: 0 02-03 14:14:26.949 5016 5016 I system_server: Successfully loaded /system/framework/oat/arm64/ethernet-service.odex with executable: 0 02-03 14:14:30.480 5016 5016 I system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1 02-03 14:14:30.481 5016 5016 I system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1 Bug: 141785760 Bug: 140451054 Bug: 148944771 Change-Id: Idac678dbd1f5fe0e381ce8e0e3561423f8a31389
2019-12-03 16:39:23 +01:00
if !dexpreoptDisabled(ctx, global, module) {
if valid, err := validateClassLoaderContext(module.ClassLoaderContexts); err != nil {
android.ReportPathErrorf(ctx, err.Error())
} else if valid {
fixClassLoaderContext(module.ClassLoaderContexts)
appImage := (generateProfile || module.ForceCreateAppImage || global.DefaultAppImages) &&
!module.NoCreateAppImage
generateDM := shouldGenerateDM(module, global)
for archIdx, _ := range module.Archs {
Move CLC construction to Ninja phase. Before this change, dexpreopt was often broken with optional libraries. This was because the CLC construction was done in Soong at an early stage, where we don't have sufficient information to determine whether an optional library is installed or not. For example, the "Settings" package uses an optional library called "androidx.window.extensions". On some devices, the library is installed, but on some other devices, it's not. Soong always adds the library to the CLC, meaning the CLC is wrong for devices which don't have the library. This change fixes the problem. See the tests below. After this change, the CLC construction is done by a Python script invoked at a very late stage. It uses product_packages.txt, which is generated by Make, to determine whether an optional library is installed or not, and filter out libraries that are not installed. Note that optional libraries are still added as dependencies by Soong. This is because dependencies have to be added at an early stage. This means what dex2oat eventually uses will be a subset of the dependencies, which is fine. Bug: 282877248 Test: m Test: atest construct_context_test Test: - 1. lunch aosp_cf_x86_64_phone-userdebug && m 2. Check the .invocation file of the "Settings" package (defined in .bp file) 3. See androidx.window.extensions Test: - 1. lunch aosp_redfin-userdebug && m 2. Check the .invocation file of the "Settings" package (defined in .bp file) 3. Don't see androidx.window.extensions Test: Check the .invocation file of the "Dialer" package (defined in .mk file) Test: - 1. Build a Pixel 5 system image and flash it to a Pixel 5 device. 2. adb shell pm art dump 3. See "reason=prebuilt" instead of "reason=vdex". (https://diff.googleplex.com/#key=fB6Ls9q2QGSN, before: left, after: right) Change-Id: Ia112bd7c2328373e68db6bffb74bf34030f683d8
2023-05-17 17:57:30 +02:00
dexpreoptCommand(ctx, globalSoong, global, module, rule, archIdx, profile, appImage,
Allow dexpreopt of source sdklib in prebuilt apex builds aosp/2984037 disabled dexperopt of the source apex system server jars when prebuilts are selected. This was done to prevent duplicate installation and dex2oat deps. AOSP art has some additional variants like com.android.art.debug. In source builds, this apex should contain service-art.jar and service-art.jar.prof (a dexpeopt artifact). We have a test to check this (`art_apex_test.py`). If we disable dexpreopt of source sdlib when prebuilts are selected, this test needs to be disabled. This is the behavior at ToT. This CL changes the behavior to enable running this test even when com.google.android.art prebuilt is active in a specific release configuraiton. To prevent collisions that prompted aosp/2984037, this CL special-cases the installation and dex2oat rules instead of disabling dexpreopt of the source sdklib altogether. b/331665856 tracks the principled solution to prevent duplicate dexpreopt rules. Implementation: Add a new copyApexSystemServerJarDex arg to GenerateDexpreoptRule API. If true, the dexjar file will be copied to out/soong/system_server_jars/. For soong modules, the value of this will be the inverse of disableSourceApexVariant. Since none of the apex system server jars are in make, this will be a noop in dexpreopt_gen Bug: 331665856 Test: modified trunk_staging.scl locally to select art.google.contributions.prebuilt; mmma art; (with the sibling CL in topic) Change-Id: Idb59e424f83d126cdc8b1671dde358745979bd8d
2024-04-17 00:03:26 +02:00
generateDM, productPackages, copyApexSystemServerJarDex)
}
}
}
return rule, nil
}
// If dexpreopt is applicable to the module, returns whether dexpreopt is disabled. Otherwise, the
// behavior is undefined.
// When it returns true, dexpreopt artifacts will not be generated, but profile will still be
// generated if profile-guided compilation is requested.
func dexpreoptDisabled(ctx android.PathContext, global *GlobalConfig, module *ModuleConfig) bool {
if ctx.Config().UnbundledBuild() {
return true
}
if global.DisablePreopt {
return true
}
if contains(global.DisablePreoptModules, module.Name) {
return true
}
// Don't preopt individual boot jars, they will be preopted together.
if global.BootJars.ContainsJar(module.Name) {
return true
}
if global.OnlyPreoptArtBootImage {
return true
}
return false
}
func profileCommand(ctx android.PathContext, globalSoong *GlobalSoongConfig, global *GlobalConfig,
module *ModuleConfig, rule *android.RuleBuilder) android.WritablePath {
profilePath := module.BuildPath.InSameDir(ctx, "profile.prof")
profileInstalledPath := module.DexLocation + ".prof"
if !module.ProfileIsTextListing {
rule.Command().Text("rm -f").Output(profilePath)
rule.Command().Text("touch").Output(profilePath)
}
cmd := rule.Command().
Text(`ANDROID_LOG_TAGS="*:e"`).
Tool(globalSoong.Profman)
if module.ProfileIsTextListing {
// The profile is a test listing of classes (used for framework jars).
// We need to generate the actual binary profile before being able to compile.
cmd.FlagWithInput("--create-profile-from=", module.ProfileClassListing.Path())
} else {
// The profile is binary profile (used for apps). Run it through profman to
// ensure the profile keys match the apk.
cmd.
Flag("--copy-and-update-profile-key").
FlagWithInput("--profile-file=", module.ProfileClassListing.Path())
}
cmd.
Flag("--output-profile-type=app").
FlagWithInput("--apk=", module.DexPath).
Flag("--dex-location="+module.DexLocation).
FlagWithOutput("--reference-profile-file=", profilePath)
if !module.ProfileIsTextListing {
cmd.Text(fmt.Sprintf(`|| echo "Profile out of date for %s"`, module.DexPath))
}
rule.Install(profilePath, profileInstalledPath)
return profilePath
}
func bootProfileCommand(ctx android.PathContext, globalSoong *GlobalSoongConfig, global *GlobalConfig,
module *ModuleConfig, rule *android.RuleBuilder) android.WritablePath {
profilePath := module.BuildPath.InSameDir(ctx, "profile.bprof")
profileInstalledPath := module.DexLocation + ".bprof"
if !module.ProfileIsTextListing {
rule.Command().Text("rm -f").Output(profilePath)
rule.Command().Text("touch").Output(profilePath)
}
cmd := rule.Command().
Text(`ANDROID_LOG_TAGS="*:e"`).
Tool(globalSoong.Profman)
// The profile is a test listing of methods.
// We need to generate the actual binary profile.
cmd.FlagWithInput("--create-profile-from=", module.ProfileBootListing.Path())
cmd.
Flag("--output-profile-type=bprof").
FlagWithInput("--apk=", module.DexPath).
Flag("--dex-location="+module.DexLocation).
FlagWithOutput("--reference-profile-file=", profilePath)
if !module.ProfileIsTextListing {
cmd.Text(fmt.Sprintf(`|| echo "Profile out of date for %s"`, module.DexPath))
}
rule.Install(profilePath, profileInstalledPath)
return profilePath
}
// Returns the dex location of a system server java library.
func GetSystemServerDexLocation(ctx android.PathContext, global *GlobalConfig, lib string) string {
if apex := global.AllApexSystemServerJars(ctx).ApexOfJar(lib); apex != "" {
return fmt.Sprintf("/apex/%s/javalib/%s.jar", apex, lib)
}
if apex := global.AllPlatformSystemServerJars(ctx).ApexOfJar(lib); apex == "system_ext" {
return fmt.Sprintf("/system_ext/framework/%s.jar", lib)
}
return fmt.Sprintf("/system/framework/%s.jar", lib)
}
// Returns the location to the odex file for the dex file at `path`.
func ToOdexPath(path string, arch android.ArchType) string {
if strings.HasPrefix(path, "/apex/") {
return filepath.Join("/system/framework/oat", arch.String(),
strings.ReplaceAll(path[1:], "/", "@")+"@classes.odex")
}
return filepath.Join(filepath.Dir(path), "oat", arch.String(),
pathtools.ReplaceExtension(filepath.Base(path), "odex"))
}
Move CLC construction to Ninja phase. Before this change, dexpreopt was often broken with optional libraries. This was because the CLC construction was done in Soong at an early stage, where we don't have sufficient information to determine whether an optional library is installed or not. For example, the "Settings" package uses an optional library called "androidx.window.extensions". On some devices, the library is installed, but on some other devices, it's not. Soong always adds the library to the CLC, meaning the CLC is wrong for devices which don't have the library. This change fixes the problem. See the tests below. After this change, the CLC construction is done by a Python script invoked at a very late stage. It uses product_packages.txt, which is generated by Make, to determine whether an optional library is installed or not, and filter out libraries that are not installed. Note that optional libraries are still added as dependencies by Soong. This is because dependencies have to be added at an early stage. This means what dex2oat eventually uses will be a subset of the dependencies, which is fine. Bug: 282877248 Test: m Test: atest construct_context_test Test: - 1. lunch aosp_cf_x86_64_phone-userdebug && m 2. Check the .invocation file of the "Settings" package (defined in .bp file) 3. See androidx.window.extensions Test: - 1. lunch aosp_redfin-userdebug && m 2. Check the .invocation file of the "Settings" package (defined in .bp file) 3. Don't see androidx.window.extensions Test: Check the .invocation file of the "Dialer" package (defined in .mk file) Test: - 1. Build a Pixel 5 system image and flash it to a Pixel 5 device. 2. adb shell pm art dump 3. See "reason=prebuilt" instead of "reason=vdex". (https://diff.googleplex.com/#key=fB6Ls9q2QGSN, before: left, after: right) Change-Id: Ia112bd7c2328373e68db6bffb74bf34030f683d8
2023-05-17 17:57:30 +02:00
func dexpreoptCommand(ctx android.BuilderContext, globalSoong *GlobalSoongConfig,
global *GlobalConfig, module *ModuleConfig, rule *android.RuleBuilder, archIdx int,
Allow dexpreopt of source sdklib in prebuilt apex builds aosp/2984037 disabled dexperopt of the source apex system server jars when prebuilts are selected. This was done to prevent duplicate installation and dex2oat deps. AOSP art has some additional variants like com.android.art.debug. In source builds, this apex should contain service-art.jar and service-art.jar.prof (a dexpeopt artifact). We have a test to check this (`art_apex_test.py`). If we disable dexpreopt of source sdlib when prebuilts are selected, this test needs to be disabled. This is the behavior at ToT. This CL changes the behavior to enable running this test even when com.google.android.art prebuilt is active in a specific release configuraiton. To prevent collisions that prompted aosp/2984037, this CL special-cases the installation and dex2oat rules instead of disabling dexpreopt of the source sdklib altogether. b/331665856 tracks the principled solution to prevent duplicate dexpreopt rules. Implementation: Add a new copyApexSystemServerJarDex arg to GenerateDexpreoptRule API. If true, the dexjar file will be copied to out/soong/system_server_jars/. For soong modules, the value of this will be the inverse of disableSourceApexVariant. Since none of the apex system server jars are in make, this will be a noop in dexpreopt_gen Bug: 331665856 Test: modified trunk_staging.scl locally to select art.google.contributions.prebuilt; mmma art; (with the sibling CL in topic) Change-Id: Idb59e424f83d126cdc8b1671dde358745979bd8d
2024-04-17 00:03:26 +02:00
profile android.WritablePath, appImage bool, generateDM bool, productPackages android.Path, copyApexSystemServerJarDex bool) {
arch := module.Archs[archIdx]
// HACK: make soname in Soong-generated .odex files match Make.
base := filepath.Base(module.DexLocation)
if filepath.Ext(base) == ".jar" {
base = "javalib.jar"
} else if filepath.Ext(base) == ".apk" {
base = "package.apk"
}
odexPath := module.BuildPath.InSameDir(ctx, "oat", arch.String(), pathtools.ReplaceExtension(base, "odex"))
odexInstallPath := ToOdexPath(module.DexLocation, arch)
if odexOnSystemOther(module, global) {
odexInstallPath = filepath.Join(SystemOtherPartition, odexInstallPath)
}
vdexPath := odexPath.ReplaceExtension(ctx, "vdex")
vdexInstallPath := pathtools.ReplaceExtension(odexInstallPath, "vdex")
invocationPath := odexPath.ReplaceExtension(ctx, "invocation")
systemServerJars := global.AllSystemServerJars(ctx)
systemServerClasspathJars := global.AllSystemServerClasspathJars(ctx)
Simplify the construction of class loader contexts for system server jars. This reworks CL https://r.android.com/1180134 as follows: 1) Do not reorder the list of system server jars passed from Make to Soong via the product variable PRODUCT_SYSTEM_SERVER_JARS. This means that for some products the order of jars on the system server classpath may be non-optimal: a jar X that depends on Y may be dexpreopted before Y, so that all references to the classes and methods from Y wil be unresolved. Unfortunately for such products, fixing the order is not a simple matter of rearranging their PRODUCT_SYSTEM_SERVER_JARS, because the conflicts may arise when the product-specific variable gets merged with the common variable. 2) As a consequence of 1), do not add artificial dependencies between system server jars: this is now impossible, as it would create circular dependencies for those products that have non-optimal order of jars. 3) Copy dex files for system server jars to a predefined build location. This is necessary because at the time when Soong generates class loader context for k-th jar, it needs to know the paths to jars 1 .. (k-1), and it might have not processed those jars yet (so Soong can't query the paths from the modules). This approach is similar to the way Soong handles bootclasspath jars. 4) Do not exclude from dexpreopting system server jars that are not Soong modules (those that are Make modules). The previous CL excluded them because Make modules do not have ModuleContext. But it turns out that ModuleContext is not necessary, as all the information is passed via the dexpreopt config. Test: aosp_walleye-userdebug boots and there are no messages in the logcat regarding class loader context mismatch: $ adb logcat | grep 'mismatch' # empty Test: Class loader contexts in the oat files for system server jars match expectations: $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/com.android.location.provider.odex 2>/dev/null | grep '^classpath' classpath = PCL[] $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/services.odex 2>/dev/null | grep '^classpath' classpath = PCL[/system/framework/com.android.location.provider.jar*1989208671] $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/ethernet-service.odex 2>/dev/null | grep '^classpath' classpath = PCL[/system/framework/com.android.location.provider.jar*1989208671:/system/framework/services.jar*4040443083:/system/framework/services.jar!classes2.dex*2210087472] Test: The phone boots and logcat has no scary messages related to class loader contexts: $ lunch aosp_walleye-userdebug && m $ adb reboot bootloader && fastboot flashall -w && adb wait-for-device $ adb root $ adb shell stop $ adb logcat -c $ adb shell setprop dalvik.vm.extra-opts -verbose:oat $ adb shell start $ adb logcat | egrep -io 'system_server: .*load.*/system/framework.*' system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 0 system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 0 system_server: Loading /system/framework/oat/arm64/services.odex with executable: 0 system_server: Successfully loaded /system/framework/oat/arm64/services.odex with executable: 0 system_server: Loading /system/framework/oat/arm64/ethernet-service.odex with executable: 0 system_server: Successfully loaded /system/framework/oat/arm64/ethernet-service.odex with executable: 0 system_server: Loading /system/framework/oat/arm64/wifi-service.odex with executable: 0 system_server: Successfully loaded /system/framework/oat/arm64/wifi-service.odex with executable: 0 system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1 system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1 system_server: Loading /system/framework/oat/arm64/services.odex with executable: 1 system_server: Successfully loaded /system/framework/oat/arm64/services.odex with executable: 1 system_server: Loading /system/framework/oat/arm64/ethernet-service.odex with executable: 1 system_server: Successfully loaded /system/framework/oat/arm64/ethernet-service.odex with executable: 1 system_server: Loading /system/framework/oat/arm64/wifi-service.odex with executable: 1 system_server: Successfully loaded /system/framework/oat/arm64/wifi-service.odex with executable: 1 system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1 system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1 Bug: 141785760 Bug: 140451054 Bug: 148944771 Bug: 147017252 Change-Id: I33c4087f8bfacd0ecb89877aa150b75360d06710 Merged-In: I33c4087f8bfacd0ecb89877aa150b75360d06710 (cherry picked from commit a4a83b0ef9b9769681ad285290fdbbf043eff3d5) Exempt-From-Owner-Approval: cherry-pick.
2020-03-11 12:59:34 +01:00
rule.Command().FlagWithArg("mkdir -p ", filepath.Dir(odexPath.String()))
rule.Command().FlagWithOutput("rm -f ", odexPath)
if jarIndex := systemServerJars.IndexOfJar(module.Name); jarIndex >= 0 {
// System server jars should be dexpreopted together: class loader context of each jar
// should include all preceding jars on the system server classpath.
var clcHost android.Paths
var clcTarget []string
endIndex := systemServerClasspathJars.IndexOfJar(module.Name)
if endIndex < 0 {
// The jar is a standalone one. Use the full classpath as the class loader context.
endIndex = systemServerClasspathJars.Len()
}
for i := 0; i < endIndex; i++ {
lib := systemServerClasspathJars.Jar(i)
clcHost = append(clcHost, SystemServerDexJarHostPath(ctx, lib))
clcTarget = append(clcTarget, GetSystemServerDexLocation(ctx, global, lib))
}
Allow dexpreopt of source sdklib in prebuilt apex builds aosp/2984037 disabled dexperopt of the source apex system server jars when prebuilts are selected. This was done to prevent duplicate installation and dex2oat deps. AOSP art has some additional variants like com.android.art.debug. In source builds, this apex should contain service-art.jar and service-art.jar.prof (a dexpeopt artifact). We have a test to check this (`art_apex_test.py`). If we disable dexpreopt of source sdlib when prebuilts are selected, this test needs to be disabled. This is the behavior at ToT. This CL changes the behavior to enable running this test even when com.google.android.art prebuilt is active in a specific release configuraiton. To prevent collisions that prompted aosp/2984037, this CL special-cases the installation and dex2oat rules instead of disabling dexpreopt of the source sdklib altogether. b/331665856 tracks the principled solution to prevent duplicate dexpreopt rules. Implementation: Add a new copyApexSystemServerJarDex arg to GenerateDexpreoptRule API. If true, the dexjar file will be copied to out/soong/system_server_jars/. For soong modules, the value of this will be the inverse of disableSourceApexVariant. Since none of the apex system server jars are in make, this will be a noop in dexpreopt_gen Bug: 331665856 Test: modified trunk_staging.scl locally to select art.google.contributions.prebuilt; mmma art; (with the sibling CL in topic) Change-Id: Idb59e424f83d126cdc8b1671dde358745979bd8d
2024-04-17 00:03:26 +02:00
if DexpreoptRunningInSoong && copyApexSystemServerJarDex {
// Copy the system server jar to a predefined location where dex2oat will find it.
dexPathHost := SystemServerDexJarHostPath(ctx, module.Name)
rule.Command().Text("mkdir -p").Flag(filepath.Dir(dexPathHost.String()))
rule.Command().Text("cp -f").Input(module.DexPath).Output(dexPathHost)
} else {
// For Make modules the copy rule is generated in the makefiles, not in dexpreopt.sh.
// This is necessary to expose the rule to Ninja, otherwise it has rules that depend on
// the jar (namely, dexpreopt commands for all subsequent system server jars that have
// this one in their class loader context), but no rule that creates it (because Ninja
// cannot see the rule in the generated dexpreopt.sh script).
}
clcHostString := "PCL[" + strings.Join(clcHost.Strings(), ":") + "]"
clcTargetString := "PCL[" + strings.Join(clcTarget, ":") + "]"
if systemServerClasspathJars.ContainsJar(module.Name) {
checkSystemServerOrder(ctx, jarIndex)
} else {
// Standalone jars are loaded by separate class loaders with SYSTEMSERVERCLASSPATH as the
// parent.
clcHostString = "PCL[];" + clcHostString
clcTargetString = "PCL[];" + clcTargetString
}
rule.Command().
Text(`class_loader_context_arg=--class-loader-context="` + clcHostString + `"`).
Implicits(clcHost).
Text(`stored_class_loader_context_arg=--stored-class-loader-context="` + clcTargetString + `"`)
} else {
// There are three categories of Java modules handled here:
//
// - Modules that have passed verify_uses_libraries check. They are AOT-compiled and
// expected to be loaded on device without CLC mismatch errors.
//
// - Modules that have failed the check in relaxed mode, so it didn't cause a build error.
// They are dexpreopted with "verify" filter and not AOT-compiled.
// TODO(b/132357300): ensure that CLC mismatch errors are ignored with "verify" filter.
//
// - Modules that didn't run the check. They are AOT-compiled, but it's unknown if they
// will have CLC mismatch errors on device (the check is disabled by default).
//
// TODO(b/132357300): enable the check by default and eliminate the last category, so that
// no time/space is wasted on AOT-compiling modules that will fail CLC check on device.
var manifestOrApk android.Path
if module.ManifestPath.Valid() {
// Ok, there is an XML manifest.
manifestOrApk = module.ManifestPath.Path()
} else if filepath.Ext(base) == ".apk" {
// Ok, there is is an APK with the manifest inside.
manifestOrApk = module.DexPath
}
// Generate command that saves target SDK version in a shell variable.
if manifestOrApk == nil {
// There is neither an XML manifest nor APK => nowhere to extract targetSdkVersion from.
// Set the latest ("any") version: then construct_context will not add any compatibility
// libraries (if this is incorrect, there will be a CLC mismatch and dexopt on device).
rule.Command().Textf(`target_sdk_version=%d`, AnySdkVersion)
} else {
rule.Command().Text(`target_sdk_version="$(`).
Tool(globalSoong.ManifestCheck).
Flag("--extract-target-sdk-version").
Input(manifestOrApk).
FlagWithInput("--aapt ", globalSoong.Aapt).
Text(`)"`)
}
// Generate command that saves host and target class loader context in shell variables.
_, paths := ComputeClassLoaderContextDependencies(module.ClassLoaderContexts)
Add non-fatal mode for verify_uses_libraries check. The new mode is enabled with environment variable RELAX_USES_LIBRARY_CHECK. If the variable is set to true, then a verify_uses_libraries check failure does not fail the build, instead it sets a special compiler filter "extract" for dexpreopt, which means that the DEX file will be extracted, but it won't be compiled to native code. Class loader context will be set to empty in this case (not &, as it is going to be deprecated soon). If the variable RELAX_USES_LIBRARY_CHECK is unset or set to something other than "true", then the old behaviour of the verify_uses_libraries check is preserved. The intended use case for this flag is to have a smoother migration path for the Java modules that need to add <uses-library> information in the build files. The flag allows to quickly silence build errors. This flag should be used with caution and only as a temporary measure, as it masks real errors and affects performance. verify_uses_libraries check is reworked so that it writes the error message to a status file (which is used instead of the former timestamp file). Currently the stored error message is not used, but it may be used later to produce a warning. Dexpreopt command checks if the status file exists and is nonempty; if that is the case, then compiler filter is set to "extract". Bug: 132357300 Test: Manually add some mismatch between the libraries in the Android.bp and Android.mk files for dexpreopted apps, build with RELAX_USES_LIBRARY_CHECK=true and obsserve that the build doesn't fail and they are compiled with compiler-filter "extract". Unset RELAX_USES_LIBRARY_CHECK and observe that the build fails. Change-Id: Ibb5d993a25b1df1d2e70b7d5aafc6997f9d64e67
2021-02-17 17:23:28 +01:00
rule.Command().
Text(`eval "$(`).Tool(globalSoong.ConstructContext).
Text(` --target-sdk-version ${target_sdk_version}`).
Move CLC construction to Ninja phase. Before this change, dexpreopt was often broken with optional libraries. This was because the CLC construction was done in Soong at an early stage, where we don't have sufficient information to determine whether an optional library is installed or not. For example, the "Settings" package uses an optional library called "androidx.window.extensions". On some devices, the library is installed, but on some other devices, it's not. Soong always adds the library to the CLC, meaning the CLC is wrong for devices which don't have the library. This change fixes the problem. See the tests below. After this change, the CLC construction is done by a Python script invoked at a very late stage. It uses product_packages.txt, which is generated by Make, to determine whether an optional library is installed or not, and filter out libraries that are not installed. Note that optional libraries are still added as dependencies by Soong. This is because dependencies have to be added at an early stage. This means what dex2oat eventually uses will be a subset of the dependencies, which is fine. Bug: 282877248 Test: m Test: atest construct_context_test Test: - 1. lunch aosp_cf_x86_64_phone-userdebug && m 2. Check the .invocation file of the "Settings" package (defined in .bp file) 3. See androidx.window.extensions Test: - 1. lunch aosp_redfin-userdebug && m 2. Check the .invocation file of the "Settings" package (defined in .bp file) 3. Don't see androidx.window.extensions Test: Check the .invocation file of the "Dialer" package (defined in .mk file) Test: - 1. Build a Pixel 5 system image and flash it to a Pixel 5 device. 2. adb shell pm art dump 3. See "reason=prebuilt" instead of "reason=vdex". (https://diff.googleplex.com/#key=fB6Ls9q2QGSN, before: left, after: right) Change-Id: Ia112bd7c2328373e68db6bffb74bf34030f683d8
2023-05-17 17:57:30 +02:00
FlagWithArg("--context-json=", module.ClassLoaderContexts.DumpForFlag()).
FlagWithInput("--product-packages=", productPackages).
Implicits(paths).
Text(`)"`)
}
// Devices that do not have a product partition use a symlink from /product to /system/product.
// Because on-device dexopt will see dex locations starting with /product, we change the paths
// to mimic this behavior.
dexLocationArg := module.DexLocation
if strings.HasPrefix(dexLocationArg, "/system/product/") {
dexLocationArg = strings.TrimPrefix(dexLocationArg, "/system")
}
cmd := rule.Command().
Text(`ANDROID_LOG_TAGS="*:e"`).
Tool(globalSoong.Dex2oat).
Flag("--avoid-storing-invocation").
FlagWithOutput("--write-invocation-to=", invocationPath).ImplicitOutput(invocationPath).
Flag("--runtime-arg").FlagWithArg("-Xms", global.Dex2oatXms).
Flag("--runtime-arg").FlagWithArg("-Xmx", global.Dex2oatXmx).
Flag("--runtime-arg").FlagWithInputList("-Xbootclasspath:", module.PreoptBootClassPathDexFiles, ":").
Flag("--runtime-arg").FlagWithList("-Xbootclasspath-locations:", module.PreoptBootClassPathDexLocations, ":").
Flag("${class_loader_context_arg}").
Simplify the construction of class loader contexts for system server jars. This reworks CL https://r.android.com/1180134 as follows: 1) Do not reorder the list of system server jars passed from Make to Soong via the product variable PRODUCT_SYSTEM_SERVER_JARS. This means that for some products the order of jars on the system server classpath may be non-optimal: a jar X that depends on Y may be dexpreopted before Y, so that all references to the classes and methods from Y wil be unresolved. Unfortunately for such products, fixing the order is not a simple matter of rearranging their PRODUCT_SYSTEM_SERVER_JARS, because the conflicts may arise when the product-specific variable gets merged with the common variable. 2) As a consequence of 1), do not add artificial dependencies between system server jars: this is now impossible, as it would create circular dependencies for those products that have non-optimal order of jars. 3) Copy dex files for system server jars to a predefined build location. This is necessary because at the time when Soong generates class loader context for k-th jar, it needs to know the paths to jars 1 .. (k-1), and it might have not processed those jars yet (so Soong can't query the paths from the modules). This approach is similar to the way Soong handles bootclasspath jars. 4) Do not exclude from dexpreopting system server jars that are not Soong modules (those that are Make modules). The previous CL excluded them because Make modules do not have ModuleContext. But it turns out that ModuleContext is not necessary, as all the information is passed via the dexpreopt config. Test: aosp_walleye-userdebug boots and there are no messages in the logcat regarding class loader context mismatch: $ adb logcat | grep 'mismatch' # empty Test: Class loader contexts in the oat files for system server jars match expectations: $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/com.android.location.provider.odex 2>/dev/null | grep '^classpath' classpath = PCL[] $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/services.odex 2>/dev/null | grep '^classpath' classpath = PCL[/system/framework/com.android.location.provider.jar*1989208671] $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/ethernet-service.odex 2>/dev/null | grep '^classpath' classpath = PCL[/system/framework/com.android.location.provider.jar*1989208671:/system/framework/services.jar*4040443083:/system/framework/services.jar!classes2.dex*2210087472] Test: The phone boots and logcat has no scary messages related to class loader contexts: $ lunch aosp_walleye-userdebug && m $ adb reboot bootloader && fastboot flashall -w && adb wait-for-device $ adb root $ adb shell stop $ adb logcat -c $ adb shell setprop dalvik.vm.extra-opts -verbose:oat $ adb shell start $ adb logcat | egrep -io 'system_server: .*load.*/system/framework.*' system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 0 system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 0 system_server: Loading /system/framework/oat/arm64/services.odex with executable: 0 system_server: Successfully loaded /system/framework/oat/arm64/services.odex with executable: 0 system_server: Loading /system/framework/oat/arm64/ethernet-service.odex with executable: 0 system_server: Successfully loaded /system/framework/oat/arm64/ethernet-service.odex with executable: 0 system_server: Loading /system/framework/oat/arm64/wifi-service.odex with executable: 0 system_server: Successfully loaded /system/framework/oat/arm64/wifi-service.odex with executable: 0 system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1 system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1 system_server: Loading /system/framework/oat/arm64/services.odex with executable: 1 system_server: Successfully loaded /system/framework/oat/arm64/services.odex with executable: 1 system_server: Loading /system/framework/oat/arm64/ethernet-service.odex with executable: 1 system_server: Successfully loaded /system/framework/oat/arm64/ethernet-service.odex with executable: 1 system_server: Loading /system/framework/oat/arm64/wifi-service.odex with executable: 1 system_server: Successfully loaded /system/framework/oat/arm64/wifi-service.odex with executable: 1 system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1 system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1 Bug: 141785760 Bug: 140451054 Bug: 148944771 Bug: 147017252 Change-Id: I33c4087f8bfacd0ecb89877aa150b75360d06710 Merged-In: I33c4087f8bfacd0ecb89877aa150b75360d06710 (cherry picked from commit a4a83b0ef9b9769681ad285290fdbbf043eff3d5) Exempt-From-Owner-Approval: cherry-pick.
2020-03-11 12:59:34 +01:00
Flag("${stored_class_loader_context_arg}").
FlagWithArg("--boot-image=", strings.Join(module.DexPreoptImageLocationsOnHost, ":")).Implicits(module.DexPreoptImagesDeps[archIdx].Paths()).
FlagWithInput("--dex-file=", module.DexPath).
FlagWithArg("--dex-location=", dexLocationArg).
FlagWithOutput("--oat-file=", odexPath).ImplicitOutput(vdexPath).
// Pass an empty directory, dex2oat shouldn't be reading arbitrary files
FlagWithArg("--android-root=", global.EmptyDirectory).
FlagWithArg("--instruction-set=", arch.String()).
FlagWithArg("--instruction-set-variant=", global.CpuVariant[arch]).
FlagWithArg("--instruction-set-features=", global.InstructionSetFeatures[arch]).
Flag("--no-generate-debug-info").
Flag("--generate-build-id").
Flag("--abort-on-hard-verifier-error").
Flag("--force-determinism").
Determine GC type based on BUILT_KERNEL_VERSION_FILE. How it works: 1. build/make/core/Makefile generates a txt file with the kernel version, which is taken from an explicit BOARD_KERNEL_VERSION value, or extracted from the kernel image on the source tree, or extracted from the kernel image extracted from the prebuilt boot.img. The file is saved at $ANDROID_PRODUCT_OUT/obj/PACKAGING/check_vintf_all_intermediates/kernel_version.txt. 2. If PRODUCT_ENABLE_UFFD_GC is "default", meaning the GC type needs to be determined by the kernel version, build/make/core/Makefile copies kernel_version.txt to out/soong/dexpreopt/kernel_version_for_uffd_gc.txt. 3. build/soong/dexpreopt/config.go writes the the UFFD GC flag to out/soong/dexpreopt/uffd_gc_flag.txt. The flag is determined by an explicit PRODUCT_ENABLE_UFFD_GC value or by contruct_uffd_gc_flag.py, which reads kernel_version_for_uffd_gc.txt and determines the flag accordingly. 4. dex2oat takes the UFFD GC flag from uffd_gc_flag.txt. 5. post_process_props.py mangles ro.dalvik.vm.enable_uffd_gc based on the same logic. Bug: 321751629 Bug: 319554951 Bug: 318763448 Bug: 319648491 Test: m --no-skip-soong-tests nothing Test: atest uffd_gc_utils_test Test: Build with `OVERRIDE_ENABLE_UFFD_GC=default m` for device with no UFFD support - 1. Check the existence of `-Xgc:CMC` in out/soong/dexpreopt_arm64/dex_bootjars/android/system/framework/arm64/boot.invocation (dex2oat invocation for a boot image) 2. Check the existence of `-Xgc:CMC` in out/soong/.intermediates/packages/apps/Settings/Settings/android_common/dexpreopt/Settings/oat/arm64/package.invocation (dex2oat invocation for an app defined in .bp) 3. Check the existence of `-Xgc:CMC` in $ANDROID_PRODUCT_OUT/obj/APPS/Dialer_intermediates/oat/arm64/package.invocation (dex2oat invocation for an app defined in .mk) 4. Check the value of ro.dalvik.vm.enable_uffd_gc in $ANDROID_PRODUCT_OUT/product/etc/build.prop Test: Build with `OVERRIDE_ENABLE_UFFD_GC=default m` for device with UFFD support, and do the steps above. Test: Build with `OVERRIDE_ENABLE_UFFD_GC=true m`, and do the steps above. Test: Build with `OVERRIDE_ENABLE_UFFD_GC=false m`, and do the steps above. Change-Id: I035ad32233e49e2a30ce11f6c7c318a648511ef8
2024-01-18 18:27:42 +01:00
FlagWithArg("--no-inline-from=", "core-oj.jar").
Text("$(cat").Input(globalSoong.UffdGcFlag).Text(")")
var preoptFlags []string
if len(module.PreoptFlags) > 0 {
preoptFlags = module.PreoptFlags
} else if len(global.PreoptFlags) > 0 {
preoptFlags = global.PreoptFlags
}
if len(preoptFlags) > 0 {
cmd.Text(strings.Join(preoptFlags, " "))
}
if module.UncompressedDex {
cmd.FlagWithArg("--copy-dex-files=", "false")
}
if !android.PrefixInList(preoptFlags, "--compiler-filter=") {
var compilerFilter string
if systemServerJars.ContainsJar(module.Name) {
if global.SystemServerCompilerFilter != "" {
// Use the product option if it is set.
compilerFilter = global.SystemServerCompilerFilter
} else if profile != nil {
// Use "speed-profile" for system server jars that have a profile.
compilerFilter = "speed-profile"
} else {
// Use "speed" for system server jars that do not have a profile.
compilerFilter = "speed"
}
} else if contains(global.SpeedApps, module.Name) || contains(global.SystemServerApps, module.Name) {
// Apps loaded into system server, and apps the product default to being compiled with the
// 'speed' compiler filter.
compilerFilter = "speed"
} else if profile != nil {
// For non system server jars, use speed-profile when we have a profile.
compilerFilter = "speed-profile"
} else if global.DefaultCompilerFilter != "" {
compilerFilter = global.DefaultCompilerFilter
} else {
compilerFilter = "quicken"
}
Add non-fatal mode for verify_uses_libraries check. The new mode is enabled with environment variable RELAX_USES_LIBRARY_CHECK. If the variable is set to true, then a verify_uses_libraries check failure does not fail the build, instead it sets a special compiler filter "extract" for dexpreopt, which means that the DEX file will be extracted, but it won't be compiled to native code. Class loader context will be set to empty in this case (not &, as it is going to be deprecated soon). If the variable RELAX_USES_LIBRARY_CHECK is unset or set to something other than "true", then the old behaviour of the verify_uses_libraries check is preserved. The intended use case for this flag is to have a smoother migration path for the Java modules that need to add <uses-library> information in the build files. The flag allows to quickly silence build errors. This flag should be used with caution and only as a temporary measure, as it masks real errors and affects performance. verify_uses_libraries check is reworked so that it writes the error message to a status file (which is used instead of the former timestamp file). Currently the stored error message is not used, but it may be used later to produce a warning. Dexpreopt command checks if the status file exists and is nonempty; if that is the case, then compiler filter is set to "extract". Bug: 132357300 Test: Manually add some mismatch between the libraries in the Android.bp and Android.mk files for dexpreopted apps, build with RELAX_USES_LIBRARY_CHECK=true and obsserve that the build doesn't fail and they are compiled with compiler-filter "extract". Unset RELAX_USES_LIBRARY_CHECK and observe that the build fails. Change-Id: Ibb5d993a25b1df1d2e70b7d5aafc6997f9d64e67
2021-02-17 17:23:28 +01:00
if module.EnforceUsesLibraries {
// If the verify_uses_libraries check failed (in this case status file contains a
// non-empty error message), then use "verify" compiler filter to avoid compiling any
Add non-fatal mode for verify_uses_libraries check. The new mode is enabled with environment variable RELAX_USES_LIBRARY_CHECK. If the variable is set to true, then a verify_uses_libraries check failure does not fail the build, instead it sets a special compiler filter "extract" for dexpreopt, which means that the DEX file will be extracted, but it won't be compiled to native code. Class loader context will be set to empty in this case (not &, as it is going to be deprecated soon). If the variable RELAX_USES_LIBRARY_CHECK is unset or set to something other than "true", then the old behaviour of the verify_uses_libraries check is preserved. The intended use case for this flag is to have a smoother migration path for the Java modules that need to add <uses-library> information in the build files. The flag allows to quickly silence build errors. This flag should be used with caution and only as a temporary measure, as it masks real errors and affects performance. verify_uses_libraries check is reworked so that it writes the error message to a status file (which is used instead of the former timestamp file). Currently the stored error message is not used, but it may be used later to produce a warning. Dexpreopt command checks if the status file exists and is nonempty; if that is the case, then compiler filter is set to "extract". Bug: 132357300 Test: Manually add some mismatch between the libraries in the Android.bp and Android.mk files for dexpreopted apps, build with RELAX_USES_LIBRARY_CHECK=true and obsserve that the build doesn't fail and they are compiled with compiler-filter "extract". Unset RELAX_USES_LIBRARY_CHECK and observe that the build fails. Change-Id: Ibb5d993a25b1df1d2e70b7d5aafc6997f9d64e67
2021-02-17 17:23:28 +01:00
// code (it would be rejected on device because of a class loader context mismatch).
cmd.Text("--compiler-filter=$(if test -s ").
Input(module.EnforceUsesLibrariesStatusFile).
Text(" ; then echo verify ; else echo " + compilerFilter + " ; fi)")
Add non-fatal mode for verify_uses_libraries check. The new mode is enabled with environment variable RELAX_USES_LIBRARY_CHECK. If the variable is set to true, then a verify_uses_libraries check failure does not fail the build, instead it sets a special compiler filter "extract" for dexpreopt, which means that the DEX file will be extracted, but it won't be compiled to native code. Class loader context will be set to empty in this case (not &, as it is going to be deprecated soon). If the variable RELAX_USES_LIBRARY_CHECK is unset or set to something other than "true", then the old behaviour of the verify_uses_libraries check is preserved. The intended use case for this flag is to have a smoother migration path for the Java modules that need to add <uses-library> information in the build files. The flag allows to quickly silence build errors. This flag should be used with caution and only as a temporary measure, as it masks real errors and affects performance. verify_uses_libraries check is reworked so that it writes the error message to a status file (which is used instead of the former timestamp file). Currently the stored error message is not used, but it may be used later to produce a warning. Dexpreopt command checks if the status file exists and is nonempty; if that is the case, then compiler filter is set to "extract". Bug: 132357300 Test: Manually add some mismatch between the libraries in the Android.bp and Android.mk files for dexpreopted apps, build with RELAX_USES_LIBRARY_CHECK=true and obsserve that the build doesn't fail and they are compiled with compiler-filter "extract". Unset RELAX_USES_LIBRARY_CHECK and observe that the build fails. Change-Id: Ibb5d993a25b1df1d2e70b7d5aafc6997f9d64e67
2021-02-17 17:23:28 +01:00
} else {
cmd.FlagWithArg("--compiler-filter=", compilerFilter)
}
}
if generateDM {
cmd.FlagWithArg("--copy-dex-files=", "false")
dmPath := module.BuildPath.InSameDir(ctx, "generated.dm")
dmInstalledPath := pathtools.ReplaceExtension(module.DexLocation, "dm")
tmpPath := module.BuildPath.InSameDir(ctx, "primary.vdex")
rule.Command().Text("cp -f").Input(vdexPath).Output(tmpPath)
rule.Command().Tool(globalSoong.SoongZip).
FlagWithArg("-L", "9").
FlagWithOutput("-o", dmPath).
Flag("-j").
Input(tmpPath)
rule.Install(dmPath, dmInstalledPath)
}
// By default, emit debug info.
debugInfo := true
if global.NoDebugInfo {
// If the global setting suppresses mini-debug-info, disable it.
debugInfo = false
}
// PRODUCT_SYSTEM_SERVER_DEBUG_INFO overrides WITH_DEXPREOPT_DEBUG_INFO.
// PRODUCT_OTHER_JAVA_DEBUG_INFO overrides WITH_DEXPREOPT_DEBUG_INFO.
if systemServerJars.ContainsJar(module.Name) {
if global.AlwaysSystemServerDebugInfo {
debugInfo = true
} else if global.NeverSystemServerDebugInfo {
debugInfo = false
}
} else {
if global.AlwaysOtherDebugInfo {
debugInfo = true
} else if global.NeverOtherDebugInfo {
debugInfo = false
}
}
if debugInfo {
cmd.Flag("--generate-mini-debug-info")
} else {
cmd.Flag("--no-generate-mini-debug-info")
}
// Set the compiler reason to 'prebuilt' to identify the oat files produced
// during the build, as opposed to compiled on the device.
cmd.FlagWithArg("--compilation-reason=", "prebuilt")
if appImage {
appImagePath := odexPath.ReplaceExtension(ctx, "art")
appImageInstallPath := pathtools.ReplaceExtension(odexInstallPath, "art")
cmd.FlagWithOutput("--app-image-file=", appImagePath).
FlagWithArg("--image-format=", "lz4")
if !global.DontResolveStartupStrings {
cmd.FlagWithArg("--resolve-startup-const-strings=", "true")
}
rule.Install(appImagePath, appImageInstallPath)
}
if profile != nil {
cmd.FlagWithInput("--profile-file=", profile)
}
rule.Install(odexPath, odexInstallPath)
rule.Install(vdexPath, vdexInstallPath)
}
func shouldGenerateDM(module *ModuleConfig, global *GlobalConfig) bool {
// Generating DM files only makes sense for verify, avoid doing for non verify compiler filter APKs.
// No reason to use a dm file if the dex is already uncompressed.
return global.GenerateDMFiles && !module.UncompressedDex &&
contains(module.PreoptFlags, "--compiler-filter=verify")
}
func OdexOnSystemOtherByName(name string, dexLocation string, global *GlobalConfig) bool {
if !global.HasSystemOther {
return false
}
if global.SanitizeLite {
return false
}
if contains(global.SpeedApps, name) || contains(global.SystemServerApps, name) {
return false
}
for _, f := range global.PatternsOnSystemOther {
if makefileMatch(filepath.Join(SystemPartition, f), dexLocation) {
return true
}
}
return false
}
func odexOnSystemOther(module *ModuleConfig, global *GlobalConfig) bool {
return OdexOnSystemOtherByName(module.Name, module.DexLocation, global)
}
// PathToLocation converts .../system/framework/arm64/boot.art to .../system/framework/boot.art
func PathToLocation(path android.Path, arch android.ArchType) string {
return PathStringToLocation(path.String(), arch)
}
// PathStringToLocation converts .../system/framework/arm64/boot.art to .../system/framework/boot.art
func PathStringToLocation(path string, arch android.ArchType) string {
pathArch := filepath.Base(filepath.Dir(path))
if pathArch != arch.String() {
panic(fmt.Errorf("last directory in %q must be %q", path, arch.String()))
}
return filepath.Join(filepath.Dir(filepath.Dir(path)), filepath.Base(path))
}
func makefileMatch(pattern, s string) bool {
percent := strings.IndexByte(pattern, '%')
switch percent {
case -1:
return pattern == s
case len(pattern) - 1:
return strings.HasPrefix(s, pattern[:len(pattern)-1])
default:
panic(fmt.Errorf("unsupported makefile pattern %q", pattern))
}
}
Simplify the construction of class loader contexts for system server jars. This reworks CL https://r.android.com/1180134 as follows: 1) Do not reorder the list of system server jars passed from Make to Soong via the product variable PRODUCT_SYSTEM_SERVER_JARS. This means that for some products the order of jars on the system server classpath may be non-optimal: a jar X that depends on Y may be dexpreopted before Y, so that all references to the classes and methods from Y wil be unresolved. Unfortunately for such products, fixing the order is not a simple matter of rearranging their PRODUCT_SYSTEM_SERVER_JARS, because the conflicts may arise when the product-specific variable gets merged with the common variable. 2) As a consequence of 1), do not add artificial dependencies between system server jars: this is now impossible, as it would create circular dependencies for those products that have non-optimal order of jars. 3) Copy dex files for system server jars to a predefined build location. This is necessary because at the time when Soong generates class loader context for k-th jar, it needs to know the paths to jars 1 .. (k-1), and it might have not processed those jars yet (so Soong can't query the paths from the modules). This approach is similar to the way Soong handles bootclasspath jars. 4) Do not exclude from dexpreopting system server jars that are not Soong modules (those that are Make modules). The previous CL excluded them because Make modules do not have ModuleContext. But it turns out that ModuleContext is not necessary, as all the information is passed via the dexpreopt config. Test: aosp_walleye-userdebug boots and there are no messages in the logcat regarding class loader context mismatch: $ adb logcat | grep 'mismatch' # empty Test: Class loader contexts in the oat files for system server jars match expectations: $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/com.android.location.provider.odex 2>/dev/null | grep '^classpath' classpath = PCL[] $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/services.odex 2>/dev/null | grep '^classpath' classpath = PCL[/system/framework/com.android.location.provider.jar*1989208671] $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/ethernet-service.odex 2>/dev/null | grep '^classpath' classpath = PCL[/system/framework/com.android.location.provider.jar*1989208671:/system/framework/services.jar*4040443083:/system/framework/services.jar!classes2.dex*2210087472] Test: The phone boots and logcat has no scary messages related to class loader contexts: $ lunch aosp_walleye-userdebug && m $ adb reboot bootloader && fastboot flashall -w && adb wait-for-device $ adb root $ adb shell stop $ adb logcat -c $ adb shell setprop dalvik.vm.extra-opts -verbose:oat $ adb shell start $ adb logcat | egrep -io 'system_server: .*load.*/system/framework.*' system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 0 system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 0 system_server: Loading /system/framework/oat/arm64/services.odex with executable: 0 system_server: Successfully loaded /system/framework/oat/arm64/services.odex with executable: 0 system_server: Loading /system/framework/oat/arm64/ethernet-service.odex with executable: 0 system_server: Successfully loaded /system/framework/oat/arm64/ethernet-service.odex with executable: 0 system_server: Loading /system/framework/oat/arm64/wifi-service.odex with executable: 0 system_server: Successfully loaded /system/framework/oat/arm64/wifi-service.odex with executable: 0 system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1 system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1 system_server: Loading /system/framework/oat/arm64/services.odex with executable: 1 system_server: Successfully loaded /system/framework/oat/arm64/services.odex with executable: 1 system_server: Loading /system/framework/oat/arm64/ethernet-service.odex with executable: 1 system_server: Successfully loaded /system/framework/oat/arm64/ethernet-service.odex with executable: 1 system_server: Loading /system/framework/oat/arm64/wifi-service.odex with executable: 1 system_server: Successfully loaded /system/framework/oat/arm64/wifi-service.odex with executable: 1 system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1 system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1 Bug: 141785760 Bug: 140451054 Bug: 148944771 Bug: 147017252 Change-Id: I33c4087f8bfacd0ecb89877aa150b75360d06710 Merged-In: I33c4087f8bfacd0ecb89877aa150b75360d06710 (cherry picked from commit a4a83b0ef9b9769681ad285290fdbbf043eff3d5) Exempt-From-Owner-Approval: cherry-pick.
2020-03-11 12:59:34 +01:00
// A predefined location for the system server dex jars. This is needed in order to generate
// class loader context for dex2oat, as the path to the jar in the Soong module may be unknown
// at that time (Soong processes the jars in dependency order, which may be different from the
// the system server classpath order).
func SystemServerDexJarHostPath(ctx android.PathContext, jar string) android.OutputPath {
if DexpreoptRunningInSoong {
// Soong module, just use the default output directory $OUT/soong.
return android.PathForOutput(ctx, "system_server_dexjars", jar+".jar")
} else {
// Make module, default output directory is $OUT (passed via the "null config" created
// by dexpreopt_gen). Append Soong subdirectory to match Soong module paths.
return android.PathForOutput(ctx, "soong", "system_server_dexjars", jar+".jar")
}
Simplify the construction of class loader contexts for system server jars. This reworks CL https://r.android.com/1180134 as follows: 1) Do not reorder the list of system server jars passed from Make to Soong via the product variable PRODUCT_SYSTEM_SERVER_JARS. This means that for some products the order of jars on the system server classpath may be non-optimal: a jar X that depends on Y may be dexpreopted before Y, so that all references to the classes and methods from Y wil be unresolved. Unfortunately for such products, fixing the order is not a simple matter of rearranging their PRODUCT_SYSTEM_SERVER_JARS, because the conflicts may arise when the product-specific variable gets merged with the common variable. 2) As a consequence of 1), do not add artificial dependencies between system server jars: this is now impossible, as it would create circular dependencies for those products that have non-optimal order of jars. 3) Copy dex files for system server jars to a predefined build location. This is necessary because at the time when Soong generates class loader context for k-th jar, it needs to know the paths to jars 1 .. (k-1), and it might have not processed those jars yet (so Soong can't query the paths from the modules). This approach is similar to the way Soong handles bootclasspath jars. 4) Do not exclude from dexpreopting system server jars that are not Soong modules (those that are Make modules). The previous CL excluded them because Make modules do not have ModuleContext. But it turns out that ModuleContext is not necessary, as all the information is passed via the dexpreopt config. Test: aosp_walleye-userdebug boots and there are no messages in the logcat regarding class loader context mismatch: $ adb logcat | grep 'mismatch' # empty Test: Class loader contexts in the oat files for system server jars match expectations: $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/com.android.location.provider.odex 2>/dev/null | grep '^classpath' classpath = PCL[] $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/services.odex 2>/dev/null | grep '^classpath' classpath = PCL[/system/framework/com.android.location.provider.jar*1989208671] $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/ethernet-service.odex 2>/dev/null | grep '^classpath' classpath = PCL[/system/framework/com.android.location.provider.jar*1989208671:/system/framework/services.jar*4040443083:/system/framework/services.jar!classes2.dex*2210087472] Test: The phone boots and logcat has no scary messages related to class loader contexts: $ lunch aosp_walleye-userdebug && m $ adb reboot bootloader && fastboot flashall -w && adb wait-for-device $ adb root $ adb shell stop $ adb logcat -c $ adb shell setprop dalvik.vm.extra-opts -verbose:oat $ adb shell start $ adb logcat | egrep -io 'system_server: .*load.*/system/framework.*' system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 0 system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 0 system_server: Loading /system/framework/oat/arm64/services.odex with executable: 0 system_server: Successfully loaded /system/framework/oat/arm64/services.odex with executable: 0 system_server: Loading /system/framework/oat/arm64/ethernet-service.odex with executable: 0 system_server: Successfully loaded /system/framework/oat/arm64/ethernet-service.odex with executable: 0 system_server: Loading /system/framework/oat/arm64/wifi-service.odex with executable: 0 system_server: Successfully loaded /system/framework/oat/arm64/wifi-service.odex with executable: 0 system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1 system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1 system_server: Loading /system/framework/oat/arm64/services.odex with executable: 1 system_server: Successfully loaded /system/framework/oat/arm64/services.odex with executable: 1 system_server: Loading /system/framework/oat/arm64/ethernet-service.odex with executable: 1 system_server: Successfully loaded /system/framework/oat/arm64/ethernet-service.odex with executable: 1 system_server: Loading /system/framework/oat/arm64/wifi-service.odex with executable: 1 system_server: Successfully loaded /system/framework/oat/arm64/wifi-service.odex with executable: 1 system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1 system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1 Bug: 141785760 Bug: 140451054 Bug: 148944771 Bug: 147017252 Change-Id: I33c4087f8bfacd0ecb89877aa150b75360d06710 Merged-In: I33c4087f8bfacd0ecb89877aa150b75360d06710 (cherry picked from commit a4a83b0ef9b9769681ad285290fdbbf043eff3d5) Exempt-From-Owner-Approval: cherry-pick.
2020-03-11 12:59:34 +01:00
}
// Check the order of jars on the system server classpath and give a warning/error if a jar precedes
// one of its dependencies. This is not an error, but a missed optimization, as dexpreopt won't
// have the dependency jar in the class loader context, and it won't be able to resolve any
// references to its classes and methods.
func checkSystemServerOrder(ctx android.PathContext, jarIndex int) {
mctx, isModule := ctx.(android.ModuleContext)
if isModule {
config := GetGlobalConfig(ctx)
jars := config.AllSystemServerClasspathJars(ctx)
mctx.WalkDeps(func(dep android.Module, parent android.Module) bool {
depIndex := jars.IndexOfJar(dep.Name())
if jarIndex < depIndex && !config.BrokenSuboptimalOrderOfSystemServerJars {
jar := jars.Jar(jarIndex)
dep := jars.Jar(depIndex)
mctx.ModuleErrorf("non-optimal order of jars on the system server classpath:"+
" '%s' precedes its dependency '%s', so dexpreopt is unable to resolve any"+
" references from '%s' to '%s'.\n", jar, dep, jar, dep)
}
return true
})
}
}
Add non-fatal mode for verify_uses_libraries check. The new mode is enabled with environment variable RELAX_USES_LIBRARY_CHECK. If the variable is set to true, then a verify_uses_libraries check failure does not fail the build, instead it sets a special compiler filter "extract" for dexpreopt, which means that the DEX file will be extracted, but it won't be compiled to native code. Class loader context will be set to empty in this case (not &, as it is going to be deprecated soon). If the variable RELAX_USES_LIBRARY_CHECK is unset or set to something other than "true", then the old behaviour of the verify_uses_libraries check is preserved. The intended use case for this flag is to have a smoother migration path for the Java modules that need to add <uses-library> information in the build files. The flag allows to quickly silence build errors. This flag should be used with caution and only as a temporary measure, as it masks real errors and affects performance. verify_uses_libraries check is reworked so that it writes the error message to a status file (which is used instead of the former timestamp file). Currently the stored error message is not used, but it may be used later to produce a warning. Dexpreopt command checks if the status file exists and is nonempty; if that is the case, then compiler filter is set to "extract". Bug: 132357300 Test: Manually add some mismatch between the libraries in the Android.bp and Android.mk files for dexpreopted apps, build with RELAX_USES_LIBRARY_CHECK=true and obsserve that the build doesn't fail and they are compiled with compiler-filter "extract". Unset RELAX_USES_LIBRARY_CHECK and observe that the build fails. Change-Id: Ibb5d993a25b1df1d2e70b7d5aafc6997f9d64e67
2021-02-17 17:23:28 +01:00
// Returns path to a file containing the reult of verify_uses_libraries check (empty if the check
// has succeeded, or an error message if it failed).
func UsesLibrariesStatusFile(ctx android.ModuleContext) android.WritablePath {
return android.PathForModuleOut(ctx, "enforce_uses_libraries.status")
}
func contains(l []string, s string) bool {
for _, e := range l {
if e == s {
return true
}
}
return false
}