Previously, we preopt system server jars without updatable bootclasspath
in the build system so that the artifacts can keep valid after a
mainline update. Things have been changed after then. We now re-compile
system server jars on the user's device after a mainline update
(b/189467174), so we no longer need the artifacts generated at build
time to keep valid. Therefore, we can preopt system server jars with
updatable bootclasspath to achieve better performance.
Bug: 199573699
Test: m services
Test: manual - 1. Build a system image and flash it into a device.
2. adb root && adb shell grep "services" /proc/`adb shell pidof system_server`/maps
3. See "/system/framework/oat/x86_64/services.{odex,vdex,art} in the
output.
4. adb logcat -d odrefresh:D odsign:D "*:S"
5. See "odrefresh said artifacts are VALID" and no dex2oat invocation
in the output.
Change-Id: I9c94a54aa15c46708067f4bc89ff05bc7af32a7b
Similar to the previous CL, but for jars from prebuilts.
Test: manual - 1. Patch aosp/1818020 and aosp/1810840.
2. m SOONG_CONFIG_art_module_source_build=false com.android.art
Bug: 194150908
Change-Id: Idc0c73a78045a602ad3a91cb5071d291bd611015
The path to the artifacts will in the form of
/system/framework/oat/<arch>/<encoded-jar-path>@classes.{odex,vdex,art},
where <encoded-jar-path> is the path to the jar file with "/" replaced
by "@". For example,
/system/framework/oat/x86_64/apex@com.android.art@javalib@service-art.jar@classes.odex
There will be a follow-up CL to update ART runtime to recognize
artifacts in that path.
Test: m com.android.art
Bug: 194150908
Change-Id: Ic89fd63c4b1cd565684cead83fc91dae3bc97a4c
Bug: 188179858
Test: compare dexpreopt_config.zip files from
1. TARGET_BUILD_UNBUNDLED_IMAGE=true m dexpreopt_config_zip
2. m dexpreopt_config_zip
(note that m clean should run between steps)
Change-Id: I36a6e8b10b9922cc5522accaf90af1aa05049a86
For running dex2oat on the target_files, the paths should be use the
device install path instead of the path starting with $(OUT).
So add usesTargetFiles option and basePath option which indicates
extracted path. With those options, the path is replaced with
$(basePath)/$(device path)
And also, add DexPreoptImageDeviceLocations in the config which refers
to the boot image path(without arch) on the device. Because
DexPreoptImage related device path was missing.
Bug: 158843648
Test: dexpreopt_gen -usesTargetFiles -basePath (extract path) and then
check if paths in the generated shell script are based on on-device
path.
Change-Id: I9667fadbf3b7c6f770e0d1bcbee5d67c1ecd8a3d
Add 'host' into the name of fields regarding path on the host side to
distinguish between paths on the device(which will be added in the
following commit), and paths on the host.
Bug: 158843648
Test: build and flash, and then
adb wait-for-device \
&& adb -s $S root \
&& adb -s $S logcat \
| grep -E 'ClassLoaderContext [a-z ]+ mismatch' -C 1
Change-Id: Ib2645ed51591ba2f4b726c115b401ad2bd6675da
Consistent with plumbing of boot jars. SystemServerJars only support /system/framework/ jars.
Bug: 180105615, 155630745
Test: m && launch_cvd
Change-Id: I58b005b7c4103c8e250090e995b1d9b2f9ed4a76
The field isn't used anymore.
And DexPreoptImagesDeps will have the slice of which size is the same as
Archs to avoid an error.
Bug: 158843648
Test: m
Change-Id: I520063ff7376811febbc82e1a0a43785feb5bbb2
1. Instead of 'slim' config, use full config even for libs
2. Define moduleJSONConfig for fields which cannot be converted to JSON
field directly(Path type field, ProfileBootListing,
DexPreoptImagesDeps are added in this CL) and exclude fields which is
convertible(DexPreoptImageLocations)
Bug: 158843648
Test: m dist
Change-Id: I3f9192ab5292bd079be1b686bb3b25735a836cbc
This CL handles updatable boot jars in the same hacky way as we handle
non-updatable boot jars: it creates a set of predefined paths to the dex
jars in a global config, then traverses all modules in a singleton
context, finds updatable boot jars and adds copy rules from these jars
to the predefined paths. A proper way would be to register dependencies
of the dexpreopted modules on the boot jars and extracting paths to dex
files by walking these dependencies.
Bug: 178467404
Test: lunch aosp_cf_x86_64_phone-userdebug && m
Test: added new Soong test
Change-Id: I87f764109315f79315d73bf43799b70eb010fc0b
The new mode is enabled with environment variable
RELAX_USES_LIBRARY_CHECK. If the variable is set to true, then a
verify_uses_libraries check failure does not fail the build, instead it
sets a special compiler filter "extract" for dexpreopt, which means that
the DEX file will be extracted, but it won't be compiled to native code.
Class loader context will be set to empty in this case (not &, as it is
going to be deprecated soon).
If the variable RELAX_USES_LIBRARY_CHECK is unset or set to something
other than "true", then the old behaviour of the verify_uses_libraries
check is preserved.
The intended use case for this flag is to have a smoother migration path
for the Java modules that need to add <uses-library> information in
the build files. The flag allows to quickly silence build errors. This
flag should be used with caution and only as a temporary measure, as it
masks real errors and affects performance.
verify_uses_libraries check is reworked so that it writes the error
message to a status file (which is used instead of the former timestamp
file). Currently the stored error message is not used, but it may be
used later to produce a warning. Dexpreopt command checks if the status
file exists and is nonempty; if that is the case, then compiler filter
is set to "extract".
Bug: 132357300
Test: Manually add some mismatch between the libraries in the Android.bp
and Android.mk files for dexpreopted apps, build with
RELAX_USES_LIBRARY_CHECK=true and obsserve that the build doesn't
fail and they are compiled with compiler-filter "extract".
Unset RELAX_USES_LIBRARY_CHECK and observe that the build fails.
Change-Id: Ibb5d993a25b1df1d2e70b7d5aafc6997f9d64e67
A path to an nonexistent file caused dex2oat to write an empty path in
the class loader context stored inside of the OAT/ODEX file, which
resulted in class loader context mistmatch and a failure to load
dexpreopt code on device.
An example of a faulty class loader context for GoogleDialer, can be
seen either with oatdump or by grepping logcat for class loader context
mimatch messages (the first empty PCL[] is expected, note the second
empty PCL[] in the subcontext for shared libraries):
PCL[]{PCL[]#PCL[/system/framework/org.apache.http.legacy.jar*22205474]
This CL removes the nonexistent path from the generated dexpreopt.config
file, which works in conjunction with a change in makefiles that stops
using the removed config field. This is possible because makefiles can
access DEX files using a module name and "intermediates-dir-for" macro.
Bug: 132357300
Test: lunch cf_x86_64_phone-userdebug && m && launch_cvd \
adb wait-for-device && adb root && adb logcat \
| grep -E 'ClassLoaderContext [a-z ]+ mismatch'
# empty grep output, no errors
Change-Id: I8d16f77c6f76ad6234cc4114ad3545cbe572ef13
In order to allow the hiddenapi code which expects its inputs to be
ModuleOutPath to consume the output from the deapexer module type (used
by prebuilt_apex) this change converts the hiddenapi from using
ModuleOutPath to use OutputPath instead.
This is part of a general cleanup to remove ModuleOutPath that is
separated out in order to avoid having the hiddenapi work dependent
upon and possibly delayed by issues with that general cleanup.
Bug: 178361284
Bug: 179124768
Test: m droid
Change-Id: I890f775bf5e33528dbaa62b95fa4198185ff8bf8
This is needed for Java libraries that are <uses-library> dependencies
of Java libraries and apps defined as Make modules. Each dexpreopted
module in Make generates a dexpreopt.config file, which incorporates
information from its dependencies' dexpreopt.config files. For
dependencies that are Make modules their dexpreopt.config files are
generated by Make, and for Soong modules they are generated by Soong.
Since Soong doesn't know which libraries are used by Make, it generates
build rules for a superset of the necessary libraries.
Bug: 132357300
Test: lunch aosp_cf_x86_phone-userdebug && m
Change-Id: I325b1037658736ee3c02450b08c00eca1a175962
dexpreopt doesn't strip dex files from the input jar anymore, so there's
no point returning the input jar path untouched.
Test: m nothing + TreeHugger
Change-Id: I8fde6cdc19e85a2fbc946513696417b945c5de0f
Enable the RuleBuilder and RuleBuilderCommand methods to access
the BuilderContext by passing it to NewRuleBuilder instead of
RuleBuilder.Build.
Test: genrule_test.go
Test: rule_builder_test.go
Test: m checkbuild
Change-Id: I63e6597e19167393876dc2259d6f521363b7dabc
The old representation consisted of a list of libraries (UsesLibraries),
a list of optional libraries (OptionalUsesLibraries) and a mapping from
library name to its build/install paths (LibraryPaths). The separation
into lists and map was necessary because of special handling of
compatibility libraries, which is now unified with normal libraries.
The new representation is a mapping from target SDK version to a tree
structure ClassLoaderContext. Each node of the tree represents a library
and contains library name, build/install paths and a slice of
subcontexts for dependencies. The same library may occur in the tree
multiple times in case it is a dependency of multiple libraries. The
order in which libraries are added matters (the resulting tree shape may
be different).
Test results have to be updated, as the resulting <uses-library> list is
reodered (previously it was a sorted list of map keys, and now it is
formed by a depth-first preorder traversal of the class loader tree).
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Bug: 168686456
Change-Id: I11be8cd2967f004fd58753d7c5fb99fed179cd63
A global variant was used to store the global mapping between
modules and APEXes. Replace it with storing pointers to APEX
contents inside each module so that they can query the contents
of any APEXes they belong to.
Bug: 146393795
Test: all Soong tests
Test: single line change to build.ninja host install dependency ordering
Test: no Android-${TARGET_PRODUCT}.mk, make_vars-${TARGET_PRODUCT}.mk or late-${TARGET_PRODUCT}.mk
Change-Id: Id2d7b73ea27f8c3b41d30820bdd86b65c539bfa4
Right now, it's looking for *.prof files at the top of the source
directory.
For devices that don't specify the profile dir (everything on AOSP,
etc), this saves a couple percent of the globs that we check during
every build (and would re-execute if you add or remove files from the
top directory)
Bug: 159845846
Test: treehugger
Change-Id: I10d43422a2b5ae25e6557d435ecc89fe43536dd7
Different names of the same field in Make and in Soong resulted in
the field being dropped when unmarshalling the JSON from Make to Soong.
As a result, the list of optional used libraries for Make modules was
always empty.
Test: lunch aosp_cf_x86_phone-userdebug && m
Test: Cherry-pick in internal master and observe that the logcat error
"ClassLoaderContext shared library size mismatch" for GoogleDialer
has disappeared:
terminal-1 $ lunch cf_x86_phone-userdebug && m && launch_cvd
terminal-2 $ adb wait-for-device && adb root \
&& adb logcat | grep 'size mismatch'
(nothing after the change, one error message before the change)
Bug: 132357300
Change-Id: I57820646fa7c1d8af01b2cbc8a3521da450719c0
Test: art/test/testrunner/run_build_test_target.py art-linux-bionic-x64
with TARGET_BUILD_UNBUNDLED enabled in the build scripts
Test: art/test/testrunner/testrunner.py --target --64 --optimizing
with TARGET_BUILD_UNBUNDLED enabled in chroot on device
Bug: 157549171
Change-Id: I124c341d1801c4afa22f836dd567019de3e9498e
This is a preliminary CL before fixing on-device paths to DEX jars.
It groups together the inormation about on-host build paths and
on-device install paths to library DEX jars.
This CL changes the structure of module dexpreopt.config files
generated by the build system. Aside of that, no functional changes.
Test: lunch aosp_cf_x86_phone-userdebug && m
Change-Id: I059654be7670f2ba66248d9c49b9694a0591f9c1
Previously there was a discrepancy between the generated paths for
Soong modules that used "out/soong/system_server_jars" and Make modules
that used "out/system_server_jars". This happened because the default
output directory is $OUT/soong for the normal Soong config and jsut $OUT
for the reduced "null config" created by dexpreopt_gen.
As a result, class loader context for system server jars defined in Make
was referring to nonexistent jars, which caused non-fatal dex2oat errors
(the build was not broken, so the errors remained unnoticed):
Failed to determine oat file name for dex location out/system_server_dexjars/[...]
Could not open dex files from location: out/system_server_dexjars/[...]
Test: aosp_walleye-userdebug boots
Test: cherry-picked in master-arc-dev that has a Make system server jar
arc-services, `lunch lunch cheets_x86-userdebug && m`, check that the
generated dexpreopt.sh for arc-services contains paths starting with
"out/soong/system_server_jars" rather than "out/system_server_jars".
Bug: 140712566
Change-Id: Ia7ea8ac383e32042c31d64971cdc8101ea3068cd
ART tests require that "out/host/linux-x86" is included.
This requires making the dexLocations per-variant specific.
Test: m test-art-host-gtest
Bug: 147817558
Bug: 147819342
Change-Id: I7839ee15cb6dfc62508bdd3fa0f306336af17055
Implement support of dumping host boot.art.
That was not possible before because imageLocation is different on host.
This CL makes imageLocation per-variant so can access for the host variant.
Test: m dump-oat-boot
Change-Id: Id82161e4901a6b77f220238aec460b8609fd765f
We plan to add boot image variants for host tests.
Distinguishing the variants by arch does not work,
since both host and device can have the same arch.
Change-Id: Iea73c77367affb074f97a0fc318389417ce537da
Also disable dexpreopting for host. These are necessary to avoid adding
dependencies on dex2oat in various non-platform builds where they will
break.
Since we cannot assume at least one module enables dexpreopting now,
the two dexpreopt singletons are silently disabled if there has been no
call to dexpreopt.GetGlobalSoongConfig.
Bug: 145934348
Bug: 148312086
Bug: 148319588
Bug: 148690468
Test: m
Test: env OUT_DIR=out-tools prebuilts/build-tools/build-prebuilts.sh
on the aosp-build-tools branch
Test: build/soong/soong_ui.bash --make-mode static_sdk_tools dist DIST_DIR=out-dist BUILD_HOST_static=1
on internal (cf b/148312086#comment8)
Test: build/soong/soong_ui.bash --make-mode dist DIST_DIR=out-apps TARGET_BUILD_APPS=Launcher3 TARGET_BUILD_VARIANT=userdebug
on internal without art/ and external/vixl/ (cf b/148319588)
Change-Id: I240dade7204b87fc2d12181534ab23439eca8b46
Should cut down on a bit of copying, and also required for an upcoming
CL that'll change GetCachedGlobalSoongConfig.
Test: m nothing
Bug: 145934348
Change-Id: I6bed737d9b061b5239cc603ad881f4ccd4312e43
dexpreopt package.
Preparation for a future CL that will need to get the make-written
GlobalConfig from within dexpreopt.
Also rename the Load*Config functions to Parse*Config, since they don't
actually load the config files anymore.
This relands https://r.android.com/1211982.
Bug: 145934348
Test: m
Change-Id: Icb9332a93811d77d2d8b06e983b92501b180a358
caching of it.
Introduce a Once cache for GlobalSoongConfig to allow it to get binary
tool paths from ordinary module dependencies (coming in a future CL)
that are then reused in singletons.
This relands https://r.android.com/1205729.
Bug: 145934348
Test: m
Change-Id: I039d6e204bee5ddc16d8e2d85057fbec20e326fe
Framework and other dex files are used without image.
Test: taimen-userdebug boots when built with
DEXPREOPT_USE_ART_IMAGE=true
Test: Check logcat for checksum verification failures.
(Build ART with extra logging in OatFileAssistant.)
Test: Check that bootclasspath-checksums from some prebuilt
oat files (say input.odex) contain only one image
checksum followed by dex file checksums with
grep -az -A1 -E '^bootclasspath-checksums$' <oat-file> | \
xargs -0 echo | gawk '{print $2}'
Bug: 119800099
Change-Id: I65c2f247656e41f2c37df1ecb9e06af7dabab76e
Revert submission 1211982-dex2oat-soong-dep
Reason for revert: Build failures. See b/148312086.
Reverted Changes:
Ibc427a9a8: Make dex2oat(d) visible for use as implicit dexpre...
I71df11c1e: Move the Once cache for dexpreopt.GlobalConfig int...
I38317f2d5: Get the dex2oat host tool path from module depende...
I440a09dba: Separate dexpreopt.GlobalSoongConfig to allow inde...
Bug: 148312086
Bug: 145934348
Exempt-From-Owner-Approval: Plain revert
Change-Id: Ice3990225635a737e49e9aed7373f06516fccea3
Revert submission 1211982-dex2oat-soong-dep
Reason for revert: Build failures. See b/148312086.
Reverted Changes:
Ibc427a9a8: Make dex2oat(d) visible for use as implicit dexpre...
I71df11c1e: Move the Once cache for dexpreopt.GlobalConfig int...
I38317f2d5: Get the dex2oat host tool path from module depende...
I440a09dba: Separate dexpreopt.GlobalSoongConfig to allow inde...
Bug: 148312086
Bug: 145934348
Exempt-From-Owner-Approval: Plain revert
Change-Id: I6b656afb5feaad70d958b9d38b6c6eab7b03fba1
package.
Preparation for a future CL that will need to get the make-written
GlobalConfig from within dexpreopt.
Also rename the Load*Config functions to Parse*Config, since they don't
actually load the config files anymore.
Bug: 145934348
Test: m
Change-Id: I71df11c1e042ca8135d273a7263e9539ea3cd68f
it.
Introduce a Once cache for GlobalSoongConfig to allow it to get binary
tool paths from ordinary module dependencies (coming in a future CL)
that are then reused in singletons.
Bug: 145934348
Test: m
Change-Id: I440a09dba7d337965a196527566b0966a18e3653
Previous CL Ia9b34aa92ebb1b4de96ea0f8f290d798be19b2cf introduced
asymmetry in handling "boot" and "apex" boot images: in JIT-zygote
experiment, though the "apex" boot image was installed, the "boot"
boot image was still used for generating dexpreopt configs.
It is unclear why this asymmetry was needed at that point; it seems
incorrect to use different boot images for dexpreopting and for
installing on device. After recent changes the asymmetry started
breaking walleye_jitzygote-userdebug on git_rvc-release, because APK
dexpreopt commands refer to inexistent boot image files.
Test: lunch aosp_walleye-userdebug && m
Test: cherry-pick CL in internal, walleye_jitzygote-userdebug boots
Change-Id: Id877c10269cf79caf6ae74b1dc169a31e6a2211b
This patch splits the system boot image in two parts:
- The ART boot image. This is the primary boot image that is
included in the ART apex and contains dexpreopted Core Libraries.
- The framweork boot image extension. It depends on the ART boot
image and contains framework libraries.
The third "apex" boot image (used in the JIT-zygote experiment)
remains unchanged; it is a monolithic primary boot image that
contains both libcore and framework libraries.
Dexpreopting of APKs now uses the framework boot image extension
(which in turn pulls in the ART boot image as a dependency).
Test: m
Test: phone boots:
lunch aosp_walleye-userdebug && m \
&& adb reboot bootloader && fastboot flashall -w
Bug: b/119800099
Exempt-From-Owner-Approval: rebased after getting approval.
Change-Id: Ida40dfae8c83bf7c2e737d5c7ea418e1197ad826
Stripping is incompatible with ART module updatability.
Bug: 65154345
Bug: 138851227
Test: build and observe no change in output (stripping is not used by
default).
Change-Id: I4bbc01d9979605b7dd2f73ee99a74c1e817c3e8c
Create a new type InstallPath that is similar to OutputPath to
differentiate intermediates output paths from installed output
paths.
RelPathString is a poorly defined, undocumented function that is
primarily used to get an install path relative to out/soong to
generate an equivalent install path for Make relative to $(OUT_DIR).
Move it to InstallPath for now, and fix the one remaining user on
OutputPath.
Add a method to create an NDK install path so that ndk_sysroot.go
doesn't have to do it manually with PathForOutput.
Bug: 141877526
Test: m checkbuild
Change-Id: I83c5a0bd1fd6c3dba8d3b6d20d039f64f353ddd5
Translated second architectures now go in NativeBridgeArch instead
of DeviceSecondaryArch.
This reapplies I568046330abc002d4eed582cb999b62a5eaba790 with
ctx.Config().HasMulitlibConflict() added to fix the NDK build,
which has arm64, arm, x86_64, and x86 architectures enabled.
Test: m checkbuild
Test: OUT_DIR=out_ndk build/soong/scripts/build-ndk-prebuilts.sh
Test: no change to build.ninja or Android-aosp_cf_x86_phone.mk
Change-Id: Iadcafbd64bfb9579ae7c86914927c43a062b0c8e
This reverts commit fe6330c7d2.
Reason for revert: this was accidentally propping up NDK builds
Bug: 141242600
Change-Id: Ie9f8e31479a74e8e9e14d720a5936da6b1d5d113
