Commit graph

120 commits

Author SHA1 Message Date
Ulya Trafimovich
eb26886c85 Move class loader context definitions to a separate file.
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: I1e7e9db1654d0b835276be1cfa6a8eeffc5e96ee
2020-10-29 17:47:34 +00:00
Ulya Trafimovich
8130c482ab Move part of logic from construct_context.py to Soong.
construct_context.py is a script that runs at Ninja stage and constructs
class loader context arguments for dex2oat. Previously it accepted lists
of library paths and concatenated them into one class loader context
string. The script also handled the dependency of "android.hidl.manager"
on "android.hidl.base" that is not tracked by the build system and must
be handled in a special way.

Now that class loader context representation is going to change from
flat lists to trees, passing paths to individual libraries as script
arguments is no longer possible, because a list of paths cannot
represent a class loader context tree. Passing the trees in a serialized
form is also inconvenient, because the script would have to parse them,
which would complicate it a lot.

Therefore this patch ports all the concatenation and "android.hidl.base"
handling to Soong. It is not possible to port the remaining script to
Soong because Soong has no information about the targetSdkVersiion of
the dexpreopted apps (it is in the manifest and sometimes in an APK, and
has to be extracted at Ninja time).

Test: construct_context_test.py
Test: new subtests in TestUsesLibs
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: Icdb03cf00d1e27e4cff3844b89bfaec4de502dd7
2020-10-27 17:05:30 +00:00
Ulya Trafimovich
46b3d5bd05 Fix handling of "android.test.mock" in class loader context.
Mimick the way PackageManager handles it at runtime: do add it to class
loader context for apps with targetSdkVersion < 30, but only if
"android.test.runner" is used. Previously it was not added at all.

Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: I4c06635277ab13e21069b9fa0b46eb6a2547dfdd
2020-10-27 17:05:30 +00:00
Ulya Trafimovich
24813e1d80 Unify handling of compat and normal libs in class loader contexts.
Also, add tests for compatibility libraries in class loader context.

This CL separates special-case handling of compatibility libraries into
a "fixup" step that is done after class loader context is constructed by
Soong. This allows to handle compatibility libraries and normal
libraries uniformly, which should enable further simplification of class
loader context representation (in subsequent CLs).

Currently the only "fixup" step is removal of libraries from conditional
class loader context if they already are in unconditional context. This
check cannot be done at the time when the libraries are added to
conditional context, because the full uncoditional context is not yet
known at that time. Previously construction of unconditional context was
delayed, now it is no longer delayed and handled in the same way as
unconditional context, and the "fixup" does the filtering.

Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: Ie71e9fb2d3d529b5317cd84e09ab3c853017c349
2020-10-27 17:05:30 +00:00
Paul Duffin
7ccacaedbe Remove unnecessary properties in GlobalJSONConfig
This change removes the properties in GlobalJSONConfig that are used to
hold the unmarshalled []string along with the code to convert them into
ConfiguredJarList as they are no longer necessary because the
ConfiguredJarList.UnmarshalJSON(..) method can handle that automatically.

Bug: 171479578
Test: m nothing
Change-Id: Ie297cd00cb23d12e544d922454868db6fe9eb77c
2020-10-27 10:36:29 +00:00
Colin Cross
405af07859 Revert "Make lots of tests run in parallel"
This reverts commit 323dc60712.

Reason for revert: Possible cause of test instability
Bug: 170513220
Test: soong tests

Change-Id: Iee168e9fbb4210569e6cffcc23e60d111403abb8
2020-10-09 18:34:24 -07:00
Colin Cross
45fd6ace7e Merge "Make lots of tests run in parallel" 2020-10-07 20:40:38 +00:00
Colin Cross
323dc60712 Make lots of tests run in parallel
Putting t.Parallel() in each test makes them run in parallel.
Additional t.Parallel() could be added to each subtest, although
that requires making a local copy of the loop variable for
table driven tests.

Test: m checkbuild
Change-Id: I5d9869ead441093f4d7c5757f2447385333a95a4
2020-10-06 15:12:22 -07:00
Ulya Trafimovich
a54d33be77 Fail the build if dexpreopt cannot find path to a <uses-library>.
This doesn't apply to compatibility libraries (those added to
conditional class loader context base on the targetSdkVersion in the
manifest), because Soong doesn't know if they are needed or not (the
build rule that parses the manifest runs at Ninja stage).

In the future we may need to relax this and instead of failing the build
to disable dexpreopt in such cases. But currently there are no use cases
that fail.

Also, this patch replaces hardcoded install path with a constant.

Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: Ic894bc17e8ffd8ff173111ea09366252001ef3e9
2020-10-06 14:04:49 +01:00
Anton Hansson
9b4ed7f3de Merge "Revert "Re-land "Move odexes of non-system apps into system_other""" 2020-09-15 09:31:07 +00:00
Anton Hansson
da4d9d967d Revert "Re-land "Move odexes of non-system apps into system_other""
This reverts commit 75480154e2.

Bug: 141707536
Test: Clean revert
Reason for revert: breaks some builds
Change-Id: If0cf736099ec12e35bcb190d0fc33daf7f1da98e
2020-09-15 09:29:11 +00:00
Anton Hansson
8eebee4385 Merge "Re-land "Move odexes of non-system apps into system_other"" 2020-09-14 11:01:49 +00:00
Anton Hansson
75480154e2 Re-land "Move odexes of non-system apps into system_other"
Previously, the patterns in SYSTEM_OTHER_ODEX_FILTER only matched
subpaths of /system. Broaden this to match /<p> as well as /system/<p>.

Because SYSTEM_OTHER_ODEX_FILTER includes system_ext and product apps,
the immediate effect of this is that the odex and vdex files of
apps on those partitions move to system_other for A/B devices.

This makes the product and system_ext partitions smaller, which
in turn makes OTAs smaller.

Bug: 141707536
Test: soong unit tests
Test: (adb shell dumpsys package com.android.stk &&
       adb shell dumpsys package com.android.quicksearchbox) | grep -i dexopt -A3;
      verify dexpreopt state is 'prebuilt' for both
Test: adb shell find /data/dalvik-cache -name '*QuickSearch*.vdex' | wc -l == 1
Test: adb shell find /data/dalvik-cache -name '*Stk*.vdex' | wc -l == 1
Change-Id: I0cf141014b9ddf32159cff47b0325d9c78b549f2
2020-09-11 14:45:51 +01:00
Ulya Trafimovich
e3bb01dfac Treat "org.apache.http.legacy" as optional <uses-library> by default.
This doesn't affect the cases with explicitly set `uses_libs` or
`optional_uses_libs`, only the cases when the library implicitly added
to the manifest_fixer by Soong. "org.apache.http.legacy" should be
handled in the same way as "android.test.base", except that they have
been added in different SDK versions.

Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: I4fa1f082bfdb67f227f432717731a7e3d790913c
2020-09-11 13:52:23 +01:00
Ulya Trafimovich
663dc53c94 Consistently use default install path for compatibility <uses-library>.
Previously default install path was provided only for comatibility
libraries that are added explicitly via `uses_libs`/`optional_uses_libs`
properties. This didn't take into account compatibility libraries that
are added by Soong when it computes transitive closure of SDK library
dependencies. As a result, install path in such cases remained unknown,
and the corresponding compatibility library was omitted from class
loader context, which caused 'ClassLoaderContext shared library size
mismatch' errors at first boot on device.

Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: If661272f46803299c8640f7f5ef161c8217d07e4
2020-09-11 11:31:22 +01:00
Ulya Trafimovich
045e11ad0d Be more strict about unknown install <uses-library> paths.
Allow default install paths only for compatibility libraries. For other
libraries that are explicitly specified in `uses_libs` and
`optional_uses_libs` unknown install path should be an error.

Bug: 132357300
Test: lunch cf_x86_phone-userdebug && m
Change-Id: I2209c90a939a8aa46c42e13bb42d09c07e4d0895
2020-09-07 13:58:51 +01:00
Ulya Trafimovich
fc24ad3d4e Propagate transitive SDK Java library dependencies to dexpreopt.
For some dependencies, like stubs, the SDK library may not be found at
build time (either because the implementation library is not among the
dependencies of the dexpreopted module, or because it's part of a
prebuilt, or because it's missing from the build altogether). In such
cases dexpreopt is useless, because dex2oat does not have access to the
full classpath (unless the &-classpath is used). Therefore do not
dexpreopt in such cases.

Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: If289088cfd103011ccb16165e95a97b30fd31b81
2020-08-27 11:47:15 +01:00
Martin Stjernholm
4ef4d55944 Merge "Resolve prebuilt module explicitly for dex2oat tool dependencies." 2020-08-21 09:57:40 +00:00
Martin Stjernholm
c004862309 Resolve prebuilt module explicitly for dex2oat tool dependencies.
Test: m nothing
Bug: 145934348
Change-Id: If27984be26976a3abe8cb5e21cf174d5e71beb8a
2020-08-20 16:05:45 +01:00
Ulya Trafimovich
b8063c6a86 Use common helper functions for getting sorted map keys.
Add a new helper SortedIntKeys similar to SortedStringKeys.

Test: lunch aosp_cf_x86_phone-userdebug && m
Change-Id: I08a43ec2cae7d1a82531295aca1a0658e3a0dd6f
2020-08-20 12:37:03 +01:00
Ulya Trafimovich
31e444e101 Collect paths to transitive SDK Java library dependencies.
Previously only the names were collected, and later used in the
manifest_fixer to add missing <uses-library> entries to the manifest.
Now we also need to collect build-time and on-device paths, to be used
in class loader context for dexpreopt. This commit only collects paths,
but does not pass them to dexpreopt yet.

Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: I34b229ee68f16ba215ba03770feadb4d890ec2bf
2020-08-19 13:51:21 +01:00
Ulya Trafimovich
249386ad69 Add structured representation for colon-separated jar lists.
With the addition of apexes and /system_ext some of the bootclasspath
and system server jars have moved from /system to the new locations.
This has been implemented by using lists of colon-separated strings
called "apex-jar pairs" (although "apex" was misleading as it could
refer to "platform" or "system_ext", not necessarily a real apex).

Using the colon-separated string representation is inconvenient, as it
requires splitting and reassembling the list components many times,
which harms performance and makes error handling difficult. Therefore
this patch refactors the colon-separated lists into a struct that
hides the implementation details.

Test: lunch aosp_cf_x86_phone-userdebug && m
Change-Id: Id248ce639a267076294f4d4d73971da2f2f77208
2020-08-05 09:59:30 +01:00
Ulyana Trafimovich
8fab5bcce3 Merge changes from topic "uses-libs-5"
* changes:
  Rewrite construct_context.sh in Python.
  Refactor class loader context generation.
2020-07-02 09:06:27 +00:00
Ulya Trafimovich
5f364b63b8 Rewrite construct_context.sh in Python.
This allows to reuse SDK version comparison routine from other Python
scripts. With the addition of non-numeric versions comparison has become
more complex, and the deleted shell script did it incorrectly: it used
comparisons like `[[ "S" -lt 28 ]]` which results in "true" as strings
are converted to zero in numeric context. This resulted in adding legacy
libraries to class loader context of apps targeting recent SDK versions.
The error was masked because currently there is only one non-AOSP app
that uses the script (GoogleDialer), and it targets numeric SDK version.

Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: I26e752e29b5453fd3e29d5dbfbe4d9df9c0a55b7
2020-07-01 09:44:23 +01:00
Ulya Trafimovich
c4dac26596 Refactor class loader context generation.
There are essentially three different cases: 1) system server dexpreopt,
2) the case when uses-library information is known, and 3) the case when
uses-library information is unknown and the &-classpath hack is used.
This patch reorganizes the code into three branches that correspond to
the above cases.

Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: Ife2d01a612e0a608ae1346d7007741498e938bc5
2020-06-30 12:40:54 +01:00
Thiébaud Weksteen
5e291c238f Fix bpfmt issues and add bpfmt to preupload checks
Test: m nothing
Change-Id: I10526a33685335bdd26cc7692815133379d4e0f6
2020-06-29 09:14:08 +02:00
Ulya Trafimovich
5cb30c5820 Move construct_context.sh from Make to Soong. Update OWNERS list.
This script belongs with other manifest-related scripts, and the future
plan is to rewrite it in Python and share common functionality with
other scripts.

Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: I4b90129e5023ca1a2c818fc160c34b0b7da943ef
2020-06-25 14:46:35 +01:00
Ulya Trafimovich
6e82748bd6 Rename dexpreopt.config field to bring it in sync with Make.
Different names of the same field in Make and in Soong resulted in
the field being dropped when unmarshalling the JSON from Make to Soong.
As a result, the list of optional used libraries for Make modules was
always empty.

Test: lunch aosp_cf_x86_phone-userdebug && m
Test: Cherry-pick in internal master and observe that the logcat error
    "ClassLoaderContext shared library size mismatch" for GoogleDialer
    has disappeared:
    terminal-1 $ lunch cf_x86_phone-userdebug && m && launch_cvd
    terminal-2 $ adb wait-for-device && adb root \
        && adb logcat | grep 'size mismatch'
    (nothing after the change, one error message before the change)

Bug: 132357300
Change-Id: I57820646fa7c1d8af01b2cbc8a3521da450719c0
2020-06-12 14:47:50 +01:00
Ulya Trafimovich
d4bcea4e24 Refactor library path representation in dexpreopt.
This is a preliminary CL before fixing on-device paths to DEX jars.
It groups together the inormation about on-host build paths and
on-device install paths to library DEX jars.

This CL changes the structure of module dexpreopt.config files
generated by the build system. Aside of that, no functional changes.

Test: lunch aosp_cf_x86_phone-userdebug && m
Change-Id: I059654be7670f2ba66248d9c49b9694a0591f9c1
2020-06-04 17:37:42 +01:00
Ulyana Trafimovich
eb3f9f1295 Merge "Error on suboptimal system server classpath order, unless explicitly allowed." 2020-06-04 08:56:04 +00:00
Ulya Trafimovich
cd3203fd5d Error on suboptimal system server classpath order, unless explicitly allowed.
The order is non-optimal if some jar X precedes its dependency Y. In
that case dexpreopt will be unable to resolve any rerefences from X to
Y. Raise an error unless the product sets variable
PRODUCT_BROKEN_SUBOPTIMAL_ORDER_OF_SYSTEM_SERVER_JARS to true in the
makefile. This is to prevent regressions in existing products that
currently have correct order.

Test: lunch cf_x86_phone-userdebug && m nothing
Test: lunch aosp_car_arm && m nothing

Bug: 140451054

Change-Id: Iafe8fe0e992deb628e38f2321d4601a4804a3c79
Merged-In: Iafe8fe0e992deb628e38f2321d4601a4804a3c79
(cherry picked from commit fb863c87c4)
2020-06-03 17:37:34 +01:00
Ulya Trafimovich
696c59dbfa Clean up class loader context generation.
Deduplicate code for conditional class loader contexts that depend
on the target SDK version in the manifest.

Test: lunch aosp_cf_phone-userdebug && m
Change-Id: I69d51b212bf502b2659d43f124ea3922f90e1881
2020-06-01 17:55:56 +01:00
Ulya Trafimovich
c9af538c90 Add android.test.base to class loader context for API version < 30.
Test: lunch aosp_cf_x86_phone-userdebug && m
Change-Id: Id2ece4b2a6317998e1dd177e853848cc3956f39f
2020-06-01 12:51:38 +01:00
Ulya Trafimovich
df00ddecb6 Don't use obsolete "org.apache.http.legacy.impl" in class loader context.
This library no longer exists on device and "org.apache.http.legacy"
should be used instead. The error lingered in the code for a while
because the code was not exercised at all. (The only app that sets
EnforceUsesLibraries is GoogleDialer (not in AOSP), and it specifies
"org.apache.http.legacy" explicitly as an optional used library.)

Test: lunch aosp_cf_x86_phone-userdebug && m
Change-Id: I05fa115476315d35a8877fc42bd1c2042f72f2de
2020-05-29 15:00:30 +01:00
Ulya Trafimovich
8640ab9c26 Raise an error instead of panic in SplitApexJarPairs.
Test: aosp_walleye-userdebug boots
Change-Id: I9a1c4ec5c8862b1936820b3cee6db9a85015cb0e
2020-05-12 12:27:50 +01:00
Ulya Trafimovich
50c4a4b19f Use two-column format for PRODUCT_BOOT_JARS components.
The first component is the apex name, or a special name "platform"
if the boot jar is a platform jar rather than a part of some apex.
This is a prerequisite change for moving core-icu4j to a separate
com.android.i18n apex.

Old one-column format is still supported, but all unqualified
components of PRODUCT_BOOT_JARS get "platform:" prepended to them
after reading the product makefiles.

Test: aosp_walleye-userdebug boots
Bug: 138994281
Change-Id: Ic229159fdcdaf6182210a53b63850a389dd786fc
2020-04-30 14:42:32 +01:00
Ulya Trafimovich
6cf2c0c197 Fix host path for dexpreopted system server modules defined in Make.
Previously there was a discrepancy between the generated paths for
Soong modules that used "out/soong/system_server_jars" and Make modules
that used "out/system_server_jars". This happened because the default
output directory is $OUT/soong for the normal Soong config and jsut $OUT
for the reduced "null config" created by dexpreopt_gen.

As a result, class loader context for system server jars defined in Make
was referring to nonexistent jars, which caused non-fatal dex2oat errors
(the build was not broken, so the errors remained unnoticed):

  Failed to determine oat file name for dex location out/system_server_dexjars/[...]
  Could not open dex files from location: out/system_server_dexjars/[...]

Test: aosp_walleye-userdebug boots
Test: cherry-picked in master-arc-dev that has a Make system server jar
  arc-services, `lunch lunch cheets_x86-userdebug && m`, check that the
  generated dexpreopt.sh for arc-services contains paths starting with
  "out/soong/system_server_jars" rather than "out/system_server_jars".

Bug: 140712566
Change-Id: Ia7ea8ac383e32042c31d64971cdc8101ea3068cd
2020-04-27 09:46:02 +01:00
Ulya Trafimovich
55285bab77 Don't append build directory to the output path twice.
Soong already uses build directory as a part of the generated output
path. Besides, trying to append it second time breaks if the build
directory is an absolute path.

Test: frameworks/rs/build_rs.py  (uses absolute build directory path)
Test: walleye-userdebug boots

Bug: 152235239
Change-Id: Ie03c9e688013b9a3e6bc859c936d89c538ded76a
Merged-In: Ie03c9e688013b9a3e6bc859c936d89c538ded76a
(cherry picked from commit f798c3ec32)
2020-03-24 17:02:19 +00:00
Ulya Trafimovich
b358ebb759 Fix null pointer dereference in Soong on dexpreopt arc-services.
The null pointer dereference was caused by trying to get device name
from a config passed to dexpreopt. For Make modules, the config is a
stub that doesn't have a device name. The fix is to use build directory
instead of device name (it is provided both by the full config passed
to Soong modules and by the stub config passed to Make modules).

Test: walleye-userdebug boots.
Test: aosp_bertha_arm-userdebug builds and generates arc-services.odex.

Bug: 152193534
Change-Id: I7adc173646cae6eaee188c6672629a5a942aea06
Merged-In: I7adc173646cae6eaee188c6672629a5a942aea06
(cherry picked from commit 9790f1058b)
2020-03-23 17:13:07 +00:00
Ulya Trafimovich
dacc6c56f1 Simplify the construction of class loader contexts for system server jars.
This reworks CL https://r.android.com/1180134 as follows:

1) Do not reorder the list of system server jars passed from Make to
  Soong via the product variable PRODUCT_SYSTEM_SERVER_JARS. This means
  that for some products the order of jars on the system server
  classpath may be non-optimal: a jar X that depends on Y may be
  dexpreopted before Y, so that all references to the classes and
  methods from Y wil be unresolved.

  Unfortunately for such products, fixing the order is not a simple
  matter of rearranging their PRODUCT_SYSTEM_SERVER_JARS, because the
  conflicts may arise when the product-specific variable gets merged
  with the common variable.

2) As a consequence of 1), do not add artificial dependencies between
  system server jars: this is now impossible, as it would create
  circular dependencies for those products that have non-optimal order
  of jars.

3) Copy dex files for system server jars to a predefined build location.
  This is necessary because at the time when Soong generates class
  loader context for k-th jar, it needs to know the paths to jars 1 ..
  (k-1), and it might have not processed those jars yet (so Soong can't
  query the paths from the modules).

  This approach is similar to the way Soong handles bootclasspath jars.

4) Do not exclude from dexpreopting system server jars that are not
  Soong modules (those that are Make modules). The previous CL excluded
  them because Make modules do not have ModuleContext. But it turns out
  that ModuleContext is not necessary, as all the information is passed
  via the dexpreopt config.

Test: aosp_walleye-userdebug boots and there are no messages in the
  logcat regarding class loader context mismatch:

  $ adb logcat | grep 'mismatch'
  # empty

Test: Class loader contexts in the oat files for system server jars
  match expectations:

  $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/com.android.location.provider.odex 2>/dev/null | grep '^classpath'
  classpath = PCL[]
  $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/services.odex 2>/dev/null | grep '^classpath'
  classpath = PCL[/system/framework/com.android.location.provider.jar*1989208671]
  $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/ethernet-service.odex 2>/dev/null | grep '^classpath'
  classpath = PCL[/system/framework/com.android.location.provider.jar*1989208671:/system/framework/services.jar*4040443083:/system/framework/services.jar!classes2.dex*2210087472]

Test: The phone boots and logcat has no scary messages related to
  class loader contexts:

  $ lunch aosp_walleye-userdebug && m
  $ adb reboot bootloader && fastboot flashall -w && adb wait-for-device
  $ adb root
  $ adb shell stop
  $ adb logcat -c
  $ adb shell setprop dalvik.vm.extra-opts -verbose:oat
  $ adb shell start
  $ adb logcat | egrep -io 'system_server: .*load.*/system/framework.*'
  system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 0
  system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 0
  system_server: Loading /system/framework/oat/arm64/services.odex with executable: 0
  system_server: Successfully loaded /system/framework/oat/arm64/services.odex with executable: 0
  system_server: Loading /system/framework/oat/arm64/ethernet-service.odex with executable: 0
  system_server: Successfully loaded /system/framework/oat/arm64/ethernet-service.odex with executable: 0
  system_server: Loading /system/framework/oat/arm64/wifi-service.odex with executable: 0
  system_server: Successfully loaded /system/framework/oat/arm64/wifi-service.odex with executable: 0
  system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1
  system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1
  system_server: Loading /system/framework/oat/arm64/services.odex with executable: 1
  system_server: Successfully loaded /system/framework/oat/arm64/services.odex with executable: 1
  system_server: Loading /system/framework/oat/arm64/ethernet-service.odex with executable: 1
  system_server: Successfully loaded /system/framework/oat/arm64/ethernet-service.odex with executable: 1
  system_server: Loading /system/framework/oat/arm64/wifi-service.odex with executable: 1
  system_server: Successfully loaded /system/framework/oat/arm64/wifi-service.odex with executable: 1
  system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1
  system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1

Bug: 141785760
Bug: 140451054
Bug: 148944771
Bug: 147017252

Change-Id: I33c4087f8bfacd0ecb89877aa150b75360d06710
Merged-In: I33c4087f8bfacd0ecb89877aa150b75360d06710
(cherry picked from commit a4a83b0ef9)

Exempt-From-Owner-Approval: cherry-pick.
2020-03-23 12:27:31 +00:00
Jaewoong Jung
3aff5787e2 Simple refactoring of prefix related functions.
Rename the two prefix-in-list funcs so that their usages are clearer.
Also find and replace all the code that essentially does either.

This introduces additional loops in some places, but I think the added
readability and simplicity outweighs the performance degradation, which
should be negligible anyway.

Test: m nothing
Test: TreeHugger
Change-Id: I37e2276ca0c815105ed0031f23c0b1264b480e4f
2020-02-14 14:33:49 -08:00
Martin Stjernholm
6d415273c0 Do all dexpreoptDisabled checks before registering a dex2oat host dep.
Also disable dexpreopting for host. These are necessary to avoid adding
dependencies on dex2oat in various non-platform builds where they will
break.

Since we cannot assume at least one module enables dexpreopting now,
the two dexpreopt singletons are silently disabled if there has been no
call to dexpreopt.GetGlobalSoongConfig.

Bug: 145934348
Bug: 148312086
Bug: 148319588
Bug: 148690468
Test: m
Test: env OUT_DIR=out-tools prebuilts/build-tools/build-prebuilts.sh
  on the aosp-build-tools branch
Test: build/soong/soong_ui.bash --make-mode static_sdk_tools dist DIST_DIR=out-dist BUILD_HOST_static=1
  on internal (cf b/148312086#comment8)
Test: build/soong/soong_ui.bash --make-mode dist DIST_DIR=out-apps TARGET_BUILD_APPS=Launcher3 TARGET_BUILD_VARIANT=userdebug
  on internal without art/ and external/vixl/ (cf b/148319588)
Change-Id: I240dade7204b87fc2d12181534ab23439eca8b46
2020-02-13 17:46:48 +00:00
Martin Stjernholm
8d80ceeb12 Pass dexpreopt config structs by reference.
Should cut down on a bit of copying, and also required for an upcoming
CL that'll change GetCachedGlobalSoongConfig.

Test: m nothing
Bug: 145934348
Change-Id: I6bed737d9b061b5239cc603ad881f4ccd4312e43
2020-02-13 17:46:48 +00:00
Martin Stjernholm
d90676fdde Reland: Get the dex2oat host tool path from module dependency on the
binary module.

This uses the Once cache for GlobalSoongConfig to propagate the dex2oat
path from a module dependency to the singletons (both the one that
writes out dexpreopt_soong.config and the one that creates the
dexpreopted boot images). Unless dexpreopting is disabled altogether
through DisablePreopt in dexpreopt.config, that means:

- We must ensure at least one module registers a dex2oat tool
  dependency and resolves a GlobalSoongConfig using it, or else the
  singletons fail. That means we litter dex2oat dependencies in java
  modules even when they won't get dexpreopted and hence don't really
  need them.

- We still assume there's at least one java_library or android_app in
  the build.

This relands https://r.android.com/1205730 without changes - the
necessary fixes are in the child CLs.

Bug: 145934348
Test: m
  (check that out/soong/dexpreopt_soong.config points to dex2oatd64)
Test: env USE_DEX2OAT_DEBUG=false m
  (check that out/soong/dexpreopt_soong.config points to dex2oat64)
Test: env OUT_DIR=out-tools prebuilts/build-tools/build-prebuilts.sh
  on the aosp-build-tools branch
Change-Id: I66661711b317d1e4ec434861982919bdde19b575
2020-02-13 17:46:48 +00:00
Martin Stjernholm
40f9f3c061 Reland: Move the Once cache for dexpreopt.GlobalConfig into the
dexpreopt package.

Preparation for a future CL that will need to get the make-written
GlobalConfig from within dexpreopt.

Also rename the Load*Config functions to Parse*Config, since they don't
actually load the config files anymore.

This relands https://r.android.com/1211982.

Bug: 145934348
Test: m
Change-Id: Icb9332a93811d77d2d8b06e983b92501b180a358
2020-02-13 17:46:48 +00:00
Martin Stjernholm
75a48d8ae2 Reland: Separate dexpreopt.GlobalSoongConfig to allow independent
caching of it.

Introduce a Once cache for GlobalSoongConfig to allow it to get binary
tool paths from ordinary module dependencies (coming in a future CL)
that are then reused in singletons.

This relands https://r.android.com/1205729.

Bug: 145934348
Test: m
Change-Id: I039d6e204bee5ddc16d8e2d85057fbec20e326fe
2020-02-13 17:21:13 +00:00
Ulya Trafimovich
f3ff010bd4 Use precise class loader context for system server jars.
This patch excludes non-Soong system server jars from dexpreopting.

System server jars should be dexpreopted together for better
performance: each jar should have all preceding system server jars
in its class loader context (that is passed to dex2oat and recorded
in the resulting .oat file to be used by the loader).

This means that we need a total order on system server jars. The
default order provided by PRODUCT_SYSTEM_SERVER_JARS is not good, as
it does not always respect genuine dependencies between jars (counter-
examples are rare, but non-trivial to fix: b/148219586).

This patch adds a post deps mutator pass that creates additional
dependencies and enforces global order. These are later used to
generate precise class loader contexts and system server classpath.

Test: Class loader contexts in the oat files for system server jars
  match expectations:

  $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/com.android.location.provider.odex | grep '^classpath'
  classpath = PCL[]
  $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/services.odex | grep '^classpath'
  classpath = PCL[/system/framework/com.android.location.provider.jar*1919890654]
  $ oatdump --oat-file=out/target/product/walleye/system/framework/oat/arm64/wifi-service.odex | grep '^classpath'
  classpath = PCL[/system/framework/com.android.location.provider.jar*1919890654:/system/framework/services.jar*4269704903:/system/framework/services.jar!classes2.dex*134345935]
  ...

Test: The phone boots and logcat has no scary messages related to
  class loader contexts:

  $ lunch aosp_walleye-userdebug && m
  $ adb reboot bootloader && fastboot flashall -w && adb wait-for-device
  $ adb root
  $ adb shell stop
  $ adb logcat -c
  $ adb shell setprop dalvik.vm.extra-opts -verbose:oat
  $ adb shell start
  $ adb logcat | egrep -i 'system_server: .*load.*/system/framework'
  02-03 14:14:26.912  5016  5016 I system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1
  02-03 14:14:26.914  5016  5016 I system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1
  02-03 14:14:26.914  5016  5016 I system_server: Loading /system/framework/oat/arm64/services.odex with executable: 1
  02-03 14:14:26.916  5016  5016 I system_server: Successfully loaded /system/framework/oat/arm64/services.odex with executable: 1
  02-03 14:14:26.927  5016  5016 I system_server: Loading /system/framework/oat/arm64/wifi-service.odex with executable: 1
  02-03 14:14:26.933  5016  5016 I system_server: Successfully loaded /system/framework/oat/arm64/wifi-service.odex with executable: 1
  02-03 14:14:26.933  5016  5016 I system_server: Loading /system/framework/oat/arm64/ethernet-service.odex with executable: 1
  02-03 14:14:26.934  5016  5016 I system_server: Successfully loaded /system/framework/oat/arm64/ethernet-service.odex with executable: 1
  02-03 14:14:26.946  5016  5016 I system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 0
  02-03 14:14:26.947  5016  5016 I system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 0
  02-03 14:14:26.947  5016  5016 I system_server: Loading /system/framework/oat/arm64/services.odex with executable: 0
  02-03 14:14:26.948  5016  5016 I system_server: Successfully loaded /system/framework/oat/arm64/services.odex with executable: 0
  02-03 14:14:26.948  5016  5016 I system_server: Loading /system/framework/oat/arm64/wifi-service.odex with executable: 0
  02-03 14:14:26.948  5016  5016 I system_server: Successfully loaded /system/framework/oat/arm64/wifi-service.odex with executable: 0
  02-03 14:14:26.949  5016  5016 I system_server: Loading /system/framework/oat/arm64/ethernet-service.odex with executable: 0
  02-03 14:14:26.949  5016  5016 I system_server: Successfully loaded /system/framework/oat/arm64/ethernet-service.odex with executable: 0
  02-03 14:14:30.480  5016  5016 I system_server: Loading /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1
  02-03 14:14:30.481  5016  5016 I system_server: Successfully loaded /system/framework/oat/arm64/com.android.location.provider.odex with executable: 1

Bug: 141785760
Bug: 140451054
Bug: 148944771

Change-Id: Idac678dbd1f5fe0e381ce8e0e3561423f8a31389
2020-02-13 14:39:03 +00:00
Ulya Trafimovich
4cdada2d1f Make PRODUCT_BOOT_JARS and PRODUCT_UPDATABLE_BOOT_JARS disjoint.
This is to unify boot jars with system server jars:
PRODUCT_SYSTEM_SERVER_JARS and PRODUCT_UPDATABLE_SYSTEM_SERVER_JARS
do not overlap.

Test: aosp_walleye-userdebug boots

Exempt-From-Owner-Approval: cherry-pick.
Bug: 146363577
Change-Id: If001049a46f8e9d38921dabeedbd61e971ebfe73
Merged-In: If001049a46f8e9d38921dabeedbd61e971ebfe73
(cherry picked from commit ef4358e536)
2020-02-13 13:35:39 +00:00
Nicolas Geoffray
73d5983af0 Remove UseApexImage and GenerateApexImage.
These options are not used anymore.

Bug: 119800099
Test: m
Change-Id: I4568ff23ed71a5c288ed87828aed6e4bc4f8bd4d
2020-02-07 14:19:28 +00:00
Vladimir Marko
40139d6422 Support dexpreopt against the primary boot image.
Framework and other dex files are used without image.

Test: taimen-userdebug boots when built with
      DEXPREOPT_USE_ART_IMAGE=true
Test: Check logcat for checksum verification failures.
      (Build ART with extra logging in OatFileAssistant.)
Test: Check that bootclasspath-checksums from some prebuilt
      oat files (say input.odex) contain only one image
      checksum followed by dex file checksums with
          grep -az -A1 -E '^bootclasspath-checksums$' <oat-file> | \
          xargs -0 echo | gawk '{print $2}'
Bug: 119800099
Change-Id: I65c2f247656e41f2c37df1ecb9e06af7dabab76e
2020-02-07 10:32:50 +00:00