1. Run pyformat scripts/check_boot_jars/check_boot_jars.py -s 4
--force_quote_type none -i to fix formatting.
3. use pylint: disable=<X> where fixes are not obvious
Test: m check_boot_jars
Test: pylint --rcfile tools/repohooks/tools/pylintrc
scripts/check_boot_jars/check_boot_jars.py
Bug: 195738175
Change-Id: I0f86d95e0d174c9f87a2872dedaa65cd3e814246
There is nothing in this script that is not python3, so
make it python3. Without this, a system without python will
fail when trying to run gdbclient.py.
Test: Ran gdbclient.py on a system without a python executable.
Change-Id: I9f8418932628569c445ee01bc6f9ead8aed7989f
1. Run pyformat scripts/hiddenapi -s 4
--force_quote_type none -i to fix formatting.
2. rename restricted variable names (e.g. variable name "input" has been
changed to "csvfile")
3. use pylint: disable=<X> where fixes are not obvious
Test: m merge_csv signature_patterns signature_patterns_test
Test: pylint --rcfile tools/repohooks/tools/pylintrc <file1>
<file1_test>
Bug: 195738175
Change-Id: I800a208f9c0ee1d32e68e4b20fd5933b3ab92c0e
1. Run pyformat scripts/hiddenapi/generate_hiddenapi_lists.py -s 4
--force_quote_type none -i to fix formatting. Some double quotes change
to single quotes since pyformat enforces consistent quotes
2. change map/filter to list comprehension to fix pylint's bad-builtin
warning
3. use pylint: disable=<X> where fixes are not obvious
Test: m generate_hiddenapi_lists_test
Test: pylint --rcfile tools/repohooks/tools/pylintrc <file1>
<file1_test>
Bug: 195738175
Change-Id: I15e71d8d81f0e3fa66d84e6e62eb1848963136e6
1. Run pyformat scripts/conv_linker_config.py -s 4 --force_quote_type none -i to fix formatting
2. Annotate #pylint: disable=import-error for linker_config_pb2 since it
will be provided by soong
Test: m conv_linker_config
Test: pylint --rcfile tools/repohooks/tools/pylintrc
build/soong/scripts/conv_linker_config.py
Bug: 195738175
Change-Id: I791576cf65cb053f68c804f8ec5c2fc22976fdb4
Fetching artifacts from the build server doesn't go well with large
numbers of files in nested directories.
Produce an additional rustdoc.zip artifact to make this easier.
This change also prevents ${DIST} from receiving the unpacked rustdoc
site. This can be changed if b/188822051 is fixed.
Bug: 162741284
Test: m rustdoc; check resulting zip contents manually
Change-Id: I80b6a8fa6e274d2d8c3419d8734251afd4d7dba7
Previously, the sdk snapshot would include all the entries from the
stub-flags.csv and all-flags.csv modular files generated by a single
bootclasspath_fragment. That included implementation details, i.e.
class members that are not part of a stable API or the hidden API which
meant that the sdk snapshots were implementation dependent.
This change removes the implementation details from the modular flag
files, i.e. those entries that are only marked as "blocked". When
comparing the files against the corresponding subset of the monolithic
files it assumes that any entries missing from the modular flag files
are blocked.
Bug: 194063708
Test: atest --host verify_overlaps_test signature_patterns_test
m out/soong/hiddenapi/hiddenapi-flags.csv
- manually change files to cause difference in flags to check
that it detects the differences.
Change-Id: I6b67b2253cf029d6830b58a06ebb0c8fcaa0dd71
Previously, the signature-patterns.csv file included a lot of
implementation details, e.g. the signatures of dex members or inner
classes that are not part of any API, including the hidden API.
This change will remove all member signatures and inner class names
from the file and replace them with just the outermost qualified class
name.
That will still leave some implementation details, e.g. the names of
implementation only classes and packages.
Bug: 194063708
Test: atest --host verify_overlaps_test signature_patterns_test
m out/soong/hiddenapi/hiddenapi-flags.csv
- manually change files to cause difference in flags to check
that it detects the differences.
Change-Id: I9de6a2a6129e875e19f7ded5fae578cbdb584660
Previously, a simple map from dex signature to the flags was used to
store the monolithic hidden API flags. This change replaces that with a
trie that uses packages and class names extracted from the signature
to map from the signature to the flags.
The signature is broken down into pieces consisting of package names,
class names and the remaining member signature. They are used in turn
to navigate through nodes in the trie to a Leaf that contains the flags
associated with the signature.
The flags can be retrieved from the trie in a number of ways:
* Using the complete signature to retrieve the flags for a single
member.
* Using a class name to retrieve the flags for all the members in a
class and any nested classes.
* Using a package wildcard to retrieve the flags for all the members in
the classes in that package.
* Using a recursive package wildcard to retrieve the flags for all the
members in the classes in that package and nested packages.
This will allow a bootclasspath_fragment to select a subset of the
monolithic flags to compare against itself without specifying the
signatures of every member of that set.
Before creating the trie an attempt was made to compute the subset by
iterating over all the signatures in the monolithic flags and matching
against a regular expression created from the patterns but that was too
slow. It took minutes to run whereas using the simple map or the trie
only takes a few seconds.
Bug: 194063708
Test: atest --host verify_overlaps_test
Change-Id: I36f5e319d3e7d62dd34305de1eec990a93cb3a89
Previously, the signatures used to select the subset of the monolithic
flags were simply the signatures read from the modular flags file. This
change moves the creation of the signature list into a separate script
that outputs the signatures to a file and then passes the path through
Soong from the bootclasspath_fragment modules that create it to the
platform_bootclasspath module that uses it to compare the modular
flags against the monolithic flags.
Currently, the signatures are the full signatures but follow up changes
will replace them with patterns (hence the name) that avoids having to
include implementation details in the hidden API flags that are output
as part of a bootclasspath_fragment's snapshot.
This change moves the stub flags related code next to the all flags
related code as they are treated in a similar way.
Bug: 194063708
Test: atest --host verify_overlaps_test signature_patterns_test
m out/soong/hiddenapi/hiddenapi-flags.csv
- manually change files to cause difference in flags to check
that it detects the differences.
Change-Id: I2855bf6d05c91b8a09591664185750361c7e644f
Previously, the compare_signature_flags compared the full set of
monolithic flags against the modular flags and just ignored any
monolithic flags for a signature that was not present in the modular
flags. It essentially performed two tasks, identifying the subset of
the monolithic flags against which the modular flags should be compared
and then doing the actual comparison.
This change separates those two tasks, by first creating a dict that is
the subset of the monolithic flags against which the modular flags will
be compared and then do the comparison.
At the moment the subset is simply the set of monolithic flags
associated with the signatures that are present in the modular flags.
Follow up changes will provide other ways to select the subset.
Bug: 194063708
Test: atest --host verify_overlaps_test
m out/soong/hiddenapi/hiddenapi-flags.csv
- manually change files to cause difference in flags to check
that it detects the differences.
Change-Id: Icb547f071b814dd955f31255b0d35597f7ba4aa1
Switches symbols to use monolithic... and modular... to try and clarify
the purpose of those symbols.
Bug: 194063708
Test: atest --host verify_overlaps_test
Change-Id: Ifc646edebe3d43b51dc3528fa883718399cb5254
Refactor verify_overlaps to make it testable and add tests for the
comparison. It does make one significant change in behavior which is to
read each of the files produced by a bootclasspath_fragment into a dict
before comparison, rather than reading and comparing them a row at a
time. That allows it to reuse the code to read a CSV into a dict.
Bug: 194063708
Test: atest --host verify_overlaps_test
m out/soong/hiddenapi/hiddenapi-flags.csv
- manually change files to cause difference in flags to check
that it detects the differences.
Change-Id: Ib70ac87fe089fc25e3bef18f367d4939bfc0cb8d
Major operations:
* Fix the go_package entries to be consistent, as the new tool
complains when it doesn't have a '/'.
* Regenerate with the new protoc-gen-go tool.
* github.com/golang/protobuf -> google.golang.org/protobuf
* proto.[Un]MarshalText -> prototext.[Un]Marshal
Change-Id: Ie1147bd2457fafb66ba555461b3bf14f0561a25d
atest generate_hiddenapi_lists_test would not work without setting
verbosity=2.
Bug: 194063708
Test: atest --host generate_hiddenapi_lists_test
Change-Id: Ic076ea85d58917921081ae33e72f06011c4ca49e
Normally Soong does that automatically when it handles module names
specified in Android.bp properties. However not all <uses-library>
entries in the manifest correspond to real modules: some of the
optional libraries may be missing at build time. Therefor this script
accepts raw module names as spelled in Android.bp/Amdroid.mk and trims
the optional namespace part manually.
Bug: 193891722
Test: manifest_check_test.py
Test: $ lunch aosp_cf_x86_64_phone-userdebug && m && launch_cvd
$ adb wait-for-device && \
adb root && \
adb logcat | \
grep -E 'ClassLoaderContext [a-z ]+ mismatch' -C1
# empty output, no errors
Change-Id: I184fb8a2dc26f49e90fb597ebcd6a58c00705206
Bug: http://b/187231324
This script parses soong/cc/config/global.go to return clang prebuilts
version. (It may get the version from a json config file in the
future.)
Currently it's used in development/scripts/symbol.py and is the
recommended way to get clang prebuilts version number in the future.
Test: get_clang_version_test.py
Change-Id: Id92a74fd523a5270999bb4d6454525bf2f2f0c04
Bug: 179354495
Test: m out/soong/hiddenapi/hiddenapi-stub-flags.txt
- check that an error is reported if a modular stub-flags.csv file,
i.e. one created by a fragment is not a subset of the monolithic
file, e.g. because a signature in the modular file has different
flags than it does in the monolithic or is not present there.
Change-Id: I46ebb495cb093a5e3abe7571c49933c845318549
Previously, if the --header property was not specified then merge_csv
would use a header constructed by sorting all the fields in the input
files. That required that any use of merge_csv which did not already
have headers in the required order would have to explicitly specify the
headers. That made it harder to use merge_csv as a generic tool as each
invocation needed to be aware of what headers were exported in the
output.
This change causes merge_csv to simply use the headers in the order in
which they are encountered in the input files. That removes the need to
specify the --header option when generating the index files.
Bug: 179354495
Test: m out/soong/hiddenapi/hiddenapi-index.csv out/soong/hiddenapi/hiddenapi-unsupported.csv
- make sure that they are not changed by this change.
Change-Id: I420b7d07aea85af6372cd7580a8be5e2cc82a513
Use the --soong-only flag instead to create it the proper way with the
config make step. This fixes build problems arising from the incomplete
soong.variables settings.
This partially relands https://r.android.com/1535202 so that we'll use
TARGET_PRODUCT=mainline_sdk in module SDK builds everywhere.
build-aml-prebuilts.sh isn't removed completely because we still need
the special case to create build_number.txt.
Test: art/build/build-art-module.sh --skip-apex && \
packages/modules/ArtPrebuilt/update-art-module-prebuilts.py \
--local-dist out/dist
then check that the created prebuilt dex2oatd64 runs with the repro
in b/190194345
Test: build/soong/scripts/build-mainline-modules.sh
then
prebuilts/runtime/mainline/update.py --local-dist .../out/dist
in a master-art tree and check that art/build/build-art-module.sh
works there
Bug: 190194345
Bug: 174315599
Change-Id: I86327b082b61c292f18c0a6b315d9add08002832
This script expects to build uncompressed APEX and use them as
prebuilts.
Bug: 189926000
Test: build_mainline_modules.sh
Change-Id: If0f62c2baa3db3c079b2913fcdc3dafd66394011
This makes sure that where there is overlap between the hidden API
flags generated for a module and the monolithic flags that they are
identical. That ensures that the modular hidden API flags will be
compatible with previous releases that relied on the monolithic flags.
Bug: 179354495
Test: m out/soong/.intermediates/art/build/boot/art-bootclasspath-fragment/android_common_apex10000/modular-hiddenapi/all-flags.csv
m out/soong/hiddenapi/hiddenapi-flags.csv
- Create some inconsistencies between the above two files.
m out/soong/hiddenapi/hiddenapi-flags.csv.valid
Change-Id: Iaf9e23cef63e221608955d89dc8d496bcc70c86e
The classpaths.proto file is needed at build time to generate appropriate
classpath configs for the modules. Unbundled ART branches do not include
packages/modules/SdkExtenstion in their manifest; so put the proto in a
project that should be visible to all unbundled modules.
Bug: 180105615
Test: m nothing
Change-Id: Ie5be1822fbee92f280332622a9317894cc98d7ac
The behaviour is semantically identical, however,
the tool additionally sorts the symbols by address,
compresses frame unwind information more efficiently,
and improves random-accessibility for lazy decompression.
Overall, the changes balance and the output size is same,
however, libunwindstack can access the data much faster
while using less memory (due to the lazy decompression).
It will also enable further improvements in the future.
Bug: 110133331
Test: ART unwinding tests, run prefetto on the device.
Change-Id: Id48f9fe67fb67fcf2b90cc3b217b71bb8f5147ca
Previously, if generate_hiddenapi_lists.py was invoked without any
custom flag files, e.g. hiddenapi-unsupported.txt, then it would fail.
This change allows it to be used for generating the hiddenapi lists
for modules that do not have any custom flag files,
e.g. framework-sdkextensions
Bug: 179354495
Test: verified that the monolithic out/soong/hiddenapi/... files are
unchanged by this change
Change-Id: I455d453024c9f06ed59cbc1e9838234f8b7c7317
Add a function to lint-project-xml to scan the given lint baseline file
for disallowed baseline lint issues.
Test: Manual
Bug: 182349282
Change-Id: I0db32bec0da24487b2f2b3f6704629f56ae76f56
'merge' command can merge multiple configurations.
It just delegates protobuf's msg.MergeFrom(other_msg).
usage: conv_linker_config merge -o out.pb -i a.pb -i b.pb ...
out.pb will have all keys from multiple inputs.
Bug: 181093750
Test: m
Test: MicrodroidTestCase
Change-Id: Ibf715aa0bcc1e5c82c85a9af8fe7dca8d6ab68ad
This is unused and doesn't work with our current API review process
anyway (un-drafting an API won't be flagged for API council review).
Test: treehugger
Bug: None
Change-Id: I6d8fcc9885b82dac5ada7772d9e3fb9101524ece
The proto is used at build time to define classpath fragments to be read
and parsed at runtime by `derive_classpath` service; in order to define
and set BOOTCLASSPATH, DEX2OATCLASSPATH, and SYSTEMSERVERCLASSPATH
environ variables.
The tool is used for debugging and generation of binary protos.
Bug: 180105615
Test: m && launch_cvd with a follow up changes
Change-Id: I7e8b084e5f63e9aaad11da3221b909818e69e235
Bug: 132357300
Test: lunch aosp_cf_x86_64_phone-userdebug && m
Test: temporarily patch Gallery2 to fail the check, observe the error
Change-Id: Id67e359188396f68dcecd8b3f4d1bc26271af56d
This no longer needs to be a prebuilt since stats-log-api-gen can be
built from proto_logging without extra dependencies.
Test: Treehugger
Bug: 170149255
Change-Id: I6925c2526cee1cbff3ba78bedd7f6c2f58dfff0f
Update OWNERS file under build/soong/scripts to include coverage team as owner of related scripts.
Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only
Change-Id: I6150d5bda0d59828a985cd2ab5dd0aa38a7987c3
Instead, return "any" SDK version 10000, which will cause dexpreopt to
not add any compatibility libraries to class loader context. This is as
good guess as any, and the build system already uses "any" version if
there is not manifest or APK from which to extract targetSdkVersion.
This fixes broken build on cf_x86_auto-userdebug.
Bug: 132357300
Bug: 183943146
Test: lunch cf_x86_auto-userdebug \
&& m out/target/product/vsoc_x86/obj/APPS/playback_intermediates/dexpreopt.zip
Change-Id: I5ee75639ece716c3a9800ddea346fe7c73da7c3a
Update the gen_ndk_usedby_apex.sh script to generate NDK API list with
library name.
Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only
Change-Id: I799e03ad7d81a4184229f658fd6c8824b9ba6e1f
There are modules that would have passed verify_uses_library check, but
don't run it for some reason (the check gets enabled either with an
explicit setting, or if the module has nonempty <ues-library> lists in
the build properties). Previously all such modules were assumed to have
empty CLC, which is not always true. In particular, compatibility
libraries are ignored, which affected e.g. Calendar and messaging apps.
This CL gives such apps a chance to have correct CLC. The goal for the
future is to enforce verify_library_check by default.
Bug: 132357300
Test: lunch aosp_cf_x86_64_phone-userdebug && m
Change-Id: Iea3be0fc9d7775c52950848b5a3fd3b7fcd36c53
The section is stripped from binaries (but preserved in /symbols).
However, is it also unintentionally copied into the minidebuginfo
compressed section, which is included in the final shipped binary.
(This does not affect symtab symbols, which are handled as usual)
Removing it from mini-debug-info shrinks some rust binaries by
up to 10x, saving >10MB on the system partition overall (AOSP).
Test: extract minidebuginfo and check it manually with readelf
Change-Id: I462d81455626bb2b5244e1bf3d5f8b71646f3401
With clang r377782, llvm binutils implements all the necessary
funtionaility for strip.sh. We can finally get rid of all the fallback
GNU binutils usage.
Test: m
Bug: 141010852
Bug: 135627985
Change-Id: I110f6028dab7f599decf59a5cb1b927b35e11857
Java modules that are defined in makefiles are not processed in
topological order, so it is necessary to communicate information from
dependencies via dexpreopt.config files. This has already been done in
make/core/dex_preopt_config_merger.py, and now manifest_check.py also
needs to get library names from their dexpreopt.config files.
This is to accommodate Java libraries which name differs from their
modules name. Soong properties `uses_libs`/`optional_uses_libs` and
makefile vars `LOCAL_USES_LIBRARIES`/`LOCAL_OPTIONAL_USES_LIBRARIES`
contain module names, not library names, so it is necessary to translate
them when comparing against library names in the manifest.
Bug: 132357300
Test: lunch cf_x86_64_phone-userdebug && m && launch_cvd \
adb wait-for-device && adb root && adb logcat \
| grep -E 'ClassLoaderContext [a-z ]+ mismatch'
# empty grep output, no errors
Change-Id: I7cfb17d6d74e37fc9f37d977f15f0d711bab5c8e
Previously there were two different scripts that did similar things:
1) build/soong/scripts/manifest_check.py
2) build/make/core/verify_uses_libraries.sh
Both scripts extracted <uses-library> tags and `targetSdkVersion` from
the manifests of Java modules, but 1) worked for XML manifests, and 2)
worked for APKs. This CL reimplements the functionality from 2) in 1),
so that one script can handle both XML manifests and APKs.
Bug: 132357300
Test: lunch cf_x86_64_phone-userdebug && m && launch_cvd \
adb wait-for-device && adb root && adb logcat \
| grep -E 'ClassLoaderContext [a-z ]+ mismatch'
# empty grep output, no errors
Change-Id: Id1b66e4f3f30f307dba70cb111c7571762cb546a
The new mode is enabled with environment variable
RELAX_USES_LIBRARY_CHECK. If the variable is set to true, then a
verify_uses_libraries check failure does not fail the build, instead it
sets a special compiler filter "extract" for dexpreopt, which means that
the DEX file will be extracted, but it won't be compiled to native code.
Class loader context will be set to empty in this case (not &, as it is
going to be deprecated soon).
If the variable RELAX_USES_LIBRARY_CHECK is unset or set to something
other than "true", then the old behaviour of the verify_uses_libraries
check is preserved.
The intended use case for this flag is to have a smoother migration path
for the Java modules that need to add <uses-library> information in
the build files. The flag allows to quickly silence build errors. This
flag should be used with caution and only as a temporary measure, as it
masks real errors and affects performance.
verify_uses_libraries check is reworked so that it writes the error
message to a status file (which is used instead of the former timestamp
file). Currently the stored error message is not used, but it may be
used later to produce a warning. Dexpreopt command checks if the status
file exists and is nonempty; if that is the case, then compiler filter
is set to "extract".
Bug: 132357300
Test: Manually add some mismatch between the libraries in the Android.bp
and Android.mk files for dexpreopted apps, build with
RELAX_USES_LIBRARY_CHECK=true and obsserve that the build doesn't
fail and they are compiled with compiler-filter "extract".
Unset RELAX_USES_LIBRARY_CHECK and observe that the build fails.
Change-Id: Ibb5d993a25b1df1d2e70b7d5aafc6997f9d64e67
DO NOT MERGE / RESTRICT AUTOMERGE and its cousin DO NOT MERGE
ANYWHERE are almost always misused. Make it an error to upload
commits with those merge strategies.
Bug: 156896646
Test: repo upload -n (with DNM and RM commits)
Change-Id: Iae3540fa35dbd7b87a14b53a20be0cd23a206f3a
This script will be used to merge the csv outputs from bloaty into a
single proto file.
Bug: 172339795
Test: atest --host bloaty_merger_test
Change-Id: I9a3c12d7b0c4b5e347fec04e6758bcc1fbc8ab40
Adds a new --key_field option to merge_csv.py which specifies the name
of the field that should be used to sort the input. If specified it
causes that field to be the first in each row and performs the merge
operation of a merge sort on the input files. That assumes that each
input file is already sorted into the same order.
Modifies the rules that use merge_csv.py to pass in:
--key_field signature
to sort the rows by signature.
Bug: 180387396
Test: Verified that hiddenapi files (both aggregated ones and for the
individual modules) are not affected by this change other than
changing the order.
Change-Id: Idcd5f0fea373b520b604889e1c280f21ed495660
Adds the hiddenapi_additional_annotations to allow a library to list
the libraries that provided additional hiddenapi related annotations
for a library.
Modifies merge_csv.py so it can process multiple zip files at the same
time and uses that to merge the embedded .uau files from a module and
those it depends upon.
Bug: 180102243
Test: m droid
Verified that hiddenapi files (both aggregated ones and for the
individual modules) are not affected by this change.
Change-Id: I796520021c7357398a9e2a09f1029e4a578b05b3
This is needed first for ART to enable statsd metrics reporting, but
this will be needed by more modules as they become unbundled.
Test: build-mainline-modules.sh
Bug: 178500083
Change-Id: I5af0b8db7a24c86ffd3f34732ac747a4fc6a7a55
Also, creates a python_binary_host module for generate_hiddenapi_lists
and uses that when constructing the build rule rather than using the
file directly.
Bug: 177317659
Test: m droid
Verified that hiddenapi files (both aggregated ones and for the
individual modules) are not affected by this change.
Change-Id: Ia11bb203ce5a74740d35f1b7e86716e15aad336e
Running find on the entire module directory can lead to flaky failures
when a directory is modified by another rule while find is traversing
it. Pass a list of libraries in the module into the script instead
of finding them on disk.
Fixes: 178509901
Test: TARGET_BUILD_APPS=com.android.runtime dist
Change-Id: Ie4bf288f0e3d99cbbed5ca8770687a9bbe5377e4
This allows the compression to be done on multiple threads
and more importantly, it allows libunwindstack to decompress
only the needed blocks, which reduces unwind time and memory.
Bug: 110133331
Test: verify lazy decompression in libunwindstack
Change-Id: Ia3f597928edcbc85acdf05cb5c83ac7fced39148
When we start switching vanilla platform builds to use ART prebuilts,
this script should still build from sources.
Test: TH
Bug: 172480615
Change-Id: I04cb61b484855d22f22fd9b82f6abb30eb5665c3
Revert submission 1535006-bap
Reason for revert: Looks like it breaks the build as aapt2 depends on the build_number.txt this script was generating. Error log below.
Reverted Changes:
I0f6b34310:Delete build-aml-prebuilts.sh
I4eae4153a:Replace use of build-aml-prebuilts
Change-Id: I7f5f1b3047f3d71eb8ac7bec95c5916425d5d8f2
FAILED: ninja: 'out/aml/soong/build_number.txt', needed by 'out/aml/soong/.intermediates/frameworks/base/tools/aapt2/aapt2/linux_glibc_x86_64/unstripped/aapt2', missing and no known rule to make it
This reverts commit 556b2ad77e.
Original change-id: Ib1b2f0dd2f9ae85b1545c6cc5bb4c5bbdfac1c15
Reason for revert: Add excutable permission to the gen_ndk_backedby_apex.sh.
Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only
Change-Id: Ib587ba200cd7f2f61d478452b43c329d72b2de06
Add gen_ndk_usedby_apex.sh script to generate NDK API list used by Mainlain modules when modules get build.
Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only
Change-Id: Ib1b2f0dd2f9ae85b1545c6cc5bb4c5bbdfac1c15
Dexpreopt and boot jars package check all require access to dex
implementation jars created for java_library and java_sdk_library. They
were available when building from source but not when building from
prebuilts, even though they are embedded within the .apex files that
are referenced from prebuilt_apex.
This changes adds support to prebuilt_apex to export the dex
implementation jars and updates java_import to use those exported dex
implementation jars.
In a source build dexpreopt/boot jars package check access the apex (or
platform) specific variant of a java_library, e.g. core-oj, from which
it retrieves the dex implementation jar path.
After this change in a prebuilt build dexpreopt/boot jars package check
behave in the same way except in this case they retrieve the dex
implementation jar path from the apex (or platform) specific variant of
the java_import, e.g. core-oj.
The work to export files from a `.apex` file for use by other modules
is performed by a new `deapexer` module type. It is not used directly
in an `Android.bp` file but instead is created implicitly by
`prebuilt_apex`,
In order to do that this contains the following changes:
* Adds a new `dexapexer` module type to handle the exporting of files
from the `.apex` file.
* Adds an exported_java_libs property to prebuilt_apex to specify the
set of libraries whose dex implementation jars need exporting.
* Creates apex specific variants of the libraries listed in the
exported_java_libs property.
* Adds the set of exported files to the ApexInfo to make them available
to the apex specific variants.
* Prevents the prebuilt_apex variants from being merged together as
they will not be compatible.
* Modifies java_import to use the exported file for variants of a
prebuilt_apex.
* Adds a ninja rule to unpack (using deapexer) the contents of the
prebuilt_apex's apex file, verify that the required files are present
and make them available as outputs for other rules to use.
* Some minor refactorings to support these changes.
* Adds tests to cover prebuilt only, prebuilt with source preferred,
and prebuilt preferred with source.
Test: m nothing
Bug: 171061220
Change-Id: Ic9bed81fb65b92f0d59f64c0bce168a9ed44cfac
Its usecase can now be supported natively by soong_ui via the
--skip-kati flag + the mainline_sdk product. Migrate
build-mainline-modules.sh to use this new approach, and delete the
script.
Bug: 174315599
Test: build/soong/scripts/build-mainline-modules.sh
Change-Id: I0f6b34310df51ddf060b2912d43487f4d37dd01b
manifest_fixer.py sets a lib's targetSdkVersion to '15' if it is
not explicitly set. But ManifestMerger will add dangerous
permissions CALL_LOG_READ/CALL_LOG_WRITE if the following
conditions are met:
- The app's targetSdkVersion is >16
- A linked lib's targetSdkVersion is <16
- The app has CONTACTS_READ/CONTACTS_WRITE permissions
This condition is triggered for some (OEM) app builds (of
Settings). Update manifest_fixer.py to use '16' as its default
targetSdkVersion so that manifest_merge doesn't add these
dangerous permissions.
Change-Id: Id5f41f7da98c190c8d145fceba0085cf473f4c7a
Without this Soong may run into inconsistencies due to stale prebuilts.
This amends https://r.android.com/1529658.
Test: TH build of NDK
Bug: 175918603
Bug: 172480615
Change-Id: I14680e4547d7d5f560d31ef61b71af8d159c4c53
Should fix build errors on mainline_modules on aosp-master. This script
still does not propagate all options to aapt2 correctly, but that will
be fixed by removing the script altogether (b/174315599).
Test: build/soong/scripts/build-aml-prebuilts.sh art-module-sdk \
art-module-test-exports conscrypt-module-host-exports \
conscrypt-module-sdk conscrypt-module-test-exports \
i18n-module-host-exports i18n-module-sdk \
i18n-module-test-exports platform-mainline-sdk \
platform-mainline-test-exports runtime-module-host-exports \
runtime-module-sdk
(the failing command in
build/soong/scripts/build-mainline-modules.sh)
Bug: 175386777
Change-Id: I81aa20b1abd177f0d2886ab9b04831437da72f2a
In platform builds missing libraries are defined with make variable
INTERNAL_PLATFORM_MISSING_USES_LIBRARIES via soong_config.mk. When Soong
is invoked with "--skip-make" parameter,
INTERNAL_PLATFORM_MISSING_USES_LIBRARIES is not picked up. As a result
the build fails because of missing dependencies.
This CL uses get_build_var to get the value of
INTERNAL_PLATFORM_MISSING_USES_LIBRARIES, then converts it to JSON list.
In the future would be better to migrate both scripts to use --skip-kati
introduced in https://r.android.com/1512613.
Bug: 175286760
Bug: 132357300
Test: forrest build for target "ndk" branch "aosp-master".
Test: Patch build-aml-prebuilts.sh and build-ndk-prebuilts.sh to dump
their soong.variables config, observe that MissingUsesLibraries is
coherent with INTERNAL_PLATFORM_MISSING_USES_LIBRARIES. Start running
each script and observe that they don't fail early due to missing
dependencies.
Change-Id: I5dbc1d3ea7a64de1e8be6332003acc940c2e6a76
Necessary only when platform start using i18n prebuilts, but this helps
CI testing.
Test: build/soong/scripts/build-mainline-modules.sh
Bug: 172480615
Change-Id: Ic24f934ad0b4b7ac7c38d791f80df33315b936e9
Add 'append' command for conv_linker_config so any item can be appended
to linker.config.pb file.
Bug: 174646355
Test: buildbot_build.sh succeeded
Change-Id: Ic3f12ee5e5c0e7ca7c27d4a8cff347e93061eb5f
Add gen_ndk_usedby_apex.sh script to generate NDK API list used by Mainlain modules when modules get build.
Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only
Forrest run: android-build.googleplex.com/builds/forrest/run/L97800000742565481
Change-Id: I39e5aa7c74eff46aba06808642b2bd67ae45ba1e
List of stub libraries from system (and bionic APEX) is currently
managed in static file. This change generates the list at build time and
adds to the system config so linkerconfig can use it.
Bug: 172889962
Test: Build cuttlefish and confirmed list is generated in
/system/etc/linker.config.pb
Change-Id: Ie0400e9d1098b0de19c6a7c25f261384aadde0a3
The xml output from dexdump is more stable than the text output so this
change switches to use that.
Bug: 171479578
Test: m check-boot-jars - for failing and passing cases
Change-Id: Ie347b1b685951524d5ae09db280248c61b4228ee
Necessary for some build steps. aapt2 is one example, although it still
has problems because part of its build logic is in make.
This isn't strictly required for any supported build target at the
moment, but the error when it fails can be confusing so let's add it
anyway.
Test: build/soong/scripts/build-aml-prebuilts.sh art-module-sdk \
art-module-test-exports conscrypt-module-sdk \
conscrypt-module-test-exports conscrypt-module-host-exports \
runtime-module-sdk runtime-module-host-exports \
i18n-module-test-exports i18n-module-sdk platform-mainline-sdk \
platform-mainline-test-exports
Bug: 172548608
Change-Id: I88526a85f288287c3a138f8cbb6fd04ad28c3795
The switch to use dex jars instead of class jars means that a boot jar
that is defined by a dex_import module will now be checked against the
package_allowed_list.txt so it is possible that it will detect
previously unreported problems.
Test: m check-boot-jars - for failing and passing cases
Bug: 171479578
Bug: 125517186
Change-Id: Ie614898dade0fb43c9418d7afb9138169db6f097
Filter out lines starts with "//" from json file to allow simple
comments on the contents. Original json format does not support
comments, but this reduces readability compared to txt file or other
formats. This change allows simple comments on the linker.config.json to
give more information on the contents.
Test: parse succeeded with commented contents
Change-Id: I1c734bf9a054f81f57aa2aea1038d0041297acf1
Adds a singleton that traverses the module variants finding the ones
that are in the list (updatable and non-updatable) of boot jars and
add a ninja rule to ensure that they only contain packages from an
allowed list.
Replaces a hack that ignored any prebuilt boot jars supplied as dex
file with an equivalent one to ensure that they are still ignored.
A follow up change that switches to checking dex jars will allow the
hack to be removed.
The boot jars check can be strict or lax. If strict then all the boot
jars listed in the configuration must be found, otherwise it will only
check the ones it finds. It is strict by default unless
TARGET_BUILD_UNBUNDLED=true or ALLOW_MISSING_DEPENDENCIES=true.
Moves the script and data file from build/make.
Test: m check-boot-jars - for failing and passing cases
SKIP_BOOT_JARS_CHECK=true - no check-boot-jars target created
ALLOW_MISSING_DEPENDENCIES=true - not strict
TARGET_BUILD_UNBUNDLED=true - not strict
verified manually that apart from path differences the same
files (same check sum) were checked in both old make checks and
the new Soong ones
EMMA_INSTRUMENT=true EMMA_INSTRUMENT_FRAMEWORK=true m check-boot-jars
Bug: 171479578
Change-Id: I9d81d6650ba64fc0d48d2dab4ba5a3ba8dd03dec
construct_context.py is a script that runs at Ninja stage and constructs
class loader context arguments for dex2oat. Previously it accepted lists
of library paths and concatenated them into one class loader context
string. The script also handled the dependency of "android.hidl.manager"
on "android.hidl.base" that is not tracked by the build system and must
be handled in a special way.
Now that class loader context representation is going to change from
flat lists to trees, passing paths to individual libraries as script
arguments is no longer possible, because a list of paths cannot
represent a class loader context tree. Passing the trees in a serialized
form is also inconvenient, because the script would have to parse them,
which would complicate it a lot.
Therefore this patch ports all the concatenation and "android.hidl.base"
handling to Soong. It is not possible to port the remaining script to
Soong because Soong has no information about the targetSdkVersiion of
the dexpreopted apps (it is in the manifest and sometimes in an APK, and
has to be extracted at Ninja time).
Test: construct_context_test.py
Test: new subtests in TestUsesLibs
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: Icdb03cf00d1e27e4cff3844b89bfaec4de502dd7
Handle new filetype 'linker_config' which is configuration for
linkerconfig in json type and convert into protobuf in build time.
Bug: 169634881
Test: Build succeeded and cuttlefish boot succeeded
Change-Id: I56555fc738e6d6600d15a191a24f79a2ee747f52
Instead of tracking per module and per module variant, track allowed
list of dependecies for all modules combined. This avoids issues with
different products and different downstream branches having different
build graphs.
To compare allowed_deps.txt vs head, run:
:; m -j out/soong/apex/depsinfo/new-allowed-deps.txt.check
To update source allowed_deps.txt, run:
:; build/soong/scripts/update-apex-allowed-deps.sh
Bug: 149622332
Test: m
Change-Id: Ic518fbd9ebfe1b46aaf9a58df731780a7e5a676b
Merged-In: Ic518fbd9ebfe1b46aaf9a58df731780a7e5a676b
(cherry picked from commit 453555083b)
(cherry picked from commit e5207cd9a6)
Revert submission 1312796-apex-allowed-deps
Reason for revert: b/161974327
Reverted Changes:
I52a4be72e:Add a check for apex/allowed_deps.txt to droidcore...
I56771ba3f:Track allowed transitive deps in any updatable mod...
Change-Id: I3be0a1b0dd824dafeadb485daca8e58b81a3ec5c
Instead of tracking per module and per module variant, track allowed
list of dependecies for all modules combined. This avoids issues with
different products and different downstream branches having different
build graphs.
To compare allowed_deps.txt vs head, run:
:; m -j out/soong/apex/depsinfo/new-allowed-deps.txt.check
To update source allowed_deps.txt, run:
:; build/soong/scripts/update-apex-allowed-deps.sh
Bug: 149622332
Test: m
Change-Id: I56771ba3fea748de8e9c58c80758670572f7af53
Merged-In: Ic518fbd9ebfe1b46aaf9a58df731780a7e5a676b
This allows to reuse SDK version comparison routine from other Python
scripts. With the addition of non-numeric versions comparison has become
more complex, and the deleted shell script did it incorrectly: it used
comparisons like `[[ "S" -lt 28 ]]` which results in "true" as strings
are converted to zero in numeric context. This resulted in adding legacy
libraries to class loader context of apps targeting recent SDK versions.
The error was masked because currently there is only one non-AOSP app
that uses the script (GoogleDialer), and it targets numeric SDK version.
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: I26e752e29b5453fd3e29d5dbfbe4d9df9c0a55b7
Kotlin common multiplatform sources support will require more
complexity in gen-kotlin-build-file.sh, move it to python instead.
Test: m checkbuild
Change-Id: I02312160ad781877f1fec971168331c0dcecf136
Those libraries are not in an APEX, and are provided as stubs
in the platform-mainline-sdk. Therefore compile them explicitly and make
them available in the dist directory.
Test: scripts/build-mainline-modules.sh
Bug: 142935992
Change-Id: I919d8c580e907ce9a7b54366653d0c63ab9d00c7
This script belongs with other manifest-related scripts, and the future
plan is to rewrite it in Python and share common functionality with
other scripts.
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: I4b90129e5023ca1a2c818fc160c34b0b7da943ef
Set the root dir in the project.xml for lint so that the paths in the
lint report are relative to the top of the tree instead of relative
to the project.xml.
Bug: 153485543
Test: run lint
Change-Id: Ie163a4dadd976e708f798855de73e58084931a91
This configuration is to be used for all internal Rust crates.
Test: rustfmt --print-config current --config-path build/soong/scripts/ .
Bug: 157238653
Change-Id: Ic49027dd1530f503e2c1f781453cdee95a983252
Add a rule that runs Android lint on each java and android module
and produces reports in xml, html and text formats.
Bug: 153485543
Test: m out/soong/.intermediates/packages/apps/Settings/Settings-core/android_common/lint-report.html
Change-Id: I5a530975b73ba767fef45b257d4f9ec901a19fcb
Package checks were not being run for java libraries that were in an
APEX and not on the platform. This change fixes that and updates the
script to report all failing classes to make it easier to update the
list of packages.
Test: m java
Bug: 157633658
Change-Id: I28044e08d3a40e9f3464bb2158ef6a28d57264d1
- We can pass all targets at once to build-aml-prebuilts.sh.
- Less noise from build-mainline-modules.sh.
- Default to the same out directory in both build scripts.
- Addressed post-submit comments on https://r.android.com/1170907.
- Various minor cleanups.
Test: build/soong/scripts/build-mainline-modules.sh
Change-Id: Id21ef80c2334462836e217032fc5bf63b7cd04e1
Not sure whether sort, tr or xargs returning non-zero status, but
one of them sometimes does.
Test: manually
Change-Id: I8f2b862de9d6e22dd90068252e393533ec992dfd
Empty results cause grep/egrep to return a non-zero status.
Bug: 145347092
Bug: 141258651
Bug: 69058154
Bug: 143285996
Test: manual run for notice files
Change-Id: I25d54ae307b97a07a98e99c54f283437718fe657
This is an unchanged copy of build/make/tools/generate-notice-files.py
After this change, the make implementation of generate-notice-files.py
will change to take a required parameter for $(PRODUCT_OUT) and will
only include notices for files built under $(PRODUCT_OUT).
Because soong has the ability to walk dependency trees, the soong
version will likely change in the future to use that ability causing
both versions to move in different directions.
After those changes are complete, we can look into factoring out any
remaining shared logic.
Test: run manually and system image notices checked for changes
Change-Id: Id139a66503457615548b46e7996349ca0817e831
Makes them more editor friendly, primarily when copied into
system/linkerconfig/testdata/root.
Test: m
Test: No diff between apex_manifest_full.json in out/
and in linkerconfig/testdata/root/
Bug: 140599044
Change-Id: I6dacdd4baa59e6d70586c4a6343b9b8fd1b574f6
This utility can also be used for parsing and
managing the Android Manifest for apex under
system/apex/apexer.
Test:
python manifest_fixer_test.py
python manifest_check_test.py
BUG: 148198056
Change-Id: Iffe465b5f36b2a3fbf81414d6f19e241c5f5648f
Unit test: go test ./... -test.v -run TestOverrideAndroidApp
Unit test: python manifest_fixer_test.py
BUG: 148198056
Change-Id: Ib5ff235d2a93e88b86aec1c0b16327ea938a094d
This can now discover which make module types have been deprecated and
suggest what needs to happen with the BUILD_BROKEN_USES_* flags.
Bug: 130720555
Test: go run .../build_broken_logs.go * (see header comment)
Change-Id: Iffaa32c0b0079e99b926162257a7c15c530c15a1
If the install apk name is different than the module name, use
test_config_fixer to update the test-file-name value in the config.
Test: app_test.go
Fixes: 147375216
Change-Id: I2141eeebbb3552995400b45634712306673fd812
For modules that provide API surfaces in addition to the standard
current, test and system it is useful to be able to specify the
directory containing the api's .txt files to make it easy to create
multiple API surfaces from within the same Android.bp file. e.g. This
is useful for conscrypt, icu and libcore to manage their intra core
and core platform APIs.
Bug: 145998881
Test: m checkbuild
Change-Id: I753631d9b6993fbf30019fef5c052a9429e519de
If the library does not provide system and test APIs then do not
generate/require corresponding .txt files.
Bug: 145998881
Test: m checkbuild
Change-Id: I21cfdb0b63fd575e8c8c63ea2b436e0c4aa8f3fc
prebuilts.
This runs Soong in skip-make mode, using normal in-make mode only to query
platform versions.
The same ${OUT_DIR} cannot be used for both skip-make and in-make builds,
because Soong generates a smaller build.ninja file in in-make builds where
many build targets are expected to be provided by the mk files. Thus this
script avoids using ${OUT_DIR} if it's an in-make build, defaulting instead
to out-aml/.
The script is based on build-ndk-prebuilts.sh, but uses a separate Soong
variable Aml_abis to enable the appropriate target architectures for
Mainline modules. Aml_abis is very similar to Ndk_abis, except "armeabi-v7a"
is used instead of "armeabi", which is necessary to match prebuilt
dependencies, e.g. for LLVM.
Test: build/soong/scripts/build-aml-prebuilts.sh libart libdexfile_external
(verify that libraries for arm, arm64, x86, x86_64 are built)
Test: build/soong/scripts/build-aml-prebuilts.sh \
out-aml/soong/.intermediates/external/conscrypt/conscrypt-module-sdk/android_common/conscrypt-module-sdk-current.zip
(verify that the zip file contains libconscrypt_jni.so's for all four arches)
Test: build/soong/scripts/build-aml-prebuilts.sh com.android.art.{release,debug,testing,host}
(verify that the build completes)
Test: Two identical build/soong/scripts/build-aml-prebuilts.sh runs after each other
(verify that the 2nd run completes both Soong and ninja steps quickly without any building)
Change-Id: I35712f9f8f0b1cbb77107314c5927c6720e6c3bf
If package_name is set for an android_test module, package names in its
AndroidTest.xml config need to be updated too.
Test: test_config_fixer_test.py
Fixes: 145011263
Change-Id: I52f17ef1d1902364b9bcfec4f6e246bd655dbf09
Previous solution by using objcopy uses a quirky behaviour of the GNU
objcopy and there is no equivalent option in llvm-objcopy.
Instead of removing symbols, extract and repack libgcc to only include
required objects.
Bug: 142585047
Test: presubmit
Change-Id: I58af74c18838f797e481da38c3265f0624fddf99
We rely on a weird, GNU specific behaviour of objcopy:
$ aarch64-linux-android-objcopy -w libgcc.a stripped.a --strip-unneeded-symbol=*
$ llvm-nm --defined-only stripped.a | grep multi
0000000000000000 T __multi3
$ aarch64-linux-android-objcopy -w libgcc.a stripped.a --strip-unneeded-symbol=* --keep-symbol=FOO
$ llvm-nm --defined-only stripped.a | grep multi
llvm-objcopy has a different, more consistent behaviour that does not
mark exported symbols as unneeded when "--keep-symbol" flag exists.
However this behaviour will leave most symbols not deleted in the
libgcc_stripped library.
Revert the change while I'm implementing an alternative solution.
Test: presubmit
Bug: 142585047
Change-Id: I175e811f8a1f6afc6558267fc54e159ad5e12acf
Also fix an issue with deps file introduced in ab5e5142. Now correctly
depends on llvm-readelf insteaf of GNU readelf.
Test: presubmit
Change-Id: Id287e2844626c07917348a62820f8b5edbb22c3d
We had a typo in the original checkin. We need two hypens before
"remove-section" for llvm-strip, just like we have for strip.
Bug: 140721128
Test: Clean build of Go device
Change-Id: Ifb83dc4e6ad8f8070d9ced678f9147d6a040aee7
llvm-objcopy now implements '--regex' option, which allows it to replace
GNU objcopy for creating libgcc_stripped archieve.
Test: build
Bug: 131631155
Change-Id: Ie1733ac8e9b3765f7652a562406dcedb4735cc1a
sysprop_library now checks the API stability itself, cutting dependency
on java_sdk_library. Under the directory {module_dir}/api,
{module_name}-current.txt and {module_name}-latest.txt hold API
signatures.
When sysprop_library is built, or a user run "m {module_name}-check-api"
command, API check is performed. First, current.txt must have exactly
same signature with built sysprop_library module. Second, current.txt
must be compatible with latest.txt.
Build system emits a handy error message to generate/update those API
files, in case of missing or mismatching. Also, a script file for
freezing API files is introduced.
Bug: 131637873
Test: 1) m && boot blueline
Test: 2) m {sysprop_library} performs API check
Test: 3) manual test for check-api, freezing api
Change-Id: I9d25f5dc64299e666527ca8e23d7233966901c4e
Merged-In: I9d25f5dc64299e666527ca8e23d7233966901c4e
Merged-In: Ib7ad4f17e82c90da5ef3f80e2ab88c0b53112c60
(cherry picked from commit 093f0eb133)
