Commit graph

409 commits

Author SHA1 Message Date
Sorin Basca
1bb6d6a2c9 Adding jdk.internal.misc in the allowed packages
Updating OpenJDK 11 means that there is a dependency on
jdk.internal.misc, which is renamed from sun.misc. The new
package is added to the allowed list.

Bug: 190470684
Bug: 190791083
Test: m
Change-Id: I174df7bad5c5ec4286c35998e16c26bcd0737612
2021-09-22 15:18:45 +00:00
Treehugger Robot
77cdde7d91 Merge "Fix statusfile used in dexpreopt" 2021-09-20 19:53:38 +00:00
Spandan Das
3d5cd4d109 Fix statusfile used in dexpreopt
https://android-review.googlesource.com/c/platform/build/soong/+/1820496/1/scripts/manifest_check.py#361
caused a regression wrt to statusfile creation. Specifically it inverted
the logic used to create the status file. This file is used in
dexpreopt, thereby affecting all odex files

Test: In internal git, m <an_odex_file>. File size is restored to
pre-regression value
Bug: 199921980
Bug: 200329228

Change-Id: If1e61ac8e2b95455dacc6e7b9431c8f6e1f339b3
2021-09-20 18:24:56 +00:00
Christopher Ferris
8e5feaa1df Make the script python3.
Test: Ran the python version of the script and the python3 version
Test: and verified the output is the same. This is for all three
Test: outputs (html, text, xml).
Test: Builds without an error.
Change-Id: I344504a4351c14e00caeb109c5507a60fb6be05a
2021-09-13 14:54:01 -07:00
Peter Collingbourne
e168f15ab9 Merge "Read LLVM version from $LLVM_PREBUILTS_VERSION if set." 2021-09-09 18:17:11 +00:00
Treehugger Robot
da513b1ec1 Merge "Apply pylint to scripts/manifest_check*.py" 2021-09-08 23:42:25 +00:00
Peter Collingbourne
cbfbd64823 Read LLVM version from $LLVM_PREBUILTS_VERSION if set.
This matches the behavior of the build system.

Change-Id: Idce0a48403acde100e20246896a1ec68df6b73c7
2021-09-08 13:46:01 -07:00
Spandan Das
96d5424440 Merge "Apply pylint to check_boot_jars.py" 2021-09-08 18:32:02 +00:00
Spandan Das
f880742582 Apply pylint to scripts/manifest_check*.py
1. Run pyformat scripts/<script>.py -s 4 --force_quote_type
none -i to fix formatting
2. Annotate #pylint: disable, where straightforward fix is not available

Test: m manifest_check_test
Test: pylint --rcfile tools/repohooks/tools/pylintrc
build/soong/scripts/manifest_check.py
build/soong/scripts/manifest_check_test.py
Bug: 195738175

Change-Id: I9af498c4abd6ac9f8b9df4f93cbdd4424eacff8e
2021-09-08 03:59:40 +00:00
Nikita Iashchenko
15c1387710 Add jdk.internal.math to package_allowed_list.txt (OpenJDK 11)
This is part of merging upstream changes from OpenJDK 11.28. This CL
adds jdk.internal.math to package_allowed_list.txt

Bug: 199067710
Test: atest CtsLibcoreOjTestCases
Change-Id: I343bed53ce6b52a6b373a75828b3234073b9cba8
2021-09-06 17:38:34 +01:00
Spandan Das
2c0af22ea8 Apply pylint to check_boot_jars.py
1. Run pyformat scripts/check_boot_jars/check_boot_jars.py -s 4
--force_quote_type none -i to fix formatting.
3. use pylint: disable=<X> where fixes are not obvious

Test: m check_boot_jars
Test: pylint --rcfile tools/repohooks/tools/pylintrc
scripts/check_boot_jars/check_boot_jars.py
Bug: 195738175

Change-Id: I0f86d95e0d174c9f87a2872dedaa65cd3e814246
2021-09-03 18:01:49 +00:00
Treehugger Robot
0364846a73 Merge "Apply pylint to remaining scripts in hiddenapi" 2021-09-01 14:27:13 +00:00
Treehugger Robot
deee92ca0c Merge "Apply pylint to scripts/hiddenapi/verify_overlaps*" 2021-09-01 14:27:13 +00:00
Treehugger Robot
373fe3b598 Merge "Apply pylint to scripts/hiddenapi/generate_hiddenapi_lists*" 2021-09-01 14:27:13 +00:00
Spandan Das
559132fb98 Apply pylint to scripts/hiddenapi/verify_overlaps*
1. Run pyformat scripts/hiddenapi/verify_overlaps.py -s 4
--force_quote_type none -i to fix formatting. Some double quotes change
to single quotes since pyformat enforces consistent quotes
2. Wrap #pylint: (enable|disable)=line-too-long in tests

Test: m verify_overlaps verify_overlaps_test
Test: pylint --rcfile tools/repohooks/tools/pylintrc <file1>
<file1_test>
Bug: 195738175

Change-Id: Ieb23c04aa53a8b74917f6865296dd8740ecf20ba
2021-08-31 18:24:24 +00:00
Christopher Ferris
cd989defab Merge "Make this script use python3." 2021-08-27 21:42:09 +00:00
Christopher Ferris
cce6d652c5 Make this script use python3.
There is nothing in this script that is not python3, so
make it python3. Without this, a system without python will
fail when trying to run gdbclient.py.

Test: Ran gdbclient.py on a system without a python executable.
Change-Id: I9f8418932628569c445ee01bc6f9ead8aed7989f
2021-08-27 11:54:19 -07:00
Treehugger Robot
5edc77d06c Merge "Apply pylint to conv_linker_config.py" 2021-08-27 04:04:14 +00:00
Spandan Das
2c2219b60e Apply pylint to remaining scripts in hiddenapi
1. Run pyformat scripts/hiddenapi -s 4
--force_quote_type none -i to fix formatting.
2. rename restricted variable names (e.g. variable name "input" has been
changed to "csvfile")
3. use pylint: disable=<X> where fixes are not obvious

Test: m merge_csv signature_patterns signature_patterns_test
Test: pylint --rcfile tools/repohooks/tools/pylintrc <file1>
<file1_test>
Bug: 195738175

Change-Id: I800a208f9c0ee1d32e68e4b20fd5933b3ab92c0e
2021-08-27 00:30:41 +00:00
Spandan Das
c2c31b13dc Apply pylint to scripts/hiddenapi/generate_hiddenapi_lists*
1. Run pyformat scripts/hiddenapi/generate_hiddenapi_lists.py -s 4
--force_quote_type none -i to fix formatting. Some double quotes change
to single quotes since pyformat enforces consistent quotes
2. change map/filter to list comprehension to fix pylint's bad-builtin
warning
3. use pylint: disable=<X> where fixes are not obvious

Test: m generate_hiddenapi_lists_test
Test: pylint --rcfile tools/repohooks/tools/pylintrc <file1>
<file1_test>
Bug: 195738175

Change-Id: I15e71d8d81f0e3fa66d84e6e62eb1848963136e6
2021-08-27 00:09:25 +00:00
Spandan Das
1d4bfd8ca4 Apply pylint to conv_linker_config.py
1. Run pyformat scripts/conv_linker_config.py -s 4 --force_quote_type none -i to fix formatting
2. Annotate #pylint: disable=import-error for linker_config_pb2 since it
will be provided by soong

Test: m conv_linker_config
Test: pylint --rcfile tools/repohooks/tools/pylintrc
build/soong/scripts/conv_linker_config.py
Bug: 195738175

Change-Id: I791576cf65cb053f68c804f8ec5c2fc22976fdb4
2021-08-26 16:48:59 +00:00
Spandan Das
ec555f1a99 Apply pylint to scripts/construct_context*.py
1. Run pyformat scripts/construct_context.py -s 4 --force_quote_type
none -i to fix formatting
2. Wrap #pylint: [disable|enable]=line-too-long in tests

Test: m construct_context construct_context_test
Test: pylint --rcfile tools/repohooks/tools/pylintrc
build/soong/scripts/construct_context.py
build/soong/scripts/construct_context_test.py
Bug: 195738175

Change-Id: I2e756b17aae3e70ddaa03356c78f1f2f85c0f6a5
2021-08-26 01:06:48 +00:00
Matthew Maurer
ca68c49621 rust: doc: Zip up docs when done
Fetching artifacts from the build server doesn't go well with large
numbers of files in nested directories.

Produce an additional rustdoc.zip artifact to make this easier.

This change also prevents ${DIST} from receiving the unpacked rustdoc
site. This can be changed if b/188822051 is fixed.

Bug: 162741284
Test: m rustdoc; check resulting zip contents manually
Change-Id: I80b6a8fa6e274d2d8c3419d8734251afd4d7dba7
2021-08-20 13:08:16 -07:00
Paul Duffin
280bae6d20 Filter blocked entries from modular flag files
Previously, the sdk snapshot would include all the entries from the
stub-flags.csv and all-flags.csv modular files generated by a single
bootclasspath_fragment. That included implementation details, i.e.
class members that are not part of a stable API or the hidden API which
meant that the sdk snapshots were implementation dependent.

This change removes the implementation details from the modular flag
files, i.e. those entries that are only marked as "blocked". When
comparing the files against the corresponding subset of the monolithic
files it assumes that any entries missing from the modular flag files
are blocked.

Bug: 194063708
Test: atest --host verify_overlaps_test signature_patterns_test
      m out/soong/hiddenapi/hiddenapi-flags.csv
      - manually change files to cause difference in flags to check
        that it detects the differences.
Change-Id: I6b67b2253cf029d6830b58a06ebb0c8fcaa0dd71
2021-08-11 17:29:47 +01:00
Paul Duffin
726d44a86d Merge "Remove member signature and inner classes from signature-patterns.csv" 2021-08-11 09:08:01 +00:00
Treehugger Robot
1dde1fade7 Merge "[scripts] Print full clang prebuilt dir from get_clang_version.py" 2021-08-11 06:57:27 +00:00
Paul Duffin
6ffdff853c Remove member signature and inner classes from signature-patterns.csv
Previously, the signature-patterns.csv file included a lot of
implementation details, e.g. the signatures of dex members or inner
classes that are not part of any API, including the hidden API.

This change will remove all member signatures and inner class names
from the file and replace them with just the outermost qualified class
name.

That will still leave some implementation details, e.g. the names of
implementation only classes and packages.

Bug: 194063708
Test: atest --host verify_overlaps_test signature_patterns_test
      m out/soong/hiddenapi/hiddenapi-flags.csv
      - manually change files to cause difference in flags to check
        that it detects the differences.
Change-Id: I9de6a2a6129e875e19f7ded5fae578cbdb584660
2021-08-10 14:01:51 +01:00
Paul Duffin
c11f6679d8 Use trie to store monolithic hidden API flags
Previously, a simple map from dex signature to the flags was used to
store the monolithic hidden API flags. This change replaces that with a
trie that uses packages and class names extracted from the signature
to map from the signature to the flags.

The signature is broken down into pieces consisting of package names,
class names and the remaining member signature. They are used in turn
to navigate through nodes in the trie to a Leaf that contains the flags
associated with the signature.

The flags can be retrieved from the trie in a number of ways:
* Using the complete signature to retrieve the flags for a single
  member.
* Using a class name to retrieve the flags for all the members in a
  class and any nested classes.
* Using a package wildcard to retrieve the flags for all the members in
  the classes in that package.
* Using a recursive package wildcard to retrieve the flags for all the
  members in the classes in that package and nested packages.

This will allow a bootclasspath_fragment to select a subset of the
monolithic flags to compare against itself without specifying the
signatures of every member of that set.

Before creating the trie an attempt was made to compute the subset by
iterating over all the signatures in the monolithic flags and matching
against a regular expression created from the patterns but that was too
slow. It took minutes to run whereas using the simple map or the trie
only takes a few seconds.

Bug: 194063708
Test: atest --host verify_overlaps_test
Change-Id: I36f5e319d3e7d62dd34305de1eec990a93cb3a89
2021-08-09 19:51:33 +01:00
Paul Duffin
67b9d61ac2 Separate creation of signature patterns from overlap checking
Previously, the signatures used to select the subset of the monolithic
flags were simply the signatures read from the modular flags file. This
change moves the creation of the signature list into a separate script
that outputs the signatures to a file and then passes the path through
Soong from the bootclasspath_fragment modules that create it to the
platform_bootclasspath module that uses it to compare the modular
flags against the monolithic flags.

Currently, the signatures are the full signatures but follow up changes
will replace them with patterns (hence the name) that avoids having to
include implementation details in the hidden API flags that are output
as part of a bootclasspath_fragment's snapshot.

This change moves the stub flags related code next to the all flags
related code as they are treated in a similar way.

Bug: 194063708
Test: atest --host verify_overlaps_test signature_patterns_test
      m out/soong/hiddenapi/hiddenapi-flags.csv
      - manually change files to cause difference in flags to check
        that it detects the differences.
Change-Id: I2855bf6d05c91b8a09591664185750361c7e644f
2021-08-06 13:40:34 +01:00
Pirama Arumuga Nainar
238ca381c4 [scripts] Print full clang prebuilt dir from get_clang_version.py
e.g. clang-r416183b1 instead of just r416183b1.

Test: manually run the script
Change-Id: I993202ea97433f841e3127ecef46dec66fb9a8c9
2021-08-05 23:39:00 +00:00
Paul Duffin
53a7607f7d Extract a subset of the monolithic flags for comparison
Previously, the compare_signature_flags compared the full set of
monolithic flags against the modular flags and just ignored any
monolithic flags for a signature that was not present in the modular
flags. It essentially performed two tasks, identifying the subset of
the monolithic flags against which the modular flags should be compared
and then doing the actual comparison.

This change separates those two tasks, by first creating a dict that is
the subset of the monolithic flags against which the modular flags will
be compared and then do the comparison.

At the moment the subset is simply the set of monolithic flags
associated with the signatures that are present in the modular flags.
Follow up changes will provide other ways to select the subset.

Bug: 194063708
Test: atest --host verify_overlaps_test
      m out/soong/hiddenapi/hiddenapi-flags.csv
      - manually change files to cause difference in flags to check
        that it detects the differences.
Change-Id: Icb547f071b814dd955f31255b0d35597f7ba4aa1
2021-08-05 14:19:57 +01:00
Paul Duffin
7be96337f2 Clarify symbols used in verify_overlaps.py
Switches symbols to use monolithic... and modular... to try and clarify
the purpose of those symbols.

Bug: 194063708
Test: atest --host verify_overlaps_test
Change-Id: Ifc646edebe3d43b51dc3528fa883718399cb5254
2021-08-05 14:19:57 +01:00
Paul Duffin
428c65156b Add tests for verify_overlaps script
Refactor verify_overlaps to make it testable and add tests for the
comparison. It does make one significant change in behavior which is to
read each of the files produced by a bootclasspath_fragment into a dict
before comparison, rather than reading and comparing them a row at a
time. That allows it to reuse the code to read a CSV into a dict.

Bug: 194063708
Test: atest --host verify_overlaps_test
      m out/soong/hiddenapi/hiddenapi-flags.csv
      - manually change files to cause difference in flags to check
        that it detects the differences.
Change-Id: Ib70ac87fe089fc25e3bef18f367d4939bfc0cb8d
2021-08-05 14:16:51 +01:00
Dan Willemsen
4591b6496d Upgrade to golang protobuf api v2
Major operations:
 * Fix the go_package entries to be consistent, as the new tool
   complains when it doesn't have a '/'.
 * Regenerate with the new protoc-gen-go tool.
 * github.com/golang/protobuf -> google.golang.org/protobuf
 * proto.[Un]MarshalText -> prototext.[Un]Marshal

Change-Id: Ie1147bd2457fafb66ba555461b3bf14f0561a25d
2021-07-23 11:12:05 -07:00
Ulyana Trafimovich
be67b7a7eb Merge "manifest_check.py: trim namespace part of the module, if needed." 2021-07-22 09:12:25 +00:00
Paul Duffin
2cde5f4389 Add target for generate_hiddenapi_lists_test
atest generate_hiddenapi_lists_test would not work without setting
verbosity=2.

Bug: 194063708
Test: atest --host generate_hiddenapi_lists_test
Change-Id: Ic076ea85d58917921081ae33e72f06011c4ca49e
2021-07-20 19:17:49 +01:00
Ulya Trafimovich
1b513458b1 manifest_check.py: trim namespace part of the module, if needed.
Normally Soong does that automatically when it handles module names
specified in Android.bp properties. However not all <uses-library>
entries in the manifest correspond to real modules: some of the
optional libraries may be missing at build time. Therefor this script
accepts raw module names as spelled in Android.bp/Amdroid.mk and trims
the optional namespace part manually.

Bug: 193891722
Test: manifest_check_test.py
Test: $ lunch aosp_cf_x86_64_phone-userdebug && m && launch_cvd
      $ adb wait-for-device && \
        adb root && \
        adb logcat | \
        grep -E 'ClassLoaderContext [a-z ]+ mismatch' -C1
      # empty output, no errors

Change-Id: I184fb8a2dc26f49e90fb597ebcd6a58c00705206
2021-07-20 14:30:40 +01:00
Pirama Arumuga Nainar
54f92016bc Merge "[scripts] Add script to print clang prebuilts version" 2021-06-29 18:29:09 +00:00
Pirama Arumuga Nainar
2558ce3890 [scripts] Add script to print clang prebuilts version
Bug: http://b/187231324

This script parses soong/cc/config/global.go to return clang prebuilts
version.  (It may get the version from a json config file in the
future.)

Currently it's used in development/scripts/symbol.py and is the
recommended way to get clang prebuilts version number in the future.

Test: get_clang_version_test.py
Change-Id: Id92a74fd523a5270999bb4d6454525bf2f2f0c04
2021-06-25 11:57:16 -07:00
Paul Duffin
2e88097152 Verify the modular stub flags are subsets of the monolithic stub flags
Bug: 179354495
Test: m out/soong/hiddenapi/hiddenapi-stub-flags.txt
      - check that an error is reported if a modular stub-flags.csv file,
        i.e. one created by a fragment is not a subset of the monolithic
        file, e.g. because a signature in the modular file has different
        flags than it does in the monolithic or is not present there.
Change-Id: I46ebb495cb093a5e3abe7571c49933c845318549
2021-06-25 14:49:10 +01:00
Orion Hodson
6a540c4e48 Update dex_preopt related OWNERS
Bug: none
Test: none
Change-Id: I1e6c69b6e3abb54fd37f5b59dad4ced5ce761289
2021-06-17 17:05:39 +01:00
Martin Stjernholm
2ed110c092 Merge "Clean up statsd-module-sdk-for-art." 2021-06-17 08:04:23 +00:00
Sasha Smundak
a3be792b10 Wrapper script to use Starlark-based configuration
Test: manual
Bug: 181797530
Change-Id: I51edea16982386a1022c77ddd282e95b2c5d77a2
2021-06-14 15:52:08 -07:00
Martin Stjernholm
6fe860b47f Clean up statsd-module-sdk-for-art.
No longer needed.

Test: build/soong/scripts/build-mainline-modules.sh
Bug: 188761143
Bug: 180769414
Change-Id: I3524095d0399710e5f474fcc322eba008abfb797
2021-06-14 19:50:58 +01:00
Paul Duffin
84c1cdf31f Maintain header order in merge_csv
Previously, if the --header property was not specified then merge_csv
would use a header constructed by sorting all the fields in the input
files. That required that any use of merge_csv which did not already
have headers in the required order would have to explicitly specify the
headers. That made it harder to use merge_csv as a generic tool as each
invocation needed to be aware of what headers were exported in the
output.

This change causes merge_csv to simply use the headers in the order in
which they are encountered in the input files. That removes the need to
specify the --header option when generating the index files.

Bug: 179354495
Test: m out/soong/hiddenapi/hiddenapi-index.csv out/soong/hiddenapi/hiddenapi-unsupported.csv
      - make sure that they are not changed by this change.
Change-Id: I420b7d07aea85af6372cd7580a8be5e2cc82a513
2021-06-09 14:02:03 +01:00
Martin Stjernholm
040e0443c3 Remove use of fake soong.variables.
Use the --soong-only flag instead to create it the proper way with the
config make step. This fixes build problems arising from the incomplete
soong.variables settings.

This partially relands https://r.android.com/1535202 so that we'll use
TARGET_PRODUCT=mainline_sdk in module SDK builds everywhere.
build-aml-prebuilts.sh isn't removed completely because we still need
the special case to create build_number.txt.

Test: art/build/build-art-module.sh --skip-apex && \
      packages/modules/ArtPrebuilt/update-art-module-prebuilts.py \
        --local-dist out/dist
  then check that the created prebuilt dex2oatd64 runs with the repro
  in b/190194345
Test: build/soong/scripts/build-mainline-modules.sh
  then
      prebuilts/runtime/mainline/update.py --local-dist .../out/dist
  in a master-art tree and check that art/build/build-art-module.sh
  works there
Bug: 190194345
Bug: 174315599
Change-Id: I86327b082b61c292f18c0a6b315d9add08002832
2021-06-08 11:36:53 +01:00
Lukacs T. Berki
1af9316e0a Remove sort_bss_symbols_by_size.
Its only use was removed in aosp/1717013.

Bug: 187928080
Test: Presubmits.
Change-Id: If6acbfe8ba21ec4dc34166768ba1bde3bb78123c
2021-06-07 09:23:15 +02:00
Samiul Islam
d365449fae Do not build compressed APEX
This script expects to build uncompressed APEX and use them as
prebuilts.

Bug: 189926000
Test: build_mainline_modules.sh
Change-Id: If0f62c2baa3db3c079b2913fcdc3dafd66394011
2021-06-02 15:34:42 +01:00
Dan Albert
7f0e9f970c Merge "Add script for building and archiving rustdocs." 2021-05-14 22:52:35 +00:00
Dan Albert
4d247476ac Add script for building and archiving rustdocs.
Test: OUT_DIR=out DIST_DIR=dist build/soong/scripts/build-rustdocs.sh
Bug: http://b/162741284
Change-Id: Ie94318ed9cc99024a27d414bfecc083b2fa96b57
2021-05-14 13:18:07 -07:00
Paul Duffin
dfa1083fee Validate monolithic and modular hidden API flags are consistent
This makes sure that where there is overlap between the hidden API
flags generated for a module and the monolithic flags that they are
identical. That ensures that the modular hidden API flags will be
compatible with previous releases that relied on the monolithic flags.

Bug: 179354495
Test: m out/soong/.intermediates/art/build/boot/art-bootclasspath-fragment/android_common_apex10000/modular-hiddenapi/all-flags.csv
      m out/soong/hiddenapi/hiddenapi-flags.csv
      - Create some inconsistencies between the above two files.
      m out/soong/hiddenapi/hiddenapi-flags.csv.valid
Change-Id: Iaf9e23cef63e221608955d89dc8d496bcc70c86e
2021-05-14 01:49:19 +01:00
satayev
73c7b8547a Move conv_classpaths.proto to packages/modules/common.
The classpaths.proto file is needed at build time to generate appropriate
classpath configs for the modules. Unbundled ART branches do not include
packages/modules/SdkExtenstion in their manifest; so put the proto in a
project that should be visible to all unbundled modules.

Bug: 180105615
Test: m nothing
Change-Id: Ie5be1822fbee92f280332622a9317894cc98d7ac
2021-05-10 13:14:23 +01:00
David Srbecky
69315e4ea7 Use create_minidebuginfo tool instead of bash script.
The behaviour is semantically identical, however,
the tool additionally sorts the symbols by address,
compresses frame unwind information more efficiently,
and improves random-accessibility for lazy decompression.

Overall, the changes balance and the output size is same,
however, libunwindstack can access the data much faster
while using less memory (due to the lazy decompression).
It will also enable further improvements in the future.

Bug: 110133331
Test: ART unwinding tests, run prefetto on the device.
Change-Id: Id48f9fe67fb67fcf2b90cc3b217b71bb8f5147ca
2021-05-01 00:06:07 +01:00
Jaewoong Jung
3c87b1df11 Fix minor issues in updatability lint CLs.
Test: TreeHugger
Bug: 182349282
Change-Id: Ifae282d51b088d0562605b41a09696d75c5138f1
2021-04-22 13:40:41 -07:00
Paul Duffin
2082e695ce Allow generate_hiddenapi_lists.py to work with no custom flag files
Previously, if generate_hiddenapi_lists.py was invoked without any
custom flag files, e.g. hiddenapi-unsupported.txt, then it would fail.
This change allows it to be used for generating the hiddenapi lists
for modules that do not have any custom flag files,
e.g. framework-sdkextensions

Bug: 179354495
Test: verified that the monolithic out/soong/hiddenapi/... files are
      unchanged by this change
Change-Id: I455d453024c9f06ed59cbc1e9838234f8b7c7317
2021-04-21 17:15:25 +01:00
Jaewoong Jung
7b93908d2e Add lint_project_xml_test.py
Test: lint_project_xml_test.py
Bug: 182349282
Change-Id: Ibdeb2e5aaf9dcdb02c5d30214e5ea89665145eb3
2021-04-20 11:04:51 -07:00
Jaewoong Jung
5a4202575e Rename lint-project-xml.py to remove dashes.
So that a future test can import it without a dirty syntax.

Test: m lint-check
Bug: 182349282
Change-Id: I520a5af49543801ab2a8ee888ff235876546dc74
2021-04-20 07:06:03 -07:00
Jaewoong Jung
a110439b3c Lint baseline file check in lint-project-xml
Add a function to lint-project-xml to scan the given lint baseline file
for disallowed baseline lint issues.

Test: Manual
Bug: 182349282
Change-Id: I0db32bec0da24487b2f2b3f6704629f56ae76f56
2021-04-19 17:38:17 -07:00
Treehugger Robot
7548b1ad05 Merge "Remove draft API support." 2021-04-15 22:58:15 +00:00
Jooyung Han
e134d09831 Add 'merge' command to conv_linker_config
'merge' command can merge multiple configurations.
It just delegates protobuf's msg.MergeFrom(other_msg).

  usage: conv_linker_config merge -o out.pb -i a.pb -i b.pb ...

out.pb will have all keys from multiple inputs.

Bug: 181093750
Test: m
Test: MicrodroidTestCase
Change-Id: Ibf715aa0bcc1e5c82c85a9af8fe7dca8d6ab68ad
2021-04-15 10:58:21 +09:00
Dan Albert
d05ba00415 Remove draft API support.
This is unused and doesn't work with our current API review process
anyway (un-drafting an API won't be flagged for API council review).

Test: treehugger
Bug: None
Change-Id: I6d8fcc9885b82dac5ada7772d9e3fb9101524ece
2021-04-13 15:55:47 -07:00
David Srbecky
789fe0fdeb Revert^2 "[strip.sh] Move remaining GNU binutils usage to llvm binutils"
This reverts commit 33dfba1d9c.

Bug: 141010852
Test: art/test.py -t 137-cfi
Test: llvm-objcopy --dump-section=.gnu_debugdata=mdi.xz libhwui.so && \
      unxz -fk mdi.xz && readelf -a mdi | less
Change-Id: I751b51274a961194647ec11fd3fd49886df1e1b3
2021-04-08 18:31:46 +01:00
Artur Satayev
ca6fe3453a Add a converter for classpaths.proto between proto and json/text formats.
The proto is used at build time to define classpath fragments to be read
and parsed at runtime by `derive_classpath` service; in order to define
and set BOOTCLASSPATH, DEX2OATCLASSPATH, and SYSTEMSERVERCLASSPATH
environ variables.

The tool is used for debugging and generation of binary protos.

Bug: 180105615
Test: m && launch_cvd with a follow up changes
Change-Id: I7e8b084e5f63e9aaad11da3221b909818e69e235
2021-04-06 18:37:13 +01:00
Yi Kong
6f43f54264 Fix toc.sh, llvm-nm does not recognise '-f P' option
llvm-nm only accepts full name for format option, or just '-P' alias.

Test: build
Bug: 184360305
Change-Id: Ie25f5410403f4dbca564430149a5753e0d48d60d
2021-04-03 03:21:24 +08:00
Yi Kong
948205699b Merge "Move toc.sh to use LLVM binutils" 2021-04-02 02:38:16 +00:00
Ulyana Trafimovich
400a1b1309 Merge "Emit a better error message in manifest_check.py." 2021-04-01 09:58:15 +00:00
Yi Kong
4ad44e713b Move toc.sh to use LLVM binutils
Test: build
Change-Id: Ib965b6ea3fba6ae007135ac7e167cc50f0f6e3de
2021-04-01 17:45:42 +08:00
Yi Kong
9d9e053964 Merge "Move gen_sorted_bss_symbols.sh to use LLVM binutils" 2021-04-01 03:10:08 +00:00
Treehugger Robot
fdfb9540f6 Merge "Do not build stats-log-api-gen-exports in mainline builds" 2021-03-31 18:48:18 +00:00
Ulya Trafimovich
bb7513d80a Emit a better error message in manifest_check.py.
Bug: 132357300
Test: lunch aosp_cf_x86_64_phone-userdebug && m
Test: temporarily patch Gallery2 to fail the check, observe the error
Change-Id: Id67e359188396f68dcecd8b3f4d1bc26271af56d
2021-03-31 14:31:49 +01:00
Yi Kong
bf1db7b149 Move gen_sorted_bss_symbols.sh to use LLVM binutils
Test: build
Change-Id: I496b92d963d127087daef405dafa24377d6bac97
2021-03-31 17:13:43 +08:00
Sophie Zheng
dd3797b44e Merge "NDK API coverage" 2021-03-30 20:44:44 +00:00
Sophie Zheng
a4c09e1894 Merge "NDK API coverage" 2021-03-30 20:40:41 +00:00
Eric Holk
406765b15a Do not build stats-log-api-gen-exports in mainline builds
This no longer needs to be a prebuilt since stats-log-api-gen can be
built from proto_logging without extra dependencies.

Test: Treehugger
Bug: 170149255
Change-Id: I6925c2526cee1cbff3ba78bedd7f6c2f58dfff0f
2021-03-29 15:53:32 -07:00
sophiez
d3e2742ad2 NDK API coverage
Update OWNERS file under build/soong/scripts to include coverage team as owner of related scripts.

Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only
Change-Id: I6150d5bda0d59828a985cd2ab5dd0aa38a7987c3
2021-03-29 20:29:35 +00:00
Ulya Trafimovich
9f12df9abb Don't fail the build if manifest_check cannot extract targetSdkVersion.
Instead, return "any" SDK version 10000, which will cause dexpreopt to
not add any compatibility libraries to class loader context. This is as
good guess as any, and the build system already uses "any" version if
there is not manifest or APK from which to extract targetSdkVersion.

This fixes broken build on cf_x86_auto-userdebug.

Bug: 132357300
Bug: 183943146
Test: lunch cf_x86_auto-userdebug \
  && m out/target/product/vsoc_x86/obj/APPS/playback_intermediates/dexpreopt.zip
Change-Id: I5ee75639ece716c3a9800ddea346fe7c73da7c3a
2021-03-29 15:43:38 +00:00
Ulyana Trafimovich
a8f6db972f Merge changes I1f0ab4af,Iea3be0fc
* changes:
  Fix AAPT path in dexpreopt.
  Allow modules that don't run verify_uses_libraries to have nonempty CLC.
2021-03-29 09:24:00 +00:00
sophiez
ae5da5ba78 NDK API coverage
Update the gen_ndk_usedby_apex.sh script to generate NDK API list with
library name.

Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only

Change-Id: I799e03ad7d81a4184229f658fd6c8824b9ba6e1f
2021-03-25 22:12:04 +00:00
Treehugger Robot
c29088b727 Merge "Remove .rustc ELF section from mini-debug-info." 2021-03-23 09:13:53 +00:00
Ulya Trafimovich
bd6b076f80 Allow modules that don't run verify_uses_libraries to have nonempty CLC.
There are modules that would have passed verify_uses_library check, but
don't run it for some reason (the check gets enabled either with an
explicit setting, or if the module has nonempty <ues-library> lists in
the build properties). Previously all such modules were assumed to have
empty CLC, which is not always true. In particular, compatibility
libraries are ignored, which affected e.g. Calendar and messaging apps.

This CL gives such apps a chance to have correct CLC. The goal for the
future is to enforce verify_library_check by default.

Bug: 132357300
Test: lunch aosp_cf_x86_64_phone-userdebug && m
Change-Id: Iea3be0fc9d7775c52950848b5a3fd3b7fcd36c53
2021-03-22 10:49:31 +00:00
satayev
2b077baa5e Merge changes from topic "move_allowed_deps_txt"
* changes:
  Treat allowed_deps.txt source file as optional.
  Move allowed_deps.txt to packages/modules/common.
2021-03-22 10:48:45 +00:00
David Srbecky
b474fc0ad6 Remove .rustc ELF section from mini-debug-info.
The section is stripped from binaries (but preserved in /symbols).

However, is it also unintentionally copied into the minidebuginfo
compressed section, which is included in the final shipped binary.
(This does not affect symtab symbols, which are handled as usual)

Removing it from mini-debug-info shrinks some rust binaries by
up to 10x, saving >10MB on the system partition overall (AOSP).

Test: extract minidebuginfo and check it manually with readelf
Change-Id: I462d81455626bb2b5244e1bf3d5f8b71646f3401
2021-03-20 20:31:29 +00:00
Treehugger Robot
f2be52c4dc Merge "Revert "[strip.sh] Move remaining GNU binutils usage to llvm binutils"" 2021-03-20 19:25:23 +00:00
David Srbecky
33dfba1d9c Revert "[strip.sh] Move remaining GNU binutils usage to llvm binutils"
This reverts commit ad50ce8349.

Bug: 183237575
Reason for revert: Breaks stack unwinding

Change-Id: I256dc77525687fb3e670355587b1eac72158b274
2021-03-20 14:46:15 +00:00
Treehugger Robot
8cc04e59e1 Merge "manifest_check.py: translate library names using dexpreopt configs." 2021-03-20 12:12:52 +00:00
Julien Desprez
2dc86b2956 Migrate manifest_*_test out of test mapping
Unit tests do not require an explicit TEST_MAPPING config

Change-Id: Ieccebc78a26b9555803a2278e2962d0b388a6957
Test: presubmit
Bug: 183209529
2021-03-19 17:16:16 +00:00
Yi Kong
ad50ce8349 [strip.sh] Move remaining GNU binutils usage to llvm binutils
With clang r377782, llvm binutils implements all the necessary
funtionaility for strip.sh. We can finally get rid of all the fallback
GNU binutils usage.

Test: m
Bug: 141010852
Bug: 135627985
Change-Id: I110f6028dab7f599decf59a5cb1b927b35e11857
2021-03-19 12:28:44 +08:00
Artur Satayev
fdb61edf43 Move allowed_deps.txt to packages/modules/common.
Bug: 179234385
Test: run update-apex-allowed-deps.sh locally
Change-Id: I8e8864468b87342c688d001bc5f6e6f8416863ed
Merged-In: I8e8864468b87342c688d001bc5f6e6f8416863ed
2021-03-17 13:44:16 +00:00
Ulya Trafimovich
3c902e7acc manifest_check.py: translate library names using dexpreopt configs.
Java modules that are defined in makefiles are not processed in
topological order, so it is necessary to communicate information from
dependencies via dexpreopt.config files. This has already been done in
make/core/dex_preopt_config_merger.py, and now manifest_check.py also
needs to get library names from their dexpreopt.config files.

This is to accommodate Java libraries which name differs from their
modules name. Soong properties `uses_libs`/`optional_uses_libs` and
makefile vars `LOCAL_USES_LIBRARIES`/`LOCAL_OPTIONAL_USES_LIBRARIES`
contain module names, not library names, so it is necessary to translate
them when comparing against library names in the manifest.

Bug: 132357300

Test: lunch cf_x86_64_phone-userdebug && m && launch_cvd \
      adb wait-for-device && adb root && adb logcat \
      | grep -E 'ClassLoaderContext [a-z ]+ mismatch'
      # empty grep output, no errors

Change-Id: I7cfb17d6d74e37fc9f37d977f15f0d711bab5c8e
2021-03-16 15:43:05 +00:00
Ulya Trafimovich
0aba252e15 Reimplement verify_uses_libraries.sh in manifest_check.py.
Previously there were two different scripts that did similar things:
1) build/soong/scripts/manifest_check.py
2) build/make/core/verify_uses_libraries.sh

Both scripts extracted <uses-library> tags and `targetSdkVersion` from
the manifests of Java modules, but 1) worked for XML manifests, and 2)
worked for APKs. This CL reimplements the functionality from 2) in 1),
so that one script can handle both XML manifests and APKs.

Bug: 132357300

Test: lunch cf_x86_64_phone-userdebug && m && launch_cvd \
      adb wait-for-device && adb root && adb logcat \
      | grep -E 'ClassLoaderContext [a-z ]+ mismatch'
      # empty grep output, no errors

Change-Id: Id1b66e4f3f30f307dba70cb111c7571762cb546a
2021-03-15 13:50:49 +00:00
Martin Stjernholm
4f0f3f01d9 Build heapprofd_client_api for use in the ART chroot.
Test: build/soong/scripts/build-mainline-modules.sh
Bug: 179915934
Change-Id: I4653bf705563ad2ddb077ecae7938c2dd1b818e9
2021-03-09 17:13:51 +00:00
Treehugger Robot
7ebe464748 Merge "Build statsd-module-sdk-for-art" 2021-02-24 01:08:28 +00:00
David Srbecky
bb96b56b0d Merge "Set block-size for XZ mini-debug-info compression." 2021-02-24 00:43:12 +00:00
Eric Holk
ff0529d209 Build statsd-module-sdk-for-art
Bug: 178236337
Test: build-mainline-modules.sh
Change-Id: I51dab13416be8ea1cd65234d560d84ca89b53e13
2021-02-23 22:49:57 +00:00
Treehugger Robot
7eacdbf8a6 Merge "Add non-fatal mode for verify_uses_libraries check." 2021-02-23 20:45:42 +00:00
Paul Duffin
ece454400d Merge changes from topic "hiddenapi_additional_annotations"
* changes:
  Sort hiddenapi monolithic files by signature
  Remove duplicates in monolithic hidden API files
  Remove implicit dependency from <x> -> <x>-hiddenapi
2021-02-23 12:48:24 +00:00
Ulya Trafimovich
8c35fcf4d5 Add non-fatal mode for verify_uses_libraries check.
The new mode is enabled with environment variable
RELAX_USES_LIBRARY_CHECK. If the variable is set to true, then a
verify_uses_libraries check failure does not fail the build, instead it
sets a special compiler filter "extract" for dexpreopt, which means that
the DEX file will be extracted, but it won't be compiled to native code.
Class loader context will be set to empty in this case (not &, as it is
going to be deprecated soon).

If the variable RELAX_USES_LIBRARY_CHECK is unset or set to something
other than "true", then the old behaviour of the verify_uses_libraries
check is preserved.

The intended use case for this flag is to have a smoother migration path
for the Java modules that need to add <uses-library> information in
the build files. The flag allows to quickly silence build errors. This
flag should be used with caution and only as a temporary measure, as it
masks real errors and affects performance.

verify_uses_libraries check is reworked so that it writes the error
message to a status file (which is used instead of the former timestamp
file). Currently the stored error message is not used, but it may be
used later to produce a warning. Dexpreopt command checks if the status
file exists and is nonempty; if that is the case, then compiler filter
is set to "extract".
Bug: 132357300

Test: Manually add some mismatch between the libraries in the Android.bp
      and Android.mk files for dexpreopted apps, build with
      RELAX_USES_LIBRARY_CHECK=true and obsserve that the build doesn't
      fail and they are compiled with compiler-filter "extract".
      Unset RELAX_USES_LIBRARY_CHECK and observe that the build fails.

Change-Id: Ibb5d993a25b1df1d2e70b7d5aafc6997f9d64e67
2021-02-22 15:41:03 +00:00
Anton Hansson
a6bf4a189c Merge "Disallow using error-prone merge strategies" 2021-02-19 15:27:29 +00:00
Anton Hansson
771fd524f2 Disallow using error-prone merge strategies
DO NOT MERGE / RESTRICT AUTOMERGE and its cousin DO NOT MERGE
ANYWHERE are almost always misused. Make it an error to upload
commits with those merge strategies.

Bug: 156896646
Test: repo upload -n (with DNM and RM commits)
Change-Id: Iae3540fa35dbd7b87a14b53a20be0cd23a206f3a
2021-02-19 13:05:54 +00:00
Treehugger Robot
999314fd98 Merge "Add bloaty_merger" 2021-02-19 09:45:39 +00:00
Eric Holk
277b0e7f41 Make statsd apex available as a prebuilt
Bug: 178236337
Test: build-mainline-modules.sh
Change-Id: I807eca7c5296227f4b593fc9f52d29bd20e78645
2021-02-18 17:39:51 +00:00
Thiébaud Weksteen
713db480cd Add bloaty_merger
This script will be used to merge the csv outputs from bloaty into a
single proto file.

Bug: 172339795
Test: atest --host bloaty_merger_test
Change-Id: I9a3c12d7b0c4b5e347fec04e6758bcc1fbc8ab40
2021-02-18 10:25:25 +01:00
Bob Badour
1bf6765def Add license to hiddenapi
Test: m all
Change-Id: Ic61a74d94bea0e4de2a47be7e15fcf874fb397c3
2021-02-17 09:46:26 -08:00
Paul Duffin
2c36f24082 Sort hiddenapi monolithic files by signature
Adds a new --key_field option to merge_csv.py which specifies the name
of the field that should be used to sort the input. If specified it
causes that field to be the first in each row and performs the merge
operation of a merge sort on the input files. That assumes that each
input file is already sorted into the same order.

Modifies the rules that use merge_csv.py to pass in:
    --key_field signature
to sort the rows by signature.

Bug: 180387396
Test: Verified that hiddenapi files (both aggregated ones and for the
      individual modules) are not affected by this change other than
      changing the order.
Change-Id: Idcd5f0fea373b520b604889e1c280f21ed495660
2021-02-16 17:38:08 +00:00
Paul Duffin
031d8693b3 Allow explicitly specified additional annotations for hiddenapi
Adds the hiddenapi_additional_annotations to allow a library to list
the libraries that provided additional hiddenapi related annotations
for a library.

Modifies merge_csv.py so it can process multiple zip files at the same
time and uses that to merge the embedded .uau files from a module and
those it depends upon.

Bug: 180102243
Test: m droid
      Verified that hiddenapi files (both aggregated ones and for the
      individual modules) are not affected by this change.
Change-Id: I796520021c7357398a9e2a09f1029e4a578b05b3
2021-02-16 13:28:26 +00:00
Eric Holk
7529f06707 Add statsd exports to be consumed by unbundled modules
This is needed first for ART to enable statsd metrics reporting, but
this will be needed by more modules as they become unbundled.

Test: build-mainline-modules.sh
Bug: 178500083
Change-Id: I5af0b8db7a24c86ffd3f34732ac747a4fc6a7a55
2021-02-11 19:02:40 +00:00
Treehugger Robot
b68036f44b Merge "Add LOCAL_LICENSE_KINDS to build/soong" 2021-02-10 07:18:29 +00:00
Paul Duffin
fdada68320 Move hiddenapi tools used by build/soong from frameworks/base
Also, creates a python_binary_host module for generate_hiddenapi_lists
and uses that when constructing the build rule rather than using the
file directly.

Bug: 177317659
Test: m droid
      Verified that hiddenapi files (both aggregated ones and for the
      individual modules) are not affected by this change.
Change-Id: Ia11bb203ce5a74740d35f1b7e86716e15aad336e
2021-02-09 10:46:36 +00:00
Colin Cross
69f0a24762 Pass list of module libraries to gen_ndk_backed_by_apex.sh
Running find on the entire module directory can lead to flaky failures
when a directory is modified by another rule while find is traversing
it.  Pass a list of libraries in the module into the script instead
of finding them on disk.

Fixes: 178509901
Test: TARGET_BUILD_APPS=com.android.runtime dist
Change-Id: Ie4bf288f0e3d99cbbed5ca8770687a9bbe5377e4
2021-02-08 16:52:07 -08:00
Bob Badour
02040de891 Add LOCAL_LICENSE_KINDS to build/soong
Added SPDX-license-identifier-Apache-2.0 to:
  Android.bp
  android/Android.bp
  android/soongconfig/Android.bp
  androidmk/Android.bp
  apex/Android.bp
  bazel/Android.bp
  bp2build/Android.bp
  bpf/Android.bp
  bpfix/Android.bp
  cc/Android.bp
  cc/config/Android.bp
  cc/libbuildversion/Android.bp
  cc/libbuildversion/tests/Android.bp
  cc/ndk_api_coverage_parser/Android.bp
  cc/ndkstubgen/Android.bp
  cc/symbolfile/Android.bp
  cmd/dep_fixer/Android.bp
  cmd/diff_target_files/Android.bp
  cmd/extract_apks/Android.bp
  cmd/extract_jar_packages/Android.bp
  cmd/extract_linker/Android.bp
  cmd/fileslist/Android.bp
  cmd/host_bionic_inject/Android.bp
  cmd/javac_wrapper/Android.bp
  cmd/merge_zips/Android.bp
  cmd/multiproduct_kati/Android.bp
  cmd/path_interposer/Android.bp
  cmd/pom2bp/Android.bp
  cmd/pom2mk/Android.bp
  cmd/sbox/Android.bp
  cmd/soong_build/Android.bp
  cmd/soong_env/Android.bp
  cmd/soong_ui/Android.bp
  cmd/zip2zip/Android.bp
  cmd/zipsync/Android.bp
  cuj/Android.bp
  dexpreopt/Android.bp
  dexpreopt/dexpreopt_gen/Android.bp
  env/Android.bp
  etc/Android.bp
  filesystem/Android.bp
  finder/Android.bp
  finder/cmd/Android.bp
  genrule/Android.bp
  jar/Android.bp
  java/Android.bp
  java/config/Android.bp
  kernel/Android.bp
  linkerconfig/Android.bp
  linkerconfig/proto/Android.bp
  makedeps/Android.bp
  partner/Android.bp
  phony/Android.bp
  python/Android.bp
  python/tests/Android.bp
  remoteexec/Android.bp
  rust/Android.bp
  rust/config/Android.bp
  scripts/Android.bp
  sdk/Android.bp
  sh/Android.bp
  shared/Android.bp
  symbol_inject/Android.bp
  symbol_inject/cmd/Android.bp
  sysprop/Android.bp
  tradefed/Android.bp
  ui/build/Android.bp
  ui/logger/Android.bp
  ui/metrics/Android.bp
  ui/metrics/proc/Android.bp
  ui/status/Android.bp
  ui/terminal/Android.bp
  ui/tracer/Android.bp
  xml/Android.bp
  zip/Android.bp
  zip/cmd/Android.bp

Added SPDX-license-identifier-Apache-2.0 SPDX-license-identifier-BSD to:
  finder/fs/Android.bp
  third_party/zip/Android.bp

Bug: 68860345
Bug: 151177513
Bug: 151953481

Test: m all

Exempt-From-Owner-Approval: janitorial work

Change-Id: Ia47ca14f16b8c9f84f9d533a07e5b00e2c04e8d4
2021-02-06 04:23:21 +00:00
David Srbecky
412322f2a2 Set block-size for XZ mini-debug-info compression.
This allows the compression to be done on multiple threads
and more importantly, it allows libunwindstack to decompress
only the needed blocks, which reduces unwind time and memory.

Bug: 110133331
Test: verify lazy decompression in libunwindstack
Change-Id: Ia3f597928edcbc85acdf05cb5c83ac7fced39148
2021-02-03 20:48:50 +00:00
Martin Stjernholm
bd396b6d8a Force ART modules to be built from source here, regardless of default.
When we start switching vanilla platform builds to use ART prebuilts,
this script should still build from sources.

Test: TH
Bug: 172480615
Change-Id: I04cb61b484855d22f22fd9b82f6abb30eb5665c3
2021-02-02 22:24:00 +00:00
Pete Bentley
9436be4321 Merge "Revert "Delete build-aml-prebuilts.sh"" 2021-01-29 16:19:57 +00:00
Pete Bentley
237de41b38 Revert "Delete build-aml-prebuilts.sh"
Revert submission 1535006-bap

Reason for revert: Looks like it breaks the build as aapt2 depends on the build_number.txt this script was generating.  Error log below.
Reverted Changes:
I0f6b34310:Delete build-aml-prebuilts.sh
I4eae4153a:Replace use of build-aml-prebuilts

Change-Id: I7f5f1b3047f3d71eb8ac7bec95c5916425d5d8f2
FAILED: ninja: 'out/aml/soong/build_number.txt', needed by 'out/aml/soong/.intermediates/frameworks/base/tools/aapt2/aapt2/linux_glibc_x86_64/unstripped/aapt2', missing and no known rule to make it
2021-01-29 15:14:42 +00:00
Anton Hansson
30d6f68b1a Merge "Delete build-aml-prebuilts.sh" 2021-01-29 09:11:16 +00:00
Sophie Zheng
9b6596bb60 Merge "Revert "Revert "Automated NDK API coverage used by Mainline modules build integration.""" 2021-01-26 00:13:46 +00:00
sophiez
6bde0b5f9d Revert "Revert "Automated NDK API coverage used by Mainline modules build integration.""
This reverts commit 556b2ad77e.

Original change-id: Ib1b2f0dd2f9ae85b1545c6cc5bb4c5bbdfac1c15

Reason for revert: Add excutable permission to the gen_ndk_backedby_apex.sh.

Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only

Change-Id: Ib587ba200cd7f2f61d478452b43c329d72b2de06
2021-01-25 19:52:10 +00:00
Riddle Hsu
f17b07fc52 Merge "Revert "Automated NDK API coverage used by Mainline modules build integration."" 2021-01-25 06:15:29 +00:00
Riddle Hsu
556b2ad77e Revert "Automated NDK API coverage used by Mainline modules build integration."
This reverts commit 87ff51db84.

Reason for revert: build breakage b/178327796

Change-Id: I51c56938540794097eda579ebe47fa5c0a4d8c12
2021-01-25 01:44:34 +00:00
Sophie Zheng
0bf77198b5 Merge "Automated NDK API coverage used by Mainline modules build integration." 2021-01-22 22:15:21 +00:00
sophiez
87ff51db84 Automated NDK API coverage used by Mainline modules build integration.
Add gen_ndk_usedby_apex.sh script to generate NDK API list used by Mainlain modules when modules get build.

Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only

Change-Id: Ib1b2f0dd2f9ae85b1545c6cc5bb4c5bbdfac1c15
2021-01-21 18:08:50 +00:00
Paul Duffin
fe6147eb26 Merge "Export dex implementation jars from prebuilt_apex" 2021-01-19 14:34:29 +00:00
Paul Duffin
064b70c918 Export dex implementation jars from prebuilt_apex
Dexpreopt and boot jars package check all require access to dex
implementation jars created for java_library and java_sdk_library. They
were available when building from source but not when building from
prebuilts, even though they are embedded within the .apex files that
are referenced from prebuilt_apex.

This changes adds support to prebuilt_apex to export the dex
implementation jars and updates java_import to use those exported dex
implementation jars.

In a source build dexpreopt/boot jars package check access the apex (or
platform) specific variant of a java_library, e.g. core-oj, from which
it retrieves the dex implementation jar path.

After this change in a prebuilt build dexpreopt/boot jars package check
behave in the same way except in this case they retrieve the dex
implementation jar path from the apex (or platform) specific variant of
the java_import, e.g. core-oj.

The work to export files from a `.apex` file for use by other modules
is performed by a new `deapexer` module type. It is not used directly
in an `Android.bp` file but instead is created implicitly by
`prebuilt_apex`,

In order to do that this contains the following changes:
* Adds a new `dexapexer` module type to handle the exporting of files
  from the `.apex` file.
* Adds an exported_java_libs property to prebuilt_apex to specify the
  set of libraries whose dex implementation jars need exporting.
* Creates apex specific variants of the libraries listed in the
  exported_java_libs property.
* Adds the set of exported files to the ApexInfo to make them available
  to the apex specific variants.
* Prevents the prebuilt_apex variants from being merged together as
  they will not be compatible.
* Modifies java_import to use the exported file for variants of a
  prebuilt_apex.
* Adds a ninja rule to unpack (using deapexer) the contents of the
  prebuilt_apex's apex file, verify that the required files are present
  and make them available as outputs for other rules to use.
* Some minor refactorings to support these changes.
* Adds tests to cover prebuilt only, prebuilt with source preferred,
  and prebuilt preferred with source.

Test: m nothing
Bug: 171061220
Change-Id: Ic9bed81fb65b92f0d59f64c0bce168a9ed44cfac
2021-01-15 18:14:10 +00:00
Colin Cross
0dd19125eb Merge "manifest_fixer: Set targetSdkVersion to '16' for libraries" 2021-01-15 17:13:31 +00:00
Anton Hansson
35e3fd0666 Delete build-aml-prebuilts.sh
Its usecase can now be supported natively by soong_ui via the
--skip-kati flag + the mainline_sdk product. Migrate
build-mainline-modules.sh to use this new approach, and delete the
script.

Bug: 174315599
Test: build/soong/scripts/build-mainline-modules.sh
Change-Id: I0f6b34310df51ddf060b2912d43487f4d37dd01b
2021-01-15 10:23:24 +00:00
Andrew Wheeler
18dcd046ad manifest_fixer: Set targetSdkVersion to '16' for libraries
manifest_fixer.py sets a lib's targetSdkVersion to '15' if it is
not explicitly set.  But ManifestMerger will add dangerous
permissions CALL_LOG_READ/CALL_LOG_WRITE if the following
conditions are met:

    - The app's targetSdkVersion is >16
    - A linked lib's targetSdkVersion is <16
    - The app has CONTACTS_READ/CONTACTS_WRITE permissions

This condition is triggered for some (OEM) app builds (of
Settings).  Update manifest_fixer.py to use '16' as its default
targetSdkVersion so that manifest_merge doesn't add these
dangerous permissions.

Change-Id: Id5f41f7da98c190c8d145fceba0085cf473f4c7a
2021-01-14 14:11:39 -06:00
Nicolas Geoffray
c1b1663576 Add tzdata-module-test-exports to the list of targets.
Test: build-mainline-modules.sh
Bug: 142935992
Change-Id: I2e765737a971efee837daefb5af0c272f561b36d
2021-01-11 09:45:57 +00:00
Martin Stjernholm
2a9013ee17 Set the Soong config variable to keep building ART from source.
Without this Soong may run into inconsistencies due to stale prebuilts.
This amends https://r.android.com/1529658.

Test: TH build of NDK
Bug: 175918603
Bug: 172480615
Change-Id: I14680e4547d7d5f560d31ef61b71af8d159c4c53
2020-12-21 13:37:22 +00:00
Martin Stjernholm
56b584c4b2 Merge "Change default for source_build config variable to true." 2020-12-17 22:31:04 +00:00
Martin Stjernholm
ed623eaed8 Change default for source_build config variable to true.
To reflect the new (temporary) true default in make.

Test: m
Bug: 172480615
Change-Id: I9a32578554650776c1d6b3920fa233c5613a05e3
2020-12-15 03:58:07 +00:00
Ulyana Trafimovich
7f487d4a86 Merge "Add MissingUsesLibraries to Soong vars for scripts that use --skip-make." 2020-12-14 22:44:17 +00:00
Martin Stjernholm
6f8fecd2e3 Stopgap fix to avoid aapt2 build errors.
Should fix build errors on mainline_modules on aosp-master. This script
still does not propagate all options to aapt2 correctly, but that will
be fixed by removing the script altogether (b/174315599).

Test: build/soong/scripts/build-aml-prebuilts.sh art-module-sdk \
      art-module-test-exports conscrypt-module-host-exports \
      conscrypt-module-sdk conscrypt-module-test-exports \
      i18n-module-host-exports i18n-module-sdk \
      i18n-module-test-exports platform-mainline-sdk \
      platform-mainline-test-exports runtime-module-host-exports \
      runtime-module-sdk
  (the failing command in
  build/soong/scripts/build-mainline-modules.sh)
Bug: 175386777
Change-Id: I81aa20b1abd177f0d2886ab9b04831437da72f2a
2020-12-14 15:12:23 +00:00
Ulya Trafimovich
37eb6c9448 Add MissingUsesLibraries to Soong vars for scripts that use --skip-make.
In platform builds missing libraries are defined with make variable
INTERNAL_PLATFORM_MISSING_USES_LIBRARIES via soong_config.mk. When Soong
is invoked with "--skip-make" parameter,
INTERNAL_PLATFORM_MISSING_USES_LIBRARIES is not picked up. As a result
the build fails because of missing dependencies.

This CL uses get_build_var to get the value of
INTERNAL_PLATFORM_MISSING_USES_LIBRARIES, then converts it to JSON list.

In the future would be better to migrate both scripts to use --skip-kati
introduced in https://r.android.com/1512613.

Bug: 175286760
Bug: 132357300
Test: forrest build for target "ndk" branch "aosp-master".
Test: Patch build-aml-prebuilts.sh and build-ndk-prebuilts.sh to dump
  their soong.variables config, observe that MissingUsesLibraries is
  coherent with INTERNAL_PLATFORM_MISSING_USES_LIBRARIES. Start running
  each script and observe that they don't fail early due to missing
  dependencies.
Change-Id: I5dbc1d3ea7a64de1e8be6332003acc940c2e6a76
2020-12-14 15:03:40 +00:00
Martin Stjernholm
4c13f58729 Add i18n-module-host-exports to the SDK list.
Necessary only when platform start using i18n prebuilts, but this helps
CI testing.

Test: build/soong/scripts/build-mainline-modules.sh
Bug: 172480615
Change-Id: Ic24f934ad0b4b7ac7c38d791f80df33315b936e9
2020-12-11 19:20:47 +00:00
Martin Stjernholm
f371381cac Propagate the Soong config variable for enabling ART Module building.
It is set in art/build/build-art-module.sh.

Test: art/build/build-art-module.sh
Bug: 172480617
Change-Id: Idbb3478667a2265ba115922207cb6d9aa6c4c91d
2020-12-08 22:30:03 +00:00
Sophie Zheng
8baa8b50a8 Merge "Automated NDK API coverage used by Mainline modules build integration." 2020-12-03 23:30:21 +00:00
Kiyoung Kim
4ee686ddca Add 'append' command for conv_linker_config
Add 'append' command for conv_linker_config so any item can be appended
to linker.config.pb file.

Bug: 174646355
Test: buildbot_build.sh succeeded
Change-Id: Ic3f12ee5e5c0e7ca7c27d4a8cff347e93061eb5f
2020-12-03 17:09:56 +09:00
sophiez
c80a2b392e Automated NDK API coverage used by Mainline modules build integration.
Add gen_ndk_usedby_apex.sh script to generate NDK API list used by Mainlain modules when modules get build.

Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only

Forrest run: android-build.googleplex.com/builds/forrest/run/L97800000742565481

Change-Id: I39e5aa7c74eff46aba06808642b2bd67ae45ba1e
2020-12-03 05:45:20 +00:00
Kiyoung Kim
24dfc1fbb2 Generate system stub library list at build time
List of stub libraries from system (and bionic APEX) is currently
managed in static file. This change generates the list at build time and
adds to the system config so linkerconfig can use it.

Bug: 172889962
Test: Build cuttlefish and confirmed list is generated in
/system/etc/linker.config.pb

Change-Id: Ie0400e9d1098b0de19c6a7c25f261384aadde0a3
2020-12-01 14:09:03 +09:00
Jingwen Chen
cda22c9bb9 Refactor 'in_make' to mean Kati is not skipped.
In Nougat and before, Make wrote a marker file to indicate that
soong_build was invoked from Make to change certain behaviors of Soong
at build time.
https://cs.android.com/android/platform/superproject/+/android-7.1.2_r36:build/core/soong.mk;l=70-73;drc=ae18638b0406ad107b0882a02a13cdd8b92f2a4e

Things have changed, and now soong_build is invoked from soong_ui, which
supports a --skip-make configuration flag:
https://cs.android.com/android/platform/superproject/+/master:build/soong/ui/build/build.go;l=31-33;drc=680387bf1d3ce7cbc77f535be7c42cec411b1687

Thus, the various remnants of 'EmbeddedInMake' and 'inMake'
configuration are misleading, since soong_build is no longer invoked
from Make. This CL refactors all instances to actually mean that
Kati is enabled (not skipped with --skip-make), and will run after
soong_build finishes, so Kati-specific behavior like the AndroidMk
singleton should run.

Test: TH presubmit

Change-Id: I576ab8e54f99f5c8ddf9feaf9a828019b279e266
2020-11-23 00:29:18 -05:00
Paul Duffin
cd4f673ceb Switch boot jars package check to use dexdump xml output
The xml output from dexdump is more stable than the text output so this
change switches to use that.

Bug: 171479578
Test: m check-boot-jars - for failing and passing cases
Change-Id: Ie347b1b685951524d5ae09db280248c61b4228ee
2020-11-16 17:46:16 +00:00
Martin Stjernholm
c488743e14 Set up BuildNumberFile in soong.variables.
Necessary for some build steps. aapt2 is one example, although it still
has problems because part of its build logic is in make.

This isn't strictly required for any supported build target at the
moment, but the error when it fails can be confusing so let's add it
anyway.

Test: build/soong/scripts/build-aml-prebuilts.sh art-module-sdk \
  art-module-test-exports conscrypt-module-sdk \
  conscrypt-module-test-exports conscrypt-module-host-exports \
  runtime-module-sdk runtime-module-host-exports \
  i18n-module-test-exports i18n-module-sdk platform-mainline-sdk \
  platform-mainline-test-exports
Bug: 172548608
Change-Id: I88526a85f288287c3a138f8cbb6fd04ad28c3795
2020-11-13 15:40:17 +00:00
Paul Duffin
2d8e1a7e59 Switch boot jars package check to using dex jars
The switch to use dex jars instead of class jars means that a boot jar
that is defined by a dex_import module will now be checked against the
package_allowed_list.txt so it is possible that it will detect
previously unreported problems.

Test: m check-boot-jars - for failing and passing cases
Bug: 171479578
Bug: 125517186
Change-Id: Ie614898dade0fb43c9418d7afb9138169db6f097
2020-11-10 17:37:43 +00:00
Kiyoung Kim
e52c665d7f Allow comments from linker.config.json
Filter out lines starts with "//" from json file to allow simple
comments on the contents. Original json format does not support
comments, but this reduces readability compared to txt file or other
formats. This change allows simple comments on the linker.config.json to
give more information on the contents.

Test: parse succeeded with commented contents
Change-Id: I1c734bf9a054f81f57aa2aea1038d0041297acf1
2020-11-03 16:00:23 +09:00
Paul Duffin
9a89a2a0ea Move boot jars package check from make
Adds a singleton that traverses the module variants finding the ones
that are in the list (updatable and non-updatable) of boot jars and
add a ninja rule to ensure that they only contain packages from an
allowed list.

Replaces a hack that ignored any prebuilt boot jars supplied as dex
file with an equivalent one to ensure that they are still ignored.
A follow up change that switches to checking dex jars will allow the
hack to be removed.

The boot jars check can be strict or lax. If strict then all the boot
jars listed in the configuration must be found, otherwise it will only
check the ones it finds. It is strict by default unless
TARGET_BUILD_UNBUNDLED=true or ALLOW_MISSING_DEPENDENCIES=true.

Moves the script and data file from build/make.

Test: m check-boot-jars - for failing and passing cases
      SKIP_BOOT_JARS_CHECK=true - no check-boot-jars target created
	  ALLOW_MISSING_DEPENDENCIES=true - not strict
	  TARGET_BUILD_UNBUNDLED=true - not strict
      verified manually that apart from path differences the same
      files (same check sum) were checked in both old make checks and
      the new Soong ones
      EMMA_INSTRUMENT=true EMMA_INSTRUMENT_FRAMEWORK=true m check-boot-jars
Bug: 171479578
Change-Id: I9d81d6650ba64fc0d48d2dab4ba5a3ba8dd03dec
2020-10-29 17:20:06 +00:00
Ulya Trafimovich
8130c482ab Move part of logic from construct_context.py to Soong.
construct_context.py is a script that runs at Ninja stage and constructs
class loader context arguments for dex2oat. Previously it accepted lists
of library paths and concatenated them into one class loader context
string. The script also handled the dependency of "android.hidl.manager"
on "android.hidl.base" that is not tracked by the build system and must
be handled in a special way.

Now that class loader context representation is going to change from
flat lists to trees, passing paths to individual libraries as script
arguments is no longer possible, because a list of paths cannot
represent a class loader context tree. Passing the trees in a serialized
form is also inconvenient, because the script would have to parse them,
which would complicate it a lot.

Therefore this patch ports all the concatenation and "android.hidl.base"
handling to Soong. It is not possible to port the remaining script to
Soong because Soong has no information about the targetSdkVersiion of
the dexpreopted apps (it is in the manifest and sometimes in an APK, and
has to be extracted at Ninja time).

Test: construct_context_test.py
Test: new subtests in TestUsesLibs
Test: lunch aosp_cf_x86_phone-userdebug && m
Bug: 132357300
Change-Id: Icdb03cf00d1e27e4cff3844b89bfaec4de502dd7
2020-10-27 17:05:30 +00:00
Martin Stjernholm
7f51107ee6 Rename ART release APEX to com.android.art.
Test: See https://r.android.com/1457217
Bug: 169639321
Change-Id: I6800c7a382486b4e50945cc8b789f4be16482fe6
Merged-In: I6800c7a382486b4e50945cc8b789f4be16482fe6
2020-10-21 15:41:02 +01:00
Kiyoung Kim
62abd12ba5 Handle new filetype 'linker_config'
Handle new filetype 'linker_config' which is configuration for
linkerconfig in json type and convert into protobuf in build time.

Bug: 169634881
Test: Build succeeded and cuttlefish boot succeeded
Change-Id: I56555fc738e6d6600d15a191a24f79a2ee747f52
2020-10-15 23:09:35 +09:00
Artur Satayev
4e1f2bd0d8 Track allowed transitive deps in any updatable module.
Instead of tracking per module and per module variant, track allowed
list of dependecies for all modules combined. This avoids issues with
different products and different downstream branches having different
build graphs.

To compare allowed_deps.txt vs head, run:
:; m -j out/soong/apex/depsinfo/new-allowed-deps.txt.check

To update source allowed_deps.txt, run:
:; build/soong/scripts/update-apex-allowed-deps.sh

Bug: 149622332
Test: m
Change-Id: Ic518fbd9ebfe1b46aaf9a58df731780a7e5a676b
Merged-In: Ic518fbd9ebfe1b46aaf9a58df731780a7e5a676b
(cherry picked from commit 453555083b)
(cherry picked from commit e5207cd9a6)
2020-09-30 21:17:42 +00:00
Martin Stjernholm
2829f6c09c Propagate settings for building in an unbundled tree.
Test: build/soong/scripts/build-aml-prebuilts.sh art-module-sdk
  on master-art
Change-Id: Iaff7f590d67fdd6f533f2831475deb7b92090ea2
2020-08-11 01:41:20 +01:00