Commit graph

435 commits

Author SHA1 Message Date
Artur Satayev
74cead891e Add support to max-target-s to generate_hiddenapi_lists
Test: make
Bug: 169748350
Change-Id: I2662a643a190ed531d85755d01e1be9c49c02e25
Merged-In: I2662a643a190ed531d85755d01e1be9c49c02e25
2022-05-06 08:59:39 +00:00
Ge Bian
93b542459a Revert "Add support to max-target-s to generate_hiddenapi_lists"
Revert submission 1931011-max-target-s

Reason for revert: Seems to cause build breakage
Reverted Changes:
I2b513f93a:Add support to max-target-s to generate_hiddenapi_...
Ifdd7b7a71:Add support for max-target-s
I06d05840e:Add support for max-target-s

Change-Id: I67413a3deda42c503b876727377e932f7e4652e9
2022-05-03 20:41:53 +00:00
Narayan Kamath
ad0ec2ae1f Add support to max-target-s to generate_hiddenapi_lists
Test: make
Bug: 169748350
Change-Id: I2b513f93a509bc1163b114d399895cbe9486817e
2022-04-28 18:34:58 +01:00
Treehugger Robot
7436a7121f Merge "Updates test_config_fixer to also work for SuiteApkInstaller" 2022-04-22 07:12:45 +00:00
Sasha Smundak
fe11851029 Recursively list Android image contents.
Relies on debugfs utility to inspect image contents.
The output looks something like this:
```
./
./app/
./app/Calendar/
./app/Calendar/Calendar.apk
```

Test: manual
Change-Id: Ia05f4e857dfd28ac76ac2b450b08f5cf5577d21f
2022-04-20 20:02:11 -07:00
Rahul Sabnis
48a8f0d10b Updates test_config_fixer to also work for SuiteApkInstaller
Bug: 228385580
Test: atest BluetoothInstrumentationTests
Merged-In: I878309354c4aa5a492fb3d2bcecf8b1410e011ee
Change-Id: I878309354c4aa5a492fb3d2bcecf8b1410e011ee
2022-04-20 10:36:58 -07:00
Paul Duffin
bd88c882f6 Remove implementation details from stub flags in sdk snapshot
Previously, the build applied the same filtering to remove
implementation details from the sdk snapshot's stub-flags.csv file as
it did for its all-flags.csv, i.e. removing the signatures that only
had a "blocked" flag. Unfortunately, that had no effect on the stub
flags as the implementation signatures had no flags, not a single
blocked flag. That meant that the sdk snapshot's
filtered-stub-flags.csv file contained a lot of implementation details.

This change removes signatures from stub-flags.csv that have no flags
which removes all implementation details from the sdk snapshot.

Bug: 194063708
Test: atest --host verify_overlaps_test
      m out/soong/hiddenapi/hiddenapi-flags.csv
      m art-module-sdk
      # Check contents of its filtered-stub-flags.csv file
Change-Id: I30edc77348fad118ea732e787ae8e206c8841f84
2022-04-12 17:44:15 +01:00
Paul Duffin
0c12b78ae8 Use named options for verify_overlaps
Previously, verify_overlaps used positional arguments, the first was
the monolithi flags and the rest were the module flag pairs (filtered
flags file and signature patterns file). This change makes them use
named options to make the purpose of the arguments clearer on the
command line.

Bug: 194063708
Test: atest --host verify_overlaps_test
      m out/soong/hiddenapi/hiddenapi-flags.csv
Change-Id: Ife0af0016eb0f91416e8330d5d98cb53c97d68a4
2022-04-08 10:56:21 +01:00
Paul Duffin
181b56cf4d Cleanup verify_overlaps*.py
Fix issues reported by pylint, Intellij Python checks and also try and
adhere to the Google Python Style Guide.

Bug: 194063708
Test: atest --host verify_overlaps_test
      m out/soong/hiddenapi/hiddenapi-flags.csv
      /usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/verify_overlaps*.py
      pyformat -s 4 --force_quote_type single -i scripts/hiddenapi/verify_overlaps*.py
Change-Id: I6273456abc3a4da4af780ae1a0473ca63521dff5
2022-04-08 10:49:24 +01:00
Paul Duffin
b99d480e1d analyze_bcpf: Explain why a package is split/single
Previously, the script would determine whether a package was split,
single or could be used as a prefix but did not explain why. This
change provides additional information to explain why they are split.

Bug: 202154151
Test: m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment
      m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment --fix
      atest --host analyze_bcpf_test
Change-Id: I3e2d5c0b54b5cc028013ce5ea979ebd9b9bf2c0d
2022-04-04 18:24:40 +01:00
Paul Duffin
ea836c2baf analyze_bcpf: Indent multi-line text to improve formatting
Addresses a comment on the review of the initial changes to add
analyze_bcpf script.

Bug: 202154151
Test: m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment
      m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment --fix
      atest --host analyze_bcpf_test
Change-Id: I8c030dcc1a29f106ca57dca8c97a5cb4425e7674
2022-04-04 16:59:36 +01:00
Paul Duffin
dbbb8374a2 signature_trie: Avoid unnecessary wrapping and unwrapping of values
Previously, Leaf.values() and Leaf.append_values() would wrap the
Leaf's value inside a list before appending it to the list of values.
So, the values list was actually a list of lists of values. The
get_matching_rows method would then use chain.from_iterable() to
flatten that list of list of values into a list of values.

This change removes the initial wrapping in a list and so removes the
need to flatten them into a single list. It also adds a test for the
values() method. Prior to this change the expected value would have
been [[1], ["A"], [{}]].

Bug: 202154151
Test: atest --host analyze_bcpf_test signature_trie_test verify_overlaps_test
Change-Id: Ida78500c9ab4466de127b2c36501b3606d0f3fe5
2022-04-01 15:04:23 +01:00
Bob Badour
de6a087d2d Revert "Revert "Build notice files from license metadata.""
This reverts commit 77807b3c27.

Reason for revert: fix and roll forward again

Change-Id: I80c796ca81ecf29df9522babe54d684a0d11b9e6
2022-04-01 18:00:00 +00:00
Wei Sheng Shih
77807b3c27 Revert "Build notice files from license metadata."
This reverts commit 43c2dcaef6.

Reason for revert: suspect build break

Bug: 227682036

Test: TARGET_BUILD_VARIANT=userdebug UNBUNDLED_BUILD_SDKS_FROM_SOURCE=true vendor/google/build/mainline_modules_bundles.sh -j97


Change-Id: Ibfb8b4fefc264f52f32ba661c269a9cd625d800a
2022-04-01 14:10:53 +00:00
Bob Badour
43c2dcaef6 Build notice files from license metadata.
Switch from generate-notice-files.py and mergenotice.py to htmlnotice.

Bug: 151177513
Bug: 213388645
Bug: 210912771

Test: m droid dist reportmissinglicenses
Change-Id: I6cac049d24f35ec358c6f341a04f4ba6161703bf
2022-03-31 17:29:01 +00:00
Treehugger Robot
91dac89447 Merge "analyze_bcpf: Compute hidden_api package properties" 2022-03-30 20:00:43 +00:00
Cole Faust
71350eaff2 Merge "Always build mk2rbc/rbcrun" 2022-03-30 19:43:06 +00:00
Paul Duffin
dd97fd2520 analyze_bcpf: Compute hidden_api package properties
Analyzes the signatures of the class members contained within the
bootclasspath_fragment to construct the split_packages,
single_packages and package_prefixes hidden_api properties that are
necessary to remove the internal implementation details from the
hidden API flags.

Bug: 202154151
Test: m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment
      m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment --fix
      m analyze_bcpf && analyze_bcpf --bcpf com.android.mediaprovider-bootclasspath-fragment --fix
      atest --host analyze_bcpf_test
Change-Id: I4a8e5a8bfee2a44775e714b9226cd4a7382e0123
2022-03-30 16:00:08 +01:00
Paul Duffin
26f19919ea analyze_bcpf: Add --fix option
Add a --fix option that will cause the script to automatically fix the
issues that it finds. It uses the bpmodify tool to add values to the
bootclasspath_fragment's hidden_api properties.

This adds analyze_bcpf to bp2buildModuleDoNotConvertList as
analyze_bcpf depends on bpmodify which is a blueprint_go_binary which
is not yet supported by bazel.

Bug: 202154151
Test: m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment
      m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment --fix
      atest --host analyze_bcpf_test

Change-Id: I5ee52419b4829474f6dbeb47f86ab2aeb22b1382
2022-03-30 16:00:08 +01:00
Paul Duffin
4dcf65951b Add analyze_bcpf.py script to help manage bootclasspath_fragments
The analyze_bcpf.py script can be used by bootclasspath_fragment owners
to help setup their fragment and also to debug issues that arise during
use.

It is run by specifying the name of the bootclasspath_fragment module
in the --bcpf option. If specified the --apex and --sdk options will
allow the tool to produce more helpful messages.

The script currently:
* Builds out/soong/hiddenapi/hiddenapi-stub-flags.txt and checks any
  inconsistencies between those flags and the flags generated for the
  bootclasspath_fragment.

* Builds out/soong/hiddenapi/hiddenapi-flags.csv and checks any
  inconsistencies between those flags and the flags generated for the
  bootclasspath_fragment. It will describe the changes that need to
  be made in order to fix any inconsistencies in the flags.

Bug: 202154151
Test: m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment
      atest --host analyze_bcpf_test
Change-Id: I824fd9d527d246dfe56725c073d89a0b0e72d25a
2022-03-30 12:18:56 +01:00
Cole Faust
d9932ad053 Always build mk2rbc/rbcrun
Currently, mk2rbc/rbcrun aren't built in certain phases
of the build, which causes the Starlark product configuration
to fail when using get_build_var if a regular build hasn't
been run before.

Bug: 226677850
Test: Manually
Change-Id: If85608eef63be25fddc4d916c82247c13eb0195f
2022-03-29 14:55:17 -07:00
Jooyung Han
1df464a90b Revert "Add a tool to list contents of .img file"
Revert submission 2041990-microdroid-contents

Reason for revert: b/227144320 (broken build)
Reverted Changes:
Ie2f471d39:Add the golden list of microdroid contents
I7c4fca184:Add a tool to list contents of .img file

Change-Id: If2b5d6cb1325cb17ef2bdc4f13ded003ab2dd6d7
2022-03-28 09:44:33 +00:00
Jooyung Han
b3a3a2893b Add a tool to list contents of .img file
list_image <img> lists the contents of the file.

For example:
  $ m microdroid
  $ list_image <path to microdroid.img>
  ./
  ./apex/
  ./bin
  ..

Bug: 195425111
Bug: 225121718
Test: list_image .img
Change-Id: I7c4fca184751ba20066fd25fa4c366a955dbabf6
2022-03-25 02:44:36 +09:00
Paul Duffin
846beb7f54 Make bootclasspath_fragment hidden API package checks exhaustive
Previously, the bootclasspath_fragment's hidden_api.split_packages and
hidden_api.package_prefixes properties did not specify an exhaustive
set of packages that were provided by the fragment. They excluded
packages which were either not split or which could not be used as a
package prefix because it would match sub-packages provided by other
bootclasspath modules.

This change adds the hidden_api.single_packages list to specify those
additional packages and then uses that information to verify that any
bootclasspath_fragment that specifies at least one of split_packages,
single_packages or package_prefixes properties only contains classes
from a package that matches one of those properties. That will
prevent a module from accidentally including unexpected classes, such
as might happen when statically including a common utility library.

It also adds coverage specific versions of the properties as additional
packages are added to the art-bootclasspath-fragment when building
coverage builds.

Bug: 194063708
Test: atest signature_patterns_test
      m out/soong/hiddenapi/hiddenapi-flags.csv
      m EMMA_INSTRUMENT=true EMMA_INSTRUMENT_FRAMEWORK=true out/soong/hiddenapi/hiddenapi-flags.csv
      # Breaks without corresponding change to add android.system to
      # the art-bootclasspath-fragment.
      /usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_patterns*.py
      pyformat -s 4 --force_quote_type single -i scripts/hiddenapi/signature_patterns*.py
Change-Id: Iddf6c59cd4dc8c36dde7943a9840ccef5794b320
2022-03-18 10:54:51 +00:00
Paul Duffin
1024f1d72b Cleanup signature_patterns*.py
Fix issues reported by pylint, Intellij Python checks and also try and
adhere to the Google Python Style Guide.

Bug: 194063708
Test: atest signature_patterns_test
      m out/soong/hiddenapi/hiddenapi-flags.csv
      /usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_patterns*.py
      pyformat -s 4 --force_quote_type single -i scripts/hiddenapi/signature_patterns*.py
Change-Id: I64d64e9cb269f58d65f4e10ec2f0e874154919e6
2022-03-17 10:36:29 +00:00
Paul Duffin
92532e72a1 Allow traversal over the trie structure
Previously, there was no way to traverse the trie structure and no way
to identify specific nodes in the trie. That made it impossible to
analyze the trie structure resulting from loading a set of flags. This
change adds type and selector properties to nodes as well as access to
the child nodes of a node to allow for the structure to be analyzed.

Bug: 202154151
Test: m out/soong/hiddenapi/hiddenapi-flags.csv
      atest --host signature_trie_test verify_overlaps_test
      pyformat -s 4 --force_quote_type double -i scripts/hiddenapi/signature_trie*
      /usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_trie*
Change-Id: Ia4714dbf59f6fd143aa3bf3ad1a59cd073d2175b
2022-03-15 15:33:51 +00:00
Paul Duffin
ea93542e90 Switch signature_to_elements to use tuple
Previously, it used a structured string of format <type>:<value>. A
tuple is more efficient and less prone to edge cases, such as when the
value is a field signature which itself contains a ":".

Bug: 202154151
Test: m out/soong/hiddenapi/hiddenapi-flags.csv
      atest --host signature_trie_test verify_overlaps_test
      pyformat -s 4 --force_quote_type double -i scripts/hiddenapi/signature_trie*
      /usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_trie*
Change-Id: I80abaff243d98aad325df1b5a655bba7f9d04e2c
2022-03-15 15:16:04 +00:00
Paul Duffin
19255f1d91 Make signature_to_elements stricter and more consistent
Previously, signature_to_elements would return a string array where
non-wildcard strings were of the form <type>:<value> but wildcard
strings were just * or **. This change makes it handle wildcards
consistently with the other element types and adds some extra
checking for edge cases.

Bug: 202154151
Test: m out/soong/hiddenapi/hiddenapi-flags.csv
      atest --host signature_trie_test verify_overlaps_test
      pyformat -s 4 --force_quote_type double -i scripts/hiddenapi/signature_trie*
      /usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_trie*
Change-Id: I5bfaf5e75c7da54b6241f68e03231939c9d65501
2022-03-15 15:16:04 +00:00
Paul Duffin
1f8a6b2f27 Extract Node superclass of Leaf and InteriorNode
Adds a Node superclass to declare and documented the methods that need
to be implemented by Leaf and InteriorNode. Also uses @dataclasses to
make it easy to declare and document the properties of each class.

This refactoring is in preparation for future changes that will need to
add common behavior to both Leaf and InteriorNode so having a
superclass will make that easier.

Bug: 202154151
Test: m out/soong/hiddenapi/hiddenapi-flags.csv
      atest --host signature_trie_test verify_overlaps_test
      pyformat -s 4 --force_quote_type double -i scripts/hiddenapi/signature_trie*
      /usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_trie*
Change-Id: If660855f6b1f45a8ea5b90b3cc0236da9e07d090
2022-03-09 15:43:57 +00:00
Paul Duffin
b5cd522053 Extract signature_trie.py from verify_overlaps.py
Makes the efficient pattern matching of hidden API flags that is used
by verify_overlaps.py available for use in other scripts.

As part of the move this cleans up the python to use consistent quotes,
and fix pylint issues.

Bug: 202154151
Test: m out/soong/hiddenapi/hiddenapi-flags.csv
      atest --host signature_trie_test verify_overlaps_test
      pyformat -s 4 --force_quote_type double -i scripts/hiddenapi/signature_trie*
      /usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_trie*
Change-Id: I758ca70bb5b7e6806f14b72fd04f821a069f188f

Change-Id: I73fdb7e02127a8c0171a285221d9e6024310953d
2022-03-09 14:04:23 +00:00
Anton Hansson
b850f84cb9 Generate soong.variables for ndk build
Update the ndk build script to have Make generate the soong.variables
via the config step, using a new "ndk" product.

Bug: 174315599
Test: build/soong/scripts/build-ndk-prebuilts.sh
Change-Id: I0f6fbf523cf9209ea17b7752dd7470012e0baf8c
2022-02-23 12:16:48 +00:00
Anton Hansson
3a3f169b56 Add support for sdk extensions in prebuilt_apis
This makes it possible to pass an extensions_dir containing finalized
module APIs to prebuilt_apis. The extension versions are compared to the
api level versions to figure out what the "latest" finalized API is for
each module. This is done using the base_sdk_extension_version, such
that any extension higher than than base_sdk_extension_version is
assumed to be finalized after any of the existing api level versions.

Bug: 220086085
Test: prebuilt_apis_test.go
Test: existing module in prebuilts/sdk
Change-Id: Ib792f84202d436f594ba5e8716c6a187f9cd60dc
2022-02-21 16:43:43 +00:00
Thiébaud Weksteen
8fae31138a Update rustfmt.toml to 2021 edition
Bug: 210475268
Test: $ cd system/security; find . -name \*.rs -exec rustfmt --check {} \;
Change-Id: I49044e19a17d3806570d04c3215ccd3554ce7903
2022-02-10 16:55:01 +11:00
Cole Faust
8d47c48082 Use soong's finder to find included makefiles in mk2rbc
Having soong generate a list of makefiles for mk2rbc
to look through is much faster than having mk2rbc search
itself. Profiling the readLinesFromeFile() function that
reads the list of makefiles shows it takes on the order
of 200 microseconds, much faster than the ~5 seconds
it takes for mk2rbc to search the tree itself.

This CL also allows include statements that are prefixed
with a variable. The concern with this was that there
would be a lot of load statemnts emitted for generic
include statements, causing the generated code to look
ugly, and converting and loading all those files could
cause performance issues. On the performance issues
front, there's already a check that it doesn't result
in over 150 potentially included files. We can lower
that number if necessary, but it's probably good for now.
On the generated code front, while it's true that it's
ugly, it's better to have working but ugly generated
code than refusing to generate anything working at all.

To ensure the soong finder step isn't slowed down due
to having to find a bunch of new makefiles, I profiled
the combination of newSourceFinder and FindSources in main.go:

Baseline incremental:
338.011634ms
340.853335ms
348.541762ms
333.229644ms
349.124824ms

Baseline clean:
1.003836419s
1.006203912s
996.193648ms
1.031005604s
1.03691152s

Modified incremental:
349.029285ms
349.264496ms
351.774948ms
337.63187ms
359.425306ms

Modified clean:
1.028238704s
1.053103506s
1.032757506s
1.016631201s
1.04288108s

So we can see the times are barely affected by this change.

Fixes: 213508006
Test: go test
Change-Id: Iab18bfb127ba3b7e63f2c01f69064805a8398764
2022-02-07 11:41:28 -08:00
Gurpreet Singh
75d65f360c Add testOnly attribute to AndroidManifest file of apex_test
If the build file contains the apex_test module, add the
testOnly attribute to the application element of the
corresponding AndroidManifest file and set its value to true.
If the testOnly attribute is already present and has value
false, then do nothing.

Tests added in manifest_fixer_test.py to check if the updated
AndroidManifest file has the testOnly attribute set to true or not.

Bug: 213310150
Test: atest --host manifest_fixer_test
Test: m nothing
Test: manually checked the AndroidManifest file generated
Change-Id: I36247dbe0261c342d451a4422c314fd8fe0c2369
2022-02-01 12:03:48 +00:00
satayev
83bda94c12 Remove OWNERS for deleted build-mainline-modules.sh
Bug: 180394948
Test: presubmit
Change-Id: I76a157f9292d28155a7f62acc93cf7fa7fb39cb9
2021-12-15 18:45:22 +00:00
satayev
035fba700a Remove unused build-mainline-modules.sh
This script is no longer used in any CI or in any other scripts.

Bug: 180394948
Test: presubmit
Change-Id: Ia1b9f699ae3b9263b6e1d6c28ebec449ee31ef84
2021-12-15 16:21:02 +00:00
sophiez
bc82ba55c5 Java used-by API list generation
Update the java used by script to fix the xml format when dexdeps processing resource only APKs

Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only
Change-Id: I61a58d7c739e74a571e97166a7c9b4091e6f8da0
2021-12-14 14:52:58 -08:00
sophiez
65a981569f Java used-by API list generation
Wrap the dexdeps output content with "<externals></externals>" to
simplify backend xml parsing.

Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only
Change-Id: I1457eb9558763e50911dc9c689b9bc3fb833a52c
2021-12-10 13:43:10 -08:00
Nikita Iashchenko
84bec80023 Merge "Add jdk.internal.ref to package_allowed_list.txt (OpenJDK 11)" 2021-12-08 13:40:44 +00:00
Cole Faust
864028a71b Support passing input variables to the product configuration
Since rblf_env / rblf_cli are not typed properly, accept
input variables via a file so that they can be converted
with the correct types.

Bug: 201700692
Test: go test
Change-Id: I9b56067cfe396d1bcd8d62c353ff222dd61a6c9f
2021-12-02 16:19:19 -08:00
Cole Faust
c903f00e3f Remove python 2 support from manifest_utils
Bug: 203436762
Test: Presubmits
Change-Id: If93033a428aa1aa0dd2db973666d4790af8f0912
2021-11-30 12:24:02 -08:00
Nikita Iashchenko
a4676f5de6 Add jdk.internal.ref to package_allowed_list.txt (OpenJDK 11)
This is part of merging upstream changes from OpenJDK 11.28. New classes
have been added to core-oj, so there is a new path added to
package_allowed_list.txt:

 * jdk.internal.ref

Bug: 198792576
Test: atest CtsLibcoreOjTestCases
Change-Id: I19a40447bbb060612e0c2f3ef0770568345a7405
2021-11-24 17:15:30 +00:00
Bob Badour
0174ae3ae0 Account for "validations:"
Change-Id: I72b188c8cac54aa59d31e91b2a03b1329384e686
2021-11-23 12:12:06 -08:00
Anton Hansson
c5eac1cf47 Merge "Silence unpack-prebuilt-apex" 2021-11-22 16:36:49 +00:00
Anton Hansson
ff2804e195 Silence unpack-prebuilt-apex
Build tools should not output anything on success, as it clutters
the terminal.

Test: No
Change-Id: Iadfc326be58b32d5cb4f40dd114d3f0a54dbd9b4
2021-11-22 15:07:35 +00:00
Sasha Smundak
82b6e8483c A script to update the file contents with the command output.
Used to avoid rerunning Ninja files regeneration if m2rbc conversion
generated the same makefile.

Test: treehugger
Change-Id: I1b0a619f961e6d2c7bf99a48053ecb58147c6db0
2021-11-18 15:15:45 -08:00
Victor Hsieh
83742ea9e9 Merge "Revert "Include compos-module-sdk in the SDK build."" 2021-11-18 15:47:01 +00:00
Victor Hsieh
e4f0b0a090 Revert "Include compos-module-sdk in the SDK build."
This reverts commit 5b817e642c.

Reason for revert: No longer needed

Bug: 203231943
Change-Id: I220ff6b2033cadf7b29f1bc4a68bec50900f2aef
2021-11-16 19:17:42 +00:00
Sophie Zheng
7b5fa4277f Merge "Java APIs used by Mainline modules." 2021-11-15 06:40:31 +00:00
sophiez
0234737f30 Java APIs used by Mainline modules.
Add scripts to generate java APIs used by Mainline modules. This is
aosp/1879177 with fixed build error that caused by dexdeps analyzing
resource only APKs which doesn't contains any .dex files. Now skip the
error generates in that case in gen_java_usedby_apex.sh

Forrest run:https://android-build.googleplex.com/builds/abtd/run/L38900000951898586

Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only
Change-Id: Id539cbe1b7306ace69c047f95ffc02265a467511
2021-11-12 23:39:35 +00:00
Cole Faust
4220a61349 Merge "Convert soong scripts to python 3" 2021-11-12 19:04:31 +00:00
Cole Faust
c41dd72d3b Convert soong scripts to python 3
Except manifest_utils, which is used by
apexer.

Tests run:
m construct_context_test manifest_check_test manifest_fixer_test test_config_fixer_test
out/host/linux-x86/testcases/construct_context_test/x86_64/construct_context_test
out/host/linux-x86/testcases/manifest_check_test/x86_64/manifest_check_test
out/host/linux-x86/testcases/manifest_fixer_test/x86_64/manifest_fixer_test
out/host/linux-x86/testcases/test_config_fixer_test/x86_64/test_config_fixer_test

Bug: 203436762
Test: Tests listed above + presubmits
Change-Id: Ife75b3af1efc871a2d8521d26153a262573bb706
2021-11-10 13:49:27 -08:00
Laura Tsai
33b60e4983 Merge "Revert "Add scripts to generate java APIs used by Mainline modules."" 2021-11-10 07:00:38 +00:00
Laura Tsai
be3c7eacfe Revert "Add scripts to generate java APIs used by Mainline modules."
This reverts commit c92380f001.

Reason for revert:
DroidMonitor:

DroidMonitor-triggered revert due to breakage
https://android-build.googleplex.com/builds/quarterdeck?branch=git_master&target=mainline_modules_arm64-userdebug&lkgb=7898389&lkbb=7900541&fkbb=7898474,
bug 205746850

Change-Id: Ibd84dfe68dfe150b7e05b4287364d1946375bf72
2021-11-10 06:59:37 +00:00
Sophie Zheng
4bb0b7a0c3 Merge "Add scripts to generate java APIs used by Mainline modules." 2021-11-09 17:13:33 +00:00
Treehugger Robot
7ab37e8903 Merge "Fix location of host tools" 2021-11-06 00:13:56 +00:00
sophiez
c92380f001 Add scripts to generate java APIs used by Mainline modules.
Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only

Change-Id: Icd854fd40c3026ae9aae6dc30c9e79027e1303a0
2021-11-05 15:38:09 -07:00
Colin Cross
6a4d3ba138 Fix location of host tools
Host tools are in out/host/linux-x86/bin now instead of
out/soong/host/linux-x86/bin.

Bug: 204136549
Test: forrest
Change-Id: I293accdde5ee4d1d21a471e12ba66e169f9410fc
2021-11-05 14:10:01 -07:00
Cole Faust
943ac416d2 Collect generated rbc files under one folder
Mk2rbc takes a .mk file and produces a .rbc file under the
same path, but with a different stem. For example:
build/test/foo.mk becomes out/build/test/foo.rbc

This makes it difficult to see everything that mk2rbc has
generated in the out folder. Move the generated files to
out/rbc instead, so they have a common stem that is separate
from the rest of the build outputs.

Fixes: 202249430
Test: m RBC_PRODUCT_CONFIG=1 nothing
Change-Id: If3edba3feef9c2d3631244d533b997ef0b8b4e8b
2021-11-04 16:39:59 -07:00
Lukacs T. Berki
04bd17ecb9 Fix paths rbc-run uses to access the rbc tools.
This was broken in aosp/1878498; ccross@ submitted aosp/1875758 between
writing that change and submitting it, which changed the paths to mk2rbc
and rbcrun, thereby breaking Starlark product configuration.

Test: Presubmits.
Change-Id: I185489589997e6bdc815a54a8f6f328c18f813d0
2021-11-04 18:02:22 +01:00
Lukacs T. Berki
90b4334a64 Remove the out/soong/.bootstrap directory.
It's better not to have state hidden in dotfiles (or rather,
dotdirectories) if we can help.

It's questionable whether the "linux-x86" path segment makes sense since
soong_build only ever runs on one operating system, but I didn't want to
rock the boat now.

Drive-by fixed some quoting fixes in rbc-run. Notably, I didn't wrap
`$@` into double quotes because I don't know whether the lack of double
quotes was intended or not.

Also drive-by fixed the fact that "out/soong" was added twice to the
directory name of bpglob. This turned out not to be a problem because
bpglob doesn't need to be explicitly built: if it's needed, it's
declared as an input of the glob files so it'll be built automatically
(at the cost of the first "null build" not actually being a null build)

Test: Presubmits.
Change-Id: I710d8d16cd8212059a0ca1ee95378505303eed83
2021-11-04 09:47:58 +01:00
Treehugger Robot
31ec5a0709 Merge "Adding jdk.internal in the allowed package list" 2021-11-02 08:49:31 +00:00
Treehugger Robot
0888700fe1 Merge "Make manifest_check errors colorful." 2021-11-01 17:11:52 +00:00
Ulya Trafimovich
b4c19f8a31 Make manifest_check errors colorful.
Insert ANSI escape codes in the error text. Colors are red, blue, green
and some text is in bold --- these are all bright enough on either dark
or light background.

Also, add a link to the online documentation.

Bug: 132357300
Test: manually mangle one of Android.bp files to get a manifest_check
  error and observe that it is colorful.
Change-Id: I2af2aa0415d0eb0eabc88dc5504198e11bfb91b6
2021-11-01 14:39:46 +00:00
Sorin Basca
ca4ab1a7a8 Adding jdk.internal in the allowed package list
This is needed for adding jdk.internal.HotSpotIntrinsicCandidate as a no-op

Bug: 202495224
Test: m
Change-Id: Ib97a7fe955d920aca93630d1d5b04fedff1af960
2021-11-01 14:20:33 +00:00
Colin Cross
34d60c9281 Use --skip-config --soong-only instead of --skip-make
--skip-make is deprecated because of its odd behavior around the
out/.soong.kati_enabled file.  Replace it with --skip-config --soong-only
instead, which will unambigiously skip running kati.

Bug: 204136549
Test: build/soong/tests/run_integration_tests.sh
Test: build/soong/scripts/build-mainline-modules.sh
Test: build/soong/scripts/build-ndk-prebuilts.sh
Change-Id: I629ee7542687aaafd68eea50346b0cf53eafe1ef
2021-10-29 16:34:55 -07:00
Sophie Zheng
7fe4ccafc1 Merge "Save all the detected native shared library backing by mainline modules." 2021-10-21 18:42:35 +00:00
sophiez
9a6eabf952 Save all the detected native shared library backing by mainline modules.
Test: TARGET_BUILD_APPS=com.google.android.adbd m dist apps_only

Change-Id: I510292542bf2550c9244e9374cd6f7b40a971486
2021-10-19 14:33:57 -07:00
Victor Hsieh
5b817e642c Include compos-module-sdk in the SDK build.
Test: build/soong/scripts/build-mainline-modules.sh
Bug: 203231943
Change-Id: Icbd4b15b2434c8a0a7983d52732b5462f3a8b978
2021-10-15 13:38:59 -07:00
Pirama Arumuga Nainar
811a1ae270 [get_clang_version] Make the version regex permissive
For instance, this will match `clang-dev` when using a local build of
llvm-toolchain.

Test: run script
Change-Id: I3f2a0a6adc360f0a299def79535ce139353aece2
2021-10-06 09:44:15 -07:00
Paul Duffin
1e18e98b14 Allow implementation details to be excluded from signature patterns file
Previously, the signature-patterns.csv file (which was output in an sdk
snapshot) included implementation details, e.g. the names of classes
that are not part of any API, including the hidden API. This change
allows a bootclasspath_fragment module owner to customize the generated
signature patterns file to remove as many implementation details as
possible from the sdk snapshot. That means that implementation only
changes would not require that sdk snapshots be updated in sync with
the corresponding APEX.

Module owners can remove implementation in the following ways:
1) Limit the number of packages that are treated as split packages by
   explicitly specifying the split packages in the split_packages
   property. A split package is one whose classes are provided by
   separate bootclasspath_fragment modules and so the signature
   patterns has to include every class in that package provided by a
   specific bootclasspath_fragment module, including implementation
   classes, instead of just listing the package.

   The default is to treat all packages as being split, so if no
   split_packages is specified then it defaults to ["*"] which matches
   all packages. Assuming that no package was split unless specifically
   stated would require that all the modules that had split packages
   explicitly list them before this change could be submitted as
   without them this change would break the build.

   Once all existing modules with split packages have been updated to
   include them then the default may be changed.

2) Adding package prefixes for any hierarchy of packages that are
   owned solely by that bootclasspath_fragment. This removes the need
   to list the packages in that hierarchy, including implementation
   specific packages.

Bug: 194063708
Test: atest --host verify_overlaps_test signature_patterns_test
      m out/soong/hiddenapi/hiddenapi-flags.csv
      - manually change files to cause difference in flags to check
        that it detects the differences.
Change-Id: If2f90cfc41b6fff9fa4ac8b9d4973110938b9392
2021-10-05 15:47:05 +01:00
Sorin Basca
6e06f911a5 Merge "OpenJDK 11: Adding jdk.internal.reflect to allowed package list" 2021-10-05 12:34:42 +00:00
Cole Faust
057828d348 Make rbc-run accept a makefile instead of a product
Needed for the board configuration.

rbc-run will also now pass any extra arguments it receives
to rbcrun. (The real program, not the script)

Bug: 201700692
Test: rbc_product_config.py aosp_arm64-userdebug
Change-Id: Ic9869f693f1d28bafee59050b6ae7b1fba14dbdd
2021-10-04 11:16:55 -07:00
Jerome Gaillard
03c64c87a4 Fix script for package check to work on macOS
The behaviour of shell parameter expansion on linux and macOS is
different:
- on linux "\/" in the replaced string is interpreted as simply "/"
- on macOS it is interpreted as the full string "\/"
For example, "android.package.example" would be changed to:
- "android/package/example" on linux
- "android\/package\/example" on macOS

The character / is not a special character for pattern matching in bash,
so it doesn't need to be escaped. Hence using / instead of \/ in the
replaced string works on both linux and macOS.

Change-Id: Id6a5cf32afc53d5ffd989c0ac8aa0b9e0fcbaf82
Fixes: 201947033
2021-10-04 15:29:06 +00:00
Sorin Basca
87f33ea80b OpenJDK 11: Adding jdk.internal.reflect to allowed package list
As part of the migration to OpenJDK 11, sun.reflect has moved to
jdk.internal.reflect. For now we'll have both packages available
as new code uses the jdk.internal version.

Test: m
Bug: 188889082
Bug: 190791083
Change-Id: Ib03c9c8c346daebd7d97387817b506095f5ec693
2021-09-27 15:39:06 +00:00
Sorin Basca
1bb6d6a2c9 Adding jdk.internal.misc in the allowed packages
Updating OpenJDK 11 means that there is a dependency on
jdk.internal.misc, which is renamed from sun.misc. The new
package is added to the allowed list.

Bug: 190470684
Bug: 190791083
Test: m
Change-Id: I174df7bad5c5ec4286c35998e16c26bcd0737612
2021-09-22 15:18:45 +00:00
Treehugger Robot
77cdde7d91 Merge "Fix statusfile used in dexpreopt" 2021-09-20 19:53:38 +00:00
Spandan Das
3d5cd4d109 Fix statusfile used in dexpreopt
https://android-review.googlesource.com/c/platform/build/soong/+/1820496/1/scripts/manifest_check.py#361
caused a regression wrt to statusfile creation. Specifically it inverted
the logic used to create the status file. This file is used in
dexpreopt, thereby affecting all odex files

Test: In internal git, m <an_odex_file>. File size is restored to
pre-regression value
Bug: 199921980
Bug: 200329228

Change-Id: If1e61ac8e2b95455dacc6e7b9431c8f6e1f339b3
2021-09-20 18:24:56 +00:00
Christopher Ferris
8e5feaa1df Make the script python3.
Test: Ran the python version of the script and the python3 version
Test: and verified the output is the same. This is for all three
Test: outputs (html, text, xml).
Test: Builds without an error.
Change-Id: I344504a4351c14e00caeb109c5507a60fb6be05a
2021-09-13 14:54:01 -07:00
Peter Collingbourne
e168f15ab9 Merge "Read LLVM version from $LLVM_PREBUILTS_VERSION if set." 2021-09-09 18:17:11 +00:00
Treehugger Robot
da513b1ec1 Merge "Apply pylint to scripts/manifest_check*.py" 2021-09-08 23:42:25 +00:00
Peter Collingbourne
cbfbd64823 Read LLVM version from $LLVM_PREBUILTS_VERSION if set.
This matches the behavior of the build system.

Change-Id: Idce0a48403acde100e20246896a1ec68df6b73c7
2021-09-08 13:46:01 -07:00
Spandan Das
96d5424440 Merge "Apply pylint to check_boot_jars.py" 2021-09-08 18:32:02 +00:00
Spandan Das
f880742582 Apply pylint to scripts/manifest_check*.py
1. Run pyformat scripts/<script>.py -s 4 --force_quote_type
none -i to fix formatting
2. Annotate #pylint: disable, where straightforward fix is not available

Test: m manifest_check_test
Test: pylint --rcfile tools/repohooks/tools/pylintrc
build/soong/scripts/manifest_check.py
build/soong/scripts/manifest_check_test.py
Bug: 195738175

Change-Id: I9af498c4abd6ac9f8b9df4f93cbdd4424eacff8e
2021-09-08 03:59:40 +00:00
Nikita Iashchenko
15c1387710 Add jdk.internal.math to package_allowed_list.txt (OpenJDK 11)
This is part of merging upstream changes from OpenJDK 11.28. This CL
adds jdk.internal.math to package_allowed_list.txt

Bug: 199067710
Test: atest CtsLibcoreOjTestCases
Change-Id: I343bed53ce6b52a6b373a75828b3234073b9cba8
2021-09-06 17:38:34 +01:00
Spandan Das
2c0af22ea8 Apply pylint to check_boot_jars.py
1. Run pyformat scripts/check_boot_jars/check_boot_jars.py -s 4
--force_quote_type none -i to fix formatting.
3. use pylint: disable=<X> where fixes are not obvious

Test: m check_boot_jars
Test: pylint --rcfile tools/repohooks/tools/pylintrc
scripts/check_boot_jars/check_boot_jars.py
Bug: 195738175

Change-Id: I0f86d95e0d174c9f87a2872dedaa65cd3e814246
2021-09-03 18:01:49 +00:00
Treehugger Robot
0364846a73 Merge "Apply pylint to remaining scripts in hiddenapi" 2021-09-01 14:27:13 +00:00
Treehugger Robot
deee92ca0c Merge "Apply pylint to scripts/hiddenapi/verify_overlaps*" 2021-09-01 14:27:13 +00:00
Treehugger Robot
373fe3b598 Merge "Apply pylint to scripts/hiddenapi/generate_hiddenapi_lists*" 2021-09-01 14:27:13 +00:00
Spandan Das
559132fb98 Apply pylint to scripts/hiddenapi/verify_overlaps*
1. Run pyformat scripts/hiddenapi/verify_overlaps.py -s 4
--force_quote_type none -i to fix formatting. Some double quotes change
to single quotes since pyformat enforces consistent quotes
2. Wrap #pylint: (enable|disable)=line-too-long in tests

Test: m verify_overlaps verify_overlaps_test
Test: pylint --rcfile tools/repohooks/tools/pylintrc <file1>
<file1_test>
Bug: 195738175

Change-Id: Ieb23c04aa53a8b74917f6865296dd8740ecf20ba
2021-08-31 18:24:24 +00:00
Christopher Ferris
cd989defab Merge "Make this script use python3." 2021-08-27 21:42:09 +00:00
Christopher Ferris
cce6d652c5 Make this script use python3.
There is nothing in this script that is not python3, so
make it python3. Without this, a system without python will
fail when trying to run gdbclient.py.

Test: Ran gdbclient.py on a system without a python executable.
Change-Id: I9f8418932628569c445ee01bc6f9ead8aed7989f
2021-08-27 11:54:19 -07:00
Treehugger Robot
5edc77d06c Merge "Apply pylint to conv_linker_config.py" 2021-08-27 04:04:14 +00:00
Spandan Das
2c2219b60e Apply pylint to remaining scripts in hiddenapi
1. Run pyformat scripts/hiddenapi -s 4
--force_quote_type none -i to fix formatting.
2. rename restricted variable names (e.g. variable name "input" has been
changed to "csvfile")
3. use pylint: disable=<X> where fixes are not obvious

Test: m merge_csv signature_patterns signature_patterns_test
Test: pylint --rcfile tools/repohooks/tools/pylintrc <file1>
<file1_test>
Bug: 195738175

Change-Id: I800a208f9c0ee1d32e68e4b20fd5933b3ab92c0e
2021-08-27 00:30:41 +00:00
Spandan Das
c2c31b13dc Apply pylint to scripts/hiddenapi/generate_hiddenapi_lists*
1. Run pyformat scripts/hiddenapi/generate_hiddenapi_lists.py -s 4
--force_quote_type none -i to fix formatting. Some double quotes change
to single quotes since pyformat enforces consistent quotes
2. change map/filter to list comprehension to fix pylint's bad-builtin
warning
3. use pylint: disable=<X> where fixes are not obvious

Test: m generate_hiddenapi_lists_test
Test: pylint --rcfile tools/repohooks/tools/pylintrc <file1>
<file1_test>
Bug: 195738175

Change-Id: I15e71d8d81f0e3fa66d84e6e62eb1848963136e6
2021-08-27 00:09:25 +00:00
Spandan Das
1d4bfd8ca4 Apply pylint to conv_linker_config.py
1. Run pyformat scripts/conv_linker_config.py -s 4 --force_quote_type none -i to fix formatting
2. Annotate #pylint: disable=import-error for linker_config_pb2 since it
will be provided by soong

Test: m conv_linker_config
Test: pylint --rcfile tools/repohooks/tools/pylintrc
build/soong/scripts/conv_linker_config.py
Bug: 195738175

Change-Id: I791576cf65cb053f68c804f8ec5c2fc22976fdb4
2021-08-26 16:48:59 +00:00
Spandan Das
ec555f1a99 Apply pylint to scripts/construct_context*.py
1. Run pyformat scripts/construct_context.py -s 4 --force_quote_type
none -i to fix formatting
2. Wrap #pylint: [disable|enable]=line-too-long in tests

Test: m construct_context construct_context_test
Test: pylint --rcfile tools/repohooks/tools/pylintrc
build/soong/scripts/construct_context.py
build/soong/scripts/construct_context_test.py
Bug: 195738175

Change-Id: I2e756b17aae3e70ddaa03356c78f1f2f85c0f6a5
2021-08-26 01:06:48 +00:00
Matthew Maurer
ca68c49621 rust: doc: Zip up docs when done
Fetching artifacts from the build server doesn't go well with large
numbers of files in nested directories.

Produce an additional rustdoc.zip artifact to make this easier.

This change also prevents ${DIST} from receiving the unpacked rustdoc
site. This can be changed if b/188822051 is fixed.

Bug: 162741284
Test: m rustdoc; check resulting zip contents manually
Change-Id: I80b6a8fa6e274d2d8c3419d8734251afd4d7dba7
2021-08-20 13:08:16 -07:00
Paul Duffin
280bae6d20 Filter blocked entries from modular flag files
Previously, the sdk snapshot would include all the entries from the
stub-flags.csv and all-flags.csv modular files generated by a single
bootclasspath_fragment. That included implementation details, i.e.
class members that are not part of a stable API or the hidden API which
meant that the sdk snapshots were implementation dependent.

This change removes the implementation details from the modular flag
files, i.e. those entries that are only marked as "blocked". When
comparing the files against the corresponding subset of the monolithic
files it assumes that any entries missing from the modular flag files
are blocked.

Bug: 194063708
Test: atest --host verify_overlaps_test signature_patterns_test
      m out/soong/hiddenapi/hiddenapi-flags.csv
      - manually change files to cause difference in flags to check
        that it detects the differences.
Change-Id: I6b67b2253cf029d6830b58a06ebb0c8fcaa0dd71
2021-08-11 17:29:47 +01:00