Previously, the signature_trie python library would reject classes
which started with a lower case character which in turn caused the
verify_overlaps tool to fail. That meant that it was impossible to
add obfuscated classes (which commonly used lower case characters
for their name) from being used on the bootclasspath.
This change removes that restriction and the accompanying test.
Bug: 265833521
Test: TH and partner testing
Change-Id: I70710484e427f64d79fb30301f3413f3b67b27e7
Revert submission 1931011-max-target-s
Reason for revert: Seems to cause build breakage
Reverted Changes:
I2b513f93a:Add support to max-target-s to generate_hiddenapi_...
Ifdd7b7a71:Add support for max-target-s
I06d05840e:Add support for max-target-s
Change-Id: I67413a3deda42c503b876727377e932f7e4652e9
Previously, the build applied the same filtering to remove
implementation details from the sdk snapshot's stub-flags.csv file as
it did for its all-flags.csv, i.e. removing the signatures that only
had a "blocked" flag. Unfortunately, that had no effect on the stub
flags as the implementation signatures had no flags, not a single
blocked flag. That meant that the sdk snapshot's
filtered-stub-flags.csv file contained a lot of implementation details.
This change removes signatures from stub-flags.csv that have no flags
which removes all implementation details from the sdk snapshot.
Bug: 194063708
Test: atest --host verify_overlaps_test
m out/soong/hiddenapi/hiddenapi-flags.csv
m art-module-sdk
# Check contents of its filtered-stub-flags.csv file
Change-Id: I30edc77348fad118ea732e787ae8e206c8841f84
Previously, verify_overlaps used positional arguments, the first was
the monolithi flags and the rest were the module flag pairs (filtered
flags file and signature patterns file). This change makes them use
named options to make the purpose of the arguments clearer on the
command line.
Bug: 194063708
Test: atest --host verify_overlaps_test
m out/soong/hiddenapi/hiddenapi-flags.csv
Change-Id: Ife0af0016eb0f91416e8330d5d98cb53c97d68a4
Fix issues reported by pylint, Intellij Python checks and also try and
adhere to the Google Python Style Guide.
Bug: 194063708
Test: atest --host verify_overlaps_test
m out/soong/hiddenapi/hiddenapi-flags.csv
/usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/verify_overlaps*.py
pyformat -s 4 --force_quote_type single -i scripts/hiddenapi/verify_overlaps*.py
Change-Id: I6273456abc3a4da4af780ae1a0473ca63521dff5
Previously, the script would determine whether a package was split,
single or could be used as a prefix but did not explain why. This
change provides additional information to explain why they are split.
Bug: 202154151
Test: m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment
m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment --fix
atest --host analyze_bcpf_test
Change-Id: I3e2d5c0b54b5cc028013ce5ea979ebd9b9bf2c0d
Addresses a comment on the review of the initial changes to add
analyze_bcpf script.
Bug: 202154151
Test: m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment
m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment --fix
atest --host analyze_bcpf_test
Change-Id: I8c030dcc1a29f106ca57dca8c97a5cb4425e7674
Previously, Leaf.values() and Leaf.append_values() would wrap the
Leaf's value inside a list before appending it to the list of values.
So, the values list was actually a list of lists of values. The
get_matching_rows method would then use chain.from_iterable() to
flatten that list of list of values into a list of values.
This change removes the initial wrapping in a list and so removes the
need to flatten them into a single list. It also adds a test for the
values() method. Prior to this change the expected value would have
been [[1], ["A"], [{}]].
Bug: 202154151
Test: atest --host analyze_bcpf_test signature_trie_test verify_overlaps_test
Change-Id: Ida78500c9ab4466de127b2c36501b3606d0f3fe5
Analyzes the signatures of the class members contained within the
bootclasspath_fragment to construct the split_packages,
single_packages and package_prefixes hidden_api properties that are
necessary to remove the internal implementation details from the
hidden API flags.
Bug: 202154151
Test: m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment
m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment --fix
m analyze_bcpf && analyze_bcpf --bcpf com.android.mediaprovider-bootclasspath-fragment --fix
atest --host analyze_bcpf_test
Change-Id: I4a8e5a8bfee2a44775e714b9226cd4a7382e0123
Add a --fix option that will cause the script to automatically fix the
issues that it finds. It uses the bpmodify tool to add values to the
bootclasspath_fragment's hidden_api properties.
This adds analyze_bcpf to bp2buildModuleDoNotConvertList as
analyze_bcpf depends on bpmodify which is a blueprint_go_binary which
is not yet supported by bazel.
Bug: 202154151
Test: m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment
m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment --fix
atest --host analyze_bcpf_test
Change-Id: I5ee52419b4829474f6dbeb47f86ab2aeb22b1382
The analyze_bcpf.py script can be used by bootclasspath_fragment owners
to help setup their fragment and also to debug issues that arise during
use.
It is run by specifying the name of the bootclasspath_fragment module
in the --bcpf option. If specified the --apex and --sdk options will
allow the tool to produce more helpful messages.
The script currently:
* Builds out/soong/hiddenapi/hiddenapi-stub-flags.txt and checks any
inconsistencies between those flags and the flags generated for the
bootclasspath_fragment.
* Builds out/soong/hiddenapi/hiddenapi-flags.csv and checks any
inconsistencies between those flags and the flags generated for the
bootclasspath_fragment. It will describe the changes that need to
be made in order to fix any inconsistencies in the flags.
Bug: 202154151
Test: m analyze_bcpf && analyze_bcpf --bcpf art-bootclasspath-fragment
atest --host analyze_bcpf_test
Change-Id: I824fd9d527d246dfe56725c073d89a0b0e72d25a
Previously, the bootclasspath_fragment's hidden_api.split_packages and
hidden_api.package_prefixes properties did not specify an exhaustive
set of packages that were provided by the fragment. They excluded
packages which were either not split or which could not be used as a
package prefix because it would match sub-packages provided by other
bootclasspath modules.
This change adds the hidden_api.single_packages list to specify those
additional packages and then uses that information to verify that any
bootclasspath_fragment that specifies at least one of split_packages,
single_packages or package_prefixes properties only contains classes
from a package that matches one of those properties. That will
prevent a module from accidentally including unexpected classes, such
as might happen when statically including a common utility library.
It also adds coverage specific versions of the properties as additional
packages are added to the art-bootclasspath-fragment when building
coverage builds.
Bug: 194063708
Test: atest signature_patterns_test
m out/soong/hiddenapi/hiddenapi-flags.csv
m EMMA_INSTRUMENT=true EMMA_INSTRUMENT_FRAMEWORK=true out/soong/hiddenapi/hiddenapi-flags.csv
# Breaks without corresponding change to add android.system to
# the art-bootclasspath-fragment.
/usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_patterns*.py
pyformat -s 4 --force_quote_type single -i scripts/hiddenapi/signature_patterns*.py
Change-Id: Iddf6c59cd4dc8c36dde7943a9840ccef5794b320
Fix issues reported by pylint, Intellij Python checks and also try and
adhere to the Google Python Style Guide.
Bug: 194063708
Test: atest signature_patterns_test
m out/soong/hiddenapi/hiddenapi-flags.csv
/usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_patterns*.py
pyformat -s 4 --force_quote_type single -i scripts/hiddenapi/signature_patterns*.py
Change-Id: I64d64e9cb269f58d65f4e10ec2f0e874154919e6
Previously, there was no way to traverse the trie structure and no way
to identify specific nodes in the trie. That made it impossible to
analyze the trie structure resulting from loading a set of flags. This
change adds type and selector properties to nodes as well as access to
the child nodes of a node to allow for the structure to be analyzed.
Bug: 202154151
Test: m out/soong/hiddenapi/hiddenapi-flags.csv
atest --host signature_trie_test verify_overlaps_test
pyformat -s 4 --force_quote_type double -i scripts/hiddenapi/signature_trie*
/usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_trie*
Change-Id: Ia4714dbf59f6fd143aa3bf3ad1a59cd073d2175b
Previously, it used a structured string of format <type>:<value>. A
tuple is more efficient and less prone to edge cases, such as when the
value is a field signature which itself contains a ":".
Bug: 202154151
Test: m out/soong/hiddenapi/hiddenapi-flags.csv
atest --host signature_trie_test verify_overlaps_test
pyformat -s 4 --force_quote_type double -i scripts/hiddenapi/signature_trie*
/usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_trie*
Change-Id: I80abaff243d98aad325df1b5a655bba7f9d04e2c
Previously, signature_to_elements would return a string array where
non-wildcard strings were of the form <type>:<value> but wildcard
strings were just * or **. This change makes it handle wildcards
consistently with the other element types and adds some extra
checking for edge cases.
Bug: 202154151
Test: m out/soong/hiddenapi/hiddenapi-flags.csv
atest --host signature_trie_test verify_overlaps_test
pyformat -s 4 --force_quote_type double -i scripts/hiddenapi/signature_trie*
/usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_trie*
Change-Id: I5bfaf5e75c7da54b6241f68e03231939c9d65501
Adds a Node superclass to declare and documented the methods that need
to be implemented by Leaf and InteriorNode. Also uses @dataclasses to
make it easy to declare and document the properties of each class.
This refactoring is in preparation for future changes that will need to
add common behavior to both Leaf and InteriorNode so having a
superclass will make that easier.
Bug: 202154151
Test: m out/soong/hiddenapi/hiddenapi-flags.csv
atest --host signature_trie_test verify_overlaps_test
pyformat -s 4 --force_quote_type double -i scripts/hiddenapi/signature_trie*
/usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_trie*
Change-Id: If660855f6b1f45a8ea5b90b3cc0236da9e07d090
Makes the efficient pattern matching of hidden API flags that is used
by verify_overlaps.py available for use in other scripts.
As part of the move this cleans up the python to use consistent quotes,
and fix pylint issues.
Bug: 202154151
Test: m out/soong/hiddenapi/hiddenapi-flags.csv
atest --host signature_trie_test verify_overlaps_test
pyformat -s 4 --force_quote_type double -i scripts/hiddenapi/signature_trie*
/usr/bin/pylint --rcfile $ANDROID_BUILD_TOP/tools/repohooks/tools/pylintrc scripts/hiddenapi/signature_trie*
Change-Id: I758ca70bb5b7e6806f14b72fd04f821a069f188f
Change-Id: I73fdb7e02127a8c0171a285221d9e6024310953d
Previously, the signature-patterns.csv file (which was output in an sdk
snapshot) included implementation details, e.g. the names of classes
that are not part of any API, including the hidden API. This change
allows a bootclasspath_fragment module owner to customize the generated
signature patterns file to remove as many implementation details as
possible from the sdk snapshot. That means that implementation only
changes would not require that sdk snapshots be updated in sync with
the corresponding APEX.
Module owners can remove implementation in the following ways:
1) Limit the number of packages that are treated as split packages by
explicitly specifying the split packages in the split_packages
property. A split package is one whose classes are provided by
separate bootclasspath_fragment modules and so the signature
patterns has to include every class in that package provided by a
specific bootclasspath_fragment module, including implementation
classes, instead of just listing the package.
The default is to treat all packages as being split, so if no
split_packages is specified then it defaults to ["*"] which matches
all packages. Assuming that no package was split unless specifically
stated would require that all the modules that had split packages
explicitly list them before this change could be submitted as
without them this change would break the build.
Once all existing modules with split packages have been updated to
include them then the default may be changed.
2) Adding package prefixes for any hierarchy of packages that are
owned solely by that bootclasspath_fragment. This removes the need
to list the packages in that hierarchy, including implementation
specific packages.
Bug: 194063708
Test: atest --host verify_overlaps_test signature_patterns_test
m out/soong/hiddenapi/hiddenapi-flags.csv
- manually change files to cause difference in flags to check
that it detects the differences.
Change-Id: If2f90cfc41b6fff9fa4ac8b9d4973110938b9392
1. Run pyformat scripts/hiddenapi -s 4
--force_quote_type none -i to fix formatting.
2. rename restricted variable names (e.g. variable name "input" has been
changed to "csvfile")
3. use pylint: disable=<X> where fixes are not obvious
Test: m merge_csv signature_patterns signature_patterns_test
Test: pylint --rcfile tools/repohooks/tools/pylintrc <file1>
<file1_test>
Bug: 195738175
Change-Id: I800a208f9c0ee1d32e68e4b20fd5933b3ab92c0e
1. Run pyformat scripts/hiddenapi/generate_hiddenapi_lists.py -s 4
--force_quote_type none -i to fix formatting. Some double quotes change
to single quotes since pyformat enforces consistent quotes
2. change map/filter to list comprehension to fix pylint's bad-builtin
warning
3. use pylint: disable=<X> where fixes are not obvious
Test: m generate_hiddenapi_lists_test
Test: pylint --rcfile tools/repohooks/tools/pylintrc <file1>
<file1_test>
Bug: 195738175
Change-Id: I15e71d8d81f0e3fa66d84e6e62eb1848963136e6
Previously, the sdk snapshot would include all the entries from the
stub-flags.csv and all-flags.csv modular files generated by a single
bootclasspath_fragment. That included implementation details, i.e.
class members that are not part of a stable API or the hidden API which
meant that the sdk snapshots were implementation dependent.
This change removes the implementation details from the modular flag
files, i.e. those entries that are only marked as "blocked". When
comparing the files against the corresponding subset of the monolithic
files it assumes that any entries missing from the modular flag files
are blocked.
Bug: 194063708
Test: atest --host verify_overlaps_test signature_patterns_test
m out/soong/hiddenapi/hiddenapi-flags.csv
- manually change files to cause difference in flags to check
that it detects the differences.
Change-Id: I6b67b2253cf029d6830b58a06ebb0c8fcaa0dd71
Previously, the signature-patterns.csv file included a lot of
implementation details, e.g. the signatures of dex members or inner
classes that are not part of any API, including the hidden API.
This change will remove all member signatures and inner class names
from the file and replace them with just the outermost qualified class
name.
That will still leave some implementation details, e.g. the names of
implementation only classes and packages.
Bug: 194063708
Test: atest --host verify_overlaps_test signature_patterns_test
m out/soong/hiddenapi/hiddenapi-flags.csv
- manually change files to cause difference in flags to check
that it detects the differences.
Change-Id: I9de6a2a6129e875e19f7ded5fae578cbdb584660
Previously, a simple map from dex signature to the flags was used to
store the monolithic hidden API flags. This change replaces that with a
trie that uses packages and class names extracted from the signature
to map from the signature to the flags.
The signature is broken down into pieces consisting of package names,
class names and the remaining member signature. They are used in turn
to navigate through nodes in the trie to a Leaf that contains the flags
associated with the signature.
The flags can be retrieved from the trie in a number of ways:
* Using the complete signature to retrieve the flags for a single
member.
* Using a class name to retrieve the flags for all the members in a
class and any nested classes.
* Using a package wildcard to retrieve the flags for all the members in
the classes in that package.
* Using a recursive package wildcard to retrieve the flags for all the
members in the classes in that package and nested packages.
This will allow a bootclasspath_fragment to select a subset of the
monolithic flags to compare against itself without specifying the
signatures of every member of that set.
Before creating the trie an attempt was made to compute the subset by
iterating over all the signatures in the monolithic flags and matching
against a regular expression created from the patterns but that was too
slow. It took minutes to run whereas using the simple map or the trie
only takes a few seconds.
Bug: 194063708
Test: atest --host verify_overlaps_test
Change-Id: I36f5e319d3e7d62dd34305de1eec990a93cb3a89
Previously, the signatures used to select the subset of the monolithic
flags were simply the signatures read from the modular flags file. This
change moves the creation of the signature list into a separate script
that outputs the signatures to a file and then passes the path through
Soong from the bootclasspath_fragment modules that create it to the
platform_bootclasspath module that uses it to compare the modular
flags against the monolithic flags.
Currently, the signatures are the full signatures but follow up changes
will replace them with patterns (hence the name) that avoids having to
include implementation details in the hidden API flags that are output
as part of a bootclasspath_fragment's snapshot.
This change moves the stub flags related code next to the all flags
related code as they are treated in a similar way.
Bug: 194063708
Test: atest --host verify_overlaps_test signature_patterns_test
m out/soong/hiddenapi/hiddenapi-flags.csv
- manually change files to cause difference in flags to check
that it detects the differences.
Change-Id: I2855bf6d05c91b8a09591664185750361c7e644f
Previously, the compare_signature_flags compared the full set of
monolithic flags against the modular flags and just ignored any
monolithic flags for a signature that was not present in the modular
flags. It essentially performed two tasks, identifying the subset of
the monolithic flags against which the modular flags should be compared
and then doing the actual comparison.
This change separates those two tasks, by first creating a dict that is
the subset of the monolithic flags against which the modular flags will
be compared and then do the comparison.
At the moment the subset is simply the set of monolithic flags
associated with the signatures that are present in the modular flags.
Follow up changes will provide other ways to select the subset.
Bug: 194063708
Test: atest --host verify_overlaps_test
m out/soong/hiddenapi/hiddenapi-flags.csv
- manually change files to cause difference in flags to check
that it detects the differences.
Change-Id: Icb547f071b814dd955f31255b0d35597f7ba4aa1
Switches symbols to use monolithic... and modular... to try and clarify
the purpose of those symbols.
Bug: 194063708
Test: atest --host verify_overlaps_test
Change-Id: Ifc646edebe3d43b51dc3528fa883718399cb5254
Refactor verify_overlaps to make it testable and add tests for the
comparison. It does make one significant change in behavior which is to
read each of the files produced by a bootclasspath_fragment into a dict
before comparison, rather than reading and comparing them a row at a
time. That allows it to reuse the code to read a CSV into a dict.
Bug: 194063708
Test: atest --host verify_overlaps_test
m out/soong/hiddenapi/hiddenapi-flags.csv
- manually change files to cause difference in flags to check
that it detects the differences.
Change-Id: Ib70ac87fe089fc25e3bef18f367d4939bfc0cb8d
atest generate_hiddenapi_lists_test would not work without setting
verbosity=2.
Bug: 194063708
Test: atest --host generate_hiddenapi_lists_test
Change-Id: Ic076ea85d58917921081ae33e72f06011c4ca49e
Bug: 179354495
Test: m out/soong/hiddenapi/hiddenapi-stub-flags.txt
- check that an error is reported if a modular stub-flags.csv file,
i.e. one created by a fragment is not a subset of the monolithic
file, e.g. because a signature in the modular file has different
flags than it does in the monolithic or is not present there.
Change-Id: I46ebb495cb093a5e3abe7571c49933c845318549
Previously, if the --header property was not specified then merge_csv
would use a header constructed by sorting all the fields in the input
files. That required that any use of merge_csv which did not already
have headers in the required order would have to explicitly specify the
headers. That made it harder to use merge_csv as a generic tool as each
invocation needed to be aware of what headers were exported in the
output.
This change causes merge_csv to simply use the headers in the order in
which they are encountered in the input files. That removes the need to
specify the --header option when generating the index files.
Bug: 179354495
Test: m out/soong/hiddenapi/hiddenapi-index.csv out/soong/hiddenapi/hiddenapi-unsupported.csv
- make sure that they are not changed by this change.
Change-Id: I420b7d07aea85af6372cd7580a8be5e2cc82a513
This makes sure that where there is overlap between the hidden API
flags generated for a module and the monolithic flags that they are
identical. That ensures that the modular hidden API flags will be
compatible with previous releases that relied on the monolithic flags.
Bug: 179354495
Test: m out/soong/.intermediates/art/build/boot/art-bootclasspath-fragment/android_common_apex10000/modular-hiddenapi/all-flags.csv
m out/soong/hiddenapi/hiddenapi-flags.csv
- Create some inconsistencies between the above two files.
m out/soong/hiddenapi/hiddenapi-flags.csv.valid
Change-Id: Iaf9e23cef63e221608955d89dc8d496bcc70c86e
Previously, if generate_hiddenapi_lists.py was invoked without any
custom flag files, e.g. hiddenapi-unsupported.txt, then it would fail.
This change allows it to be used for generating the hiddenapi lists
for modules that do not have any custom flag files,
e.g. framework-sdkextensions
Bug: 179354495
Test: verified that the monolithic out/soong/hiddenapi/... files are
unchanged by this change
Change-Id: I455d453024c9f06ed59cbc1e9838234f8b7c7317
Adds a new --key_field option to merge_csv.py which specifies the name
of the field that should be used to sort the input. If specified it
causes that field to be the first in each row and performs the merge
operation of a merge sort on the input files. That assumes that each
input file is already sorted into the same order.
Modifies the rules that use merge_csv.py to pass in:
--key_field signature
to sort the rows by signature.
Bug: 180387396
Test: Verified that hiddenapi files (both aggregated ones and for the
individual modules) are not affected by this change other than
changing the order.
Change-Id: Idcd5f0fea373b520b604889e1c280f21ed495660
Adds the hiddenapi_additional_annotations to allow a library to list
the libraries that provided additional hiddenapi related annotations
for a library.
Modifies merge_csv.py so it can process multiple zip files at the same
time and uses that to merge the embedded .uau files from a module and
those it depends upon.
Bug: 180102243
Test: m droid
Verified that hiddenapi files (both aggregated ones and for the
individual modules) are not affected by this change.
Change-Id: I796520021c7357398a9e2a09f1029e4a578b05b3
Also, creates a python_binary_host module for generate_hiddenapi_lists
and uses that when constructing the build rule rather than using the
file directly.
Bug: 177317659
Test: m droid
Verified that hiddenapi files (both aggregated ones and for the
individual modules) are not affected by this change.
Change-Id: Ia11bb203ce5a74740d35f1b7e86716e15aad336e
