Update the ndk build script to have Make generate the soong.variables
via the config step, using a new "ndk" product.
Bug: 174315599
Test: build/soong/scripts/build-ndk-prebuilts.sh
Change-Id: I0f6fbf523cf9209ea17b7752dd7470012e0baf8c
This makes it possible to pass an extensions_dir containing finalized
module APIs to prebuilt_apis. The extension versions are compared to the
api level versions to figure out what the "latest" finalized API is for
each module. This is done using the base_sdk_extension_version, such
that any extension higher than than base_sdk_extension_version is
assumed to be finalized after any of the existing api level versions.
Bug: 220086085
Test: prebuilt_apis_test.go
Test: existing module in prebuilts/sdk
Change-Id: Ib792f84202d436f594ba5e8716c6a187f9cd60dc
Having soong generate a list of makefiles for mk2rbc
to look through is much faster than having mk2rbc search
itself. Profiling the readLinesFromeFile() function that
reads the list of makefiles shows it takes on the order
of 200 microseconds, much faster than the ~5 seconds
it takes for mk2rbc to search the tree itself.
This CL also allows include statements that are prefixed
with a variable. The concern with this was that there
would be a lot of load statemnts emitted for generic
include statements, causing the generated code to look
ugly, and converting and loading all those files could
cause performance issues. On the performance issues
front, there's already a check that it doesn't result
in over 150 potentially included files. We can lower
that number if necessary, but it's probably good for now.
On the generated code front, while it's true that it's
ugly, it's better to have working but ugly generated
code than refusing to generate anything working at all.
To ensure the soong finder step isn't slowed down due
to having to find a bunch of new makefiles, I profiled
the combination of newSourceFinder and FindSources in main.go:
Baseline incremental:
338.011634ms
340.853335ms
348.541762ms
333.229644ms
349.124824ms
Baseline clean:
1.003836419s
1.006203912s
996.193648ms
1.031005604s
1.03691152s
Modified incremental:
349.029285ms
349.264496ms
351.774948ms
337.63187ms
359.425306ms
Modified clean:
1.028238704s
1.053103506s
1.032757506s
1.016631201s
1.04288108s
So we can see the times are barely affected by this change.
Fixes: 213508006
Test: go test
Change-Id: Iab18bfb127ba3b7e63f2c01f69064805a8398764
If the build file contains the apex_test module, add the
testOnly attribute to the application element of the
corresponding AndroidManifest file and set its value to true.
If the testOnly attribute is already present and has value
false, then do nothing.
Tests added in manifest_fixer_test.py to check if the updated
AndroidManifest file has the testOnly attribute set to true or not.
Bug: 213310150
Test: atest --host manifest_fixer_test
Test: m nothing
Test: manually checked the AndroidManifest file generated
Change-Id: I36247dbe0261c342d451a4422c314fd8fe0c2369
Update the java used by script to fix the xml format when dexdeps processing resource only APKs
Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only
Change-Id: I61a58d7c739e74a571e97166a7c9b4091e6f8da0
Wrap the dexdeps output content with "<externals></externals>" to
simplify backend xml parsing.
Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only
Change-Id: I1457eb9558763e50911dc9c689b9bc3fb833a52c
Since rblf_env / rblf_cli are not typed properly, accept
input variables via a file so that they can be converted
with the correct types.
Bug: 201700692
Test: go test
Change-Id: I9b56067cfe396d1bcd8d62c353ff222dd61a6c9f
This is part of merging upstream changes from OpenJDK 11.28. New classes
have been added to core-oj, so there is a new path added to
package_allowed_list.txt:
* jdk.internal.ref
Bug: 198792576
Test: atest CtsLibcoreOjTestCases
Change-Id: I19a40447bbb060612e0c2f3ef0770568345a7405
Used to avoid rerunning Ninja files regeneration if m2rbc conversion
generated the same makefile.
Test: treehugger
Change-Id: I1b0a619f961e6d2c7bf99a48053ecb58147c6db0
Add scripts to generate java APIs used by Mainline modules. This is
aosp/1879177 with fixed build error that caused by dexdeps analyzing
resource only APKs which doesn't contains any .dex files. Now skip the
error generates in that case in gen_java_usedby_apex.sh
Forrest run:https://android-build.googleplex.com/builds/abtd/run/L38900000951898586
Test: TARGET_BUILD_APPS=com.android.adbd m dist apps_only
Change-Id: Id539cbe1b7306ace69c047f95ffc02265a467511
Host tools are in out/host/linux-x86/bin now instead of
out/soong/host/linux-x86/bin.
Bug: 204136549
Test: forrest
Change-Id: I293accdde5ee4d1d21a471e12ba66e169f9410fc
Mk2rbc takes a .mk file and produces a .rbc file under the
same path, but with a different stem. For example:
build/test/foo.mk becomes out/build/test/foo.rbc
This makes it difficult to see everything that mk2rbc has
generated in the out folder. Move the generated files to
out/rbc instead, so they have a common stem that is separate
from the rest of the build outputs.
Fixes: 202249430
Test: m RBC_PRODUCT_CONFIG=1 nothing
Change-Id: If3edba3feef9c2d3631244d533b997ef0b8b4e8b
This was broken in aosp/1878498; ccross@ submitted aosp/1875758 between
writing that change and submitting it, which changed the paths to mk2rbc
and rbcrun, thereby breaking Starlark product configuration.
Test: Presubmits.
Change-Id: I185489589997e6bdc815a54a8f6f328c18f813d0
It's better not to have state hidden in dotfiles (or rather,
dotdirectories) if we can help.
It's questionable whether the "linux-x86" path segment makes sense since
soong_build only ever runs on one operating system, but I didn't want to
rock the boat now.
Drive-by fixed some quoting fixes in rbc-run. Notably, I didn't wrap
`$@` into double quotes because I don't know whether the lack of double
quotes was intended or not.
Also drive-by fixed the fact that "out/soong" was added twice to the
directory name of bpglob. This turned out not to be a problem because
bpglob doesn't need to be explicitly built: if it's needed, it's
declared as an input of the glob files so it'll be built automatically
(at the cost of the first "null build" not actually being a null build)
Test: Presubmits.
Change-Id: I710d8d16cd8212059a0ca1ee95378505303eed83
Insert ANSI escape codes in the error text. Colors are red, blue, green
and some text is in bold --- these are all bright enough on either dark
or light background.
Also, add a link to the online documentation.
Bug: 132357300
Test: manually mangle one of Android.bp files to get a manifest_check
error and observe that it is colorful.
Change-Id: I2af2aa0415d0eb0eabc88dc5504198e11bfb91b6
This is needed for adding jdk.internal.HotSpotIntrinsicCandidate as a no-op
Bug: 202495224
Test: m
Change-Id: Ib97a7fe955d920aca93630d1d5b04fedff1af960
--skip-make is deprecated because of its odd behavior around the
out/.soong.kati_enabled file. Replace it with --skip-config --soong-only
instead, which will unambigiously skip running kati.
Bug: 204136549
Test: build/soong/tests/run_integration_tests.sh
Test: build/soong/scripts/build-mainline-modules.sh
Test: build/soong/scripts/build-ndk-prebuilts.sh
Change-Id: I629ee7542687aaafd68eea50346b0cf53eafe1ef
For instance, this will match `clang-dev` when using a local build of
llvm-toolchain.
Test: run script
Change-Id: I3f2a0a6adc360f0a299def79535ce139353aece2
Previously, the signature-patterns.csv file (which was output in an sdk
snapshot) included implementation details, e.g. the names of classes
that are not part of any API, including the hidden API. This change
allows a bootclasspath_fragment module owner to customize the generated
signature patterns file to remove as many implementation details as
possible from the sdk snapshot. That means that implementation only
changes would not require that sdk snapshots be updated in sync with
the corresponding APEX.
Module owners can remove implementation in the following ways:
1) Limit the number of packages that are treated as split packages by
explicitly specifying the split packages in the split_packages
property. A split package is one whose classes are provided by
separate bootclasspath_fragment modules and so the signature
patterns has to include every class in that package provided by a
specific bootclasspath_fragment module, including implementation
classes, instead of just listing the package.
The default is to treat all packages as being split, so if no
split_packages is specified then it defaults to ["*"] which matches
all packages. Assuming that no package was split unless specifically
stated would require that all the modules that had split packages
explicitly list them before this change could be submitted as
without them this change would break the build.
Once all existing modules with split packages have been updated to
include them then the default may be changed.
2) Adding package prefixes for any hierarchy of packages that are
owned solely by that bootclasspath_fragment. This removes the need
to list the packages in that hierarchy, including implementation
specific packages.
Bug: 194063708
Test: atest --host verify_overlaps_test signature_patterns_test
m out/soong/hiddenapi/hiddenapi-flags.csv
- manually change files to cause difference in flags to check
that it detects the differences.
Change-Id: If2f90cfc41b6fff9fa4ac8b9d4973110938b9392
Needed for the board configuration.
rbc-run will also now pass any extra arguments it receives
to rbcrun. (The real program, not the script)
Bug: 201700692
Test: rbc_product_config.py aosp_arm64-userdebug
Change-Id: Ic9869f693f1d28bafee59050b6ae7b1fba14dbdd
The behaviour of shell parameter expansion on linux and macOS is
different:
- on linux "\/" in the replaced string is interpreted as simply "/"
- on macOS it is interpreted as the full string "\/"
For example, "android.package.example" would be changed to:
- "android/package/example" on linux
- "android\/package\/example" on macOS
The character / is not a special character for pattern matching in bash,
so it doesn't need to be escaped. Hence using / instead of \/ in the
replaced string works on both linux and macOS.
Change-Id: Id6a5cf32afc53d5ffd989c0ac8aa0b9e0fcbaf82
Fixes: 201947033
As part of the migration to OpenJDK 11, sun.reflect has moved to
jdk.internal.reflect. For now we'll have both packages available
as new code uses the jdk.internal version.
Test: m
Bug: 188889082
Bug: 190791083
Change-Id: Ib03c9c8c346daebd7d97387817b506095f5ec693
Updating OpenJDK 11 means that there is a dependency on
jdk.internal.misc, which is renamed from sun.misc. The new
package is added to the allowed list.
Bug: 190470684
Bug: 190791083
Test: m
Change-Id: I174df7bad5c5ec4286c35998e16c26bcd0737612
https://android-review.googlesource.com/c/platform/build/soong/+/1820496/1/scripts/manifest_check.py#361
caused a regression wrt to statusfile creation. Specifically it inverted
the logic used to create the status file. This file is used in
dexpreopt, thereby affecting all odex files
Test: In internal git, m <an_odex_file>. File size is restored to
pre-regression value
Bug: 199921980
Bug: 200329228
Change-Id: If1e61ac8e2b95455dacc6e7b9431c8f6e1f339b3
Test: Ran the python version of the script and the python3 version
Test: and verified the output is the same. This is for all three
Test: outputs (html, text, xml).
Test: Builds without an error.
Change-Id: I344504a4351c14e00caeb109c5507a60fb6be05a
1. Run pyformat scripts/<script>.py -s 4 --force_quote_type
none -i to fix formatting
2. Annotate #pylint: disable, where straightforward fix is not available
Test: m manifest_check_test
Test: pylint --rcfile tools/repohooks/tools/pylintrc
build/soong/scripts/manifest_check.py
build/soong/scripts/manifest_check_test.py
Bug: 195738175
Change-Id: I9af498c4abd6ac9f8b9df4f93cbdd4424eacff8e
This is part of merging upstream changes from OpenJDK 11.28. This CL
adds jdk.internal.math to package_allowed_list.txt
Bug: 199067710
Test: atest CtsLibcoreOjTestCases
Change-Id: I343bed53ce6b52a6b373a75828b3234073b9cba8
1. Run pyformat scripts/check_boot_jars/check_boot_jars.py -s 4
--force_quote_type none -i to fix formatting.
3. use pylint: disable=<X> where fixes are not obvious
Test: m check_boot_jars
Test: pylint --rcfile tools/repohooks/tools/pylintrc
scripts/check_boot_jars/check_boot_jars.py
Bug: 195738175
Change-Id: I0f86d95e0d174c9f87a2872dedaa65cd3e814246
There is nothing in this script that is not python3, so
make it python3. Without this, a system without python will
fail when trying to run gdbclient.py.
Test: Ran gdbclient.py on a system without a python executable.
Change-Id: I9f8418932628569c445ee01bc6f9ead8aed7989f
1. Run pyformat scripts/hiddenapi -s 4
--force_quote_type none -i to fix formatting.
2. rename restricted variable names (e.g. variable name "input" has been
changed to "csvfile")
3. use pylint: disable=<X> where fixes are not obvious
Test: m merge_csv signature_patterns signature_patterns_test
Test: pylint --rcfile tools/repohooks/tools/pylintrc <file1>
<file1_test>
Bug: 195738175
Change-Id: I800a208f9c0ee1d32e68e4b20fd5933b3ab92c0e
1. Run pyformat scripts/hiddenapi/generate_hiddenapi_lists.py -s 4
--force_quote_type none -i to fix formatting. Some double quotes change
to single quotes since pyformat enforces consistent quotes
2. change map/filter to list comprehension to fix pylint's bad-builtin
warning
3. use pylint: disable=<X> where fixes are not obvious
Test: m generate_hiddenapi_lists_test
Test: pylint --rcfile tools/repohooks/tools/pylintrc <file1>
<file1_test>
Bug: 195738175
Change-Id: I15e71d8d81f0e3fa66d84e6e62eb1848963136e6
1. Run pyformat scripts/conv_linker_config.py -s 4 --force_quote_type none -i to fix formatting
2. Annotate #pylint: disable=import-error for linker_config_pb2 since it
will be provided by soong
Test: m conv_linker_config
Test: pylint --rcfile tools/repohooks/tools/pylintrc
build/soong/scripts/conv_linker_config.py
Bug: 195738175
Change-Id: I791576cf65cb053f68c804f8ec5c2fc22976fdb4
Fetching artifacts from the build server doesn't go well with large
numbers of files in nested directories.
Produce an additional rustdoc.zip artifact to make this easier.
This change also prevents ${DIST} from receiving the unpacked rustdoc
site. This can be changed if b/188822051 is fixed.
Bug: 162741284
Test: m rustdoc; check resulting zip contents manually
Change-Id: I80b6a8fa6e274d2d8c3419d8734251afd4d7dba7
Previously, the sdk snapshot would include all the entries from the
stub-flags.csv and all-flags.csv modular files generated by a single
bootclasspath_fragment. That included implementation details, i.e.
class members that are not part of a stable API or the hidden API which
meant that the sdk snapshots were implementation dependent.
This change removes the implementation details from the modular flag
files, i.e. those entries that are only marked as "blocked". When
comparing the files against the corresponding subset of the monolithic
files it assumes that any entries missing from the modular flag files
are blocked.
Bug: 194063708
Test: atest --host verify_overlaps_test signature_patterns_test
m out/soong/hiddenapi/hiddenapi-flags.csv
- manually change files to cause difference in flags to check
that it detects the differences.
Change-Id: I6b67b2253cf029d6830b58a06ebb0c8fcaa0dd71
Previously, the signature-patterns.csv file included a lot of
implementation details, e.g. the signatures of dex members or inner
classes that are not part of any API, including the hidden API.
This change will remove all member signatures and inner class names
from the file and replace them with just the outermost qualified class
name.
That will still leave some implementation details, e.g. the names of
implementation only classes and packages.
Bug: 194063708
Test: atest --host verify_overlaps_test signature_patterns_test
m out/soong/hiddenapi/hiddenapi-flags.csv
- manually change files to cause difference in flags to check
that it detects the differences.
Change-Id: I9de6a2a6129e875e19f7ded5fae578cbdb584660
Previously, a simple map from dex signature to the flags was used to
store the monolithic hidden API flags. This change replaces that with a
trie that uses packages and class names extracted from the signature
to map from the signature to the flags.
The signature is broken down into pieces consisting of package names,
class names and the remaining member signature. They are used in turn
to navigate through nodes in the trie to a Leaf that contains the flags
associated with the signature.
The flags can be retrieved from the trie in a number of ways:
* Using the complete signature to retrieve the flags for a single
member.
* Using a class name to retrieve the flags for all the members in a
class and any nested classes.
* Using a package wildcard to retrieve the flags for all the members in
the classes in that package.
* Using a recursive package wildcard to retrieve the flags for all the
members in the classes in that package and nested packages.
This will allow a bootclasspath_fragment to select a subset of the
monolithic flags to compare against itself without specifying the
signatures of every member of that set.
Before creating the trie an attempt was made to compute the subset by
iterating over all the signatures in the monolithic flags and matching
against a regular expression created from the patterns but that was too
slow. It took minutes to run whereas using the simple map or the trie
only takes a few seconds.
Bug: 194063708
Test: atest --host verify_overlaps_test
Change-Id: I36f5e319d3e7d62dd34305de1eec990a93cb3a89
Previously, the signatures used to select the subset of the monolithic
flags were simply the signatures read from the modular flags file. This
change moves the creation of the signature list into a separate script
that outputs the signatures to a file and then passes the path through
Soong from the bootclasspath_fragment modules that create it to the
platform_bootclasspath module that uses it to compare the modular
flags against the monolithic flags.
Currently, the signatures are the full signatures but follow up changes
will replace them with patterns (hence the name) that avoids having to
include implementation details in the hidden API flags that are output
as part of a bootclasspath_fragment's snapshot.
This change moves the stub flags related code next to the all flags
related code as they are treated in a similar way.
Bug: 194063708
Test: atest --host verify_overlaps_test signature_patterns_test
m out/soong/hiddenapi/hiddenapi-flags.csv
- manually change files to cause difference in flags to check
that it detects the differences.
Change-Id: I2855bf6d05c91b8a09591664185750361c7e644f
Previously, the compare_signature_flags compared the full set of
monolithic flags against the modular flags and just ignored any
monolithic flags for a signature that was not present in the modular
flags. It essentially performed two tasks, identifying the subset of
the monolithic flags against which the modular flags should be compared
and then doing the actual comparison.
This change separates those two tasks, by first creating a dict that is
the subset of the monolithic flags against which the modular flags will
be compared and then do the comparison.
At the moment the subset is simply the set of monolithic flags
associated with the signatures that are present in the modular flags.
Follow up changes will provide other ways to select the subset.
Bug: 194063708
Test: atest --host verify_overlaps_test
m out/soong/hiddenapi/hiddenapi-flags.csv
- manually change files to cause difference in flags to check
that it detects the differences.
Change-Id: Icb547f071b814dd955f31255b0d35597f7ba4aa1
Switches symbols to use monolithic... and modular... to try and clarify
the purpose of those symbols.
Bug: 194063708
Test: atest --host verify_overlaps_test
Change-Id: Ifc646edebe3d43b51dc3528fa883718399cb5254
Refactor verify_overlaps to make it testable and add tests for the
comparison. It does make one significant change in behavior which is to
read each of the files produced by a bootclasspath_fragment into a dict
before comparison, rather than reading and comparing them a row at a
time. That allows it to reuse the code to read a CSV into a dict.
Bug: 194063708
Test: atest --host verify_overlaps_test
m out/soong/hiddenapi/hiddenapi-flags.csv
- manually change files to cause difference in flags to check
that it detects the differences.
Change-Id: Ib70ac87fe089fc25e3bef18f367d4939bfc0cb8d
Major operations:
* Fix the go_package entries to be consistent, as the new tool
complains when it doesn't have a '/'.
* Regenerate with the new protoc-gen-go tool.
* github.com/golang/protobuf -> google.golang.org/protobuf
* proto.[Un]MarshalText -> prototext.[Un]Marshal
Change-Id: Ie1147bd2457fafb66ba555461b3bf14f0561a25d
atest generate_hiddenapi_lists_test would not work without setting
verbosity=2.
Bug: 194063708
Test: atest --host generate_hiddenapi_lists_test
Change-Id: Ic076ea85d58917921081ae33e72f06011c4ca49e
Normally Soong does that automatically when it handles module names
specified in Android.bp properties. However not all <uses-library>
entries in the manifest correspond to real modules: some of the
optional libraries may be missing at build time. Therefor this script
accepts raw module names as spelled in Android.bp/Amdroid.mk and trims
the optional namespace part manually.
Bug: 193891722
Test: manifest_check_test.py
Test: $ lunch aosp_cf_x86_64_phone-userdebug && m && launch_cvd
$ adb wait-for-device && \
adb root && \
adb logcat | \
grep -E 'ClassLoaderContext [a-z ]+ mismatch' -C1
# empty output, no errors
Change-Id: I184fb8a2dc26f49e90fb597ebcd6a58c00705206
Bug: http://b/187231324
This script parses soong/cc/config/global.go to return clang prebuilts
version. (It may get the version from a json config file in the
future.)
Currently it's used in development/scripts/symbol.py and is the
recommended way to get clang prebuilts version number in the future.
Test: get_clang_version_test.py
Change-Id: Id92a74fd523a5270999bb4d6454525bf2f2f0c04
Bug: 179354495
Test: m out/soong/hiddenapi/hiddenapi-stub-flags.txt
- check that an error is reported if a modular stub-flags.csv file,
i.e. one created by a fragment is not a subset of the monolithic
file, e.g. because a signature in the modular file has different
flags than it does in the monolithic or is not present there.
Change-Id: I46ebb495cb093a5e3abe7571c49933c845318549
Previously, if the --header property was not specified then merge_csv
would use a header constructed by sorting all the fields in the input
files. That required that any use of merge_csv which did not already
have headers in the required order would have to explicitly specify the
headers. That made it harder to use merge_csv as a generic tool as each
invocation needed to be aware of what headers were exported in the
output.
This change causes merge_csv to simply use the headers in the order in
which they are encountered in the input files. That removes the need to
specify the --header option when generating the index files.
Bug: 179354495
Test: m out/soong/hiddenapi/hiddenapi-index.csv out/soong/hiddenapi/hiddenapi-unsupported.csv
- make sure that they are not changed by this change.
Change-Id: I420b7d07aea85af6372cd7580a8be5e2cc82a513
Use the --soong-only flag instead to create it the proper way with the
config make step. This fixes build problems arising from the incomplete
soong.variables settings.
This partially relands https://r.android.com/1535202 so that we'll use
TARGET_PRODUCT=mainline_sdk in module SDK builds everywhere.
build-aml-prebuilts.sh isn't removed completely because we still need
the special case to create build_number.txt.
Test: art/build/build-art-module.sh --skip-apex && \
packages/modules/ArtPrebuilt/update-art-module-prebuilts.py \
--local-dist out/dist
then check that the created prebuilt dex2oatd64 runs with the repro
in b/190194345
Test: build/soong/scripts/build-mainline-modules.sh
then
prebuilts/runtime/mainline/update.py --local-dist .../out/dist
in a master-art tree and check that art/build/build-art-module.sh
works there
Bug: 190194345
Bug: 174315599
Change-Id: I86327b082b61c292f18c0a6b315d9add08002832
This script expects to build uncompressed APEX and use them as
prebuilts.
Bug: 189926000
Test: build_mainline_modules.sh
Change-Id: If0f62c2baa3db3c079b2913fcdc3dafd66394011
This makes sure that where there is overlap between the hidden API
flags generated for a module and the monolithic flags that they are
identical. That ensures that the modular hidden API flags will be
compatible with previous releases that relied on the monolithic flags.
Bug: 179354495
Test: m out/soong/.intermediates/art/build/boot/art-bootclasspath-fragment/android_common_apex10000/modular-hiddenapi/all-flags.csv
m out/soong/hiddenapi/hiddenapi-flags.csv
- Create some inconsistencies between the above two files.
m out/soong/hiddenapi/hiddenapi-flags.csv.valid
Change-Id: Iaf9e23cef63e221608955d89dc8d496bcc70c86e
The classpaths.proto file is needed at build time to generate appropriate
classpath configs for the modules. Unbundled ART branches do not include
packages/modules/SdkExtenstion in their manifest; so put the proto in a
project that should be visible to all unbundled modules.
Bug: 180105615
Test: m nothing
Change-Id: Ie5be1822fbee92f280332622a9317894cc98d7ac
The behaviour is semantically identical, however,
the tool additionally sorts the symbols by address,
compresses frame unwind information more efficiently,
and improves random-accessibility for lazy decompression.
Overall, the changes balance and the output size is same,
however, libunwindstack can access the data much faster
while using less memory (due to the lazy decompression).
It will also enable further improvements in the future.
Bug: 110133331
Test: ART unwinding tests, run prefetto on the device.
Change-Id: Id48f9fe67fb67fcf2b90cc3b217b71bb8f5147ca
