Commit graph

144 commits

Author SHA1 Message Date
Colin Cross
ffbcd1d8a0 Extract primary apk from apk set zip
Extract and install the primary apk normally, and then unzip the rest
of them as a post install command.

Bug: 204136549
Test: app_set_test.go
Change-Id: I17437ff27f49df6bc91bdbbea6173b46c7d3ec4e
2021-11-17 19:55:33 -08:00
Colin Cross
a1ff7c6926 Enable compose kotlinc plugin when depending on the compose runtime
When a module depends on the compose runtime add a -Xplugin argument
to the kotlinc flags that enables the compose compiler plugin.

Bug: 196351110
Test: TestKotlinCompose
Change-Id: I423a3c4d12df42804a24b672a40a165bc8dd165f
2021-09-20 09:12:42 -07:00
Cole Faust
75fffb14b8 Add a build flag to always enable errorprone per-target
Currently, errorprone is only run if the RUN_ERROR_PRONE
enviornment variable is true. Add a flag that individual
modules can use to always enable errorprone.

In a followup cl, I plan to add another flag that will
force all errorprone checks to be errors, so that modules
can be confident that they're not ignoring any errorprone
checks.

Bug: 190944875
Test: New unit test and manually
Change-Id: Iab0c81642ed22a736add054147829e91a891d179
2021-06-14 15:31:32 -07:00
Colin Cross
a4eafddc41 Support multiple rsp files in REParams
rewrapper supports a comma separate list of rsp files, replace
REParams.RSPFile with REParmas.RSPFiles.

Test: remoteexec_test.go
Change-Id: I7850c071c23d368d6fad4480dd527d146c13c6d3
2021-03-25 11:06:45 -07:00
Colin Cross
77cdcfdeaf Move android package on top of remotexec
Remove the references to the android package in remotexec so that
the android package can reference the remoteexec package.  This
will allow RuleBuilder to integrate directly with remoteexec.

Bug: 182612695
Test: m checkbuild
Change-Id: I15be5ef126d8aacbd605518638f341daf6f31bb3
2021-03-18 16:17:34 -07:00
Sasha Smundak
b0addafd93 Limit the number of the Java source files in a single compilation unit
KYTHE_JAVA_SOURCE_BATCH_SIZE environment variable controls this setting.
The limit is 1000 if this variable is not set.

Fixes: 179932118
Test: run prebuilts/build-tools/build-prebuilts.sh, use it to build kzips, check
Change-Id: I9ad57dfd1d2c2dce5cff755b1bd61cf933420bd3
2021-02-16 15:14:52 -08:00
Colin Cross
d97cf63cdf Turn missing jarjar output files into errors
Jarjar doesn't exit with a nonzero return code when there is a syntax
error in a rules file and doesn't write the output file.  Without a
nonzero return code ninja prints a warning but continues, which leads
to stale results on incremental builds where the output file already
exists, or delayed errors on clean builds whre the output file didn't
exist.

Delete the output file before running jarjar, and then check if it
exists after running jarjar.

Fixes: 119516143
Test: m out/target/common/obj/APPS/TeleService_intermediates/classes.jar
Change-Id: I7e9f7ff34de565a986ab0dc6e375f606c91c3846
2021-02-08 10:14:04 -08:00
Treehugger Robot
5b0b94ecea Merge "Soong: remove output file before running SoongZip" 2020-12-09 08:05:06 +00:00
Ulf Adams
26d195521b Soong: remove output file before running SoongZip
It looks like there is a bug in rewrapper/reproxy that makes it upload the
output file as an input if it still exists in the output tree. This causes
unnecessary cache misses on incremental builds, and can also cause the
action to fail remotely (depending on the service). By removing the output
file at the beginning of the action, we ensure that it can't be uploaded.

Test: ran against a remote execution server; actions no longer fail remotely
Change-Id: I98fcc0cf105553a9920b0e96965a438e052d7c05
2020-11-25 23:03:28 +01:00
Colin Cross
cf371cc1f7 Replace android.WriteFile rule with android.WriteFileRule
The android.WriteFile rule takes careful escaping to produce the
right contents.  Wrap it in an android.WriteFileRule that handles
the escaping.

Test: compare all android.WriteFile outputs
Change-Id: If71a5843af47a37ca61714e1a1ebb32d08536c31
2020-11-14 16:26:00 -08:00
Ramy Medhat
16f23a4038 Check UseRBE is set before replacing any template with the RE version.
Test: presubmit
Change-Id: I6df58b4e700f0d231367af2710672d731d20a736
2020-09-03 01:29:49 -04:00
Jaewoong Jung
11c1e0f94d Output apkcerts file for android_app_set.
Soong and Make have no ways to figure out what splits will be outputted
from a given android_app_set, so it's impossible for them to provide
full PACKAGES.$(LOCAL_MODULE).CERTIFICATE entries, which are required to
build a final apkcerts.txt. This change makes extract_apks produce
apkcerts.txt files for each input modules instead. The Make-side
counterpart of this change merges all local apkcerts.txt into a final
one.

Fixes: 160119159
Test: main_test.go
Test: m apkcerts-list
Merged-In: I321e80fd636a955213761f56a3ac64bfe7f7f7c0
Change-Id: I321e80fd636a955213761f56a3ac64bfe7f7f7c0
2020-07-01 10:03:09 -07:00
Colin Cross
b549b77120 Revert "Always run package check"
This reverts commit 91206d9ed1.

Reason for revert: package-check.sh doesn't work on mac
Bug: 158081251
Bug: 157649935
Change-Id: I50af71d54d5f1f8b516bfcf1efbcf6217e89c83a
2020-06-03 17:15:15 +00:00
Paul Duffin
91206d9ed1 Always run package check
Previously, the package check was only run when building checkbuild or
the phony target created for a specific module. It would not run when
building a module that depended upon a library with the
permitted_packages property. That was because the package check was
only run when the package-check.stamp file was built and that was only
added to the phony and checkbuild targets.

Instead of touching a separate package-check.stamp file to indicate
that the check has been performed this change copies the input jar file
to the package check to a separate ../package-check/<jar> file which is
then treated as the output of the library and is the input for any dex
processing. So, any modules that depend on this library will
transitively depend on the output file produced by the package check
command and so will ensure that the package check is always run.

Test: Removed "android.net" from the permitted_packages for
      "framework-tethering", build "framework-tethering" which
      triggered and failed the package check and
      "com.android.tethering" which did not.
      Made this change.
      Built "com.android.tethering" which triggered and then failed the
      package check.
      Removed change to "framework-tethering"
Bug: 157649935
Change-Id: Ib01aa09e13f80282218049270eb7a58ec5f9f605
2020-06-02 10:29:47 +01:00
Kousik Kumar
366afc5011 Add support for remote-execution / caching of jar/zip actions
Test: Ran a build with `RBE_JAR="true" RBE_JAR_EXEC_STRATEGY="remote"
RBE_ZIP="true" RBE_ZIP_EXEC_STRATEGY="remote" ... use_rbe m` and that
succeeded.

Bug: b/156765207
Change-Id: Ia17b532fbb282be414adf879914870082dd0841b
2020-05-28 11:56:48 -07:00
Kousik Kumar
1372c1b430 Add support for remote-execution / caching of turbine actions
Test: Ran a sample turbine action with:
RBE_TURBINE_EXEC_STRATEGY="remote" RBE_TURBINE="true" ... use_rbe m out/soong/.intermediates/external/protobuf/libprotobuf-java-micro/android_common/turbine/libprotobuf-java-micro.jar
and checked in reproxy_log.txt that remote-metadata had status SUCCESS.

Bug: b/156765475
Change-Id: I85c334e6998c6dd2a68460b3580d50fffd900e6f
2020-05-20 07:57:37 -07:00
Sasha Smundak
a7856c0077 Implement android_app_set module
Bug: 152319766
Test: manual and builtin
Change-Id: Id0877476f9ae23311d92c0b59a9c568140ab4119
2020-05-14 19:01:11 -07:00
Ramy Medhat
1dcc27e47c Refactor RBE support for Javac/R8/D8 to use the remoteexec package.
Test: build with and without RBE_JAVAC/RBE_R8/RBE_D8 set.
Change-Id: I1607f8cfb4d2c3cbb3672152bbeb561d9968bc60
2020-04-28 11:37:20 -04:00
Treehugger Robot
52c0b7b35b Merge "Repeat kapt processor argument for multiple processors" 2020-04-24 15:51:13 +00:00
Colin Cross
5a11686e64 Repeat kapt processor argument for multiple processors
kapt claims to support a comma separated list of annotation
processors, but it errors if multiple annotation processors
are given.  Surrounding the the list with {} does not error,
but it also doesn't even warn if the second element in the
list is garbage, so it may not be running the second
processor.  Repeat the processor argument for each
annotation processor class instead.

Bug: 154736649
Test: TestKapt
Test: m checkbuild
Change-Id: I4c7c161dbf867d7fba1aaf16fd5e502647e3f682
2020-04-22 20:43:37 -07:00
Artur Satayev
762d9f38af Remove Android compat annotations on jarjar transform.
Set a system property to enable Android specific transformer that drops
UnsupportedAppUsage annotation for repackaged classes to avoid adding
unneeded hidden api symbols on jarjar.

Bug: 146418363
Test: m, manual diff hiddenapi-flags.csv
Exempt-From-Owner-Approval: patch approved by owner
Change-Id: I10b49feb15257301574162b5d389cbc30277f672
2020-04-20 11:17:18 +00:00
Colin Cross
1ec3fce9b6 Disable goma for javac actions in Soong
goma can't handle the --system argument that Android must pass to
javac.

Bug: b/143658984
Test: treehugger
Change-Id: I207668d8baa4ab28160938732a38f696aeac3f44
2020-03-05 20:46:15 +00:00
Ramy Medhat
8ea054a81e Add support for experimentally enabling RBE support on specific rules.
This CL adds RBE support to javac, r8, and d8 rules which is only
enabled if respective environment variables are set.

Test: an aosp_crosshatch build with and without the new variables.
Change-Id: Ic82f3627944f6a5ee7b9f3228170c2709b1bfcb8
2020-01-28 12:42:47 -05:00
Sasha Smundak
6c2d4f91f8 Save compilation units in protobuf format.
Bug: 146224091
Test: manual
Change-Id: I7a856bab13f54c78efa0061421c3fcb9341bc6e7
2020-01-15 12:40:03 -08:00
Colin Cross
c2557d1d56 Fix turbine classpath arguments
We were passing "--classpath foo.jar --classpath bar.jar" to turbine,
it now wants (and may always have expected?)
"--classpath foo.jar bar.jar".

Test: m checkbuild
Change-Id: Ib7e0ca64a34e236110f7b785d6a0fb51ed75567e
2019-11-07 02:05:06 +00:00
Ramy Medhat
75c82fb518 Merge "Run non-RBE supported actions in the local pool when USE_RBE is set." 2019-11-06 18:36:18 +00:00
Ramy Medhat
dd0418a4d4 Run non-RBE supported actions in the local pool when USE_RBE is set.
Bug: 143938974
Test: ran CTS build at -j500 successfully.
Change-Id: I55074bd67308cd716972e24fb56a20bc393d5d9d
2019-11-05 22:57:35 +00:00
Sasha Smundak
09950a4f4f Make the changes to emitXrefRule matching those in aosp/1145829
Test: build kzip
Bug: 121267023
Change-Id: Id3a70d113dd7c24071cbb7cbfbf138527991cb77
2019-11-04 16:35:32 -08:00
Colin Cross
6cef481ee7 Use java language 1.9 for sdk_version: "current"
Java language was set for 1.8 for anything building against the
current SDK because the stubs were not built in Soong, so the
system modules could not be built.  The stubs have been built
in Soong since Iabd32b30954b3f4a6d9a779fde52a032b684807e, but
I5e870c34dd0ebc8ae3f888ec627da590c846a76f missed updating
this TODO.

Use 1.9 when building against the stubs, but continue using
1.8 for unbundled builds until we have prebuilt system modules.

Always use the core-current-stubs-system-modules to avoid
splitting android.* packages between the system modules and
the classpath, which would cause new classes in android.*
packages in classpath jars to be ignored.  Add a new
java9Classpath field to sdkDep that will contain the stubs jar
when targeting Java language level 1.9, and plumb it through
to javac and turbine.  Rename the modules field to bootclasspath.

Bug: 142896162
Test: m checkbuild
Change-Id: Icfd32d0a863b2303a997c7cf03cb3708aade4724
2019-10-29 13:35:39 -07:00
Colin Cross
bf3119ee35 Use system modules for turbine
turbine supports taking system modules on the command line,
now that we target Java language level 1.9 and use system modules
by default switch turbine to match javac.

Test: m javac-check
Change-Id: Ieee07502151da0d5693bb8929213d495c039106b
2019-10-29 13:35:39 -07:00
Colin Cross
1e7438524b Make javaVersion an enum
Remove the hardcoded checks against "1.9" by making javaVersion
an enum and implementing javaVersion.usesJavaModules().

Test: TestClasspath
Change-Id: I559eeb1f45880bb8177269c6d977ee4dfbadce57
2019-10-28 14:55:03 -07:00
Sasha Smundak
651436460a Pass filename mappings to C++ and Java extractors.
Android builds by default put artifacts into out/ subdirectory of
the source tree, causing the extractor to record their names as
relative. The indexer considers such files as sources, which is wrong.
Fortunately, the extractor can be fed a set of filename rewriting
rules (see build/tools/vnames.json).
Also, undo previous unsuccessful attempt use to absolute path for the
output directory to distinguish between source code and artifacts.

Bug: 141385476
Test: run the build, inspect compilation units of the kzip file
Change-Id: I89ec3aed8fd14f43ea6e0b226d54f643346f6125
2019-09-27 10:28:11 -07:00
Colin Cross
3b706fde7f Shard kythe invocations when javac is sharded
The kythe pipeline has trouble injesting all 6000 files of
framework.jar at once.  Shared invocations of the kythe extractor
when javac is sharded to produce multiple smaller kzip files.

Bug: 140426870
Test: no change to build.ninja
Test: m out/soong/.intermediates/frameworks/base/framework/android_common/framework0.kzip
Test: TestSharding in java_test.go
Change-Id: I867db4ef5cb1e7f3ce8359a46aac2c00ed8a8912
2019-09-05 16:49:11 -07:00
Sasha Smundak
2a4549ec98 Support source code cross-referencing for C++ and Java
Use Kythe (https://kythe.io) to build cross reference for the Android
source code. ~generate the input for it during the build. This is done
on demand: if XREF_CORPUS environment variable is set, build emits a
Ninja rule to generate Kythe input for each compilation rule. It
also emits two consolidation rules (`xref_cxx` and `xref_java`),
that depend on all Kythe input generation rules for C++ and Java.

The value of the XREF_CORPUS environment variable is recorded in the
generated files and thus passed to Kythe. For the AOSP master branch it is
`android.googlesource.com/platform/superproject`, so the command to build
all input for Kythe on that branch is:
```
XREF_CORPUS=android.googlesource.com/platform/superproject m xref_cxx xref_java
```

Each Kythe input generation rule generates a single file with .kzip
extension. Individual .kzip files have a lot of common information, so
there will be a post-build consolidation step run to combine them.
The consolidated .kzip file is then passed to Kythe backend.

The tools to generate .kzip files are provided by Kythe (it calls them
'extractors'). We are going to build them in toolbuilding branches
(clang-tools and build-tools) and check them in as binaries into master
and other PDK branches:
For C++,  `prebuilts/clang-tools/linux-x86/bin/cxx_extractor`
for Java, `prebuilts/build-tools/common/framework/javac_extractor.jar`

Bug: 121267023
Test: 1) When XREF_CORPUS is set, build generates Ninja rules to create
.kzip files; 2) When XREF_CORPUS is set, building
`xref_cxx`/`xref_java` creates .kzip files; 3) Unless XREF_CORPUS is
set, build generates the same Ninja rules as before

Change-Id: If957b35d7abc82dbfbb3665980e7c34afe7c789e
2019-07-26 09:16:47 -07:00
Colin Cross
b77043e207 Add a systemModules utility type
Storing system modules in a classpath is clumsy, as there should
only ever be one system modules, and it needs to store both a
directory to pass as the argument and a set of generated files
to use as dependencies.  Store them in a separate systemModules
type instead.

Test: m checkbuild
Change-Id: I020556c736bd5091865bcca51dc0fb9e4db6b45b
2019-07-18 14:28:17 -07:00
Colin Cross
33961b54e6 Convert droidstubs to android.RuleBuilder
The droiddoc rules are ripe for converting to android.RuleBuilder,
they conditionally use many input files which is hard to track
with the standard blueprint.Rule style, as the argument and the
dependency have to be handled independently.  Start converting
to android.RuleBuilder by converting the droidstubs module.

Test: m docs
Test: m checkapi
Test: m updateapi
Change-Id: I08713e91149471e88a40115f69824cf5eaf88fb6
2019-07-18 14:28:14 -07:00
Dan Willemsen
ff60a73d89 Depend on all the files from system modules
Instead of just one of the files that we pass into javac.

Test: treehugger
Change-Id: I8478e88656487c9f667893d7c17839f0ea63c78f
2019-06-18 01:29:02 +00:00
Sasha Smundak
26c6d9e9fb Tune JVM, print full diagnostics on crash.
JVM occasionally (0.5% time) crashes on entering a method with SIGSEGV.
It might be due to excessive load on the host machine. Reduce the number
of the compiler threads to 6, and do not start all the GC threads
immediately. If JVM crashes, copy full diagnostics to stderr.

Bug: 132766811
Test: treehugger
Change-Id: Ib7bfd568f6d32456bf623cbb615ffc990c326a34
2019-06-11 13:30:13 -07:00
Jimmy Chen
e88eaa7db7 Revert "Revert "Revert "Print full diagnostics file when Java process crashes."""
This reverts commit b4be092855.

Reason for revert: break build

Change-Id: I11fc5eb5f635a5d99eba962f5817851629e50ab2
2019-06-05 06:47:11 +00:00
Alexander Smundak
b4be092855 Revert "Revert "Print full diagnostics file when Java process crashes.""
This reverts commit d71dc6b5e0.

Reason for revert: ag/974065 fixes the problem that caused the revert

Change-Id: I3ba9eecdab34602aaf19fa518520ac932c9b97af
2019-06-03 16:53:22 +00:00
Alexander Smundak
d71dc6b5e0 Revert "Print full diagnostics file when Java process crashes."
This reverts commit 9d6df2246c.

Reason for revert: This change set the value of JAVA make variable to a string containing "foo bar". When soong_ui is asked to evaluate such string, it drops the quotes, causing bash to treat 'foo' and 'bar' as separate command line arguments.

Change-Id: If4e3c4c950c305bb67a572de3d86119c2da81dc1
2019-05-31 03:15:55 +00:00
Sasha Smundak
9d6df2246c Print full diagnostics file when Java process crashes.
Bug: 132766811
Test: treehugger
Change-Id: I8cca2486f2194a8643007a53e4a695d2a18abbf8
2019-05-29 17:46:22 -07:00
Colin Cross
3047fa23da Build framework.aidl in Soong
Move the rules to build framework.aidl into Soong, and use it
when compiling aidl files with sdk_version: "current".

Also fixes incorrectly using the aidl includes exported by the
"framework" module when the proguardRaiseDep dependency was
added.

Bug: 130798034
Test: sdk_test.go
Change-Id: I126adf1d9e7b6acb528875ff62b974ba7ad9a337
2019-04-18 13:51:58 -07:00
Vladimir Marko
0975ee0de3 Check package restrictions for Java libs.
Test: m checkbuild; inspect verbose log.
Test: Manual - compile with unmet restrictions.
Bug: 122937705
Change-Id: I9360ae8b6d9ce016b7827be5e8ffc6eb521809b7
2019-04-08 14:02:14 +01:00
Colin Cross
cc0ce80ed5 Fix package path of android/soong/android pctx
android/soong/common was renamed to android/soong/android long
ago, but the pctx package path was still "android/soong/common".
This required all users of rules defined in android/soong/android
to import "android/soong/android" and then
pctx.Import("android/soong/common").

Test: m checkbuild
Change-Id: I20d096522760538f7cfc2bec3d4bfeba99b275d4
2019-04-04 03:59:34 +00:00
Colin Cross
19878da6a0 Move proto compilation to RuleBuilder
Using blueprint.Rule for protoc commands was causing code duplication
because there was no good way to run the same protoc for cc, java and
python but then run custom source packaging steps for java and python.
Move most of the code into a common function that returns a
RuleBuilder, and then let java and python add their own commands at
the end of the rule.

Bug: 70706119
Test: All Soong tests
Test: m checkbuild
Change-Id: Ic692136775d273bcc4f4de99620ab4878667c83a
2019-04-02 16:38:47 +00:00
Colin Cross
0b9f31fb08 Replace *Escape with *EscapeList
Follow the change to blueprint to make *Escape take and return a string
and add *EscapeList that take and return slices of strings.  Fix up
a few places that were unnecessarily converting a string to a slice
and back to a string.

Test: m nothing
Change-Id: I3fa87de175522205f36544ef76aa2f04aef1b936
2019-03-04 18:11:53 +00:00
Colin Cross
e4246abd7f Make manifest and APK agree on uncompressed native libs
Only put uncompressed native libs in an APK if the min_sdk_version
supports it (>= 23, Marshmallow), and set
android:extractNativeLibs="false" in the AndroidManifest.xml so
that the platform won't extract them anyways.

Bug: 117618214
Test: m checkbuild
Change-Id: I760017e48bf3c6b618aabde0982df45995765d48
2019-02-08 15:24:47 +00:00
Colin Cross
44c29a8fe1 Clean up srcjar directories after each rule
The srcjar directories just confuse people looking for generated
files, clean them up after each rule.

Test: m checkbuild
Change-Id: If712ce7d4922619fd8420ae8f89830fe7824114d
2019-01-25 00:53:05 +00:00
Colin Cross
7788c1283f Remove annotation_processors property
Now that there are no uses left of annotation_processors remove
the property.

If there are no annotation processor classes for javac for a module,
due to an empty "plugins" property or using kapt for annotation
processors, pass -proc:none to javac to ensure it does not try
to run any annotation processors found in the classpath.

Bug: 77284273
Test: plugin_test.go, kotlin_test.go
Change-Id: I3823d9fec8d3d07d2e49b1d97839f7fcbdd35647
2019-01-24 23:48:04 +00:00