Commit graph

202 commits

Author SHA1 Message Date
Colin Cross
cb9335932f Add Jacoco support
Add support for instrumenting jars with jacoco.  Unlike in Make,
Jacoco in Soong is done entirely using jars.

Instrumentation is enabled by EMMA_INSTRUMENT=true, and affects
all apps.  If EMMA_INSTRUMENT_FRAMEWORK=true then it also affects
any java libraries listed in InstrumentFrameworkModules.

Bug: 69629238
Test: m EMMA_INSTRUMENT=true EMMA_INSTRUMENT_FRAMEWORK=true SKIP_BOOT_JARS_CHECK=true WITH_DEXPREOPT=false
Change-Id: If699715b277529cd7322ffca67c23b0746e1cccd
2017-11-22 16:08:53 -08:00
Jeff Gaston
f5f6e49478 Prohibit accidental renames of product config json fields
Can be caused by `json:"omitempty"` instead of `json:",omitempty"`

Bug: 69076024
Test: m -j # which runs unit tests
Change-Id: I92e3193d00a740c72d36a56748e0b0a8ad1d772e
2017-11-20 16:28:53 -08:00
Colin Cross
f18e11074d Fix using aidl files from filegroups
Compute sources including from filegroup and genrule dependencies
before determining if any sources will cause flags to be added.

Test: gen_test.go
Change-Id: I0434b003bbda07a58bb2ce1a0a72997918c8fae2
2017-11-17 11:22:08 -08:00
Colin Cross
ad59e75a56 Add cc_library tests
Add tests around reusing objects between static and shared
libraries.

Test: library_test.go
Change-Id: I1a1a01c5ea9f9edfbcaa5b29c39c281630e04f70
2017-11-17 11:22:04 -08:00
Colin Cross
b671544973 Move first/last unique elements utility functions to android package
Move firstUniqueElements to android.FirstUniqueStrings,
lastUniqueElements to android.LastUniqueStrings, and lastUniquePaths
to android.LastUniquePaths.

Test: m checkbuild
Change-Id: Ieac840405126c7f8f98afb4a4ef35c01a18fe7fb
2017-10-24 12:12:32 -07:00
Jeff Gaston
11b5c51d4e split soong_zip into a library and a binary
to make it faster/easier to invoke from other Go programs
(such as multiproduct_kati)

Bug: 67478260
Test: m -j
Change-Id: Idd2671a44290550197c88f53dd11a6dd39c85cc5
2017-10-19 18:18:04 +00:00
Colin Cross
93e8595044 Initial kotlin support
Allow java libraries to specify .kt sources, precompile them with
kotlin, and then pass them in the classpath to javac.

Bug: 65219535
Test: java_test.go
Change-Id: Ife22b6ef82ced9ec26a9e5392b2dadacbb16546f
2017-10-17 16:25:22 -07:00
Colin Cross
1369cdb280 Initial support for converting jars to java9 system modules
Adds a java_system_modules module type that (when
EXPERIMENTAL_USE_OPENJDK9 is set to true) converts a list of
java library modules and prebuilt jars into system modules,
and plumbs the system modules through to the javac command
line.

Also exports the location of the system modules to make
variables, as well as the name of the default system module.

Test: TestClasspath in java_test.go, runs automatically as part of the build
Bug: 63986449
Change-Id: I27bd5d2010092422a27b69c91568e49010e02f40
2017-10-16 15:00:02 -07:00
Colin Cross
6af17aa022 Add support for .proto files in java modules
Test: m -j checkbuild
Change-Id: Ia03429948baebff85164a91a34507866c97a08ef
2017-10-03 10:25:15 -07:00
Dan Willemsen
9d6c6722a7 Rename target.linux[_x86[_64]] to target.linux_glibc[_x86[_64]]
In the future, target.linux will apply to all targets running a linux kernel
(android, linux_glibc, linux_bionic). So move all current users to the specific
linux_glibc.

There will be another cleanup pass later that will move some instances back to
target.linux if the properties should be shared with target.android and
target.linux_bionic, but target.linux needs to be removed first.

Test: out/soong/build.ninja identical before/after
Change-Id: I04e4f984781e60feb355de7adf80c54300e0d1f5
2017-10-02 10:44:42 -07:00
Pirama Arumuga Nainar
ada83ec0a6 Add PGO support to soong
Bug: http://b/63768402
Bug: http://b/65598278

Add support for the 'pgo' property to specify how a module is processed
under PGO.  A sample property is below:

pgo: {
    instrumentation: true, // could be "sampling: true" when supported
    profile_file: "pgo_simple.profdata",
    benchmarks: ["pgo_simple"],
}

1. Runtime profiles can be gathered using "sampling" or
"instrumentation".  Sampling is not supported initially.

2. If 'toolchain/pgo-profiles' project is found,
'toolchain/pgo-profiles/${profile_file}' is passed to the compiler and
linker when building this module.

3. If ANDROID_PGO_INSTRUMENT environment variable is set, and includes a
benchmark in the 'benchmarks' list, appropriate flags (for e.g.
-fprofile-generate for instrumentation) are passed to the compiler and
linker when building this module.

Test: Add example modules that specify the pgo property and verify
appropriate flags and dependencies in the Ninja file.  Some
tests/examples are in https://android-review.googlesource.com/474805

Change-Id: I6242e0c904497a115e367dea6927ba1c4b906355
2017-09-21 12:37:19 -07:00
Dan Willemsen
c77a0b3f9c Extract the linker and embed it into host bionic binaries
The linux kernel requires that the ELF interpreter (runtime linker)
that's referenced by PT_INTERP be either an absolute path, or a relative
path from the current working directory. We'd prefer a relative path
from the binary, similarly to how we handle looking up shared libraries,
but that's not supported.

Instead, extract the load sections from the runtime linker ELF binary
and embed them into each host bionic binary, omitting the PT_INTERP
declaration. The kernel will treat it as a static binary, and we'll use
a special entry point (linker_wrapper) to fix up the arguments passed by
the kernel before jumping to the embedded linker. From the linker's
point of view, it looks like the kernel loaded the linker like normal.

Bug: 31559095
Test: Enable host bionic,
      out/soong/host/linux_bionic-x86/nativetest64/libdemangle_test/libdemangle_test
Change-Id: I8d0aea9790b5e86fcc3ea6e2d00cfa33907e2853
2017-09-20 15:34:33 -07:00
Dan Willemsen
3e5bdf29ba Add cc_genrule
cc_genrule is the same as a normal genrule, but can depend on other cc
modules (like cc_object).

Test: mmma external/minijail
Change-Id: I8df87665c7bdc76ce89c92755c054f967a818e57
2017-09-18 23:33:25 -07:00
Colin Cross
38f794ee49 Refactor proto in preparation for java proto support
Test: m -j checkbuild
Change-Id: Idf00ea0bacb2777458f9af2c7eb47e1e1854eeba
2017-09-11 12:41:58 -07:00
Colin Cross
fee57cb33e Move error_prone config to external/error_prone
The error_prone config will likely need to be updated at the same
time as the error_prone prebuilts, so move the config into the same
project.  This requires jumping through some hoops because
external/error_prone may not be present in all manifests.

Bug: 64489631
Test: m -j RUN_ERROR_PRONE=true
Change-Id: I2c59df1148134ffedac0e00d32ac7082a9fb5330
2017-09-05 22:08:13 -07:00
Nan Zhang
a3fc4ba733 Add support for python_defaults modules
Test: python_test.go
Change-Id: I2077414a5b06da5e660a1b48bfdb2eb926fb702f
2017-09-05 17:31:24 -07:00
Colin Cross
c6bbef326f Add error-prone support
Add support for compiling java sources with the error-prone tool.

Test: m -j checkbuild
Change-Id: Ieb4ee0e05f8f34a52ed7bcf1c7cbacf1c9c4d0b5
2017-08-28 11:12:38 -07:00
Jeff Gaston
01547b23d2 Extract jar-sorting to be accessible to soong_zip
Bug: 64536066
Test: m -j # which runs unit tests
Change-Id: I4830bd331c9dab0b1d300a18aefaf25a6af4cfdc
2017-08-24 14:43:35 -07:00
Stephen Crane
ba090d16c2 Add LTO support to soong
Enabling the lto property for a module builds that module and all static
dependencies with LTO.

LTO (link-time optimization) allows the compiler to optimize and
generate code for the entire module at link time, rather than
per-compilation unit. LTO is required for Clang CFI and other
whole-program optimization techniques. LTO also allows cross-compilation
unit optimizations that should result in faster and smaller code, at the
expense of additional compilation time.

Test: make -j12 libc with lto: true for libc

Change-Id: Ib8baefedf60e02701d44673a7c473e0845730101
2017-08-15 12:54:12 -07:00
Jeff Gaston
f1fd45e784 Revert "Revert "Cacheable, multithreaded finder.""
Bug: 62455338
Test: m -j

This reverts commit d1abeb9d98.

Change-Id: I9f73031636157511b5f1c6ce8a205e9bc91669ff
2017-08-10 12:20:30 -07:00
Jeff Gaston
d1abeb9d98 Revert "Cacheable, multithreaded finder."
This reverts commit b6d161bf16.

Reason for revert: New Build Breakage: aosp-master/sdk_mac @ 4260825

Change-Id: I8bda8c50c5e5c9f84621d11a4c15b168833bcd21
2017-08-10 01:16:34 +00:00
Jeff Gaston
b6d161bf16 Cacheable, multithreaded finder.
It can find every Android.bp in internal master in
about 2.5 sec the first time and 0.3 sec subsequent times

Bug: 62455338

Test: m -j blueprint_tools # which runs the unit tests

Test: m -j blueprint_tools && \
  out/soong/host/linux-x86/bin/finder \
    -v
    --db /tmp/mydb \
    --names Android.mk \
    --prune-files .android-out-dir \
    --exclude-dirs .git,.repo \
    . \
    >/tmp/finder-log 2>&1

Change-Id: I5ab2650459a1dae0d5d076faf411ec2d053c743d
2017-08-08 14:19:55 -07:00
Colin Cross
72bb363608 Add basic tests for java modules
Java modules have been unused for a few years, add a few very basic
tests to exercise the modules as they are brought up to date.

Test: java_test.go
Change-Id: Ib23fc065674a61308333130ff1a992525b92e705
2017-07-19 23:17:43 +00:00
Justin Yun
8effde47de Install VNDK libs in /system instead of /vendor
If BOARD_VNDK_VERSION is set, and a module is set to
`vendor_available: true` it is installed in /system and /vendor.

However, if the module is a VNDK library, it must be
installed at `/system/${LIB}/vndk` instead of /vendor/${LIB}.
For those modules, need following to set.

vendor_available: true,
vndk: {
    enabled: true,
    support_system_process: true,
},

`support_system_process` is optional to define.
If it is defined to true, the module is regarded as vndk-sp.

link-type check for VNDK modules is added to make sure that VNDK
modules only link to other VNDK shared libraries or LL-NDKs.

move the ABI checks to VNDK from all of vendor_available.

Bug: 38304436
Test: attempt to compile with BOARD_VNDK_VERSION:=current
Test: Use `vendor_available_vndk: true` for VNDK modules and compile
      with BOARD_VNDK_VERSION:=current
Change-Id: I409268e0b7f05a9d01697bf9f9f4726b5aac631f
2017-07-18 14:48:29 +09:00
Colin Cross
cec8171420 Add integration testing infrastructure
Fix mutator registration for tests to allow different tests
in the same package to register different mutators.

Allow tests to track the resulting ModuleBuildParams objects
to use in assertions, and provide helpers for getting them.
For example:
    config := android.TestConfig(buildDir)
    ctx := android.NewTestContext()
    ctx.RegisterModuleType(...)
    ctx.MockFileSystem(...)
    ctx.ParseBlueprintsFile("Android.bp")
    ctx.PrepareBuildActions(config)
    ctx.Register()
    // Get the Inputs value passed to the javac rule for the foo module
    inputs := ctx.ModuleForTests("foo".Rule("javac").Inputs

Test: java_test.go
Change-Id: I10c82967f5f3586d2c176f169906b571ed82fc73
2017-07-14 14:19:51 -07:00
Jiyong Park
d773eb3e86 add device_kernel_headers module for vendor-specific kernel headers
device_kernel_headers is a built-in heder-only lib that provides device-specific
kernel headers. The header path is configured via a new product variable
SystemIncludeDirs, which is currently the mirror of
TARGET_PROJECT_SYSTEM_INCLUDES in the make world.

Note: generic kernel headers (bionic/libc/kernel) have been added to the
include path by default. "device_kernel_headers" module is for device-specific
kernel headers such as /device/*/*/kernel-headers.

Note 2: this is opt-in for Android.bp modules (i.e. header_libs :
["device_kernel_headers"] required.) while it is always provided to
Android.mk modules.

Bug: 62939405
Test: choosecombo to aosp_sailfish (or any other Pixel/Nexus targets)
BOARD_VNDK_VERSION=current m -j gralloc.msm8996 (or any other vendor
libs using vendor-specific kernel headers)

Change-Id: I81c60abc13942c89fff723d1544b27a81b300db0
2017-07-08 09:29:04 +09:00
Colin Cross
3e3e72da90 Add java config and share it with make
Add a java/config package to hold config information, and share
it with make through makevars.

Test: builds
Change-Id: I46c088bda0fe97a1823bfdd80fa692d0bf61da1b
2017-06-23 11:24:54 -07:00
Jeff Gaston
aff66e55a9 Revert "Revert "Initial implementation of bpfix""
Bug: 38351765
Test: bpfix Android.bp

This reverts commit a8cc9c53fa.

Change-Id: I60f02a8dd920346aa17b9044f834ffe94fa693c6
2017-06-19 15:52:15 -07:00
Jeff Gaston
efc1b412f1 Have Soong try to enforce that genrules declare all their outputs.
This causes Soong to put the outputs of each genrule into a temporary
location and copy the declared outputs back to the output directory.
This gets the process closer to having an actual sandbox.

Bug: 35562758
Test: make

Change-Id: I8048fbf1a3899a86fb99d71b60669b6633b07b3e
2017-06-09 17:57:18 +00:00
Jeff Gaston
a8cc9c53fa Revert "Initial implementation of bpfix"
This reverts commit de4b0463f2.

Reason for revert: postsubmit build broken

Change-Id: If68a4af22e517b7be0876343a81c1319fa4d7046
2017-05-26 00:29:26 +00:00
Jeff Gaston
de4b0463f2 Initial implementation of bpfix
Bug: 38351765
Test: bpfix Android.bp

Change-Id: Ia94602f93bc40db3d006174c39102ac5fd29a64c
2017-05-24 13:06:35 -07:00
Nan Zhang
5323f8e32f Add python_test_host module.
bug: 31676493
Test: created py_test modules in real folder. and ran 'mma'.
Change-Id: I22aa2fad74b11e4a31ea7a4c4a4f0ea64cd3fc94
2017-05-17 11:26:00 -07:00
Colin Cross
2a252bef50 Support .rs and .fs files in cc_* module srcs lists
Translate .rs and .fs files to .cpp files using llvm-rs-cc.

Test: builds
Change-Id: I242cea0d09c9985730a512cec7705c3f1479f4ed
2017-05-05 18:22:30 -07:00
Colin Cross
7e0eaf15b9 Support %s in product variable properties
Support using strings as product variable substitutions, and
add tests for printfIntoProperty.

Test: varaible_test.go
Change-Id: I06cfadfb1d3fc81da72fb71323706df20426c8b7
2017-05-05 18:22:30 -07:00
Nan Zhang
db0b9a3cf3 Supported python build in host side.
The base module handles all the common functionalites, such as version
compatibilty check, version variations split, source file format check,
source/data file duplicate check.

The library/binary module focuses on how to generate binary build actions,
such as setting up stub script, zipping, filling in __init__.py in
runfiles dir tree.

Bug: b/31676493
Test: go test under python package

Change-Id: I06608369f350f7195873d459e1c8d1bdb811e77e
2017-05-05 13:27:56 -07:00
Jayant Chowdhary
3e231fd8bd Add header-abi-checker for Vndk abi checks.
header-abi-dumper: dumps abi exported by source files for Vndk.
header-abi-linker: links abi dumps produced by header-abi-dumper.
header-abi-diff: compares linked dumps.

Test: mm -j64 showcommands > make_log in bionic/libc.
      This produced linked dumps in out/soong/.intermediates.
      Copied these dumps to
      prebuilts/abi-dumps/ndk/current/arm64/source-based/.
      Changed the abi and re-ran mm -j64 showcommands > make_log
      confirmed that the build reported compatibility breakge without
      actually failing (advisory mode).

Change-Id: Iccad6908fe68a80f47230751671d156893b96ead
2017-04-14 19:48:10 -07:00
Dan Willemsen
4416e5db69 Split /system and /vendor modules, allow multi-installation
Nothing changes if BOARD_VNDK_VERSION isn't set.

When the VNDK is enabled (BOARD_VNDK_VERSION in Make), this will split
/system and /vendor modules into two different variant spaces that can't
link to each other. There are a few interfaces between the two variant
spaces:

The `llndk_library` stubs will be available in the /vendor variant, but
won't be installed, so at runtime the /system variant will be used.

Setting `vendor_available: true` will split a module into both variants.
The /system (or "core") variant will compile just like today. The
/vendor ("vendor") variant will compile against everything else in the
vendor space (so LL-NDK instead of libc/liblog/etc). There will be two
copies of these libraries installed onto the final device.

Since the available runtime interfaces for vendor modules may be
reduced, and your dependencies may not expose their private interfaces,
we allow the vendor variants to reduce their compilation set, and export
a different set of headers:

  cc_library {
      name: "libfoo",
      srcs: ["common.cpp", "private_impl.cpp"],
      export_include_dirs: ["include"],
      target: {
          vendor: {
	      export_include_dirs: ["include_vndk"],
	      exclude_srcs: ["private_impl.cpp"],
	      srcs: ["vendor_only.cpp"],
	  },
      },
  }

So the "core" variant would compile with both "common.cpp" and
"private_impl.cpp", and export "include".

The "vendor" variant would compile "common.cpp" and "vendor_only.cpp",
and export "include_vndk".

Bug: 36426473
Bug: 36079834
Test: out/soong/build.ninja, out/soong/Android- only changes due to _core addition and
      .llndk -> .vendor
Test: attempt to compile with BOARD_VNDK_VERSION:=current
Change-Id: Idef28764043bf6c33dc0d2e7e2026c38867ff769
2017-04-11 12:27:41 -07:00
Dan Willemsen
b916b80bf3 Add LLNDK support for the VNDK
Instead of using the NDK headers and libraries, add LL-NDK specific
headers and library stubs for VNDK users. This allows us to provide an
expanded liblog interface.

Test: aosp_arm; m -j
Test: Enable BOARD_VNDK_VERSION on aosp_arm; m -j
Test: Inspect out/soong/build.ninja before/after (w/o vndk)
Change-Id: Ic85f07fa10c695b5baab10c41f5e0ad38700bf3d
2017-04-07 10:18:56 -07:00
Dan Albert
30c9d6e771 Generate file to map codenames to API levels.
The NDK stub library generator needs to know what unreleased API
levels exist and what the ordering is between them. This singleton
will later be expanded to generate api-level.h to cut down on
duplication.

Test: make out/soong/api_levels.json && cat out/soong/api_levels.json
Bug: None
Change-Id: I53126dd2cacb67c331c44f7d7c77c98b176b93cd
2017-03-31 12:27:42 -07:00
Dan Willemsen
615609854f Revert "install *.so in different paths for their types"
This reverts commit 4c48f724e1. It's
causing test failures, warnings and complaints, so backing it out and
we'll resolve those before putting it back in.

Bug: 33681361
Test: m -j
2017-03-16 19:48:32 -07:00
Dan Willemsen
01a405a481 Add LinuxBionic toolchain and switch
This adds a toolchain definition for LinuxBionic that only supports
Clang/64-bit. It pulls pieces from the x86_linux_host and x86_64_device
configs, and uses the android clang triple, with some manual overrides.

To enable building this, set your soong.config file to:

 {"Host_bionic": true}

Bug: 31559095
Test: out/soong/{Android,make-vars}-aosp_arm64.mk the same with or
      without host bionic turned on
Test: No change to out/soong/build.ninja before/after this change
Change-Id: Id97dda8bd9aa670c32aed31fbe6aaa8175e70b59
2017-03-15 14:18:42 -07:00
Jiyong Park
4c48f724e1 install *.so in different paths for their types
Shared libraries are now installed to different directories depending on
their types.

* NDK libraries: /system/lib/ndk
* VNDK libraries: /system/lib/vndk
* VNDK-ext libraries: /system/lib/vndk-ext
* Framework-only libraries: /system/lib
* Vendor-only libraries: /vendor/lib
* Same-process HALs: /vendor/lib/sameprocess

In addition, a new module type vndk_ext_library is added. It is almost
identical to cc_shared_library but it introduces another attribute
'extends'. This is use to reference the vndk library that this vndk-ext
library is extending.

For example, in order to extend a vndk library libFoo:

cc_library {
  name: "libFoo",
  srcs: [...]
}
---------------------
vndk_ext_library {
  name: "libFoo-extended",
  srcs: [...]
  extends: "libFoo"
}

Then, libFoo will be installed as /system/lib/vndk/libFoo.so and
libFoo-extended will be installed as /system/lib/vndk-ext/libFoo.so.
Note that file name of the latter is libFoo.so, not libFoo-extended.so:
file name of an extending module is automatically set to that of the
extended module.

Bug: 33681361
Test: build & run. Libraries must be in the correct directories.
Change-Id: Ia1eb3940605d582a252c78da0f3a5b36fdab062b
2017-03-14 09:13:44 +09:00
Josh Gao
7bd4c5c339 Add libwinpthread, link it into win32 binaries.
Bug: http://b/31665213
Test: wine adb.exe
Test: wine fastboot.exe
Change-Id: I6d6ff69f0c016e2654119a09161685841cbccc7e
2017-02-23 17:57:03 -08:00
Dan Willemsen
581341d4f2 Native Coverage support in Soong (gcov)
This is configured the same as make -- a global NATIVE_COVERAGE=true
flag to allow native coverage, then COVERAGE_PATHS=path1,path2,... to
turn it on for certain paths.

There are .gcnodir files exported to Make and saved in $OUT/coverage/...
files which are `ar` archives containing all of the compiler-produced
.gcno files for a particular executable / shared library.

Unlike the Make implementation, this only passes links the helper
library (automatically through --coverage) when one of the object files
or static libraries being used actually has coverage enabled.

Host support is currently disabled, since we set -nodefaultlibs, which
prevents libclang_rt.profile-*.a from being picked up automatically.

Bug: 32749731
Test: NATIVE_COVERAGE=true COVERAGE_PATHS=system/core/libcutils m -j libbacktrace libutils tombstoned
      $OUT/coverage/system/lib*/libcutils.gcnodir looks correct (self)
      $OUT/coverage/system/lib*/libbacktrace.gcnodir looks correct (static)
      $OUT/coverage/system/lib*/libutils.gcnodir doesn't exist (shared)
      $OUT/coverage/system/bin/tombstoned.gcnodir looks correct (executable)
Test: NATIVE_COVERAGE=true COVERAGE_PATHS=external/libcxxabi m -j libc++
      Confirm that $OUT/coverage/system/lib*/libc++.gcnodir looks correct (whole_static_libs)
Change-Id: I48aaa0ba8d76e50e9c2d1151421c0c6dc8ed79a9
2017-02-14 13:05:48 -08:00
Fabien Sanglard
d61f1f4559 Add support for CMakefile generation
Test: Manually generated CMakeLists.txt for gui/ui/aapt2.

Change-Id: I7dedc300c1e50b8e39bc58091b650c0bbe2c62da
2017-02-07 09:43:02 -08:00
Nan Zhang
6d34b308ff Added phony module type for soong.
The "required" field within "phony" module in .bp file has to be
non-empty. And "phony" module doesn't contain any "src" files.
Bug: b/32981153
Test: make -j out/soong/build.ninja

Change-Id: Iaa2009467c44702572b97ca9cbd02c1d368c8d83
2017-02-06 22:23:19 -08:00
Dan Willemsen
1e70446251 Add a Go replacement for our top-level Make wrapper
Right now this mostly just copies what Make is doing in
build/core/ninja.mk and build/core/soong.mk. The only major feature it
adds is a rotating log file with some verbose logging.

There is one major functional difference -- you cannot override random
Make variables during the Make phase anymore. The environment variable
is set, and if Make uses ?= or the equivalent, it can still use those
variables. We already made this change for Kati, which also loads all of
the same code and actually does the build, so it has been half-removed
for a while.

The only "UI" this implements is what I'll call "Make Emulation" mode --
it's expected that current command lines will continue working, and
we'll explore alternate user interfaces later.

We're still using Make as a wrapper, but all it does is call into this
single Go program, it won't even load the product configuration. Once
this is default, we can start moving individual users over to using this
directly (still in Make emulation mode), skipping the Make wrapper.

Ideas for the future:
* Generating trace files showing time spent in Make/Kati/Soong/Ninja
  (also importing ninja traces into the same stream). I had this working
  in a previous version of this patch, but removed it to keep the size
  down and focus on the current features.
* More intelligent SIGALRM handling, once we fully remove the Make
  wrapper (which hides the SIGALRM)
* Reading the experimental binary output stream from Ninja, so that we
  can always save the verbose log even if we're not printing it out to
  the console

Test: USE_SOONG_UI=true m -j blueprint_tools
Change-Id: I884327b9a8ae24499eb6c56f6e1ad26df1cfa4e4
2017-02-06 14:05:07 -08:00
Colin Cross
faeb7aa135 Support data properties for test binaries
Allow tests to specify a data property that lists files or filegroup
modules that will be packaged alongside the test.  Also add a path
property to filegroup modules to allow shifting the path of the
packaged files, and add ExpandSourcesSubDir to expand the filegroup
sources while including a shifted relative path in the Paths objects.

Test: soong tests, manually adding data to a module
Change-Id: I52a48942660e12755d313ef13279313361b4fc35
2017-02-02 16:23:30 -08:00
Colin Cross
068e0feace Support filegroups
filegroup is a module that contains a list of files, and can be used
to export files across package boundaries.  filegroups (and genrules)
can be referenced from srcs properties of other modules using the
syntax ":module".

Test: m -j
Change-Id: I3d6fc4819c0b4225b474e0ad42f0d947f55a5961
2016-12-17 16:34:03 +00:00
Dan Willemsen
2e47b34435 Add Toolchain.Bionic()
Move some cc ctx.Host() / ctx.Device() checks over to using
ctx.toolchain().Bionic(). There will be more changes, these are just the
obvious ones dealing with host_ldlibs / crt / system libraries.

Bug: 31559095
Test: out/soong/build.ninja is identical
Change-Id: Ibba60483b4ab0e01f6996eb7d883120e4acc1830
2016-11-17 01:50:43 -08:00
Dan Willemsen
00fcbde401 Add LinuxBionic OsType
Actual support will come in later changes.

Bug: 31559095
Test: out/soong/build.ninja is identical
Change-Id: I31392fb78a51f43567d16aef89e32cd5137cdbc1
2016-11-17 01:49:50 -08:00
Colin Cross
6bde0948d2 Add android.Expand
Add android.Expand to expand $() variables in properties.

Test: expand_test
Bug: 31948427
Change-Id: Id30856a1d21d02e8997fcf2358e4a5feeede05be
2016-11-04 15:59:23 -07:00
Colin Cross
7f19f37443 Move globbing to Blueprint
Move Soong's globbing-with-dependencies support into Blueprint so it can
be used for subdirs= lines in Android.bp files.

Blueprint has a slight change in behavior around subname= lines, it now
always uses the subname and doesn't fall back to Blueprints.  To support
the Blueprints files in build/blueprint, use them directly with build=.

Test: build, add source file that matches glob, rebuild
Change-Id: Ifd0b0d3bc061aae0a16d6c7ca9a1cd8672656b4d
2016-11-04 04:54:16 +00:00
Dan Willemsen
a03cf6d322 Add clang-tidy support
For every file which we can run clang-tidy (C/C++ clang-built), we add a
new build node that depends on the object file (since clang-tidy does
not export a depfile), and is depended on by the link step. This is
better than how we're doing it in make, since calling tidy can be turned
on or off without needing to rebuild the object files.

This does not attempt to port WITH_TIDY_ONLY from Make, since the way
that it works is broken (due to the lack of a depfile).

Bug: 32244182
Test: WITH_TIDY=true mmma -j bionic/libc
Test: ./soong (Setting ClangTidy: true)
Change-Id: I40bbb5bb00d292d72bf1c293b93080b5f9f6d8ea
2016-10-31 16:21:01 -07:00
Colin Cross
0c461f1f6e Add support for .proto files
.proto files are translated to .pb.cc and .pb.h files, which are then
compiled normally.

Bug: 32286026
Test: mmma -j system/extras/perfprofd
Change-Id: I538071424d667aacf35b4b8bfebe217f5f092726
2016-10-27 15:28:09 -07:00
Colin Cross
0d614dd222 Re-enable prebuilt_test
Use a temporary directory as the build directory during tests so files
don't get written to the source tree.

Also add a few more tests for prebuilts with no file specified.

Bug: 31800129
Test: m -j, make sure .soong.environment is not written to the source tree
Change-Id: I623bc114b2ff534c8df9fb3ce273e804711f8f05
2016-10-14 17:07:49 -07:00
Colin Cross
b778a9cb4e Disable prebuilt_test
The prebuilt tests are accidentally writing a .soong.environment file to
the build/soong/android directory, disable them until TestConfig() is
fixed to prevent writing to the source tree.

Bug: 31800129
Test: m -j, make sure .soong.environment is not written to the source tree
Change-Id: I3b7d47e42111ecababe275b7a9ff0d08bcebfc7a
2016-10-13 18:09:51 -07:00
Colin Cross
ce75d2c6a2 Initial prebuilt support
Support prebuilt shared libraries as an initial proof-of-concept of
prebuilts.  Future changes will support binaries and static libraries,
and the ability to select which to use based on something besides
blueprint properties.

Test: TestPrebuilts run during m -j
Change-Id: I6e84da667e9005ae11844bad01d25cbe4ced1ce3
2016-10-13 14:52:47 -07:00
Colin Cross
798bfce9d0 Move registration into android package
Mutator registration is tightly coupled with the android package, move
all registration from the soong package to the android package.

Test: build.ninja identical
Change-Id: Ie183d0b52cc7431c9e05b231934d189208ef1efe
2016-10-12 14:30:44 -07:00
Dan Willemsen
394e9dc876 Add Relocation Packer support
This only applies to shared libraries on the device, and like stripping,
we'll let make do the actual packing if we're embedded in Make.

Change-Id: I1585d74ecfc41e18dcbb5ffb70005adf007cc941
2016-09-15 03:39:05 +00:00
Colin Cross
178a509855 Replace PropertyCustomizer with hooks
Replace PropertyCustomizer with a more extensible hooks mechanism.
Instead of passing an object that satisifies an interface, pass a
function pointer that takes a context interface as an argument.  Callers
can use lambdas to capture any other necessary parameters like property
structs.

Also add two new hooks, arch hooks that occur after splitting modules
into arch variants, and install hooks that occur each time a file is
installed to the output directory.

Change-Id: I3a3e34aa97f1a92d3a31e5004b4b1ba68869d242
2016-09-13 17:37:32 -07:00
Dan Willemsen
377737a01c Move Android.bp definitions into subdirs
Test: m blueprint_tools still produces the same tools
Change-Id: Ia1e2e43c62cb6035616bef9fbef56417b46cf3a4
2016-08-25 15:50:21 -07:00
Colin Cross
a120ec1c5e Add PropertyCustomizer
Allow any module factory to insert a PropertyCustomizer on the module,
which will be called before any other mutators.  The PropertyCustomizer
can append or prepend to any properties, allowing module types to extend
other module types by modifying the public, stable interface provided by
the properties.

Change-Id: Idff02be80d939a70df1c6bbccffdd1f04ff975d2
2016-08-22 17:08:06 -07:00
Colin Cross
9272ade7a8 Add DeviceConfig and OncePer objects
Add DeviceConfig to store per-device configuration information.  Put a
OncePer object inside Config and DeviceConfig, which computes a value
once per key per object to allow build logic to store arbitrary
per-build or per-device computed values.

Change-Id: I1a38b426f29d223ef5e803e0d4d9604500de2fd2
2016-08-17 16:39:06 -07:00
Dan Willemsen
47ec28f3b1 Rename soong_jar to soong_zip
This is a general purpose tool that happens to contain some jar specific
features.

Change-Id: I05f4654d4517c245ad7a3c15492e0d2368bbf64f
2016-08-11 01:30:16 -07:00
Dan Willemsen
017d8937fa soong_jar: Parallel compression
This compresses multiple files in parallel, and will split up larger
files (5MB+) into smaller chunks (1MB) to compress in parallel.

There is a small size overhead to recombine the chunks, but it's only a
few bytes per chunk, so for a 1MB chunk, it's minimal.

Rough numbers, with everything in the page cache, this can compress
~4GB (1000 files) down to 1GB in 6.5 seconds, instead of 120 seconds with
the non-parallel soong_jar and 150 seconds with zip.

Go's DEFLATE algorithm is still a bit worse than zip's -- about 3.5%
larger file sizes, but for most of our "dist" targets that is fine.

Change-Id: Ie4886c7d0f954ace46e599156e35fea7e74d6dd7
2016-08-11 01:30:16 -07:00
Dan Willemsen
3bf1a08505 Add zip2zip tool to copy zip entries from one file to another
This doesn't do any decompression / recompression, but just copies over
the already compressed contents. So it's similar to zip -U, but allows
rewriting of the paths.

The first expected usecase is to replace img_from_target_files during
the build, since it does the equivalent of this:

zip2zip -i <target-files.zip> -o <img.zip> OTA/android-info.txt:android-info.txt IMAGES/*:.

Except it decompresses and recompresses the images, which takes over a
minute instead of a few seconds.

Change-Id: I88d0df188635088783223873f78e193272dbdf1c
2016-08-10 16:57:03 -07:00
Colin Cross
b916a38233 Refactor cc modules to use decorators instead of inheritance
For example , instead of trying to have libraryLinker inherit from
baseLinker and libraryCompiler inherit from baseCompiler, create a
single decorator object that wraps both baseLinker and baseCompiler.

Test: Builds, no unexpected changes to build.ninja
Change-Id: I2468adaea8466c203a240259ba5694b8b1df7a52
2016-08-05 10:25:09 -07:00
Colin Cross
b98c8b0595 Move toolchain and global variables into separate package
Move all of the configuration into a cc/config package

Change-Id: If56fc7242062ed1ce3cb297f78a1e0ef7537373c
2016-08-01 13:37:01 -07:00
Colin Cross
4d9c2d17c3 Separate cc.go
Split cc.go into files per module type, plus files for the compiler,
linker, and installer stages.

Change-Id: Id44c03f42fcd180950ccd008d4de0c144ea3597b
2016-07-29 15:17:58 -07:00
Dan Albert
914449fca8 Generate NDK sysroots from the platform build.
The list of migrated libraries is currently empty. Libraries will be
migrated as follow up patches.

Test: Migrated libc to this system and everything still builds.
      build.ninja shows libraries being built and used and headers are
      collected for the sysroot.
Bug: http://b/27533932
Change-Id: Iaba00543c1390f432befe0eed768ed3fbb8a9b96
2016-07-28 00:42:05 -07:00
Colin Cross
a43f9b3cbf Rename androidmk/test.go so go test can find it
Change-Id: If18c749fe9bc883552952284a469ea8e8f3e8f6d
2016-06-08 14:49:55 -07:00
Dan Willemsen
6c2ac0673d Add tests for androidmk
Change-Id: Ic31bf68f75c8dd4d7737d9ea01f2d93637cdaca2
2016-06-03 00:00:50 -07:00
Dan Willemsen
c2aa4a9c52 Start using blueprint_go_binary
And install the tools into a more obvious location. soong_env is not
moved, since we need it to exist early, so that we can use it in
soong.bash in case there's a build failure.

Change-Id: I9bd1fa320d84d180b2cf3deb90782d380666f7a6
2016-05-31 18:31:56 -07:00
Colin Cross
08693d2bf2 Improve makefile parser
Improve the androidmk makefile parser based on ideas from go/ast and
friends:
   - Use type switching instead of the As* mess
   - Don't store endPos for every node, compute it based on the last
     known position in the node plus the length of the last token
   - Store positions as only the offset into the file, and then unpack
     them into Line/Column scanner.Position objects later

Change-Id: I87eb6661859951e6c2ea5a85db6229fa5561d615
2016-05-26 03:34:30 +00:00
Dan Willemsen
20acc5c520 Add flag property checking
Some checks for common errors with user-provided compiler and linker
flags:

* Using -I instead of include_dirs
* Using -l<lib> in ldflags instead of host_ldlibs (or shared_libs)
* Using -L in ldflags
* Splitting a multi-word flag into two flags
* Combining two flags into one list entry
* Using a path that could search outside the source or output trees
* Using a non-whitelisted library in host_ldlibs

Maybe some of the flag checks should happen during a static analysis
pass, but we don't have one right now, and this only adds ~1/2 second to
our 73 second Mega_device runs (recompile the changed code, run
soong_build, then report unknown target).

Change-Id: Icea7436260f1caa62c0cec29817a1cfea27b3e7c
2016-05-25 17:50:05 -07:00
Colin Cross
635c3b0157 Rename common to android
Rename the "common" package to "android", because common is too
generic.  Also removes all android.Android naming stutter.

Ran:
gomvpkg -from 'android/soong/common' -to 'android/soong/android'
gorename -from '"android/soong/android".AndroidModuleContext' -to 'ModuleContext'
gorename -from '"android/soong/android".AndroidBaseContext' -to 'BaseContext'
gorename -from '"android/soong/android".AndroidModuleBase' -to 'ModuleBase'
gorename -from '"android/soong/android".AndroidBottomUpMutatorContext' -to 'BottomUpMutatorContext'
gorename -from '"android/soong/android".AndroidTopDownMutatorContext' -to 'TopDownMutatorContext'
gorename -from '"android/soong/android".AndroidModule' -to 'Module'

Change-Id: I3b23590b8ce7c8a1ea1139411d84a53163288da7
2016-05-18 15:37:25 -07:00
Dan Willemsen
e8b1bc0a68 Lock down environment variable access after all other singletons
To allow other singletons to use new environment variables, move env.go
to the end of the source list, so that its singleton is run last.

Change-Id: Ic92308f72fc0c92a0fedbc5546e337940e32eaab
2016-05-12 19:03:10 -07:00
Dan Willemsen
4b7d5deb84 Publish Soong configuration to Make variables
For configuration like TARGET_GLOBAL_CFLAGS, it would be good to have
Make and Soong use the same values. This change adds an interface for
packages like cc to verify the current value of make variables match the
Soong equivalents, and once they're satisfied with the results, remove
the make definition and use the value from Soong.

The cc implementation exports a few variables currently that I expect to
match between both implementations in all cases. It also checks
{CLANG,}{HOST,TARGET}_GLOBAL_{C,LD}FLAGS, which should be mostly similar
between make and soong now.

Bug: 23566674
Change-Id: Idc8582ef31ace11a8baefcf525c3683f08a573aa
2016-05-12 13:36:55 -07:00
Colin Cross
16b2349190 Add support for sanitizer property
Add a new feature to cc modules that adds the cflags and libraries
necessary to implement AddressSanitizer, ThreadSanitizer, and
UndefinedBehaviorSanitizer.

Change-Id: Ibe5ffadc7ece56080a2521f2c7c00da9ef712584
2016-04-21 16:42:08 -07:00
Colin Cross
ca860ac720 Refactor cc
Refactor all of cc in order to use composition instead of inheritance.
All cc module types exported by cc are now *cc.Module objects, with
compilation, linking, and installing steps delegated to different
objects in order to form the full module type.  Additional features that
modify dependencies and flags can be inserted in a features object list,
and custom module types can be created by adding a Customizer object
that can modify properties.

Change-Id: Ie1283d14920f7856f6947b0530606b2f4d58fab0
2016-03-21 17:31:04 -07:00
Dan Willemsen
34cc69e4bf Use Path instead of string for file paths
This centralizes verification and common operations, like converting the
path to a source file to the path for a built object.

It also embeds the configuration knowledge into the path, so that we can
remove "${SrcDir}/path" from the ninja file. When SrcDir is '.', that
leads to paths like './path' instead of just 'path' like make is doing,
causing differences in compiled binaries.

Change-Id: Ib4e8910a6e867ce1b7b420d927c04f1142a7589e
2015-12-09 14:29:12 -08:00
Dan Willemsen
218f65628e Hybrid soong/mk build using prebuilts
Change-Id: I96daa69af27bd19c8bfbe327a1318f51c738fc03
2015-12-01 15:08:01 -08:00
Dan Willemsen
490fd49557 Support cross-compiling Windows binaries on Linux
This defines another mutator between HostOrDevice and Arch that will
expand host modules into a module for each host type
(Darwin/Linux/Windows) that is currently being built.

Change-Id: I4c8ac6b616c229f6bd45ad8a35902652fb6a4fff
2015-11-30 15:28:31 -08:00
Colin Cross
3b336c2056 Add mips64
Change-Id: I2b4adbc1a1568b66515243005beda2c56a4a0519
2015-11-24 21:34:54 +00:00
Colin Cross
023f1e8e8f Add mips
Change-Id: Icb05292877a60939542ce09d9774e4b9d1353502
2015-11-24 21:34:46 +00:00
Colin Cross
6371b387b7 Add x86_64
Change-Id: Ib0f31d147546ae187697867c6a0b937266bb9b39
2015-11-24 21:34:36 +00:00
Colin Cross
b0cba6a00f Add x86
Change-Id: I28e78cd49b184e0aa50c1c562b6bf719300e0832
2015-11-24 21:34:26 +00:00
Colin Cross
b9ec7b43ca Delete androidbp
androidbp is gone, replaced with combining kati ninja output for
makefiles with blueprint ninja output for soong.

Change-Id: I00b9e28877abf2ceb2223d3ccf0637bc1b7571bb
2015-11-19 15:42:32 -08:00
Colin Cross
cfad119eaf Add support for defaults modules
Add a cc_defaults module, which will prepend any matching properties to
modules that reference it with a "defaults" property.

Change-Id: I5908dd98d204f71b29ad95a4ab85403aa1621ca2
2015-11-03 15:46:08 -08:00
Colin Cross
6362e27848 Remove EarlyMutators and DynamicDependencies
EarlyMutators are identical to BottomUpMutators, except they run before
DynamicDependencies.  DynamicDependencies can be replaced with a
BottomUpMutator.  Replace both EarlyMutators and DynamicDependencies
with BottomUpMutators, which allows setting the order between all
mutators through registration order.

Change-Id: Id1305d798d3d2da592061c89d7c10a71780b71a3
2015-11-03 15:46:08 -08:00
Colin Cross
06a931bdb6 Replace extendProperties with pathtools.AppendProperties
Blueprint has a generic AppendProperties/AppendMatchingProperties now,
use it, and replace all bool properties that might be modified by a
mutator with *bool, which provides the correct replace-if-set semantics
for append.

Also remove uses of ContainsProperty except when explicitly checking if
a property was set in a blueprints file.

Change-Id: If523af61d6b4630e79504d7fc2840f36e98571cc
2015-11-03 15:46:08 -08:00
Dan Willemsen
98b26c576a Use the blueprint plugin infrastructure
Instead of using this hardcoded list, use the blueprint plugin
infrastructure to find them at build time and load them.

Bug: 23567252
Change-Id: Iae87e8c9bf020f4beb42c418b9a6bf8b0fe8b601
2015-09-15 00:31:30 +00:00
Colin Cross
7f64b6de31 Support product variables
Allow modules to vary their properties based on product variables.
For now, DEVICE_USES_LOGD, DEVICE_USES_JEMALLOC, and DEVICE_USES_DLMALLOC,
and BOARD_MALLOC_ALIGNMENT are supported.

Product variables can provide a value (only bool and int supported for
now), and if any of the product variable properties contains a "%d"
then Sprintf will be called with the property value as the format
and the product variable value convert to an int as the only argument.

For example:

    product_variables: {
        dlmalloc_alignment: {
            cflags: ["-DMALLOC_ALIGNMENT=%d"],
        },
    },

will cause -DMALLOC_ALIGNMENT=16 to be added to any top level
properties called "cflags".

Change-Id: I74882a6ab4914d3e222f8d06cfac371b7b829ae5
2015-08-24 16:20:08 -07:00
Colin Cross
463a90e587 use init functions to register module types, etc.
Instead of putting all the blueprint registrations in soong_build,
put them all in init() functions.  This puts the registration next
to the implementation.

Change-Id: Ide1a749518f5e9d1367a18ab3bb1d91da3310c76
2015-07-09 17:57:18 -07:00
Dan Willemsen
eba363ed67 Use same libgcc as Make for arm
The current make system uses the arm libgcc.a, not the thumb version.

Change-Id: I38efb80b0d14bbe9c56beeb50f3817307f076d89
2015-07-08 18:11:14 -07:00
Dan Willemsen
2701212d05 androidbp: Implement static/shared property copying
During mutation, make deep copies of the blueprint modules, then start
modifying the property lists to apply the static/shared properties.

Change-Id: Idc7063f4d1cf7d173ae10418e3437f3e2b914f59
2015-07-06 13:52:26 -07:00
Colin Cross
b3245e9cf6 androidbp: simplify translation by evaluating all expressions in Blueprint
Translation is getting complicated because the expressions supported
by Blueprint are difficult to support in Make.  Modify androidbp
to use context aware parsing so it can evaluate all expressions at
parse time, so it only needs to deal with constant values.

Change-Id: I57047645fb48475baecd0361f78a93ec0a26011e
2015-06-30 17:51:25 -07:00
Dan Willemsen
f33877b0e9 androidbp: Test valueToString
Change-Id: I358cf4bb020fc4db14792e2cdffc18bc2f89f4d4
2015-06-29 21:46:22 +00:00
Dan Willemsen
e5e2033146 Build and run tests during bootstrap
Change-Id: If40489d74136af4d15d4ff26210a50975a462497
2015-06-26 11:51:03 -07:00
Colin Cross
1f8c52be73 Add per-directory build targets
Build a map of blueprint directory to modules built from that
directory, and then add phony rules to build.ninja that emulate
the behavior of mma in the current build system.

Also fixes checkbuild to depend on checkbuild files and installable
files, but not installed files.

Change-Id: I8bad6e93387940df7439dbd4554f6d79f924c65f
2015-06-17 10:18:35 -07:00
Colin Cross
e5ee41274a Rename Blueprints to Android.bp
Rename module definition files to Android.bp to avoid conflicts
with another project called Blueprint.

Change-Id: I105a07555eb7890f56120deab9036cf9ae5d8525
2015-05-20 13:10:32 -07:00
Renamed from Blueprints (Browse further)