We need to have a way to see the list of modules that directly or
indirectly contribute to an APEX. People find it difficult to determine
whether a module is included in which APEXes because APEX tracks
indirect dependencies as well as direct dependencies. Therefore, just
looking at Android.bp for the APEX itself doesn't give the answer.
This change adds a new make target <apex_name>-deps-info, which
generates out/soong/<apex_name>-deps-info.txt file that shows the
internal and external dependencies of the said APEX.
Here, internal means the dependencies are actually part of the
APEX, while external means the dependencies are still external to the
APEX.
Bug: 146323213
Test: m (apex_test amended)
Change-Id: I33d1ccf5d1ca335d71cd6ced0f5f66b8c3886d13
The cc.GatherRequiredDepsForTest() method returns some default module
definitions that are required when using cc module types like
cc_library. Previously, the registration of the module types and
mutators needed to process those default definitions was duplicated
in the test config initialization.
This change removes that duplicated code and replaces it with calls
to cc.RegisterRequiredBuildComponentsForTest(ctx) which registers all
the required build components.
Test: m checkbuild
Bug: 146540677
Change-Id: I80b6913c5691ff164ce9d308b9e1da24940f2d42
The filesystem object was available through ModuleContext.Fs(), but
gives too much access to the filesystem without enforicing correct
dependencies. In order to support sandboxing the soong_build
process move the filesystem into the Config. The next change will
make it private.
Bug: 146437378
Test: all Soong tests
Change-Id: I5d3ae9108f120fd335b21efd612aefa078378813
The clang-r370808 upgrade contains a change to LLD allow PT_LOAD
segments to reside at non-multiples of the page size in the resulting
object file. https://reviews.llvm.org/rL369344
While this helps reduce the alignment waste and resulting image size, it
has interesting implications for execute only memory (XOM): The runtime
loader will now load code or data from other segments into pages with
different protections than intended.
This would partially defeat execute only (XOM) text sections as the
segment could now overlap with previous and following sections. This
might allow for code or data from the preceding and following sections
(like .eh_frame, and .data.rel.ro) to be executable, and either ends of
.text to be readable.
When the runtime loader (linker[64]) `mmap`s segments from *.so files,
the file offset parameter (see `man 2 mmap`) MUST be a multiple of the
page size. Since the updated LLD can now pack segments in a file (which
helps minimize resulting object file size) (previously, the segment
offsets were page aligned), this has interesting implications.
To appreciate the current bug, consider the following output from
`readelf` before this patch is applied, but after the toolchain upgrade:
```
$ readelf -lSW $OUT/symbols/apex/com.android.runtime/lib64/bionic/libc.so
...
[Nr] Name Type Address Off Size ES Flg Lk Inf Al
...
[13] .eh_frame PROGBITS 000000000002e7c0 02e7c0 013374 00 A 0 0 8
[14] .text PROGBITS 0000000000042b40 041b40 09ecb4 00 AX 0 0 64
[15] .plt PROGBITS 00000000000e1800 0e0800 001f30 00 AX 0 0 16
[16] .data.rel.ro PROGBITS 00000000000e4740 0e2740 005208 00 WA 0 0 32
...
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
PHDR 0x000040 0x0000000000000040 0x0000000000000040 0x000230 0x000230 R 0x8
LOAD 0x000000 0x0000000000000000 0x0000000000000000 0x041b34 0x041b34 R 0x1000
LOAD 0x041b40 0x0000000000042b40 0x0000000000042b40 0x0a0bf0 0x0a0bf0 E 0x1000
LOAD 0x0e2740 0x00000000000e4740 0x00000000000e4740 0x006720 0x006720 RW 0x1000
...
01 .note.android.ident .note.gnu.build-id .dynsym .gnu.version .gnu.version_d .gnu.version_r .gnu.hash .dynstr .rela.dyn .rela.plt .rodata .eh_frame_hdr .eh_frame
02 .text .plt
03 .data.rel.ro .fini_array .init_array .dynamic .got .got.plt
...
The above output tells us:
1. .text will wind up in the third (02) segment.
2. The third segment will be (LOAD)'ed as (E)xecutable.
3. Because the file (Offset) of the first segment (0x41b40) is NOT a
multiple of the page size, it cannot be passed as the `offset` to
`mmap`. As such it will be rounded down to the first multiple of the
page size, 0x41000.
4. The preceding section (.eh_frame) will be loaded in the preceding
segment (01). It occupies file (Off)set range [(0x2e7c0):0x41b34].
0x41b34 is not explicit in the output, instead you must use the
formula:
Off + Size == End
ie.
0x2e7c0 + 0x13374 == 0x41b34
(This happens to match (FileSiz) of the second segment, which makes
sense as .eh_frame is the final section in the second segment.)
5. mmap'ing file offset 0x41000 when loading the second segment will
include 0x4c0 bytes (0x42000 - 0x41b40) from .text, now mapped as
readable (oops). Suddenly code from .text is now readable (and thus
scannable for gadgets for ROP chains).
6. mmap'ing file offset 0x41000 when loading the third segment will
include 0xb34 bytes (0x41b34 - 0x41000) from .eh_frame, now mapped as
executable (oops). Suddenly data from .eh_frame is now executable
(and thus a potential gadget for ROP chains).
7. mmap'ing file offset 0xe2000 when loading the third segment will
include 0x8CO bytes (0xe3000 - 0xe2740) from .data.rel.ro, now mapped
as executable (oops). Suddenly data from .data.rel.ro is now
executable (and thus a potential gadget for ROP chains).
8. mmap'ing file offset 0xe2000 when loading the fourth segment will
include 0x730 bytes (0xe0800 + 0x1f30 - 0xe2000) from .plt, now
mapped as readable (oops). Suddenly data from .plt is now readable
(and thus scannable for gadgets for ROP chains).
All these oops' could be avoided if the linker placed .text+.plt at page
size aligned file offsets, which is what `-Wl,-z,separate-code` code
does. After this patch, we have:
```
$ readelf -lSW $OUT/symbols/apex/com.android.runtime/lib64/bionic/libc.so
...
Type Offset VirtAddr PhysAddr FileSiz MemSiz Flg Align
PHDR 0x000040 0x0000000000000040 0x0000000000000040 0x000230 0x000230 R 0x8
LOAD 0x000000 0x0000000000000000 0x0000000000000000 0x041b34 0x041b34 R 0x1000
LOAD 0x042000 0x0000000000042000 0x0000000000042000 0x0a0be0 0x0a0be0 E 0x1000
LOAD 0x0e3000 0x00000000000e3000 0x00000000000e3000 0x006720 0x006720 RW 0x1000
```
In the future, we could go back to tightly packing segments in the
binary if the runtime loader was improved to detect the previously
stated problem, and `memset` over the problematic ranges of the freshly
`mmap`ed pages (implying additional startup cost for reduced binary
size). This might save ~6 KB from each native binary, which adds up to
~17 MB for an AOSP image.
Also, prefer
-Wl,--execute-only
rather than
-Wl,-execute-only
Bug: 139945549
Bug: 146144180
Test: readelf -lSW $OUT/symbols/apex/com.android.runtime/lib64/bionic/libc.so
Change-Id: I64527e034ca3c71565ea52ed06f81f75d5216627
Reported-by: Ryan Prichard <rprichard@google.com>
Suggested-by: Fangrui Song <maskray@google.com>
Signed-off-by: Nick Desaulniers <ndesaulniers@google.com>
Use the empty string for the core image variant so that modules
added to imageMutator do not change their build directory.
Bug: 142286466
Test: m checkbuild
Change-Id: Ida4534d9a4d6176236aaa480fed359ce27acfaa1
Merged-In: Ida4534d9a4d6176236aaa480fed359ce27acfaa1
(cherry picked from commit 72d685ee7f45e5393be44ae4159edf083ac918de)
* Now the installation directories match those for C/C++ tests:
* The relative_install_path refers to path under nativetest(64).
* Device test files are installed in data/nativetest(64).
* Automatically generated configuration files and copied test binaries
are still in the "testcases" directory.
* Change host test configuration to run test binary files
in testcases/<mutated_module_name>/<arch_type>/<stem_name>
Bug: 140938178
Test: atest --include-subdirs under external/rust/crates
Change-Id: I4b29afb897f4ba8749e87f79857c5b1a959bb2b0
* Copy current configuration from
rust/config/x86_64_device.go and
cc/config/x86_device.go.
Bug: 145608833
Test: build rust modules for aosp_x86-userdebug
Change-Id: Ie4951dd4703596f8e503d813a6c4ac03c1f20980
Move the ImageMutator to be registered just after the archMutator
in preparation for moving it between osMutator and archMutator.
Requries updating variants in a few tests that now run the
ImageMutator.
Bug: 142286466
Test: no change to build.ninja
Test: all soong tests
Change-Id: Ia9d2a7bc0e225bedec3c9a83ea04f471a931bf47
* The RustBinaryTest runner and device test config template
are derived from GTest. Device tests are pushed to /data/local/tmp.
Bug: 140938178
Test: make unicode-xid projects; run atest
Change-Id: Idb4cab7872b48c6e25bc59b85aa6d9dece4383ec
Avoid having to pass ModuleFactoryAdaptor to every call to
RegisterModuleType in a test by wrapping RegisterModuleType.
Test: all soong tests
Change-Id: If8847d16487de0479cc3020b728256922b3cadba
Rust C libraries cannot currently depend on dylibs. This prevents device
libraries from attempting to link the stdlib dynamically, instead using
rlibs.
Bug: 144861059
Test: Simple Rust library compiles as a shared/static library.
Change-Id: I38e5aae90e50ddcc5fc72d5a8d1820d92955cf48
cc.LinkableInterfaces should be able to request if an empty variant is
required from the LinkageMutator. We were previously assuming that if only
static or shared libraries were being built, then we didn't need the
empty variant. However this should be explicit to handle cases where the
default set of variants being created may not include both static and
shared libraries.
Bug: 144861059
Test: m -j crosvm.experimental
Change-Id: I66724bdc16ff350b06dfa4d049c82f33019979a5
Prepare for making the image mutator available to all modules and
moving it between the os and arch mutators by moving it into the
android package and using an interface implemented by the module
types to control it.
Bug: 142286466
Test: No unexpected changes to out/soong/build.ninja
Change-Id: I0dcc9c7b5ec80edffade340c367f6ae4da34151b
* Rust test modules should have NATIVE_TESTS class,
not EXECUTABLES, to work with atest --host.
Bug: 140938178
Test: make rust projects; run atest --host .
Change-Id: Ie9e237ee8c17b6e2d41f08a77c2b79d462a2c193
* Rename testBinaryDecorator to testDecorator
* Add TestProperties
* Add install function for testDecorator to install config files
* Add tradefed.AutoGenRustHostTestConfig
* Depend on new build/make/core/rust_host_test_config_template.xml
and new tradefed.testtype.rust.RustBinaryHostTest class
* Add autogenTemplateWithName in tradefed/autogen.go
to generate config files with customized(mutated) executable name.
* Make rust_test module names more robust and easy to use.
* Use crate name instead of source file name as the Stem
for single source file modules, to match original user
specified output file name in Cargo.toml.
* Do not set up test module SubName when Stem is empty
or when the module name already contains Stem suffix.
That happens when TestPerSrcMutator is disabled or when
there is only one source file with renamed output file name.
* In TEST_MAPPING, references to mutated rust_test modules should be
(1) <module_name> for single source file modules without mutation, or
(2) <module_name>_<crate_name> for single source file modules, or
(3) <module_name>_<source_file_base_name> for multi-file modules.
Bug: 140938178
Test: mm in rust projects, check output test file names
Change-Id: Ifdbfa14d5eed4f10b4fb983f82c93bbb9be3f899
Make sure LinkabeInterfaces can export include dirs to cc modules. This
fixes the Rust implementation and makes sure these are pulled in for cc
modules.
Bug: 144052093
Test: cc module depending on a rust library includes dirs correctly.
Change-Id: I6b5d5e6ab6afb507178b4f2cbdc45f01031dbfe4
Now that we have sysroots built in Soong, we can enable these devices
and they just work.
Bug: 141251907
Bug: 141381044
Change-Id: I562256fee372d331db883f7c1b46405945295c24
In order to ensure we are using current platform Bionic for any platform
Rust binaries, we need to build the sysroot in Soong. This will also
enable us too hook the "test" crate if necessary.
While both a dynamic and static sysroot are available, on device only a
dynamic sysroot will be injected. On host, we continue using the sysroot
used to build the compiler as before.
Bug: 139486496
Change-Id: I127377e5b056610ceb5015a34d266250320fbc31
Break up some of the longer lines in the Rust source and refactor the
TransformSrcto* functions to take fewer arguments.
Bug: N/A
Test: m -j crosvm.experimental
Change-Id: Ia0deb8bfe3e1bfd07ef35633e9a7e6f5e799b3a3
rustc expects libraries and proc_macro filenames to conform to
a particular format, alphanumeric with underscores and lib${crate_name}.*.
Enforce this with a check when getStem() is called.
This makes the crate_name property required for proc_macros and
libraries. This also removes the notion of a default crate name derived
from the module name. It's not needed for binaries, so this won't impact
them.
Bug: 143579265
Test: m -j crosvm.experimental
Change-Id: I2770cf7d02dd4291c3d240d58d242b940098dcee
* Rust tests are like binary files compiled with --test.
New test.go follows binary.go code patterns and reuses
some code in binary.go.
* Generate one test per source file as testPerSrc in cc/test.go.
The "all tests" variation feature of cc/test.go is not copied yet.
Fix some Stem and SubName settings to make testPerSrc work.
* Move cc.CheckDuplicate to android.CheckDuplicate,
which is now shared by cc and rust.
* Refactor tests in binary_test.go and add new test_test.go.
Bug: 140938178
Test: mm in rust projects, added rust_test and rust_test_host
Change-Id: Ia6fec8b4cf2572fd352ab1938a1f3c7b5cca2212
Adds the ability for rust modules to be compiled as C libraries, and
allows cc modules to depend on these rust-generated modules. This also
means that soong-rust should not have any dependencies on soong-cc aside
from what's required for testing.
There's a couple small fixes included as well:
- A bug in libNameFromFilePath that caused issues when library's had
"lib" in their name.
- VariantName is removed from rust library MutatedProperties since this
was unused.
Bug: 140726209
Test: Soong tests pass.
Test: Example cc_binary can include a rust shared library as a dep.
Test: m crosvm.experimental
Change-Id: Ia7deed1345d2423001089014cc65ce7934123da4
Adds an interface, CcLinkableInterface, for cc linkable dependencies
which come from other toolchains such as Rust.
Bug: 140726209
Test: Soong tests pass, rust modules still compile.
Change-Id: I7378a46fad94fd0b735746aaf4e265fd2c2c04d8
Support for building Rust modules on darwin hosts.
Bug: 140640858
Test: m libremain works on darwin
Change-Id: Ieb1ff9167de34cffbebebab31fa48da07081c8a6
The arch variants are hardcoded in every module type. Refactor
them out into a Target.Variations() method in preparation for
splitting the arch mutator into two, which will require using
different variations.
Test: m checkbuild
Change-Id: I28ef7cd5168095ac888fe77f04e27f9ad81978c0
Add a ToMakePath() method that returns a new path that points out
out/ instead of out/soong/, and replace the
"$(OUT_DIR)/" + path.RelPathString()
pattern with
path.ToMakePath().String()
Bug: 141877526
Test: m checkbuild
Change-Id: I391b9f2ed78c83a58d905d48355ce9b01d610d16
Create a new type InstallPath that is similar to OutputPath to
differentiate intermediates output paths from installed output
paths.
RelPathString is a poorly defined, undocumented function that is
primarily used to get an install path relative to out/soong to
generate an equivalent install path for Make relative to $(OUT_DIR).
Move it to InstallPath for now, and fix the one remaining user on
OutputPath.
Add a method to create an NDK install path so that ndk_sysroot.go
doesn't have to do it manually with PathForOutput.
Bug: 141877526
Test: m checkbuild
Change-Id: I83c5a0bd1fd6c3dba8d3b6d20d039f64f353ddd5
* Edition and Deny_warnings should not be set when
constructing a BaseCompilerProperties, or the
initialized values will reject values inherited from rust_defaults.
* Use getEdition and getDenyWarnings to retrieve those properties
with defaults from config.
Bug: 141699953
Test: mm in rust projects
Change-Id: Id1ae357caeaf656cd33732bf4e54920e206f4ead
We need to pass --remap-path-prefix to Rust builds to ensure that they
are reproducible across different paths. This adds that for both host
and device builds.
Test: Verify host and device builds are reproducible with different
directories.
Change-Id: I68ba31537b2332fd05613aa0fb00a1502d78e410
* "-D warnings" means "deny all warnings" and make them errors.
* Modules with warnings should fix all warnings or use
deny_warnings: false
Bug: 141699953
Test: mm in projects with Rust modules
Change-Id: I6310dee8e34b7780937e8fc1834016a04a943a2f
Ensure that devices can include proc_macros, which are host-only
and may include host-only dependencies.
Bug: 141491501
Test: Soong tests pass.
Test: Test fails as expected when removing CL 1126496
Change-Id: I3ae7ab80283cd1fd4b800a533cb3205b3c108d45
Currently proc_macros don't pull in the correct arch dependencies when
the host arch differs from the target arch.
This fixes how proc_macro dependencies are handled by defining them as
always being host-only and including them as dependencies for
device-modules by using AddFarVariationDependencies.
Bug: 141491501
Test: Example device rust module builds with proc_macro dependency.
Change-Id: Ic037dc406ce90526f8b68c92fffc0d93a498a4ff
Add libtest to our list of stdlib libraries, and remove some others that
are only used when generating proc_macros (which aren't dynamic).
Bug: 141207434
Test: m -j crosvm.experimental
Change-Id: I39cb030940adf1993e861d0142378eeea8b1ff5e
Translated second architectures now go in NativeBridgeArch instead
of DeviceSecondaryArch.
This reapplies I568046330abc002d4eed582cb999b62a5eaba790 with
ctx.Config().HasMulitlibConflict() added to fix the NDK build,
which has arm64, arm, x86_64, and x86 architectures enabled.
Test: m checkbuild
Test: OUT_DIR=out_ndk build/soong/scripts/build-ndk-prebuilts.sh
Test: no change to build.ninja or Android-aosp_cf_x86_phone.mk
Change-Id: Iadcafbd64bfb9579ae7c86914927c43a062b0c8e
Splits out the Rust paths whitelist into a separate file
under config/ so that OWNERS can be defined for it.
Bug: 141207129
Test: m -j crosvm.experimental
Change-Id: I5effa6783e5c47560b4b1eae12ad0eb9e9ba96fe
The Target checks against the os type and arch type occur before we
determine a dependency is a rust.Module / cc.Module. This meant
rust_defaults would fail this check, preventing them from being used.
Bug: 140963307
Test: make a rust_binary_host using rust_defaults.
Change-Id: Ibf43635f14ad367b4ce8016a2196f8c645b36bfe
Both Rust and cc use this function, so move it over to android
package's util.go and export it.
Bug: 140734195
Test: m -j
Change-Id: Ibe8b7a94592e402468a027ad6027b187f29c8e07
The Rust toolchain only supports Linux hosts, so skip Rust tests
that depend on host toolchains when building on other platforms.
Test: Tests still run on Linux hosts
Bug: 140435149
Change-Id: I298b52589ab5c6a7ec3acc4db1111596cd995d76
Adds support to Soong for building rust modules. This currently only
supports x86_64 device and x86 linux host targets. The functionality
is sufficient to build crosvm.
Bug: 136189233
Test: Test module builds.
Test: crosvm builds.
Change-Id: I6ea04615834a6d673578ab10ea1a2eb04259fe09
