ReadPadSegmentNote() finds the elf note of type
NT_ANDROID_TYPE_PAD_SEGMENT and checks that the desc value
is 1, to decided whether the LOAD segment mappings should
be extended (padded) to avoid gaps.
Cache the result of this operation in ElfReader and soinfo
for use in the subsequent patch which handles the extension
of the segment mappings.
Test: atest -c linker-unit-tests [Later patch]
Test: m && launch_cvd
Bug: 316403210
Change-Id: I32c05cce741d221c3f92835ea09d932c40bdf8b1
Signed-off-by: Kalesh Singh <kaleshsingh@google.com>
This patch adds the necessary bionic code for the linker to protect
global data using MTE.
The implementation is described in the MemtagABI addendum to the
AArch64 ELF ABI:
https://github.com/ARM-software/abi-aa/blob/main/memtagabielf64/memtagabielf64.rst
In summary, this patch includes:
1. When MTE globals is requested, the linker maps writable SHF_ALLOC
sections as anonymous pages with PROT_MTE (copying the file contents
into the anonymous mapping), rather than using a file-backed private
mapping. This is required as file-based mappings are not necessarily
backed by the kernel with tag-capable memory. For sections already
mapped by the kernel when the linker is invoked via. PT_INTERP, we
unmap the contents, remap a PROT_MTE+anonymous mapping in its place,
and re-load the file contents from disk.
2. When MTE globals is requested, the linker tags areas of global memory
(as defined in SHT_AARCH64_MEMTAG_GLOBALS_DYNAMIC) with random tags,
but ensuring that adjacent globals are never tagged using the same
memory tag (to provide detemrinistic overflow detection).
3. Changes to RELATIVE, ABS64, and GLOB_DAT relocations to load and
store tags in the right places. This ensures that the address tags are
materialized into the GOT entries as well. These changes are a
functional no-op to existing binaries and/or non-MTE capable hardware.
Bug: N/A
Test: atest bionic-unit-tests CtsBionicTestCases --test-filter=*Memtag*
Change-Id: Id7b1a925339b14949d5a8f607dd86928624bda0e
Adds support for the dynamic entries to specify MTE enablement. This is
now the preferred way for dynamically linked executables to specify to
the loader what mode MTE should be in, and whether stack MTE should be
enabled. In future, this is also needed for MTE globals support.
Leave the existing ELF note parsing as a backup option because dynamic
entries are not supported for fully static executables, and there's
still a bunch of glue sitting around in the build system and tests that
explicitly include the note. When -fsanitize=memtag* is specified, lld
will create the note implicitly (along with the new dynamic entries),
but at some point once we've cleaned up all the old references to the
note, we can remove the notegen from lld.
Bug: N/A
Test: atest bionic-unit-tests CtsBionicTestCases --test-filter=*Memtag*
Test: Build/boot the device under _fullmte.
Change-Id: I954b7e78afa5ff4274a3948b968cfad8eba94d88
Once upon a time (and, indeed, to this very day if you're on LP32) the
soinfo struct used a fixed-length buffer for the soname. This caused
some issues, mainly with app developers who accidentally included a full
Windows "C:\My Computer\...\libfoo.so" style path. To avoid all this we
switched to just pointing into the ELF file itself, where the DT_SONAME
is already stored as a NUL-terminated string. And all was well for many
years.
Now though, we've seen a bunch of slow startup traces from dogfood where
`dlopen("libnativebridge.so")` in a cold start takes 125-200ms on a recent
device, despite no IO contention. Even though libnativebridge.so is only
20KiB.
Measurement showed that every library whose soname we check required
pulling in a whole page just for the (usually) very short string. Worse,
there's readahead. In one trace we saw 18 pages of libhwui.so pulled
in just for `"libhwui.so\0"`. In fact, there were 3306 pages (~13MiB)
added to the page cache during `dlopen("libnativebridge.so")`. 13MiB for
a 20KiB shared library!
This is the obvious change to use a std::string to copy the sonames
instead. This will dirty slightly more memory, but massively improve
locality.
Testing with the same pathological setup took `dlopen("libnativebridge.so")`
down from 192ms to 819us.
Bug: http://b/177102905
Test: tested with a pathologically modified kernel
Change-Id: I33837f4706adc25f93c6fa6013e8ba970911dfb9
Add inaccessible gaps between shared libraries to make it harder for the
attackers to defeat ASLR by random probing.
To avoid excessive page table bloat, only do this when a library is
about to cross a huge page boundary, effectively allowing several
smaller libraries to be lumped together.
Bug: 158113540
Test: look at /proc/$$/maps
Change-Id: I39c0100b81f72447e8b3c6faafa561111492bf8c
This reverts commit a8cf3fef2a.
Reason for revert: memory regression due to the fragmentation of the page tables
Bug: 159810641
Bug: 158113540
Change-Id: I6212c623ff440c7f6889f0a1e82cf7a96200a411
Improve ASLR by increasing the randomly sized gaps between shared
library mappings, and keep them mapped PROT_NONE.
Bug: 158113540
Test: look at /proc/$$/maps
Change-Id: Ie72c84047fb624fe2ac8b7744b2a2d0d255ea974
Symbol lookup is O(L) where L is the number of libraries to search (e.g.
in the global and local lookup groups). Factor out the per-DSO work into
soinfo_do_lookup_impl, and optimize for the situation where all the DSOs
are using DT_GNU_HASH (rather than SysV hashes).
To load a set of libraries, the loader first constructs an auxiliary list
of libraries (SymbolLookupList, containing SymbolLookupLib objects). The
SymbolLookupList is reused for each DSO in a load group. (-Bsymbolic is
accommodated by modifying the SymbolLookupLib at the front of the list.)
To search for a symbol, soinfo_do_lookup_impl has a small loop that first
scans a vector of GNU bloom filters looking for a possible match.
There was a slight improvement from templatizing soinfo_do_lookup_impl
and skipping the does-this-DSO-lack-GNU-hash check.
Rewrite the relocation processing loop to be faster. There are specialized
functions that handle the expected relocation types in normal relocation
sections and in PLT relocation sections.
This CL can reduce the initial link time of large programs by around
40-50% (e.g. audioserver, cameraserver, etc). On the linker relocation
benchmark (64-bit walleye), it reduces the time from 131.6ms to 71.9ms.
Bug: http://b/143577578 (incidentally fixed by this CL)
Test: bionic-unit-tests
Change-Id: If40a42fb6ff566570f7280b71d58f7fa290b9343
Validate the list of defined versions explicitly, during library
prelinking, rather than implicitly as part of constructing the
VersionTracker in soinfo::link_image.
Doing the validation upfront allows removing the symbol lookup failure
code paths, which only happen on a library with invalid version
information.
Helps on the walleye 64-bit linker relocation benchmark (146.2ms ->
131.6ms)
Bug: none
Test: bionic unit tests
Change-Id: Id17508aba3af2863909f0526897c4277419322b7
Previously, during a find_libraries call that loaded a library, a
library was prelinked once for each DT_NEEDED reference to the library.
This CL has a negligible effect on the linker relocation benchmark
(146.9ms -> 146.2ms).
Bug: none
Test: bionic unit tests
Change-Id: I385f312b8acf8d35aa0af9722131fe367b5edd9b
The run-time of dl_iterate_phdr can be noticeable for exception-handling.
e.g. I tested extest on the GitHub issue by:
- disabling the dlpi_subs/dlpi_adds-based caching in libgcc
- adding a weaker "load base hint" caching in libgcc that doesn't need
the dlpi_subs/dlpi_adds fields, but still has to iterate over every
module to validate a cache hit
extest throws 10000 exceptions, and I saw a regression from ~1550ms
runtime on Q to about ~1950ms on master. This CL reduces the regression to
about ~1700ms.
Bug: https://github.com/android/ndk/issues/1062
Test: bionic unit tests
Change-Id: I099e97e1a20f5b2aa6737789e49d965170eb85a8
Ordinary executables have a PT_INTERP path of /system/bin/linker[64], but:
- executables using bootstrap Bionic use /system/bin/bootstrap/linker[64]
- ASAN executables use /system/bin/linker_asan[64]
gdb appears to use the PT_INTERP path for debugging the dynamic linker
before the linker has initialized the r_debug module list. If the linker's
l_name differs from PT_INTERP, then gdb assumes that the linker has been
unloaded and searches for a new solib using the linker's l_name path.
gdb may print a warning like:
warning: Temporarily disabling breakpoints for unloaded shared library "$OUT/symbols/system/bin/linker64"
If I'm currently debugging the linker when this happens, gdb apparently
doesn't load debug symbols for the linker. This can be worked around with
gdb's "sharedlibrary" command, but it's better to avoid it.
Previously, when PT_INTERP was the bootstrap linker, but l_name was
"/system/bin/linker[64]", gdb would find the default non-bootstrap linker
binary and (presumably) get confused about symbol addresses.
(Also, remove the "static std::string exe_path" variable because the
soinfo::realpath_ field is a std::string that already lasts until exit. We
already use it for link_map_head.l_name in notify_gdb_of_load.)
Bug: http://b/134183407
Test: manual
Change-Id: I9a95425a3a5e9fd01e9dd272273c6ed3667dbb9a
Introduce a new flag ANDROID_DLEXT_RESERVED_ADDRESS_RECURSIVE which
instructs the linker to use the reserved address space to load all of
the newly-loaded libraries required by a dlopen() call instead of only
the main library. They will be loaded consecutively into that region if
they fit. The RELRO sections of all the loaded libraries will also be
considered for reading/writing shared RELRO data.
This will allow the WebView implementation to potentially consist of
more than one .so file while still benefiting from the RELRO sharing
optimisation, which would otherwise only apply to the "root" .so file.
Test: bionic-unit-tests (existing and newly added)
Bug: 110790153
Change-Id: I61da775c29fd5017d9a1e2b6b3757c3d20a355b3
Each TLSDESC relocation relocates a 2-word descriptor in the GOT that
contains:
- the address of a TLS resolver function
- an argument to pass (indirectly) to the resolver function
(Specifically, the address of the 2-word descriptor is passed to the
resolver.)
The loader resolves R_GENERIC_TLSDESC relocations using one of three
resolver functions that it defines:
- tlsdesc_resolver_static
- tlsdesc_resolver_dynamic
- tlsdesc_resolver_unresolved_weak
The resolver functions are written in assembly because they have a
restrictive calling convention. They're only allowed to modify x0 and
(apparently) the condition codes.
For a relocation to memory in static TLS (i.e. the executable or an solib
loaded initially), the loader uses a simple resolver function,
tlsdesc_resolver_static, that returns the static offset it receives from
the loader.
For relocations to dynamic TLS memory (i.e. memory in a dlopen'ed solib),
the loader uses tlsdesc_resolver_dynamic, which allocates TLS memory on
demand. It inlines the fast path of __tls_get_addr, then falls back to
__tls_get_addr when it needs to allocate memory. The loader handles these
dynamic TLS relocations in two passes:
- In the first pass, it allocates a table of TlsDynamicResolverArg
objects, one per dynamic TLSDESC relocation.
- In the second pass, once the table is finalized, it writes the
addresses of the TlsDynamicResolverArg objects into the TLSDESC
relocations.
tlsdesc_resolver_unresolved_weak returns a negated thread pointer so that
taking the address of an unresolved weak TLS symbols produces NULL.
The loader handles R_GENERIC_TLSDESC in a target-independent way, but
only for arm64, because Bionic has only implemented the resolver functions
for arm64.
Bug: http://b/78026329
Test: bionic unit tests
Test: check that backtrace works inside a resolver function and inside
__tls_get_addr called from a resolver
(gdbclient.py, b __tls_get_addr, bt)
Merged-In: I752e59ff986292449892c449dad2546e6f0ff7b6
Change-Id: I752e59ff986292449892c449dad2546e6f0ff7b6
Initialize a thread's DTV to an empty zeroed DTV. Allocate the DTV and
any ELF module's TLS segment on-demand in __tls_get_addr. Use a generation
counter, incremented in the linker, to signal when threads should
update/reallocate their DTV objects.
A generation count of 0 always indicates the constant zero DTV.
Once a DTV is allocated, it isn't freed until the thread exits, because
a signal handler could interrupt the fast path of __tls_get_addr between
accessing the DTV slot and reading a field of the DTV. Bionic keeps a
linked list of DTV objects so it can free them at thread-exit.
Dynamic TLS memory is allocated using a BionicAllocator instance in
libc_shared_globals. For async-signal safety, access to the
linker/libc-shared state is protected by first blocking signals, then by
acquiring the reader-writer lock, TlsModules::rwlock. A write lock is
needed to allocate or free memory.
In pthread_exit, unconditionally block signals before freeing dynamic
TLS memory or freeing the shadow call stack.
ndk_cruft.cpp: Avoid including pthread_internal.h inside an extern "C".
(The header now includes a C++ template that doesn't compile inside
extern "C".)
Bug: http://b/78026329
Bug: http://b/123094171
Test: bionic unit tests
Change-Id: I3c9b12921c9e68b33dcc1d1dd276bff364eff5d7
The field was pointing into an element of an std::vector, but the address
of a vector element is invalidated when the vector is resized.
This bug was caught by the new elftls.shared_ie and
elftls_dl.dlopen_shared_var_ie tests.
Bug: http://b/78026329
Test: bionic unit tests
Change-Id: I7232f6d703a9e339fe8966a95b7a68bae2c9c420
* Initialize the exe's l_ld correctly, and initialize its l_addr field
earlier.
* Copy the phdr/phnum fields from the linker's temporary soinfo to its
final soinfo. This change ensures that dl_iterate_phdr shows the phdr
table for the linker.
* Change init_linker_info_for_gdb a little: use an soinfo's fields to
init the soinfo::link_map_head field, then reuse the new
init_link_map_head function to handle the linker and the executable.
Test: manual
Test: bionic-unit-tests
Bug: https://issuetracker.google.com/112627083
Bug: http://b/110967431
Change-Id: I40fad2c4d48f409347aaa1ccb98d96db89da1dfe
The tls_nodelete state should apply to load_group not
isolated soinfo. This actually also means that multiple
soinfos may have tls_counter on their dso_handles.
This change replaces TLS_NODELETE flag with secondary counter.
Note that access to the secondary counter (located inside soinfo)
is pretty expensive because it requires soinfo lookup by dso_handle
whereas dso_handle counter is much faster. This is why it is updated
only when dso_handle counter starts or hits 0.
Bug: http://b/80278285
Test: bionic-unit-tests --gtest_filter=dl*
Change-Id: I535583f6714e45fa2a7eaf7bb3126da20ee7cba9
We've been using #pragma once for new internal files, but let's be more bold.
Bug: N/A
Test: builds
Change-Id: I7e2ee2730043bd884f9571cdbd8b524043030c07
This change modifies the encoding used in SHT_RELR sections to a simpler
version that gives better results. This encoding was suggested by Andrew
Grieve and is described in this post on generic-abi@googlegroups.com:
https://groups.google.com/d/msg/generic-abi/bX460iggiKg/Pi9aSwwABgAJ
Bug: None
Test: Built image for marlin, flashed on device, ran arm and
aarch64 binaries containing '.relr.dyn' sections using
the new encoding.
Change-Id: I266affe0fbad91dc375995985a221cb02499447b
This change adds experimental support for SHT_RELR sections, proposed
here: https://groups.google.com/forum/#!topic/generic-abi/bX460iggiKg
Definitions for the new ELF section type and dynamic array tags, as well
as the encoding used in the new section are all under discussion and are
subject to change. Use with caution!
Bug: None
Test: Built image for marlin, flashed on device, ran arm and
aarch64 binaries containing '.relr.dyn' sections.
Change-Id: I2953ae932d3c42ae394e71f8fa058013758a1778
Introduce new flag to mark soinfo as TLS_NODELETE when
there are thread_local dtors associated with dso_handle
belonging to it.
Test: bionic-unit-tests --gtest_filter=dl*
Test: bionic-unit-tests-glibc --gtest_filter=dl*
Bug: https://github.com/android-ndk/ndk/issues/360
Change-Id: I724ef89fc899788f95c47e6372c38b3313f18fed
This change addresses multiple problems introduced by
02586a2a34
1. In the case of unsuccessful dlopen the failure guard is triggered
for two namespaces which leads to double unload.
2. In the case where load_tasks includes libraries from 3 and more
namespaces it results in incorrect linking of libraries shared between
second and third/forth and so on namespaces.
The root cause of these problems was recursive call to find_libraries.
It does not do what it is expected to do. It does not form new load_tasks
list and immediately jumps to linking local_group. Not only this skips
reference counting it also will include unlinked but accessible library
from third (and fourth and fifth) namespaces in invalid local group. The
best case scenario here is that for 3 or more namesapces this will
fail to link. The worse case scenario it will link the library
incorrectly with will lead to very hard to catch bugs.
This change removes recursive call and replaces it with explicit list of
local_groups which should be linked. It also revisits the way we do
reference counting - with this change the reference counts are updated after
after libraries are successfully loaded.
Also update soinfo_free to abort in case when linker tries to free same
soinfo for the second time - this makes linker behavior less undefined.
Test: bionic-unit-tests
Bug: http://b/69787209
Change-Id: Iea25ced181a98c6503cce6e2b832c91d697342d5
This change also replaces elf-hash with gnu-hash.
Test: make
Test: bionic-unit-tests --gtest_filter=dl*:Dl*
Change-Id: Ibc4026f7abc7e8002f69c33eebaf6a193f1d22eb
This commit updates interface of libdl.c.
1. android_init_namespaces is replaces with android_init_anonymous_namespace
2. added 2 arguments to android_create_namespace to specify linked namespace
and the list of shared libraries sonames.
3. symbol lookup does not get past boundary libraries (added check and test for it).
Bug: http://b/26833548
Bug: http://b/21879602
Test: bionic-unit-tests --gtest_filter=dl*:Dl*
Change-Id: I32921da487a02e5bd0d2fc528904d1228394bfb9
Do not hijack libdl.so methods but make libdl proxy calls to
loader instead. This will be replaces by calls to libc.so
once loader functionality is migrated.
Also add a lock to dl_unwind_find_exidx function call.
Test: bionic-unit-tests --gtest_filter=dl*:Dl*
Bug: http://b/27106625
Change-Id: Ic33a7109a86f4262798d63a35f4c61d15b0068bb
Do not hijack libdl.so methods but make libdl proxy calls to
loader instead. This will be replaces by calls to libc.so
once loader functionality is migrated.
Also add a lock to dl_unwind_find_exidx function call.
Test: bionic-unit-tests --gtest_filter=dl*:Dl*
Bug: http://b/27106625
Change-Id: I9e666e771e4bbca52151cfa7fc4c8677e1480818
Extract linker executable specific code to linker_main.cpp;
this part of code does not have a place in libdl.a/so.
Bug: http://b/14998480
Bug: http://b/30706810
Test: mm && run bionic-unit-tests
Change-Id: I90f7475e93a919b0f9525da22928089ad35b8f6c
Move soinfo and globals out of linker.cpp to
separate files.
Breaking up huge linker.cpp into smaller peaces
in order to make it easier to extract part of the
code that belongs to libdl.so and remove parts of
the code that do not belong to linker
(refactoring part 2 of many)
Change-Id: I868417f4b8d2b84d0e8265e354bc7977161497e2
