This is the second attempt to purge linker block allocators. Unlike the
previously reverted change which purge allocators whenever all objects
are freed, we only purge right before control leaves the linker. This
limits the performance impact to one munmap() call per dlopen(), in
most cases.
Bug: 112073665
Test: Boot and check memory usage with 'showmap'.
Test: Run camear cold start performance test.
Change-Id: I02c7c44935f768e065fbe7ff0389a84bd44713f0
The dynamic linker installed to /system partition is only for
bootstraping, i.e. to be used only by early processes. The existing path
/system/bin/linker[64] will be a symlink to the mount pointpoint
/bionic/bin/linker[64] which will be bind-mounted with either the
bootstrap linker or the default linker in the runtime APEX. Since the
path is taken by the symlink, bootstrap linker is moved to ./boostrap
subdirectory.
Bug: 120266448
Test: m linker; the linker is found under /system/bin/bootstrap/
Change-Id: I163e0c8a1738b94565a6b60f48e65841ba058a91
This change adds following files and symlinks:
Files:
/bionic/lib[64]/lib{c|dl|m}.so
/bionic/bin/linker[64]
Symlinks:
/system/lib[64]/lib{c|dl|m}.so -> /bionic/lib[64]/lib{c|dl|m}.so
/system/bin/linker[64] -> /bionic/bin/linker[64]
/system/bin/linker_asan[64] -> /bionic/bin/linker[64]
The files serve as mount points for either the bootstrap Bionic or the
default Bionic from the runtime APEX. init does the bind-mounting during
booting.
The symlinks are there to not change the ordinary paths to the bionic
files; there are many places that the paths are implied or hard-coded,
e.g., dlopen("/system/lib/libc.so") or DT_INTERP pointing to
/system/bin/linker in the vendor prebuilts.
Bug: 120266448
Test: m blueline, cf_x86, aosp_arm
The aforementioned files and symlinks are found
Change-Id: I97e38c29409ac0610dde285db8df6e94a7930094
LIB is now expanded to either lib or lib64. Add a new lib
libtest_dt_runpath_y which is under lib or lib64 directory to the tests.
Test: run bionic-unit-tests
Merged-In: Ib051d8cae4d50b452759d594c87126b45bc9e959
Change-Id: Ib051d8cae4d50b452759d594c87126b45bc9e959
Each TLSDESC relocation relocates a 2-word descriptor in the GOT that
contains:
- the address of a TLS resolver function
- an argument to pass (indirectly) to the resolver function
(Specifically, the address of the 2-word descriptor is passed to the
resolver.)
The loader resolves R_GENERIC_TLSDESC relocations using one of three
resolver functions that it defines:
- tlsdesc_resolver_static
- tlsdesc_resolver_dynamic
- tlsdesc_resolver_unresolved_weak
The resolver functions are written in assembly because they have a
restrictive calling convention. They're only allowed to modify x0 and
(apparently) the condition codes.
For a relocation to memory in static TLS (i.e. the executable or an solib
loaded initially), the loader uses a simple resolver function,
tlsdesc_resolver_static, that returns the static offset it receives from
the loader.
For relocations to dynamic TLS memory (i.e. memory in a dlopen'ed solib),
the loader uses tlsdesc_resolver_dynamic, which allocates TLS memory on
demand. It inlines the fast path of __tls_get_addr, then falls back to
__tls_get_addr when it needs to allocate memory. The loader handles these
dynamic TLS relocations in two passes:
- In the first pass, it allocates a table of TlsDynamicResolverArg
objects, one per dynamic TLSDESC relocation.
- In the second pass, once the table is finalized, it writes the
addresses of the TlsDynamicResolverArg objects into the TLSDESC
relocations.
tlsdesc_resolver_unresolved_weak returns a negated thread pointer so that
taking the address of an unresolved weak TLS symbols produces NULL.
The loader handles R_GENERIC_TLSDESC in a target-independent way, but
only for arm64, because Bionic has only implemented the resolver functions
for arm64.
Bug: http://b/78026329
Test: bionic unit tests
Test: check that backtrace works inside a resolver function and inside
__tls_get_addr called from a resolver
(gdbclient.py, b __tls_get_addr, bt)
Merged-In: I752e59ff986292449892c449dad2546e6f0ff7b6
Change-Id: I752e59ff986292449892c449dad2546e6f0ff7b6
* changes:
Implement dynamic TLS accesses and allocation
Implement TLS_DTPMOD and TLS_DTPREL relocations
Ignore DT_TLSDESC_GOT / DT_TLSDESC_PLT
Disable the dlfcn.dlopen_library_with_ELF_TLS test
Add BionicAllocator::memalign
Move the linker allocator into libc
Replace some of linker_allocator's header includes
Add | to make bionic's trace end print match other trace end prints.
Test: took systrace with bionic tag enabled
Change-Id: Ieabb139dd224aa8045be914f21c0432d42a93755
This is used to prevent the additional indirection even after heap
profiling has finished, preventing any performance impact on processes
that are not currently being profiled.
Test: m
Test: flash sailfish
Test: try tearing down & re-enabling hooks
Bug: 120186127
Change-Id: Idc5988111a47870d2c093fd6a017b47e65f5616b
Initialize a thread's DTV to an empty zeroed DTV. Allocate the DTV and
any ELF module's TLS segment on-demand in __tls_get_addr. Use a generation
counter, incremented in the linker, to signal when threads should
update/reallocate their DTV objects.
A generation count of 0 always indicates the constant zero DTV.
Once a DTV is allocated, it isn't freed until the thread exits, because
a signal handler could interrupt the fast path of __tls_get_addr between
accessing the DTV slot and reading a field of the DTV. Bionic keeps a
linked list of DTV objects so it can free them at thread-exit.
Dynamic TLS memory is allocated using a BionicAllocator instance in
libc_shared_globals. For async-signal safety, access to the
linker/libc-shared state is protected by first blocking signals, then by
acquiring the reader-writer lock, TlsModules::rwlock. A write lock is
needed to allocate or free memory.
In pthread_exit, unconditionally block signals before freeing dynamic
TLS memory or freeing the shadow call stack.
ndk_cruft.cpp: Avoid including pthread_internal.h inside an extern "C".
(The header now includes a C++ template that doesn't compile inside
extern "C".)
Bug: http://b/78026329
Bug: http://b/123094171
Test: bionic unit tests
Change-Id: I3c9b12921c9e68b33dcc1d1dd276bff364eff5d7
Generalize the omitted symbol and missing-TLS-segment behaviors to all TLS
relocations.
R_GENERIC_TLS_DTPMOD is a module ID, which starts at 1 for the executable.
R_GENERIC_TLS_DTPREL is an offset from the start of a module to a specific
TLS symbol.
binutils currently disagrees with Bionic about the values of
R_AARCH64_TLS_DTPMOD64 and R_AARCH64_TLS_DTPREL64, so disable
DTPMOD/DTPREL for now on arm64.
Bug: http://b/78026329
Test: bionic unit tests (Tests for dynamic TLS will be added later)
Change-Id: I05c28d6a1036bdd6127f605036679b7475689445
These dynamic table entries are used for lazy relocation of TLSDESC
relocations. Bionic relocates everything eagerly and can ignore these
entries.
Bug: http://b/78026329
Test: bionic unit tests
Test: manual
Change-Id: I788c4a95041fae66de2fbeab3babe0205631999c
This test is verifying that an ELF TLS solib can't be loaded, but once
ELF TLS is implemented, it *can* be loaded. Removing the test requires
coordinated changes with internal test suites.
I could modify the test to verify that the solib was successfully loaded,
but support for dynamic ELF TLS appears across a few different CLs.
Bug: http://b/78026329
Test: bionic unit tests
Change-Id: If8cd798f456568a5c76310b754cea48ca8edaa82
A static executable is almost entirely statically relocated by the
linker, with the exception of IRELATIVE relocations, which must be
resolved by libc by enumerating the relocations using the special
linker-defined symbols __rela?_iplt_{start,end}. This patch implements
ifunc support by enumerating the relocations in this way.
Bug: 112482891
Test: /data/nativetest{,64}/bionic-unit-tests-static/bionic-unit-tests-static on walleye_hwasan-userdebug
Change-Id: Ia5522a190da0b86e095b141d5d4e68dd7dd4b695
Bionic needs this functionality to allocate a TLS segment with greater
than 16-byte alignment. For simplicity, this allocator only supports up
to one page of alignment.
The memory layout changes slightly when allocating an object of exactly
PAGE_SIZE alignment. Instead of allocating the page_info header at the
start of the page containing the pointer, it is allocated at the start
of the preceding page.
Bug: http://b/78026329
Test: linker-unit-tests{32,64}
Change-Id: I1c8d1cd7ca72d113bced5ee15ba8d831426b0081
Rename LinkerMemoryAllocator -> BionicAllocator
Rename LinkerSmallObjectAllocator -> BionicSmallObjectAllocator
libc and the linker need to share an instance of the allocator for
allocating and freeing dynamic ELF TLS memory (DTVs and segments). The
linker also continues to use this allocator.
Bug: http://b/78026329
Test: /data/nativetest/bionic-unit-tests-static
Test: /data/nativetest64/bionic-unit-tests-static
Test: /data/nativetest/linker-unit-tests/linker-unit-tests32
Test: /data/nativetest64/linker-unit-tests/linker-unit-tests64
Change-Id: I2da037006ddf8041a75f3eba2071a8fcdcc223ce
The <async_safe/log.h> include in linker_allocator.h was made
unnecessary when the STL dependency was removed.
Replace the linker_*.h header includes with private/bionic_*.h
includes.
This change prepares LinkerMemoryAllocator for use in libc allocating
dynamic ELF TLS memory.
Bug: http://b/78026329
Test: bionic unit tests
Change-Id: I6ad62bd513e71dd3f91a4b724a92b19a1eb6c3b4
Suppress the SELinux denial log spam by not calling realpath(3) when the
path does not exist or is not accessible for read, and then not auditing
access(2) failure.
Bug: 120996057
Test: copy ping to /data/local/tmp, run it, verify no errors
Test: run bionic-unit-tests, the tests pass
Change-Id: Ie6058bfc9524a9b5c50fc7183fdddea6a8fb9200
If a signal handler is blocking all of their signals, we should
probably respect that and not silently unblock bionic's reserved
signals for them. Otherwise, user code can deadlock, run out of stack,
etc. through no fault of their own, if one of the reserved signals
comes in while they've pivoted onto their signal stack.
Bug: http://b/122939726
Test: treehugger
Change-Id: I6425a3e7413edc16157b35dffe632e1ab1d76618
In addition to ORIGIN, LIB is now expanded to either lib or lib64
depending on the ABI in the DT_RUNPATH.
Test: To be added (can't be added to AOSP directly since there are
internal tests that have to be updated as well)
Change-Id: I34a8316f1765a27c731bc270ee25c0ce287d7eed
Given that the block allocator does not free allocated memory unless
all objects are freed, we can allocate a large chunk of memory at a
time. This gives us:
1. Higher space efficiency, especially when the objects being
allocated has a large size.
2. Much less mmap()/prctl() calls.
3. Much less vm_area_struct objects.
While this increases the size of virtual memory allocated, the number
of dirty pages would be unchanged or less.
Test result on a Go device:
- Zygote starts 24ms faster on average.
- Kernel vm_area_struct usage 138KB less, across the entire system.
Bug: 112073665
Test: Boot and check the result from 'showmap'.
Change-Id: Iece631a5081151dd1e5bf25dbddac9b264bcdcff
If all allocated memory from a block allocator is freed, it is a good
opportunity to purge all the pages allocated to reduce lingering dirty
pages.
Memory saving varies with the platform and what processes are running.
Measuring right after boot, this saves ~1.8MB on cuttelfish and ~1.3MB
on a 32-bit ARM device.
Bug: 112073665
Test: Boot and check memory usage with 'showmap'.
Change-Id: I53769e0ec9699f0b3645cdf281a2c0bbffb98676
Addressing Elliott's remaining comments on the android_mallopt change.
Intending to let this get merged in normally (should be clean).
Test: blueline-userdebug still builds.
Change-Id: I4f00191091b8af367f84d087432a5af5f83036ee
On user builds, heapprofd should only be allowed to profile apps that
are either debuggable, or profileable (according to the manifest). This
change exposes extra zygote-specific knowledge to bionic, and makes the
dedicated signal handler check for the special case of being in a zygote child.
With this & the corresponding framework change, we should now be
handling the 4 combinations of:
{java, native} x {profile_at_runtime, profile_at_startup}.
See internal go/heapprofd-java-trigger for further context.
Test: on-device unit tests (shared & static) on blueline-userdebug.
Test: flashed blueline-userdebug, confirmed that java profiling activates from startup and at runtime.
Bug: 120409382
Change-Id: Ic251afeca4324dc650ac1d4f46976b526eae692a
(cherry picked from commit 998792e2b6)
Merged-In: Ic251afeca4324dc650ac1d4f46976b526eae692a
