Use the real page size from getauxval() for memtag stack
MTE protection.
Bug: 296275298
Test: atest -c bionic-unit-tests
Change-Id: I1711291b918b09e5464f1d15358dd1ff7fa2f371
Signed-off-by: Kalesh Singh <kaleshsingh@google.com>
When I added %m to async_safe_* too, we never followed up and cleaned up
callers.
Test: treehugger
Change-Id: If81943c4c45de49f0fb4bc29cfbd3fc53d4a47fe
To enable experiments with non-4KiB page sizes, introduce
an inline page_size() function that will either return the runtime
page size (if PAGE_SIZE is not 4096) or a constant 4096 (elsewhere).
This should ensure that there are no changes to the generated code on
unaffected platforms.
Test: source build/envsetup.sh
lunch aosp_cf_arm64_16k_phone-userdebug
m -j32 installclean
m -j32
Test: launch_cvd \
-kernel_path /path/to/out/android14-5.15/dist/Image \
-initramfs_path /path/to/out/android14-5.15/dist/initramfs.img \
-userdata_format=ext4
Bug: 277272383
Bug: 230790254
Change-Id: Ic0ed98b67f7c6b845804b90a4e16649f2fc94028
Map all stacks (primary, thread, and sigaltstack) as PROT_MTE when the
binary requests it through the ELF note.
For the reference, the note is produced by the following toolchain changes:
https://reviews.llvm.org/D118948https://reviews.llvm.org/D119384https://reviews.llvm.org/D119381
Bug: b/174878242
Test: fvp_mini with ToT LLVM (more tests in a separate change)
Change-Id: I04a4e21c966e7309b47b1f549a2919958d93a872
This patch introduces GWP-ASan system properties and environment
variables to control the internal sampling rates of GWP-ASan. This can
be used for:
1. "Torture testing" the system, i.e. running it under an extremely
high sampling rate under GWP-ASan.
2. Increasing sampling remotely to allow further crash report
collection of rare issues.
There are three sets of system properites:
1. libc.debug.gwp_asan.*.system_default: Default values for native
executables and system apps.
2. libc.debug.gwp_asan.*.app_default: Default values for non-system
apps, and
3. libc.debug.gwp_asan.*.<basename/app_name>: Default values for an
individual app or native process.
There are three variables that can be changed:
1. The allocation sampling rate (default: 2500) - using the environment
variable GWP_ASAN_SAMPLE_RATE or the libc.debug.gwp_asan.sample_rate.*
system property.
2. The process sampling rate (default: 128 for system apps/processes, 1
for opted-in apps) - using the environment variable
GWP_ASAN_PROCESS_SAMPLING or the libc.debug.gwp_asan.process_sampling.*
system property,
3. The number of slots available (default: 32) - using the environment
variable GWP_ASAN_MAX_ALLOCS or the libc.debug.gwp_asan.max_allocs.*
system property.
If not specified, #3 will be calculated as a ratio of the default
|2500 SampleRate : 32 slots|. So, a sample rate of "1250" (i.e. twice as
frequent sampling) will result in a doubling of the max_allocs to "64".
Bug: 219651032
Test: atest bionic-unit-tests
Change-Id: Idb40a2a4d074e01ce3c4e635ad639a91a32d570f
Currently, the ELF note parsing in the loader is not permissive. This
patch relaxes the restrictions on the bits of the ELF note that could be
extended in the future.
This may allow more MTE options in apexes. If we add some extra metadata
bit (say, in bit 5) in the future, and then build MTE into every apex,
we don't want it to crash on Android 13 devices (we'd much rather it
just be a no-op).
Remove the (unused anywhere) NT_MEMTAG_DEFAULT and call it
NT_MEMTAG_NONE.
And finally, make the tests work on bionic-unit-tests-static. We
previously didn't deploy the test binaries, so add them as a data
dependency.
Bug: N/A
Test: atest bionic-unit-tests-static
Change-Id: I13530faad55c719c6eb848297f8ce378e18afbfc
Normally, platform-specific note types in the toolchain are prefixed
with the platform name. Because we're exposing the NT_TYPE_MEMTAG and
synthesizing the note in the toolchain in an upcoming patch
(https://reviews.llvm.org/D118948), it's been requested that we change
the name to include the platform prefix.
While NT_TYPE_IDENT and NT_TYPE_KUSER aren't known about or synthesized
by the toolchain, update those references as well for consistency.
Bug: N/A
Test: Build Android
Change-Id: I7742e4917ae275d59d7984991664ea48028053a1
Strictly this still isn't quite the same, because they won't actually be
profiled, but at least they won't *crash* now if they're sent a
profiling signal.
Bug: http://b/201497662
Test: treehugger
Change-Id: I0728492eed77584cd850d28897056996387e6671
On devices where the performance of ASYNC mode is similar to SYNC
mode on certain CPUs, OEMs may choose to configure the kernel to
prefer SYNC mode on those CPUs by writing the value "sync" to the
sysfs node: /sys/devices/system/cpu/cpu<N>/mte_tcf_preferred
The kernel will only respect the per-CPU preference if the user program
allows this by specifying the preferred mode as a member of a set of
allowed modes. Since only kernels with r.android.com/1754670 support
specifying multiple modes, fall back to trying to specify a single
mode if that doesn't work.
Bug: 189966263
Change-Id: Ie7ada3b073178b7967f0819cbdadc2d8e3a2c648
This works around buggy applications that read a few bytes past the
end of their allocation, which would otherwise cause a segfault with
the concurrent Scudo change that aligns large allocations to the right.
Because the implementation of
android_set_application_target_sdk_version() lives in the linker,
we need to introduce a hook so that libc is notified when the target
SDK version changes.
Bug: 181344545
Change-Id: Id4be6645b94fad3f64ae48afd16c0154f1de448f
These two options allow for ARM MTE to be enabled dynamically (instead
of at compile time via the ELF note). They are settable from a non-root
shell to allow device owners to test system binaries with MTE.
The following values may be set:
1. 'off' -> No MTE, uses TBI on compatible devices.
2. 'sync' -> SYNC MTE.
3. 'async' -> ASYNC MTE.
The following methods can be used to launch a process (sanitizer-status)
with configurable values:
1. adb shell MEMTAG_OPTIONS=async sanitizer-status
2. adb shell setprop arm64.memtag.process.sanitizer-status async && \
adb shell sanitizer-status
Note: The system server will require some special handing in the zygote
pre-fork to check the sysprops. The zygote should always have the ELF
note. TODO in a subsequent patch.
Bug: 135772972
Bug: 172365548
Test: Launching sanitizer-status above using both the settings.
Change-Id: Ic1dbf3985a3f23521ec86725ec482c8f6739c182
Use a note in executables to specify
(none|sync|async) heap tagging level. To be extended with (heap x stack x
globals) in the future. A missing note disables all tagging.
Bug: b/135772972
Test: bionic-unit-tests (in a future change)
Change-Id: Iab145a922c7abe24cdce17323f9e0c1063cc1321
A constructor or destructor function with an integral priority is
placed in an .init_array or .fini_array section with the priority
suffixed to the section name:
- __attribute__((constructor)) ==> .init_array
- __attribute__((constructor(42))) ==> .init_array.42
The suffixed init/fini sections appear before the unsuffixed sections,
so the prioritized functions appeared before the __{INIT,FINI}_ARRAY__
symbols and were dropped when the symbols were used.
The (static) linker doesn't recognize priority suffixes on
.preinit_array.
This bug affected .init_array and .fini_array for static executables.
For dynamic executables, only .fini_array was affected, because
.init_array is handled by the dynamic loader instead, which uses
DT_INIT_ARRAY[SZ]. For DSOs, neither is affected, because the two
sections are only handled by the dynamic loader.
This patch also fixes a minor inconsistency where dynamic init/preinit
were passed argc/argv/envp, but static were not.
Bug: http://b/170983066
Test: bionic-unit-tests
Change-Id: I0fffa776e5d9bdb6f8af06b4c1af148236742fef
(Based on proposal at https://sourceware.org/glibc/wiki/ThreadPropertiesAPI)
This includes API to:
- locate static and dynamic TLS
- register thread-exit and dynamic TLS creation/destruction callbacks
Change-Id: Icd9d29a5b2f47495395645e19d3b2c96826f19c8
pthread_atfork may call malloc() during its once-init. This causes
problems with allocators (GWP-ASan) that require explicit initialisation
before calls to malloc().
Bug: 135634846
Test: atest bionic
Change-Id: I1810a00465db99d5aa34fa6f74dea5908a628d3a
Using ifuncs allows the linker to select faster versions of libc functions
like strcmp, making linking faster.
The linker continues to first initialize TLS, then call the ifunc
resolvers. There are small amounts of code in Bionic that need to avoid
calling functions selected using ifuncs (generally string.h APIs). I've
tried to compile those pieces with -ffreestanding. Maybe it's unnecessary,
but maybe it could help avoid compiler-inserted memset calls, and maybe
it will be useful later on.
The ifuncs are called in a special early pass using special
__rel[a]_iplt_start / __rel[a]_iplt_end symbols. The linker will encounter
the ifuncs again as R_*_IRELATIVE dynamic relocations, so they're skipped
on the second pass.
Break linker_main.cpp into its own liblinker_main library so it can be
compiled with -ffreestanding.
On walleye, this change fixes a recent 2.3% linker64 start-up time
regression (156.6ms -> 160.2ms), but it also helps the 32-bit time by
about 1.9% on the same benchmark. I'm measuring the run-time using a
synthetic benchmark based on loading libandroid_servers.so.
Test: bionic unit tests, manual benchmarking
Bug: none
Merged-In: Ieb9446c2df13a66fc0d377596756becad0af6995
Change-Id: Ieb9446c2df13a66fc0d377596756becad0af6995
(cherry picked from commit 772bcbb0c2)
In order for an ifunc resolver to detect the presence of certain CPU features,
access to getauxval(AT_HWCAP) or getauxval(AT_HWCAP2) may be required. In order
for getauxval() to work, it needs to access the pointer to the auxiliary vector
stored by the linker in the libc shared globals data structure. Accessing the
shared globals requires libc to call the __libc_shared_globals() function
exported by the linker. However, in order to call this function, libc must
be fully relocated, which is not guaranteed to be the case at the point when
ifunc resolvers are called.
glibc solves this problem by passing the values of getauxval(AT_HWCAP)
(and getauxval(AT_HWCAP2) on aarch64) as arguments to the ifunc resolver.
Since this seems to be not only the most straightforward way to solve the
problem but also improves our compatibility with glibc, we adopt their
calling convention.
This change is ABI compatible with old resolvers because the arguments are
passed in registers, so the old resolvers will simply ignore the new arguments.
Bug: 135772972
Change-Id: Ie65bd6e7067f0c878df3d348c815fda61dc12de2
__hwasan_init() was segfaulting when called from here because it
was calling into libc functions which required more of libc to be
initialized. Instead, call __hwasan_init_static(), which does a
minimal amount of initialization for statically linked executables,
just enough that we can run instrumented code. __hwasan_init() itself
will end up being called later (most likely from a global ctor)
after libc is fully initialized.
We'll need to wait for LLVM r352816+r352823 to land in our toolchain
before landing this.
Change-Id: I12ffc7e08f6dd161e4ff2088f8d56265af7baedf
Gold isn't emitting these symbols, so we don't necessarily have the
support for them (gold is still the default for most architectures in
the NDK).
Test: bionic static unit tests
Bug: None
Change-Id: Ifc360cb6c26571fb3f0309adb0faf0af7ee5b36f
Initialize a thread's DTV to an empty zeroed DTV. Allocate the DTV and
any ELF module's TLS segment on-demand in __tls_get_addr. Use a generation
counter, incremented in the linker, to signal when threads should
update/reallocate their DTV objects.
A generation count of 0 always indicates the constant zero DTV.
Once a DTV is allocated, it isn't freed until the thread exits, because
a signal handler could interrupt the fast path of __tls_get_addr between
accessing the DTV slot and reading a field of the DTV. Bionic keeps a
linked list of DTV objects so it can free them at thread-exit.
Dynamic TLS memory is allocated using a BionicAllocator instance in
libc_shared_globals. For async-signal safety, access to the
linker/libc-shared state is protected by first blocking signals, then by
acquiring the reader-writer lock, TlsModules::rwlock. A write lock is
needed to allocate or free memory.
In pthread_exit, unconditionally block signals before freeing dynamic
TLS memory or freeing the shadow call stack.
ndk_cruft.cpp: Avoid including pthread_internal.h inside an extern "C".
(The header now includes a C++ template that doesn't compile inside
extern "C".)
Bug: http://b/78026329
Bug: http://b/123094171
Test: bionic unit tests
Change-Id: I3c9b12921c9e68b33dcc1d1dd276bff364eff5d7
A static executable is almost entirely statically relocated by the
linker, with the exception of IRELATIVE relocations, which must be
resolved by libc by enumerating the relocations using the special
linker-defined symbols __rela?_iplt_{start,end}. This patch implements
ifunc support by enumerating the relocations in this way.
Bug: 112482891
Test: /data/nativetest{,64}/bionic-unit-tests-static/bionic-unit-tests-static on walleye_hwasan-userdebug
Change-Id: Ia5522a190da0b86e095b141d5d4e68dd7dd4b695
If the alignment of a TLS segment in a shared object is invalid, return
an error through dlerror() rather than aborting the process.
Bug: http://b/78026329
Test: bionic unit tests
Change-Id: I60e589ddd8ca897f485d55af089f08bd3ff5b1fa
Replace reserve_tcb with reserve_exe_segment_and_tcb, which lays out both
the TCB and the executable's TLS segment, accounting for the difference in
layout between variant 1 and variant 2 targets.
The function isn't actually called with a non-null TlsSegment* yet.
Bug: http://b/78026329
Test: bionic unit tests
Change-Id: Ibd6238577423a7d0451f36da7e64912046959796
For ELF TLS "local-exec" accesses, the static linker assumes that an
executable's TLS segment is located at a statically-known offset from the
thread pointer (i.e. "variant 1" for ARM and "variant 2" for x86).
Because these layouts are incompatible, Bionic generally needs to allocate
its TLS slots differently between different architectures.
To allow per-architecture TLS slots:
- Replace the TLS_SLOT_xxx enumerators with macros. New ARM slots are
generally negative, while new x86 slots are generally positive.
- Define a bionic_tcb struct that provides two things:
- a void* raw_slots_storage[BIONIC_TLS_SLOTS] field
- an inline accessor function: void*& tls_slot(size_t tpindex);
For ELF TLS, it's necessary to allocate a temporary TCB (i.e. TLS slots),
because the runtime linker doesn't know how large the static TLS area is
until after it has loaded all of the initial solibs.
To accommodate Golang, it's necessary to allocate the pthread keys at a
fixed, small, positive offset from the thread pointer.
This CL moves the pthread keys into bionic_tls, then allocates a single
mapping per thread that looks like so:
- stack guard
- stack [omitted for main thread and with pthread_attr_setstack]
- static TLS:
- bionic_tcb [exec TLS will either precede or succeed the TCB]
- bionic_tls [prefixed by the pthread keys]
- [solib TLS segments will be placed here]
- guard page
As before, if the new mapping includes a stack, the pthread_internal_t
is allocated on it.
At startup, Bionic allocates a temporary bionic_tcb object on the stack,
then allocates a temporary bionic_tls object using mmap. This mmap is
delayed because the linker can't currently call async_safe_fatal() before
relocating itself.
Later, Bionic allocates a stack-less thread mapping for the main thread,
and copies slots from the temporary TCB to the new TCB.
(See *::copy_from_bootstrap methods.)
Bug: http://b/78026329
Test: bionic unit tests
Test: verify that a Golang app still works
Test: verify that a Golang app crashes if bionic_{tls,tcb} are swapped
Merged-In: I6543063752f4ec8ef6dc9c7f2a06ce2a18fc5af3
Change-Id: I6543063752f4ec8ef6dc9c7f2a06ce2a18fc5af3
(cherry picked from commit 1e660b70da)
Having a constexpr constructor should guarantee that the static
`globals` variable in __libc_shared_globals is initialized statically
(as opposed to dynamically), which is important because
__libc_shared_globals is called very early (before the linker has
relocated itself). With the constructor, though, the fields can safely
have in-line default initializers.
Bug: none
Test: bionic unit tests
Change-Id: Icde821557369625734a4d85d7ff55428bad5c247
Instead of passing the address of a KernelArgumentBlock to libc.so for
initialization, use __loader_shared_globals() to initialize globals.
Most of the work happened in the previous CLs. This CL switches a few
KernelArgumentBlock::getauxval calls to [__bionic_]getauxval and stops
routing the KernelArgumentBlock address through the libc init functions.
Bug: none
Test: bionic unit tests
Change-Id: I96c7b02c21d55c454558b7a5a9243c682782f2dd
Merged-In: I96c7b02c21d55c454558b7a5a9243c682782f2dd
(cherry picked from commit 746ad15912)
Split __libc_init_main_thread into __libc_init_main_thread_early and
__libc_init_main_thread_late. The early function is called very early in
the startup of the dynamic linker and static executables. It initializes
the global auxv pointer and enough TLS memory to do system calls, access
errno, and run -fstack-protector code (but with a zero cookie because the
code for generating a cookie is complex).
After the linker is relocated, __libc_init_main_thread_late finishes
thread initialization.
Bug: none
Test: bionic unit tests
Change-Id: I6fcd8d7587a380f8bd649c817b40a3a6cc1d2ee0
Merged-In: I6fcd8d7587a380f8bd649c817b40a3a6cc1d2ee0
(cherry picked from commit 39bc44bb0e)
Initialize the __progname and environ global variables using
libc_shared_globals rather than KernelArgumentBlock.
Also: suppose the linker is invoked on an executable:
linker prog [args...]
The first argument passed to main() and constructor functions is "prog"
rather than "linker". For consistency, this CL changes the BSD
__progname global from "linker" to "prog".
Bug: none
Test: bionic unit tests
Change-Id: I376d76953c9436706dbc53911ef6585c1acc1c31
__libc_shared_globals() is available in dynamic modules as soon as
relocation has finished (i.e. after ifuncs run). Before ifuncs have run,
the android_set_abort_message() function already doesn't work because it
calls public APIs via the PLT. (If this matters, we can use a static
bool variable to enable android_set_abort_message after libc
initialization).
__libc_shared_globals() is hidden, so it's available in the linker
immediately (i.e. before relocation). TLS memory (e.g. errno) currently
isn't accessible until after relocation, but a later patch fixes that.
Bug: none
Test: bionic unit tests
Change-Id: Ied4433758ed2da9ee404c6158e319cf502d05a53
Previously, the address of the global variable was communicated from the
dynamic linker to libc.so using a field of KernelArgumentBlock, which is
communicated using the TLS_SLOT_BIONIC_PREINIT slot.
As long as this function isn't called during relocations (i.e. while
executing an ifunc), it always return a non-NULL value. If it's called
before its PLT entry is relocated, I expect a crash.
I removed the __libc_init_shared_globals function. It's currently empty,
and I don't think there's one point in libc's initialization where
shared globals should be initialized.
Bug: http://b/25751302
Test: bionic unit tests
Change-Id: I614d25e7ef5e0d2ccc40d5c821dee10f1ec61c2e
Also move this and android_get_device_api_level into <android/api-level.h>
so that they're always available.
This involves cleaning up <sys/cdefs.h> slightly.
Bug: N/A
Test: builds
Change-Id: I25435c55f3549cd0d827a7581bee75ea8228028b
Make sure that TLS_SLOT_TSAN is always available and correctly set up in
HWASan-instrumented functions by setting up the tls register and running hwasan
initialization (__hwasan_init in the main thread and __hwasan_thread_enter in
secondary) early enough.
This is needed to accomodate a change in HWASan: https://reviews.llvm.org/D52249
Bug: 112438058
Test: boot with SANITIZE_TARGET=hwaddress, run bionic-tests
Change-Id: Icd909a4ea0da6c6c1095522bcc28debef5f2c63d
* Allow sanitization of libc (excluding existing global sanitizers)
and disallow sanitization of linker. The latter has not been
necessary before because HWASan is the first sanitizer to support
static binaries (with the exception of CFI, which is not used
globally).
* Static binary startup: initialize HWASan shadow very early so that
almost entire libc can be sanitized. The rest of initialization is
done in a global constructor; until that is done sanitized code can
run but can't report errors (will simply crash with SIGTRAP).
* Switch malloc_common from je_* to __sanitizer_*.
* Call hwasan functions when entering and leaving threads. We can not
intercept pthread_create when libc depends on libclang_rt.hwasan.
An alternative to this would be a callback interface like requested
here:
https://sourceware.org/glibc/wiki/ThreadPropertiesAPI
All of the above is behind a compile-time check
__has_feature(hwaddress_sanitizer). This means that HWASan actually
requires libc to be instrumented, and would not work otherwise. It's
an implementation choice that greatly reduces complexity of the tool.
Instrumented libc also guarantees that hwasan is present and
initialized in every process, which allows piecemeal sanitization
(i.e. library w/o main executable, or even individual static
libraries), unlike ASan.
Change-Id: If44c46b79b15049d1745ba46ec910ae4f355d19c
Add two functions to allow objects that own a file descriptor to
enforce that only they can close their file descriptor.
Use them in FILE* and DIR*.
Bug: http://b/110100358
Test: bionic_unit_tests
Test: aosp/master boots without errors
Test: treehugger
Change-Id: Iecd6e8b26c62217271e0822dc3d2d7888b091a45
