* Bionic benchmarks results at the bottom
* This is a squash of the following commits:
libc: ARM64: optimize memset.
This is an optimized memset for AArch64. Memset is split into 4 main
cases: small sets of up to 16 bytes, medium of 16..96 bytes which are
fully unrolled. Large memsets of more than 96 bytes align the
destination and use an unrolled loop processing 64 bytes per
iteration. Memsets of zero of more than 256 use the dc zva
instruction, and there are faster versions for the common ZVA sizes 64
or 128. STP of Q registers is used to reduce codesize without loss of
performance.
Change-Id: I0c5b5ec5ab8a1fd0f23eee8fbacada0be08e841f
libc: ARM64: improve performance in strlen
Change-Id: Ic20f93a0052a49bd76cd6795f51e8606ccfbf11c
libc: ARM64: Optimize memcpy.
This is an optimized memcpy for AArch64. Copies are split into 3 main
cases: small copies of up to 16 bytes, medium copies of 17..96 bytes
which are fully unrolled. Large copies of more than 96 bytes align
the destination and use an unrolled loop processing 64 bytes per
iteration. In order to share code with memmove, small and medium
copies read all data before writing, allowing any kind of overlap. On
a random copy test memcpy is 40.8% faster on A57 and 28.4% on A53.
Change-Id: Ibb9483e45bbc0e8ca3d5ce98a31c55dfd8a5ac28
libc: AArch64: Tune memcpy
* Further tuning for performance.
Change-Id: Id08eaab885f9743fa7575077924a947c1b88e4ff
libc: ARM64: optimize memmove for Cortex-A53
* Sadly does not work on Denver or Kryo, so can't go to generic
This is an optimized memmove for AArch64. All copies of up to 96
bytes and all backward copies are done by the new memcpy. The only
remaining case is large forward copies which are done in the same way
as the memcpy loop, but copying from the end rather than the start.
Tested on the Nextbit Robin with MSM8992 (Snapdragon 808):
Before
BM_string_memcmp/8 1000k 27 0.286 GiB/s
BM_string_memcmp/64 50M 20 3.053 GiB/s
BM_string_memcmp/512 20M 126 4.060 GiB/s
BM_string_memcmp/1024 10M 234 4.372 GiB/s
BM_string_memcmp/8Ki 1000k 1726 4.745 GiB/s
BM_string_memcmp/16Ki 500k 3711 4.415 GiB/s
BM_string_memcmp/32Ki 200k 8276 3.959 GiB/s
BM_string_memcmp/64Ki 100k 16351 4.008 GiB/s
BM_string_memcpy/8 1000k 13 0.612 GiB/s
BM_string_memcpy/64 1000k 8 7.187 GiB/s
BM_string_memcpy/512 50M 38 13.311 GiB/s
BM_string_memcpy/1024 20M 86 11.858 GiB/s
BM_string_memcpy/8Ki 5M 620 13.203 GiB/s
BM_string_memcpy/16Ki 1000k 1265 12.950 GiB/s
BM_string_memcpy/32Ki 500k 2977 11.004 GiB/s
BM_string_memcpy/64Ki 500k 8003 8.188 GiB/s
BM_string_memmove/8 1000k 11 0.684 GiB/s
BM_string_memmove/64 1000k 16 3.855 GiB/s
BM_string_memmove/512 50M 57 8.915 GiB/s
BM_string_memmove/1024 20M 117 8.720 GiB/s
BM_string_memmove/8Ki 2M 853 9.594 GiB/s
BM_string_memmove/16Ki 1000k 1731 9.462 GiB/s
BM_string_memmove/32Ki 500k 3566 9.189 GiB/s
BM_string_memmove/64Ki 500k 7708 8.501 GiB/s
BM_string_memset/8 1000k 16 0.487 GiB/s
BM_string_memset/64 1000k 16 3.995 GiB/s
BM_string_memset/512 50M 37 13.489 GiB/s
BM_string_memset/1024 50M 58 17.405 GiB/s
BM_string_memset/8Ki 5M 451 18.160 GiB/s
BM_string_memset/16Ki 2M 883 18.554 GiB/s
BM_string_memset/32Ki 1000k 2181 15.022 GiB/s
BM_string_memset/64Ki 500k 4563 14.362 GiB/s
BM_string_strlen/8 1000k 8 0.965 GiB/s
BM_string_strlen/64 1000k 16 3.855 GiB/s
BM_string_strlen/512 20M 92 5.540 GiB/s
BM_string_strlen/1024 10M 167 6.111 GiB/s
BM_string_strlen/8Ki 1000k 1237 6.620 GiB/s
BM_string_strlen/16Ki 1000k 2765 5.923 GiB/s
BM_string_strlen/32Ki 500k 6135 5.341 GiB/s
BM_string_strlen/64Ki 200k 13168 4.977 GiB/s
After
BM_string_memcmp/8 1000k 21 0.369 GiB/s
BM_string_memcmp/64 1000k 28 2.272 GiB/s
BM_string_memcmp/512 20M 128 3.983 GiB/s
BM_string_memcmp/1024 10M 234 4.375 GiB/s
BM_string_memcmp/8Ki 1000k 1732 4.728 GiB/s
BM_string_memcmp/16Ki 500k 3485 4.701 GiB/s
BM_string_memcmp/32Ki 500k 7031 4.660 GiB/s
BM_string_memcmp/64Ki 200k 14296 4.584 GiB/s
BM_string_memcpy/8 1000k 5 1.458 GiB/s
BM_string_memcpy/64 1000k 7 8.952 GiB/s
BM_string_memcpy/512 50M 36 13.907 GiB/s
BM_string_memcpy/1024 20M 80 12.750 GiB/s
BM_string_memcpy/8Ki 5M 572 14.307 GiB/s
BM_string_memcpy/16Ki 1000k 1165 14.053 GiB/s
BM_string_memcpy/32Ki 500k 3141 10.430 GiB/s
BM_string_memcpy/64Ki 500k 7008 9.351 GiB/s
BM_string_memmove/8 50M 7 1.074 GiB/s
BM_string_memmove/64 1000k 9 6.593 GiB/s
BM_string_memmove/512 50M 37 13.502 GiB/s
BM_string_memmove/1024 20M 80 12.656 GiB/s
BM_string_memmove/8Ki 5M 573 14.281 GiB/s
BM_string_memmove/16Ki 1000k 1168 14.018 GiB/s
BM_string_memmove/32Ki 1000k 2825 11.599 GiB/s
BM_string_memmove/64Ki 500k 6548 10.008 GiB/s
BM_string_memset/8 1000k 7 1.038 GiB/s
BM_string_memset/64 1000k 8 7.151 GiB/s
BM_string_memset/512 1000k 29 17.272 GiB/s
BM_string_memset/1024 50M 53 18.969 GiB/s
BM_string_memset/8Ki 5M 424 19.300 GiB/s
BM_string_memset/16Ki 2M 846 19.350 GiB/s
BM_string_memset/32Ki 1000k 2028 16.156 GiB/s
BM_string_memset/64Ki 500k 4514 14.517 GiB/s
BM_string_strlen/8 1000k 7 1.120 GiB/s
BM_string_strlen/64 1000k 16 3.918 GiB/s
BM_string_strlen/512 50M 64 7.894 GiB/s
BM_string_strlen/1024 20M 104 9.815 GiB/s
BM_string_strlen/8Ki 5M 664 12.337 GiB/s
BM_string_strlen/16Ki 1000k 1291 12.682 GiB/s
BM_string_strlen/32Ki 1000k 2940 11.143 GiB/s
BM_string_strlen/64Ki 500k 6440 10.175 GiB/s
Change-Id: I635bd2798a755256f748b2af19b1a56fb85a40c6
These directories all have Android.bp files that are always used now,
delete the Android.mk files.
Change-Id: Ib0ba2d28bff88483b505426ba61606da314e03ab
It turns out that at least the Nexus 9 kernel is built without CONFIG_QUOTA.
If we decide we're going to mandate quota functionality, I'm happy for us to
be a part of CTS that ensures that happens, but I don't want to be first, so
there's not much to test here other than "will it compile?". The strace
output looks right though.
Bug: http://b/27948821
Bug: http://b/27952303
Change-Id: If667195eee849ed17c8fa9110f6b02907fc8fc04
{get,set}domainname aren't in POSIX but are widely-implemented
extensions.
The Linux kernel provides a setdomainname syscall but not a symmetric
getdomainname syscall, since it expects userspace to get the domain name
from uname(2).
Change-Id: I96726c242f4bb646c130b361688328b0b97269a0
Signed-off-by: Greg Hackmann <ghackmann@google.com>
This involves actually implementing assembler __memset_chk for arm64,
but that's easily done.
Obviously I'd like this for all architectures (and all the string functions),
but this is low-hanging fruit...
Change-Id: I70ec48c91aafd1f0feb974a2555c51611de9ef82
Our FORTIFY _chk functions' implementations were very repetitive and verbose
but not very helpful. We'd also screwed up and put the SSIZE_MAX checks where
they would never fire unless you actually had a buffer as large as half your
address space, which probably doesn't happen very often.
Factor out the duplication and take the opportunity to actually show details
like how big the overrun buffer was, or by how much it was overrun.
Also remove the obsolete FORTIFY event logging.
Also remove the unused __libc_fatal_no_abort.
This change doesn't improve the diagnostics from the optimized assembler
implementations.
Change-Id: I176a90701395404d50975b547a00bd2c654e1252
Exactly which functions get a stack protector is up to the compiler, so
let's separate the code that sets up the environment stack protection
requires and explicitly build it with -fno-stack-protector.
Bug: http://b/26276517
Change-Id: I8719e23ead1f1e81715c32c1335da868f68369b5
The mremap definition was incorrect (unsigned long instead of int) and
it was missing the optional new_address parameter.
Change-Id: Ib9d0675aaa098c21617cedc9b2b8cf267be3aec4
This moves the generic arm/arm64/x86 settings into the main makefiles
and makes the rest of them derivatives. This better aligns with how
soong handles arch/cpu variants.
Also updates the Android.bp to make it consistent with the make
versions.
Change-Id: I5a0275d992bc657459eb6fe1697ad2336731d122
A continuation of commit 2825f10b7f.
Add O_PATH compatibility support for flistxattr(). This allows
a process to list out all the extended attributes associated with
O_PATH file descriptors.
Change-Id: Ie2285ac7ad2e4eac427ddba6c2d182d41b130f75
Support O_PATH file descriptors when handling fgetxattr and fsetxattr.
This avoids requiring file read access to pull extended attributes.
This is needed to support O_PATH file descriptors when calling
SELinux's fgetfilecon() call. In particular, this allows the querying
and setting of SELinux file context by using something like the following
code:
int dirfd = open("/path/to/dir", O_DIRECTORY);
int fd = openat(dirfd, "file", O_PATH | O_NOFOLLOW);
char *context;
fgetfilecon(fd, &context);
This change was motivated by a comment in
https://android-review.googlesource.com/#/c/152680/1/toys/posix/ls.c
Change-Id: Ic0cdf9f9dd0e35a63b44a4c4a08400020041eddf
All arch-arm and arch-arm64 .S files were compiled
by gcc with and without this patch. The output object files
were identical. When compiled with llvm and this patch,
the output files were also identical to gcc's output.
BUG: 18061004
Change-Id: I458914d512ddf5496e4eb3d288bf032cd526d32b
The visibility control in pthread_atfork.h is incorrect.
It breaks 64bit libc.so by hiding pthread_atfork.
This reverts commit 6df122f852.
Change-Id: I21e4b344d500c6f6de0ccb7420b916c4e233dd34
This doesn't affect code like Chrome that correctly ignores EINTR on
close, makes code that tries TEMP_FAILURE_RETRY work (where before it might
have closed a different fd and appeared to succeed, or had a bogus EBADF),
and makes "goto fail" code work (instead of mistakenly assuming that EINTR
means that the close failed).
Who loses? Anyone actively trying to detect that they caught a signal while
in close(2). I don't think those people exist, and I think they have better
alternatives available.
Bug: https://code.google.com/p/chromium/issues/detail?id=269623
Bug: http://b/20501816
Change-Id: I11e2f66532fe5d1b0082b2433212e24bdda8219b
Apparently clang really doesn't want you to take the address of a builtin.
Since this is only a temporary hack, let's just shrug and accept that
clang-built volantis images won't work until we have new NVIDIA blobs.
Bug: http://b/20065774
Change-Id: I4c8e893b15a1af8f9c54d3f89bfef112b63d09b4
NVIDIA binary blobs are assuming that __cache_clear, _Unwind_Backtrace,
and _Unwind_GetIP are all in some library that they link, but now we've
cleaned up this leakage, they're no longer getting it. Deliberately leak
the symbols from libc.so until we get new blobs.
Bug: http://b/20065774
Change-Id: I92ef07b2bce8d1ad719bf40dab41d745cd6904d4
The kernel system call faccessat() does not have any flags arguments,
so passing flags to the kernel is currently ignored.
Fix the kernel system call so that no flags argument is passed in.
Ensure that we don't support AT_SYMLINK_NOFOLLOW. This non-POSIX
(http://pubs.opengroup.org/onlinepubs/9699919799/functions/access.html)
flag is a glibc extension, and has non-intuitive, error prone behavior.
For example, consider the following code:
symlink("foo.is.dangling", "foo");
if (faccessat(AT_FDCWD, "foo", R_OK, AT_SYMLINK_NOFOLLOW) == 0) {
int fd = openat(AT_FDCWD, "foo", O_RDONLY | O_NOFOLLOW);
}
The faccessat() call in glibc will return true, but an attempt to
open the dangling symlink will end up failing. GLIBC documents this
as returning the access mode of the symlink itself, which will
always return true for any symlink on Linux.
Some further discussions of this are at:
* http://lists.landley.net/pipermail/toybox-landley.net/2014-September/003617.html
* http://permalink.gmane.org/gmane.linux.lib.musl.general/6952
AT_SYMLINK_NOFOLLOW seems broken by design. I suspect this is why this
function was never added to POSIX. (note that "access" is pretty much
broken by design too, since it introduces a race condition between
check and action). We shouldn't support this until it's clearly
documented by POSIX or we can have it produce intuitive results.
Don't support AT_EACCESS for now. Implementing it is complicated, and
pretty much useless on Android, since we don't have setuid binaries.
See http://git.musl-libc.org/cgit/musl/commit/?id=0a05eace163cee9b08571d2ff9d90f5e82d9c228
for how an implementation might look.
Bug: 18867827
Change-Id: I25b86c5020f3152ffa3ac3047f6c4152908d0e04
The overflow's actually in the generic C implementation of memchr.
While I'm here, let's switch our generic memrchr to the OpenBSD version too.
Bug: https://code.google.com/p/android/issues/detail?id=147048
Change-Id: I296ae06a1ee196d2c77c95a22f11ee4d658962da
* changes:
Use LOCAL_LDFLAGS_64 instead of enumerating 64-bit architectures
Fix typo in cpu variant makefile depenendency for arm64
Remove libc_static_common_src_files
Share LP32 makefile settings between arches
In https://android-review.googlesource.com/#/c/127908/5/libc/SYSCALLS.TXT@116
Elliott said:
for LP64 these will be hidden. for LP32 we were cowards and left
them all public for compatibility (though i don't think we ever
dremeled to see whether it was needed). we don't have an easy
way to recognize additions, though, so we can't prevent adding
new turds.
Add a mechanism to prevent the adding of new turds, and use that
mechanism on the fchmod/fchmodat system calls.
Bug: 19233951
Change-Id: I98f98345970b631a379f348df57858f9fc3d57c0
Many libc functions have an option to not follow symbolic
links. This is useful to avoid security sensitive code
from inadvertantly following attacker supplied symlinks
and taking inappropriate action on files it shouldn't.
For example, open() has O_NOFOLLOW, chown() has
lchown(), stat() has lstat(), etc.
There is no such equivalent function for chmod(), such as lchmod().
To address this, POSIX introduced fchmodat(AT_SYMLINK_NOFOLLOW),
which is intended to provide a way to perform a chmod operation
which doesn't follow symlinks.
Currently, the Linux kernel doesn't implement AT_SYMLINK_NOFOLLOW.
In GLIBC, attempting to use the AT_SYMLINK_NOFOLLOW flag causes
fchmodat to return ENOTSUP. Details are in "man fchmodat".
Bionic currently differs from GLIBC in that AT_SYMLINK_NOFOLLOW
is silently ignored and treated as if the flag wasn't present.
This patch provides a userspace implementation of
AT_SYMLINK_NOFOLLOW for bionic. Using open(O_PATH | O_NOFOLLOW),
we can provide a way to atomically change the permissions on
files without worrying about race conditions.
As part of this change, we add support for fchmod on O_PATH
file descriptors, because it's relatively straight forward
and could be useful in the future.
The basic idea behind this implementation comes from
https://sourceware.org/bugzilla/show_bug.cgi?id=14578 , specifically
comment #10.
Change-Id: I1eba0cdb2c509d9193ceecf28f13118188a3cfa7
Specifically, use the argument to sigsetjmp as a flag in the buffer
to indicate whether or not the signal mask is valid.
Bug: 16918359
Change-Id: I5bb1f1220f14c105c6bc57e0c28c1dc366d1438f
Note that this doesn't address the fact that we don't save/restore the
real-time signals. But it does let us pass the tests we currently fail.
Bug: 16918359
Change-Id: I063a6926164289a71026a412da7f5dd2ca9a74b3
They'd drifted slightly which led to a compilation error in toybox,
which was assuming pid_t was defined. arm and arm64 were picking it
up via <endian.h> but x86 wasn't.
Change-Id: I58401e6c0066959dfc3b305b020876aaf7074bbf
Add the missing prototypes, fix the existing prototypes to use clockid_t
rather than int, fix clock_nanosleep's failure behavior, and add simple
tests.
Bug: 17644443
Bug: https://code.google.com/p/android/issues/detail?id=77372
Change-Id: I03fba369939403918abcabae9551a7123953d780
Signed-off-by: Haruki Hasegawa <h6a.h4i.0@gmail.com>
gdb was happy with what we had, but libgcc and libunwind weren't.
libgcc is happy with the kernel's restorer (because of the extra nop),
though libunwind looks like it's going to need code changes regardless.
We could make our restorer more like the kernel's one, but why bother
when we can just let the kernel supply the canonical one?
Bug: 17436734
Change-Id: I330fa5e68f23b1cf8133aa552896657b0b873ed3
* LP32 should use sa_restorer too. gdb expects this, and future (>= 3.15) x86
kernels will apparently stop supporting the case where SA_RESTORER isn't
set.
* gdb and libunwind care about the exact instruction sequences, so we need to
modify the code slightly in a few cases to match what they're looking for.
* gdb also cares about the exact function names (for some architectures),
so we need to use __restore and __restore_rt rather than __sigreturn and
__rt_sigreturn.
* It's possible that we don't have a VDSO; dl_iterate_phdr shouldn't assume
that getauxval(AT_SYSINFO_EHDR) will return a non-null pointer.
This fixes unwinding through a signal handler in gdb for all architectures.
It doesn't fix libunwind for arm and arm64. I'll keep investigating that...
Bug: 17436734
Change-Id: Ic1ea1184db6655c5d96180dc07bcc09628e647cb
There are number of changes in the way IFUNC related relocations are done:
1. IRELATIVE relocations are now supported for x86/x86_64 and arm64.
2. IFUNC relocations are now relying on static linker to generate
them in correct order - this removes necessety of additional
relocation pass for ifuncs.
3. Related to 2: rela?.dyn relocations are preformed before .plt ones.
4. Ifunc are resolved on symbol lookup this approach allowed to avoid
mprotect(PROT_WRITE) call on r-x program segments.
Bug: 17399706
Bug: 17177284
Change-Id: I414dd3e82bd47cc03442c5dfc7c279949aec51ed
The use of the .hidden directive to avoid going via the PLT for
__set_errno had the side-effect of actually making __set_errno
hidden (which is odd because assembler directives don't usually
affect symbols defined in a different file --- you can't even
create a weak reference to a symbol that's defined in a different
file).
This change switches the system call stubs over to a new always-hidden
__set_errno_internal and has a visible __set_errno on LP32 just for
binary compatibility with old NDK apps.
(cherry-pick of 7efad83d430f4d824f2aaa75edea5106f6ff8aae.)
Bug: 17423135
Change-Id: I6b6d7a05dda85f923d22e5ffd169a91e23499b7b
On most architectures the kernel subtracts a random offset to the stack
pointer in create_elf_tables by calling arch_align_stack before writing
the auxval table and so on. On all but x86 this doesn't cause a problem
because the random offset is less than a page, but on x86 it's up to two
pages. This means that our old technique of rounding the stack pointer
doesn't work. (Our old implementation of that technique was wrong too.)
It's also incorrect to assume that the main thread's stack base and size
are constant. Likewise to assume that the main thread has a guard page.
The main thread is not like other threads.
This patch switches to reading /proc/self/maps (and checking RLIMIT_STACK)
whenever we're asked.
Bug: 17111575
Signed-off-by: Fengwei Yin <fengwei.yin@intel.com>
Change-Id: I1d4dbffe7bc7bda1d353c3a295dbf68d29f63158
The memchr implementation for 64 bit fails if these conditions occur:
- The buffer is 32 byte aligned.
- The buffer contains the character in the first byte.
- The count sent in is zero.
The function should return NULL, but it's not.
Bug: 16676625
Change-Id: Iab33cc7a8b79920350c72f054dff0e0a3cde69ce
Also fix a few formatting issues in copyright headers that were confusing
the script (though obviously it would be better if the script were smarter).
Change-Id: I7f561bef4f84fdcbd84f375ee226bd65db0e507b
In practice, with this implementation we never need to make a system call.
We get the main thread's tid (which is the same as our pid) back from
the set_tid_address system call we have to make during initialization.
A new pthread will have the same pid as its parent, and a fork child's
main (and only) thread will have a pid equal to its tid, which we get for
free from the kernel before clone returns.
The only time we'd actually have to make a getpid system call now is if
we take a signal during fork and the signal handler calls getpid. (That,
or we call getpid in the dynamic linker while it's still dealing with its
own relocations and hasn't even set up the main thread yet.)
Bug: 15387103
Change-Id: I6d4718ed0a5c912fc75b5f738c49a023dbed5189
Add 64-bit bionic implementation for denver. memcpy/memset are
denver-specific optimized. Use generic version of other routines.
Change-Id: I44a830e07b82b2986001d73d1540b4080aaa839b
vfork() was removed from POSIX 2008, so this replaces its implementation
with a call to fork().
Bug: 13935372
Change-Id: I6d99ac9e52a2efc5ee9bda1cab908774b830cedc
This patch removes the frame record created on svc calls.
Change-Id: I67cf926ba59540e824fb9749d30538e332df7c1e
Signed-off-by: Serban Constantinescu <serban.constantinescu@arm.com>
