Ryan Prichard
07440a8773
Replace TLS_SLOT_BIONIC_PREINIT w/ shared globals
...
Instead of passing the address of a KernelArgumentBlock to libc.so for
initialization, use __loader_shared_globals() to initialize globals.
Most of the work happened in the previous CLs. This CL switches a few
KernelArgumentBlock::getauxval calls to [__bionic_]getauxval and stops
routing the KernelArgumentBlock address through the libc init functions.
Bug: none
Test: bionic unit tests
Change-Id: I96c7b02c21d55c454558b7a5a9243c682782f2dd
Merged-In: I96c7b02c21d55c454558b7a5a9243c682782f2dd
(cherry picked from commit 746ad15912
2018-12-04 13:51:56 -08:00
Elliott Hughes
f1515f6408
Clean up the time(2) implementation.
...
This is also slightly faster for the no VDSO case (56ns vs 66ns).
Bug: N/A
Test: ran tests, benchmarks
Change-Id: I2b0edd06ee6942eb57c32678279278a53ca5ee9b
2018-01-12 15:20:28 -08:00
Mark Salyzyn
4473ccd5b0
bionic: add vdso time()
...
time() can be a hot call, and it currently uses __vdso_gettimeofday,
which is already pretty fast (~3 times faster than the syscall),
but with a __vdso_time call it is ~3 times even faster, in part
because __vdso_time does not require interlocking with updates,
and the read for just the seconds is atomic. __vdso_time is
always available, whereas __vdso_gettimeofday is gated on access
to the physical timers. arm improvement is compelling (x10),
x86 improvement is even more pronounced (x100).
[TL;DR]
w/vdso32 kernel patches, locked cores to MAX, little cores only.
BEFORE:
hikey960 vdso (aarch64):
----------------------------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------------------------
BM_time_clock_gettime 48 ns 48 ns 15414753
BM_time_clock_gettime_syscall 175 ns 175 ns 4062031
BM_time_clock_gettime_REALTIME 44 ns 44 ns 15897875
BM_time_clock_gettime_BOOTTIME 47 ns 47 ns 14307903
BM_time_clock_gettime_TAI 210 ns 210 ns 3341372
BM_time_clock_gettime_unsupported 100 ns 100 ns 7030649
BM_time_gettimeofday 47 ns 47 ns 14969643
BM_time_gettimeofday_syscall 163 ns 163 ns 4283542
BM_time_time 59 ns 59 ns 11815385
hikey960 vdso32 (aarch32):
----------------------------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------------------------
BM_time_clock_gettime 90 ns 90 ns 7572898
BM_time_clock_gettime_syscall 251 ns 251 ns 2763442
BM_time_clock_gettime_REALTIME 81 ns 80 ns 8699536
BM_time_clock_gettime_BOOTTIME 97 ns 97 ns 7256667
BM_time_clock_gettime_TAI 272 ns 272 ns 2570419
BM_time_clock_gettime_unsupported 160 ns 160 ns 4379819
BM_time_gettimeofday 73 ns 73 ns 9608922
BM_time_gettimeofday_syscall 200 ns 199 ns 3527957
BM_time_time 123 ns 123 ns 5651095
x86_64 (glibc):
--------------------------------------------------------------------
Benchmark Time CPU Iterations
--------------------------------------------------------------------
BM_time_clock_gettime 21 ns 21 ns 28873070
BM_time_clock_gettime_syscall 224 ns 224 ns 3095370
BM_time_clock_gettime_REALTIME 17 ns 17 ns 42083086
BM_time_clock_gettime_BOOTTIME 239 ns 239 ns 2924015
BM_time_clock_gettime_TAI 236 ns 236 ns 2961423
BM_time_clock_gettime_unsupported 221 ns 221 ns 3357696
BM_time_gettimeofday 22 ns 22 ns 27975154
BM_time_gettimeofday_syscall 238 ns 238 ns 2882032
BM_time_time 2 ns 2 ns 340354885
BM_time_time_syscall 207 ns 207 ns 3383073
imx7d_pico IOT nyc (w/arm,cpu-registers-not-fw-configured) (armv7a):
(virtual timers)
Benchmark Time(ns) CPU(ns) Iterations
------------------------------------------------------------------
BM_time_clock_gettime 20 477 1489362
BM_time_clock_gettime_syscall 20 487 1458333
BM_time_clock_gettime_REALTIME 19 464 1400000
BM_time_clock_gettime_BOOTTIME 29 700 1000000
BM_time_clock_gettime_TAI 29 690 1000000
BM_time_clock_gettime_unsupported 9 227 3043478
BM_time_gettimeofday 18 444 1555556
BM_time_gettimeofday_syscall 19 456 1555556
BM_time_time 21 497 1166667
imx7d_pico IOT nyc (wo/arm,cpu-registers-not-fw-configured) (armv7a):
(physical timers)
Benchmark Time(ns) CPU(ns) Iterations
------------------------------------------------------------------
BM_time_clock_gettime 6 144 4666667
BM_time_clock_gettime_syscall 20 486 1400000
BM_time_clock_gettime_REALTIME 6 136 5000000
BM_time_clock_gettime_BOOTTIME 6 153 4375000
BM_time_clock_gettime_TAI 31 760 1000000
BM_time_clock_gettime_unsupported 10 233 3043478
BM_time_gettimeofday 6 140 5000000
BM_time_gettimeofday_syscall 19 450 1555556
BM_time_time 9 203 3500000
AFTER:
hikey960 vdso (aarch64):
--------------------------------------------------------------------
Benchmark Time CPU Iterations
--------------------------------------------------------------------
BM_time_clock_gettime 48 ns 48 ns 15414753
BM_time_clock_gettime_syscall 175 ns 175 ns 4062031
BM_time_clock_gettime_REALTIME 44 ns 44 ns 15897875
BM_time_clock_gettime_BOOTTIME 47 ns 47 ns 14307903
BM_time_clock_gettime_TAI 210 ns 210 ns 3341372
BM_time_clock_gettime_unsupported 100 ns 100 ns 7030649
BM_time_gettimeofday 47 ns 47 ns 14975314
BM_time_gettimeofday_syscall 164 ns 164 ns 4278797
BM_time_time 16 ns 16 ns 42932165
hikey960 vdso32 (aarch32):
--------------------------------------------------------------------
Benchmark Time CPU Iterations
--------------------------------------------------------------------
BM_time_clock_gettime 90 ns 90 ns 7572898
BM_time_clock_gettime_syscall 251 ns 251 ns 2763442
BM_time_clock_gettime_REALTIME 81 ns 80 ns 8699536
BM_time_clock_gettime_BOOTTIME 97 ns 97 ns 7256667
BM_time_clock_gettime_TAI 272 ns 272 ns 2570419
BM_time_clock_gettime_unsupported 160 ns 160 ns 4379819
BM_time_gettimeofday 73 ns 73 ns 9596230
BM_time_gettimeofday_syscall 199 ns 199 ns 3575428
BM_time_time 35 ns 35 ns 19798801
imx7d_pico IOT nyc (w/arm,cpu-registers-not-fw-configured) (armv7a):
Benchmark Time(ns) CPU(ns) Iterations
------------------------------------------------------------------
BM_time_clock_gettime 20 477 1489362
BM_time_clock_gettime_syscall 20 487 1458333
BM_time_clock_gettime_REALTIME 19 464 1400000
BM_time_clock_gettime_BOOTTIME 29 700 1000000
BM_time_clock_gettime_TAI 29 690 1000000
BM_time_clock_gettime_unsupported 9 227 3043478
BM_time_gettimeofday 18 444 1555556
BM_time_gettimeofday_syscall 19 456 1555556
BM_time_time 2 50 11666667
imx7d_pico IOT nyc (wo/arm,cpu-registers-not-fw-configured) (armv7a):
Benchmark Time(ns) CPU(ns) Iterations
------------------------------------------------------------------
BM_time_clock_gettime 6 144 4666667
BM_time_clock_gettime_syscall 20 486 1400000
BM_time_clock_gettime_REALTIME 6 136 5000000
BM_time_clock_gettime_BOOTTIME 6 153 4375000
BM_time_clock_gettime_TAI 31 760 1000000
BM_time_clock_gettime_unsupported 10 233 3043478
BM_time_gettimeofday 6 140 5000000
BM_time_gettimeofday_syscall 19 450 1555556
BM_time_time 2 50 10000000
Test: bionic-unit-tests --gtest_filter=time.time
taskset F bionic-benchmarks --bionic_xml=vdso.xml \
--benchmark_filter='BM_time_(time*|clock_gettime*|gettimeofday*)'
Bug: 63737556
Change-Id: I81b088a12ca41a6c4733d46c5477527777138efa
2018-01-11 15:46:11 +00:00
Mark Salyzyn
79249b0897
bionic: add vdso clock_getres
...
clock_getres() should not be a hot call, nevertheless it is
~6-7 times faster for supported clock ids if it uses
__vdso_clock_getres if available. There is a 3% performance
penalty for unsupported clock ids via __vdso_clock_getres with
respect to a direct syscall.
[TL;DR]
w/vdso32 kernel patches, locked cores to MAX, little cores only.
BEFORE:
hikey960 vdso (aarch64):
----------------------------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------------------------
BM_time_clock_getres 126 ns 126 ns 5577874
BM_time_clock_getres_syscall 127 ns 127 ns 5505016
BM_time_clock_getres_REALTIME 126 ns 126 ns 5574682
BM_time_clock_getres_BOOTTIME 126 ns 126 ns 5575237
BM_time_clock_getres_TAI 126 ns 126 ns 5576810
BM_time_clock_getres_unsupported 128 ns 128 ns 5480189
hikey960 vdso32 (aarch32):
----------------------------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------------------------
BM_time_clock_getres 199 ns 199 ns 3508708
BM_time_clock_getres_syscall 220 ns 220 ns 3184676
BM_time_clock_getres_REALTIME 199 ns 199 ns 3509697
BM_time_clock_getres_BOOTTIME 199 ns 199 ns 3513551
BM_time_clock_getres_TAI 200 ns 199 ns 3512412
BM_time_clock_getres_unsupported 196 ns 196 ns 3575609
x86_64 (glibc):
---------------------------------------------------------------------
Benchmark Time CPU Iterations
---------------------------------------------------------------------
BM_time_clock_getres 252 ns 252 ns 2370263
BM_time_clock_getres_syscall 215 ns 215 ns 3287497
BM_time_clock_getres_REALTIME 214 ns 214 ns 3294228
BM_time_clock_getres_BOOTTIME 213 ns 213 ns 3277519
BM_time_clock_getres_TAI 213 ns 213 ns 3294991
BM_time_clock_getres_unsupported 206 ns 206 ns 3450654
imx7d_pico IOT nyc (w/arm,cpu-registers-not-fw-configured) (armv7a):
(Virtual Timers)
Benchmark Time(ns) CPU(ns) Iterations
------------------------------------------------------------------
BM_time_clock_getres 16 345 2000000
BM_time_clock_getres_syscall 16 339 2121212
BM_time_clock_getres_REALTIME 17 350 2058824
BM_time_clock_getres_BOOTTIME 17 345 2000000
BM_time_clock_getres_TAI 16 350 2000000
BM_time_clock_getres_unsupported 13 284 2500000
AFTER:
hikey960 vdso (aarch64):
---------------------------------------------------------------------
Benchmark Time CPU Iterations
---------------------------------------------------------------------
BM_time_clock_getres 18 ns 18 ns 37880389
BM_time_clock_getres_syscall 127 ns 127 ns 5520029
BM_time_clock_getres_REALTIME 18 ns 18 ns 37879962
BM_time_clock_getres_BOOTTIME 19 ns 18 ns 37878361
BM_time_clock_getres_TAI 131 ns 131 ns 5368484
BM_time_clock_getres_unsupported 97 ns 97 ns 7182864
hikey960 vdso32 (aarch32):
---------------------------------------------------------------------
Benchmark Time CPU Iterations
---------------------------------------------------------------------
BM_time_clock_getres 36 ns 36 ns 19205240
BM_time_clock_getres_syscall 212 ns 212 ns 3297100
BM_time_clock_getres_REALTIME 36 ns 36 ns 19219109
BM_time_clock_getres_BOOTTIME 36 ns 36 ns 19222490
BM_time_clock_getres_TAI 206 ns 206 ns 3402868
BM_time_clock_getres_unsupported 159 ns 159 ns 4409492
imx7d_pico IOT nyc (wo/arm,cpu-registers-not-fw-configured) (armv7a):
(Physical Timers)
Benchmark Time(ns) CPU(ns) Iterations
------------------------------------------------------------------
BM_time_clock_getres 2 48 14000000
BM_time_clock_getres_syscall 14 335 2058824
BM_time_clock_getres_REALTIME 2 49 14583333
BM_time_clock_getres_BOOTTIME 2 48 14000000
BM_time_clock_getres_TAI 14 350 2058824
BM_time_clock_getres_unsupported 8 203 3500000
Test: taskset F \
/data/benchmarktest{64}/bionic-benchmarks/bionic-benchmarks \
--bionic_xml=vdso.xml --benchmark_filter=BM_time_clock_getres*
Bug: 63737556
Change-Id: I80c0a5106625d76720287f715fcf145d2aad1705
2017-12-07 09:41:48 -08:00
Elliott Hughes
9591df5294
Fix vdso system call fallback failures.
...
When a vdso call falls back to making a regular system call, the inline
code for the system call doesn't know about errno and just leaves the
usual kernel result to be translated. Add the missing translation.
Also fix the defaults for non-vdso systems so we actually take the
fallback path (and so avoid unintentionally doing the errno translation
twice in those cases).
Bug: http://b/69626243
Test: ran new tests from http://b/63737556
Change-Id: If379632ea2e059e3d3bc3ff41bf3608dc05fb0a3
2017-11-28 09:09:01 -08:00
Elliott Hughes
c51a404b1a
All architectures have AT_SYSINFO_EHDR now.
...
Bug: N/A
Test: builds
Change-Id: Ibc894be98ed0781c8b991ffadff34f616b934aa7
2017-09-22 13:08:50 -07:00
Goran Ferenc
996f6decfa
MIPS: 32/64-bit VDSO support
...
File libc/SYSCALLS.TXT is updated to generate bionic's system call wrappers
for clock_gettime() & gettimeofday() that will be called if kernel vdso
implementations fail to execute.
The system call wrappers are generated using a python script gensyscalls.py.
Since all architectures support vdso now, there is no more need for conditional
statements regarding supported architectures in libc/bionic/vdso.cpp &
libc/private/bionic_vdso.h files.
Test: builds
Change-Id: I7213f29c179a7929851499d78a72900638ae861a
Signed-off-by: Goran Ferenc <goran.ferenc@imgtec.com>
2017-06-01 08:50:57 -07:00
Elliott Hughes
4ce902c30e
Support 32-bit ARM vdso.
...
Linux 4.1 added this.
Bug: http://b/19198045
Change-Id: I28be802ff403a61dd6733a001411b3ff05fef5a5
2016-07-12 13:55:51 -07:00
Josh Gao
93c0f5ee00
Move VDSO pointers to a shared globals struct.
...
Change-Id: I01cbc9cf0917dc1fac52d9205bda2c68529d12ef
2015-10-09 15:59:04 -07:00
Robert Jarzmik
10726d52ac
libc: arch-x86: implement kernel vdso time functions
...
This patch give the possibility of time vdso support on 32bit kernel.
If the 32bit x86 kernel provides gettimeofday() and clock_gettime()
primitives in vdso. In this case make bionic use them. If the kernel
doesn't provide them, fallback to the legacy system call versions.
Change-Id: I87b772a9486fa356903e1f98f486ab9eb0b6f6f7
Signed-off-by: Robert Jarzmik <robert.jarzmik@intel.com>
Signed-off-by: Mingwei Shi <mingwei.shi@intel.com>
2015-07-22 01:31:38 +00:00
Elliott Hughes
613f814508
Revert "Revert "make vdso function pointers read-only at runtime""
...
This reverts commit 1946856b1fhttp://b/22568628
Change-Id: I3e7241c8b54c27ea4a898bc952375c1e9ae38c80
2015-07-20 15:59:33 -07:00
Elliott Hughes
1946856b1f
Revert "make vdso function pointers read-only at runtime"
...
This reverts commit df1a3c6d21http://b/22568628 ).
Change-Id: I071d6d6a0ae7881d65641839e665acdcf58462b4
2015-07-20 17:30:33 +00:00
Daniel Micay
df1a3c6d21
make vdso function pointers read-only at runtime
...
Global, writable function pointers are low-hanging fruit for hijacking
control flow with an overflow from a global buffer or an arbitrary write
vulnerability. This moves the function pointer table into a dedicated
page and makes it read-only at runtime, similar to RELRO.
This increases the memory usage of the library by just under one page.
This could be avoided by having the linker load the vdso by replacing
weak symbols. It's not significant within the Zygote spawning model
though because it's read-only after early init.
Change-Id: Id7a49c96c1b15c2e1926528304b3c54a81e78caf
2015-07-17 11:11:42 -07:00
Elliott Hughes
05fc1d7050
Add missing includes.
...
Change-Id: Ibf549266a19a67eb9158d341a69dddfb654be669
2015-01-28 19:23:11 -08:00
Elliott Hughes
8b5df3920f
Turn on -Wold-style-cast and fix the errors.
...
A couple of dodgy cases where we cast away const, but otherwise pretty boring.
Change-Id: Ibc39ebd525377792b5911464be842121c20f03b9
2015-01-21 17:09:58 -08:00
Elliott Hughes
625993dfbb
Use VDSO for clock_gettime(2) and gettimeofday(2).
...
Bug: 15387103
Change-Id: Ifc3608ea65060c1dc38120b10b6e79874f182a36
2014-07-16 14:27:43 -07:00
