diff --git a/benchmarks/README.md b/benchmarks/README.md index fe447d170..2616a51f8 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -1,44 +1,48 @@ -# Bionic Benchmarks +Bionic Benchmarks +================= -[TOC] - -## libc benchmarks (bionic-benchmarks) - -`bionic-benchmarks` is a command line tool for measuring the runtimes of libc functions. It is built -on top of [Google Benchmark](https://github.com/google/benchmark) with some additions to organize +Bionic benchmarks is a command line tool for measuring the runtimes of libc functions. It is built +on top of [Google benchmarks](https://github.com/google/benchmark) with some additions to organize tests into suites. +Running the benchmarks +---------------------- + ### Device benchmarks - $ mmma bionic/benchmarks - $ adb root - $ adb sync data + $ mma + $ adb remount + $ adb sync $ adb shell /data/benchmarktest/bionic-benchmarks/bionic-benchmarks $ adb shell /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks -By default, `bionic-benchmarks` runs all of the benchmarks in alphabetical order. Pass -`--benchmark_filter=getpid` to run just the benchmarks with "getpid" in their name. +When operated without specifying an xml file, the default is to run all +of the benchmarks in alphabetical order. + +You can use `--benchmark_filter=getpid` to just run benchmarks with "getpid" +in their name. ### Host benchmarks -See the `benchmarks/run-on-host.sh` script. The host benchmarks can be run with 32-bit or 64-bit -Bionic, or the host glibc. +See the benchmarks/run-on-host.sh script. The host benchmarks can be run +with 32 bit or 64 bit bionic, or the host glibc. -### XML suites +## Suites Suites are stored in the `suites/` directory and can be chosen with the command line flag -`--bionic_xml`. +'--bionic_xml'. -To choose a specific XML file, use the `--bionic_xml=FILE.XML` option. By default, this option -searches for the XML file in the `suites/` directory. If it doesn't exist in that directory, then -the file will be found as relative to the current directory. If the option specifies the full path -to an XML file such as `/data/nativetest/suites/example.xml`, it will be used as-is. +To choose a specific xml file, use the `--bionic_xml=FILE.XML` option. By default, this +option searches for the xml file in the `suites/` directory. If it doesn't exist +in that directory then the file will be found as relative to the current +directory. If the option specifies the full path to an xml file such as +`/data/nativetest/suites/example.xml`, it will be used as is. -If no XML file is specified through the command-line option, the default is to use `suites/full.xml`. -However, for the host bionic benchmarks (`bionic-benchmarks-glibc`), the default is to use -`suites/host.xml`. +If no xml file is specified through the command-line option, the default is to use `suites/full.xml`. +However, for the host bionic benchmarks (bionic-benchmarks-glibc), the default +is to use `suites/host.xml`. -### XML suite format +### Format The format for a benchmark is: @@ -51,19 +55,19 @@ The format for a benchmark is: ``` -XML-specified values for iterations and cpu take precedence over those specified via command line -(via `--bionic_iterations` and `--bionic_cpu`, respectively.) +xml-specified values for iterations and cpu take precedence over those specified via command line +(via '--bionic_iterations' and '--bionic_cpu', respectively.) To make small changes in runs, you can also schedule benchmarks by passing in their name and a -space-separated list of arguments via the `--bionic_extra` command line flag, e.g. -`--bionic_extra="BM_string_memcpy AT_COMMON_SIZES"` or `--bionic_extra="BM_string_memcmp 32 8 8"` +space-separated list of arguments via the 'bionic_extra' command line flag, e.g. +'--bionic_extra="BM_string_memcpy AT_COMMON_SIZES"' or '--bionic_extra="BM_string_memcmp 32 8 8"' Note that benchmarks will run normally if extra arguments are passed in, and it will fail with a segfault if too few are passed in. ### Shorthand -For the sake of brevity, multiple runs can be scheduled in one XML element by putting one of the +For the sake of brevity, multiple runs can be scheduled in one xml element by putting one of the following in the args field: NUM_PROPS @@ -77,112 +81,5 @@ the suites directory. ### Unit Tests -`bionic-benchmarks` also has its own set of unit tests, which can be run from the binary in +Bionic benchmarks also has its own set of unit tests, which can be run from the binary in `/data/nativetest[64]/bionic-benchmarks-tests` - -## Process startup time (bionic-spawn-benchmarks) - -The `spawn/` subdirectory has a few benchmarks measuring the time used to start simple programs -(e.g. Toybox's `true` and `sh -c true`). Run it on a device like so: - - m bionic-spawn-benchmarks - adb root - adb sync data - adb shell /data/benchmarktest/bionic-spawn-benchmarks/bionic-spawn-benchmarks - adb shell /data/benchmarktest64/bionic-spawn-benchmarks/bionic-spawn-benchmarks - -Google Benchmark reports both a real-time figure ("Time") and a CPU usage figure. For these -benchmarks, the CPU measurement only counts time spent in the thread calling `posix_spawn`, not that -spent in the spawned process. The real-time is probably more useful, and it is the figure used to -determine the iteration count. - -Locking the CPU frequency seems to improve the results of these benchmarks significantly, and it -reduces variability. - -## Google Benchmark notes - -### Repetitions - -Google Benchmark uses two settings to control how many times to run each benchmark, "iterations" and -"repetitions". By default, the repetition count is one. Google Benchmark runs the benchmark a few -times to determine a sufficiently-large iteration count. - -Google Benchmark can optionally run a benchmark run repeatedly and report statistics (median, mean, -standard deviation) for the runs. To do so, pass the `--benchmark_repetitions` option, e.g.: - - # ./bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll --benchmark_repetitions=4 - ... - ------------------------------------------------------------------- - Benchmark Time CPU Iterations - ------------------------------------------------------------------- - BM_stdlib_strtoll 27.7 ns 27.7 ns 25290525 - BM_stdlib_strtoll 27.7 ns 27.7 ns 25290525 - BM_stdlib_strtoll 27.7 ns 27.7 ns 25290525 - BM_stdlib_strtoll 27.8 ns 27.7 ns 25290525 - BM_stdlib_strtoll_mean 27.7 ns 27.7 ns 4 - BM_stdlib_strtoll_median 27.7 ns 27.7 ns 4 - BM_stdlib_strtoll_stddev 0.023 ns 0.023 ns 4 - -There are 4 runs, each with 25290525 iterations. Measurements for the individual runs can be -suppressed if they aren't needed: - - # ./bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll --benchmark_repetitions=4 --benchmark_report_aggregates_only - ... - ------------------------------------------------------------------- - Benchmark Time CPU Iterations - ------------------------------------------------------------------- - BM_stdlib_strtoll_mean 27.8 ns 27.7 ns 4 - BM_stdlib_strtoll_median 27.7 ns 27.7 ns 4 - BM_stdlib_strtoll_stddev 0.043 ns 0.043 ns 4 - -### CPU frequencies - -To get consistent results between runs, it can sometimes be helpful to restrict a benchmark to -specific cores, or to lock cores at specific frequencies. Some phones have a big.LITTLE core setup, -or at least allow some cores to run at higher frequencies than others. - -A core can be selected for `bionic-benchmarks` using the `--bionic_cpu` option or using the -`taskset` utility. e.g. A Pixel 3 device has 4 Kryo 385 Silver cores followed by 4 Gold cores: - - blueline:/ # /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll --bionic_cpu=0 - ... - ------------------------------------------------------------ - Benchmark Time CPU Iterations - ------------------------------------------------------------ - BM_stdlib_strtoll 64.2 ns 63.6 ns 11017493 - - blueline:/ # /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll --bionic_cpu=4 - ... - ------------------------------------------------------------ - Benchmark Time CPU Iterations - ------------------------------------------------------------ - BM_stdlib_strtoll 21.8 ns 21.7 ns 33167103 - -A similar result can be achieved using `taskset`. The first parameter is a bitmask of core numbers -to pass to `sched_setaffinity`: - - blueline:/ # taskset f /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll - ... - ------------------------------------------------------------ - Benchmark Time CPU Iterations - ------------------------------------------------------------ - BM_stdlib_strtoll 64.3 ns 63.6 ns 10998697 - - blueline:/ # taskset f0 /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll - ... - ------------------------------------------------------------ - Benchmark Time CPU Iterations - ------------------------------------------------------------ - BM_stdlib_strtoll 21.3 ns 21.2 ns 33094801 - -To lock the CPU frequency, use the sysfs interface at `/sys/devices/system/cpu/cpu*/cpufreq/`. -Changing the scaling governor to `performance` suppresses the warning that Google Benchmark -otherwise prints: - - ***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead. - -Some devices have a `perf-setup.sh` script that locks CPU and GPU frequencies. Some TradeFed -benchmarks appear to be using the script. For more information: - * run `get_build_var BOARD_PERFSETUP_SCRIPT` - * run `m perf-setup.sh` to install the script into `${OUT}/data/local/tmp/perf-setup.sh` - * see: https://android.googlesource.com/platform/platform_testing/+/refs/heads/master/scripts/perf-setup/ diff --git a/benchmarks/spawn/Android.bp b/benchmarks/spawn/Android.bp deleted file mode 100644 index 1295e2b30..000000000 --- a/benchmarks/spawn/Android.bp +++ /dev/null @@ -1,99 +0,0 @@ -// -// Copyright (C) 2019 The Android Open Source Project -// All rights reserved. -// -// Redistribution and use in source and binary forms, with or without -// modification, are permitted provided that the following conditions -// are met: -// * Redistributions of source code must retain the above copyright -// notice, this list of conditions and the following disclaimer. -// * Redistributions in binary form must reproduce the above copyright -// notice, this list of conditions and the following disclaimer in -// the documentation and/or other materials provided with the -// distribution. -// -// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS -// FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE -// COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, -// INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, -// BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS -// OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED -// AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, -// OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT -// OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF -// SUCH DAMAGE. -// - -cc_benchmark { - name: "bionic-spawn-benchmarks", - srcs: ["spawn_benchmarks.cpp"], - static_libs: ["libbase"], - - // Install these binaries in the same directory as the main benchmark binary. - data: [ - ":bench_noop", - ":bench_noop_nostl", - ":bench_noop_static", - ], - - host_supported: true, - target: { - darwin: { enabled: false }, - windows: { enabled: false }, - }, -} - -cc_defaults { - name: "noop_binary_defaults", - - compile_multilib: "both", - multilib: { - lib32: { suffix: "32" }, - lib64: { suffix: "64" }, - }, - - host_supported: true, - target: { - darwin: { enabled: false }, - windows: { enabled: false }, - }, -} - -cc_binary { - defaults: ["noop_binary_defaults"], - name: "bench_noop", - srcs: ["noop.cpp"], - - // When this binary is installed to host/linux-x86/bin, its runpath is ${ORIGIN}/../lib[64], - // which is fine for finding host/linux-x86/lib[64]/libc++.so. When it's installed to - // host/linux-x86/benchmarktest[64]/bionic-spawn-benchmarks, the runpath needs an extra "..". - target: { - linux_glibc_x86: { - ldflags: [ - "-Wl,--rpath,${ORIGIN}/../../lib", - ], - }, - linux_glibc_x86_64: { - ldflags: [ - "-Wl,--rpath,${ORIGIN}/../../lib64", - ], - }, - } -} - -cc_binary { - defaults: ["noop_binary_defaults"], - name: "bench_noop_nostl", - srcs: ["noop.cpp"], - stl: "none", -} - -cc_binary { - defaults: ["noop_binary_defaults"], - name: "bench_noop_static", - srcs: ["noop.cpp"], - static_executable: true, - stl: "libc++_static", -} diff --git a/benchmarks/spawn/AndroidTest.xml b/benchmarks/spawn/AndroidTest.xml deleted file mode 100644 index 93316754d..000000000 --- a/benchmarks/spawn/AndroidTest.xml +++ /dev/null @@ -1,53 +0,0 @@ - - - - diff --git a/benchmarks/spawn/noop.cpp b/benchmarks/spawn/noop.cpp deleted file mode 100644 index 1f1cac037..000000000 --- a/benchmarks/spawn/noop.cpp +++ /dev/null @@ -1,31 +0,0 @@ -/* - * Copyright (C) 2019 The Android Open Source Project - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in - * the documentation and/or other materials provided with the - * distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS - * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE - * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, - * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, - * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS - * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED - * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, - * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT - * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF - * SUCH DAMAGE. - */ - -int main() { - return 0; -} diff --git a/benchmarks/spawn/spawn_benchmarks.cpp b/benchmarks/spawn/spawn_benchmarks.cpp deleted file mode 100644 index e8f7c1792..000000000 --- a/benchmarks/spawn/spawn_benchmarks.cpp +++ /dev/null @@ -1,101 +0,0 @@ -/* - * Copyright (C) 2019 The Android Open Source Project - * All rights reserved. - * - * Redistribution and use in source and binary forms, with or without - * modification, are permitted provided that the following conditions - * are met: - * * Redistributions of source code must retain the above copyright - * notice, this list of conditions and the following disclaimer. - * * Redistributions in binary form must reproduce the above copyright - * notice, this list of conditions and the following disclaimer in - * the documentation and/or other materials provided with the - * distribution. - * - * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS - * "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT - * LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS - * FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE - * COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, - * INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, - * BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS - * OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED - * AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, - * OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT - * OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF - * SUCH DAMAGE. - */ - -#include -#include -#include -#include -#include -#include -#include - -#include -#include -#include - -static std::string test_program(const char* name) { -#if defined(__LP64__) - return android::base::GetExecutableDirectory() + "/" + name + "64"; -#else - return android::base::GetExecutableDirectory() + "/" + name + "32"; -#endif -} - -extern char** environ; - -static void BM_spawn_test(benchmark::State& state, const char* const* argv) { - for (auto _ : state) { - pid_t child = 0; - if (int spawn_err = posix_spawn(&child, argv[0], nullptr, nullptr, const_cast(argv), - environ)) { - state.SkipWithError(android::base::StringPrintf( - "posix_spawn of %s failed: %s", argv[0], strerror(spawn_err)).c_str()); - break; - } - - int wstatus = 0; - const pid_t wait_result = TEMP_FAILURE_RETRY(waitpid(child, &wstatus, 0)); - if (wait_result != child) { - state.SkipWithError(android::base::StringPrintf( - "waitpid on pid %d for %s failed: %s", - static_cast(child), argv[0], strerror(errno)).c_str()); - break; - } - if (WIFEXITED(wstatus) && WEXITSTATUS(wstatus) == 127) { - state.SkipWithError(android::base::StringPrintf("could not exec %s", argv[0]).c_str()); - break; - } - } -} - -#define SPAWN_BENCHMARK(name, ...) \ - BENCHMARK_CAPTURE(BM_spawn_test, name, (const char*[]) { __VA_ARGS__, nullptr }) \ - ->UseRealTime() \ - ->Unit(benchmark::kMicrosecond) \ - -SPAWN_BENCHMARK(noop, test_program("bench_noop").c_str()); -SPAWN_BENCHMARK(noop_nostl, test_program("bench_noop_nostl").c_str()); -SPAWN_BENCHMARK(noop_static, test_program("bench_noop_static").c_str()); - -// Android has a /bin -> /system/bin symlink, but use /system/bin explicitly so we can more easily -// compare Bionic-vs-glibc on a Linux desktop machine. -#if defined(__GLIBC__) - -SPAWN_BENCHMARK(bin_true, "/bin/true"); -SPAWN_BENCHMARK(sh_true, "/bin/sh", "-c", "true"); - -#elif defined(__ANDROID__) - -SPAWN_BENCHMARK(system_bin_true, "/system/bin/true"); -SPAWN_BENCHMARK(vendor_bin_true, "/vendor/bin/true"); -SPAWN_BENCHMARK(system_sh_true, "/system/bin/sh", "-c", "true"); -SPAWN_BENCHMARK(vendor_sh_true, "/vendor/bin/sh", "-c", "true"); - -#endif - -BENCHMARK_MAIN();