tequilaOS/platform_bionic
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Merge "Revert "Add benchmarks that run simple programs""

Browse source 
am: 0878ac991a

Change-Id: I7462a2870711af912cdb4733fc6d7e19692691f1
This commit is contained in:
Raman Tenneti 2019-09-26 11:33:35 -07:00 committed by android-build-merger
commit 93d9329cbf
5 changed files with 34 additions and 421 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

171
benchmarks/README.md
View file

@ -1,44 +1,48 @@
# Bionic Benchmarks
Bionic Benchmarks
=================
[TOC]
## libc benchmarks (bionic-benchmarks)
`bionic-benchmarks` is a command line tool for measuring the runtimes of libc functions. It is built
on top of [Google Benchmark](https://github.com/google/benchmark) with some additions to organize
Bionic benchmarks is a command line tool for measuring the runtimes of libc functions. It is built
on top of [Google benchmarks](https://github.com/google/benchmark) with some additions to organize
tests into suites.
Running the benchmarks
----------------------
### Device benchmarks
$ mmma bionic/benchmarks
$ adb root
$ adb sync data
$ mma
$ adb remount
$ adb sync
$ adb shell /data/benchmarktest/bionic-benchmarks/bionic-benchmarks
$ adb shell /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks
By default, `bionic-benchmarks` runs all of the benchmarks in alphabetical order. Pass
`--benchmark_filter=getpid` to run just the benchmarks with "getpid" in their name.
When operated without specifying an xml file, the default is to run all
of the benchmarks in alphabetical order.
You can use `--benchmark_filter=getpid` to just run benchmarks with "getpid"
in their name.
### Host benchmarks
See the `benchmarks/run-on-host.sh` script. The host benchmarks can be run with 32-bit or 64-bit
Bionic, or the host glibc.
See the benchmarks/run-on-host.sh script. The host benchmarks can be run
with 32 bit or 64 bit bionic, or the host glibc.
### XML suites
## Suites
Suites are stored in the `suites/` directory and can be chosen with the command line flag
`--bionic_xml`.
'--bionic_xml'.
To choose a specific XML file, use the `--bionic_xml=FILE.XML` option. By default, this option
searches for the XML file in the `suites/` directory. If it doesn't exist in that directory, then
the file will be found as relative to the current directory. If the option specifies the full path
to an XML file such as `/data/nativetest/suites/example.xml`, it will be used as-is.
To choose a specific xml file, use the `--bionic_xml=FILE.XML` option. By default, this
option searches for the xml file in the `suites/` directory. If it doesn't exist
in that directory then the file will be found as relative to the current
directory. If the option specifies the full path to an xml file such as
`/data/nativetest/suites/example.xml`, it will be used as is.
If no XML file is specified through the command-line option, the default is to use `suites/full.xml`.
However, for the host bionic benchmarks (`bionic-benchmarks-glibc`), the default is to use
`suites/host.xml`.
If no xml file is specified through the command-line option, the default is to use `suites/full.xml`.
However, for the host bionic benchmarks (bionic-benchmarks-glibc), the default
is to use `suites/host.xml`.
### XML suite format
### Format
The format for a benchmark is:
@ -51,19 +55,19 @@ The format for a benchmark is:
</fn>
```
XML-specified values for iterations and cpu take precedence over those specified via command line
(via `--bionic_iterations` and `--bionic_cpu`, respectively.)
xml-specified values for iterations and cpu take precedence over those specified via command line
(via '--bionic_iterations' and '--bionic_cpu', respectively.)
To make small changes in runs, you can also schedule benchmarks by passing in their name and a
space-separated list of arguments via the `--bionic_extra` command line flag, e.g.
`--bionic_extra="BM_string_memcpy AT_COMMON_SIZES"` or `--bionic_extra="BM_string_memcmp 32 8 8"`
space-separated list of arguments via the 'bionic_extra' command line flag, e.g.
'--bionic_extra="BM_string_memcpy AT_COMMON_SIZES"' or '--bionic_extra="BM_string_memcmp 32 8 8"'
Note that benchmarks will run normally if extra arguments are passed in, and it will fail
with a segfault if too few are passed in.
### Shorthand
For the sake of brevity, multiple runs can be scheduled in one XML element by putting one of the
For the sake of brevity, multiple runs can be scheduled in one xml element by putting one of the
following in the args field:
NUM_PROPS
@ -77,112 +81,5 @@ the suites directory.
### Unit Tests
`bionic-benchmarks` also has its own set of unit tests, which can be run from the binary in
Bionic benchmarks also has its own set of unit tests, which can be run from the binary in
`/data/nativetest[64]/bionic-benchmarks-tests`
## Process startup time (bionic-spawn-benchmarks)
The `spawn/` subdirectory has a few benchmarks measuring the time used to start simple programs
(e.g. Toybox's `true` and `sh -c true`). Run it on a device like so:
m bionic-spawn-benchmarks
adb root
adb sync data
adb shell /data/benchmarktest/bionic-spawn-benchmarks/bionic-spawn-benchmarks
adb shell /data/benchmarktest64/bionic-spawn-benchmarks/bionic-spawn-benchmarks
Google Benchmark reports both a real-time figure ("Time") and a CPU usage figure. For these
benchmarks, the CPU measurement only counts time spent in the thread calling `posix_spawn`, not that
spent in the spawned process. The real-time is probably more useful, and it is the figure used to
determine the iteration count.
Locking the CPU frequency seems to improve the results of these benchmarks significantly, and it
reduces variability.
## Google Benchmark notes
### Repetitions
Google Benchmark uses two settings to control how many times to run each benchmark, "iterations" and
"repetitions". By default, the repetition count is one. Google Benchmark runs the benchmark a few
times to determine a sufficiently-large iteration count.
Google Benchmark can optionally run a benchmark run repeatedly and report statistics (median, mean,
standard deviation) for the runs. To do so, pass the `--benchmark_repetitions` option, e.g.:
# ./bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll --benchmark_repetitions=4
...
-------------------------------------------------------------------
Benchmark Time CPU Iterations
-------------------------------------------------------------------
BM_stdlib_strtoll 27.7 ns 27.7 ns 25290525
BM_stdlib_strtoll 27.7 ns 27.7 ns 25290525
BM_stdlib_strtoll 27.7 ns 27.7 ns 25290525
BM_stdlib_strtoll 27.8 ns 27.7 ns 25290525
BM_stdlib_strtoll_mean 27.7 ns 27.7 ns 4
BM_stdlib_strtoll_median 27.7 ns 27.7 ns 4
BM_stdlib_strtoll_stddev 0.023 ns 0.023 ns 4
There are 4 runs, each with 25290525 iterations. Measurements for the individual runs can be
suppressed if they aren't needed:
# ./bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll --benchmark_repetitions=4 --benchmark_report_aggregates_only
...
-------------------------------------------------------------------
Benchmark Time CPU Iterations
-------------------------------------------------------------------
BM_stdlib_strtoll_mean 27.8 ns 27.7 ns 4
BM_stdlib_strtoll_median 27.7 ns 27.7 ns 4
BM_stdlib_strtoll_stddev 0.043 ns 0.043 ns 4
### CPU frequencies
To get consistent results between runs, it can sometimes be helpful to restrict a benchmark to
specific cores, or to lock cores at specific frequencies. Some phones have a big.LITTLE core setup,
or at least allow some cores to run at higher frequencies than others.
A core can be selected for `bionic-benchmarks` using the `--bionic_cpu` option or using the
`taskset` utility. e.g. A Pixel 3 device has 4 Kryo 385 Silver cores followed by 4 Gold cores:
blueline:/ # /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll --bionic_cpu=0
...
------------------------------------------------------------
Benchmark Time CPU Iterations
------------------------------------------------------------
BM_stdlib_strtoll 64.2 ns 63.6 ns 11017493
blueline:/ # /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll --bionic_cpu=4
...
------------------------------------------------------------
Benchmark Time CPU Iterations
------------------------------------------------------------
BM_stdlib_strtoll 21.8 ns 21.7 ns 33167103
A similar result can be achieved using `taskset`. The first parameter is a bitmask of core numbers
to pass to `sched_setaffinity`:
blueline:/ # taskset f /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll
...
------------------------------------------------------------
Benchmark Time CPU Iterations
------------------------------------------------------------
BM_stdlib_strtoll 64.3 ns 63.6 ns 10998697
blueline:/ # taskset f0 /data/benchmarktest64/bionic-benchmarks/bionic-benchmarks --benchmark_filter=BM_stdlib_strtoll
...
------------------------------------------------------------
Benchmark Time CPU Iterations
------------------------------------------------------------
BM_stdlib_strtoll 21.3 ns 21.2 ns 33094801
To lock the CPU frequency, use the sysfs interface at `/sys/devices/system/cpu/cpu*/cpufreq/`.
Changing the scaling governor to `performance` suppresses the warning that Google Benchmark
otherwise prints:
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.
Some devices have a `perf-setup.sh` script that locks CPU and GPU frequencies. Some TradeFed
benchmarks appear to be using the script. For more information:
* run `get_build_var BOARD_PERFSETUP_SCRIPT`
* run `m perf-setup.sh` to install the script into `${OUT}/data/local/tmp/perf-setup.sh`
* see: https://android.googlesource.com/platform/platform_testing/+/refs/heads/master/scripts/perf-setup/

99
benchmarks/spawn/Android.bp
View file

@ -1,99 +0,0 @@
//
// Copyright (C) 2019 The Android Open Source Project
// All rights reserved.
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions
// are met:
// * Redistributions of source code must retain the above copyright
// notice, this list of conditions and the following disclaimer.
// * Redistributions in binary form must reproduce the above copyright
// notice, this list of conditions and the following disclaimer in
// the documentation and/or other materials provided with the
// distribution.
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
// FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
// COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
// INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
// BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
// OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
// AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
// OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
// OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
// SUCH DAMAGE.
//
cc_benchmark {
name: "bionic-spawn-benchmarks",
srcs: ["spawn_benchmarks.cpp"],
static_libs: ["libbase"],
// Install these binaries in the same directory as the main benchmark binary.
data: [
":bench_noop",
":bench_noop_nostl",
":bench_noop_static",
],
host_supported: true,
target: {
darwin: { enabled: false },
windows: { enabled: false },
},
}
cc_defaults {
name: "noop_binary_defaults",
compile_multilib: "both",
multilib: {
lib32: { suffix: "32" },
lib64: { suffix: "64" },
},
host_supported: true,
target: {
darwin: { enabled: false },
windows: { enabled: false },
},
}
cc_binary {
defaults: ["noop_binary_defaults"],
name: "bench_noop",
srcs: ["noop.cpp"],
// When this binary is installed to host/linux-x86/bin, its runpath is ${ORIGIN}/../lib[64],
// which is fine for finding host/linux-x86/lib[64]/libc++.so. When it's installed to
// host/linux-x86/benchmarktest[64]/bionic-spawn-benchmarks, the runpath needs an extra "..".
target: {
linux_glibc_x86: {
ldflags: [
"-Wl,--rpath,${ORIGIN}/../../lib",
],
},
linux_glibc_x86_64: {
ldflags: [
"-Wl,--rpath,${ORIGIN}/../../lib64",
],
},
}
}
cc_binary {
defaults: ["noop_binary_defaults"],
name: "bench_noop_nostl",
srcs: ["noop.cpp"],
stl: "none",
}
cc_binary {
defaults: ["noop_binary_defaults"],
name: "bench_noop_static",
srcs: ["noop.cpp"],
static_executable: true,
stl: "libc++_static",
}

53
benchmarks/spawn/AndroidTest.xml
View file

@ -1,53 +0,0 @@
<?xml version="1.0" encoding="utf-8"?>
<!-- Copyright (C) 2019 The Android Open Source Project
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in
the documentation and/or other materials provided with the
distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.
-->
<configuration description="Runs bionic-spawn-benchmarks.">
<option name="test-suite-tag" value="apct" />
<target_preparer class="com.android.tradefed.targetprep.PushFilePreparer">
<option name="cleanup" value="true" />
<option name="push" value="bionic-spawn-benchmarks->/data/local/tmp/bionic-spawn-benchmarks" />
</target_preparer>
<!-- TODO(b/120549168): This seems necessary for consistent results on a walleye, but it's not working with atest
<target_preparer class="com.android.tradefed.targetprep.PushFilePreparer">
<option name="push" value="perf-setup.sh->/data/local/tmp/perf-setup.sh" />
<option name="post-push" value="chmod 755 /data/local/tmp/perf-setup.sh;/data/local/tmp/perf-setup.sh" />
</target_preparer>
-->
<test class="com.android.tradefed.testtype.GoogleBenchmarkTest" >
<option name="native-benchmark-device-path" value="/data/local/tmp" />
<option name="benchmark-module-name" value="bionic-spawn-benchmarks" />
<!-- The GoogleBenchmarkTest class ordinarily expects every file in the benchmark's
directory (recursively) to be a google-benchmark binary, so we need this setting to
avoid failing on the bench_* noop programs, which don't output benchmark results. -->
<option name="file-exclusion-filter-regex" value=".*/bench_[^/]*$" />
</test>
</configuration>

31
benchmarks/spawn/noop.cpp
View file

@ -1,31 +0,0 @@
/*
* Copyright (C) 2019 The Android Open Source Project
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
* FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
* COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
* BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
* OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
* AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
* OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
* OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
int main() {
return 0;
}

101
benchmarks/spawn/spawn_benchmarks.cpp
View file

@ -1,101 +0,0 @@
/*
* Copyright (C) 2019 The Android Open Source Project
* All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
* FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
* COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
* INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,
* BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS
* OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED
* AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
* OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
* OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include <errno.h>
#include <spawn.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <sys/wait.h>
#include <unistd.h>
#include <android-base/file.h>
#include <android-base/stringprintf.h>
#include <benchmark/benchmark.h>
static std::string test_program(const char* name) {
#if defined(__LP64__)
return android::base::GetExecutableDirectory() + "/" + name + "64";
#else
return android::base::GetExecutableDirectory() + "/" + name + "32";
#endif
}
extern char** environ;
static void BM_spawn_test(benchmark::State& state, const char* const* argv) {
for (auto _ : state) {
pid_t child = 0;
if (int spawn_err = posix_spawn(&child, argv[0], nullptr, nullptr, const_cast<char**>(argv),
environ)) {
state.SkipWithError(android::base::StringPrintf(
"posix_spawn of %s failed: %s", argv[0], strerror(spawn_err)).c_str());
break;
}
int wstatus = 0;
const pid_t wait_result = TEMP_FAILURE_RETRY(waitpid(child, &wstatus, 0));
if (wait_result != child) {
state.SkipWithError(android::base::StringPrintf(
"waitpid on pid %d for %s failed: %s",
static_cast<int>(child), argv[0], strerror(errno)).c_str());
break;
}
if (WIFEXITED(wstatus) && WEXITSTATUS(wstatus) == 127) {
state.SkipWithError(android::base::StringPrintf("could not exec %s", argv[0]).c_str());
break;
}
}
}
#define SPAWN_BENCHMARK(name, ...) \
BENCHMARK_CAPTURE(BM_spawn_test, name, (const char*[]) { __VA_ARGS__, nullptr }) \
->UseRealTime() \
->Unit(benchmark::kMicrosecond) \
SPAWN_BENCHMARK(noop, test_program("bench_noop").c_str());
SPAWN_BENCHMARK(noop_nostl, test_program("bench_noop_nostl").c_str());
SPAWN_BENCHMARK(noop_static, test_program("bench_noop_static").c_str());
// Android has a /bin -> /system/bin symlink, but use /system/bin explicitly so we can more easily
// compare Bionic-vs-glibc on a Linux desktop machine.
#if defined(__GLIBC__)
SPAWN_BENCHMARK(bin_true, "/bin/true");
SPAWN_BENCHMARK(sh_true, "/bin/sh", "-c", "true");
#elif defined(__ANDROID__)
SPAWN_BENCHMARK(system_bin_true, "/system/bin/true");
SPAWN_BENCHMARK(vendor_bin_true, "/vendor/bin/true");
SPAWN_BENCHMARK(system_sh_true, "/system/bin/sh", "-c", "true");
SPAWN_BENCHMARK(vendor_sh_true, "/vendor/bin/sh", "-c", "true");
#endif
BENCHMARK_MAIN();