While doing this, refactor the intercept code to be easier to understand.
The primary use case for this is to perform a parallel stack dump (both Java and native) for specific ANRs.
Add tests for all of the different intercept conditions.
Modify the tests to display the error message from the intercept
response if there is an error.
Bug: 254634348
Test: All unit tests pass.
Test: Ran debuggerd on native and java processes.
Test: Created a bugreport without error.
Change-Id: Ic531ccee05b9a470748b815cf109e0076150a0b6
The long term plan is to completely remove tombstoned from microdroid (b/243494912), however it might take time some time to implement it.
In the meantime, we've recently removed cgroups support from the microdroid kernel. This means that starting a tombstoned results in a bunch of non-fatal errors in the logs that are related to the fact that tombstoned service specifies task_profiles.
To get rid of these error messages we temporary add a microdroid variant of the tombstoned (tombstoned.microdroid) that doesn't specify task_profiles.
Bug: 239367015
Test: microdroid presubmit
Change-Id: Ia7d37ede2276790008702e48fdfaf37f4c1fd251
The output.text.fd value is only ever -1 when there is a failure.
There is no need to check both < 0 or -1, so only check for -1.
Test: Unit tests pass.
Test: Verified the message is seen on intercept and not on
Test: regular crashes.
Change-Id: I1eddcd5d2342b268ceb261b246c98b10cee85bb4
The "missing output fd" message can seem like an error, so modify
the message to indicate what is really happening. This message
will occur normally when running the debuggerd command, or when
a bugreport is generated, or when an ANR occurs. In all of those
cases, this is not an error, but an expected action.
Bug: 196189981
Test: Ran debuggerd -b and debuggerd and verified this message is seen.
Test: Ran unit tests.
Change-Id: I6e3d5a76d92b972c77fca301ea7147745bc67c37
writepid command usage to join a cgroup has been deprecated in favor
of a more flexible approach using task_profiles. This way cgroup path
is not hardcoded and cgroup changes can be easily made. Replace
writepid with task_profiles command to migrate between cgroups.
Bug: 191283136
Test: build and boot
Signed-off-by: Suren Baghdasaryan <surenb@google.com>
Change-Id: I945c634dfa7621437d8ea3981bce370d680b7371
In order to test the platform in emulators that are orders of magnitude
slower than real hardware we need to be able to avoid hitting timeouts
that prevent it from coming up properly. For this purpose introduce
a system property, ro.hw_timeout_multiplier, which may be set to
an integer value that acts as a multiplier for various timeouts on
the system.
Bug: 178231152
Change-Id: I6d7710beed0c4c5b1720e74e7abe3a586778c678
Merged-In: I6d7710beed0c4c5b1720e74e7abe3a586778c678
We were already doing this for the text tombstones but not for protos,
which meant that we stopped producing protos once we hit the limit
on the number of tombstones. Move the code for the text tombstones
into a common location and call it for both types.
Change-Id: I4951150da51a32d50821d147458fc5c18200c9d4
There's no way to atomically unlink a specific file for which we have an fd from
a path, which means that we can't safely delete a tombstone without coordination
with tombstoned, which is risky. For example, if we use flock on the directory,
and system_server crashes while holding the lock, we risk deadlock.
We do the next best thing, and keep a file descriptor around for every
tombstone, and truncate it, which requires system_server to be able to
write to tombstones (which are owned by the system group).
Test: treehugger
Change-Id: I6ba7f1fe87ee1a4b57bdb3741e8ec9fbc80788c9
Respect ro.timeout_multiplier property. Some of these are required for
tombstone writing to work on MTE QEMU, the rest are done speculatively.
Test: add crashing code to system_server, observe the tombstone
Bug: 178231152
Change-Id: Ic86e494af571301df7af07d13a6c046a0da6bda7
Revert "Let crash_dump read /proc/$PID."
Revert submission 1556807-tombstone_proto
Reason for revert: b/178455196, Broken test: android.seccomp.cts.SeccompHostJUnit4DeviceTest#testAppZygoteSyscalls on git_master on cf_x86_64_phone-userdebug
Reverted Changes:
Ide6811297:tombstoned: switch from goto to RAII.
I8d285c4b4:tombstoned: make it easier to add more types of ou...
Id0f0fa285:tombstoned: support for protobuf fds.
I6be6082ab:Let crash_dump read /proc/$PID.
Id812ca390:Make protobuf vendor_ramdisk_available.
Ieeece6e6d:libdebuggerd: add protobuf implementation.
Change-Id: I8a77f6b9e1b42902ef7ee250cc3f1fd341ea0e2b
Revert "Let crash_dump read /proc/$PID."
Revert submission 1556807-tombstone_proto
Reason for revert: b/178455196, Broken test: android.seccomp.cts.SeccompHostJUnit4DeviceTest#testAppZygoteSyscalls on git_master on cf_x86_64_phone-userdebug
Reverted Changes:
Ide6811297:tombstoned: switch from goto to RAII.
I8d285c4b4:tombstoned: make it easier to add more types of ou...
Id0f0fa285:tombstoned: support for protobuf fds.
I6be6082ab:Let crash_dump read /proc/$PID.
Id812ca390:Make protobuf vendor_ramdisk_available.
Ieeece6e6d:libdebuggerd: add protobuf implementation.
Change-Id: Ib2403c1b61f6cf0513b76361440fbc5909d7554a
Revert "Let crash_dump read /proc/$PID."
Revert submission 1556807-tombstone_proto
Reason for revert: b/178455196, Broken test: android.seccomp.cts.SeccompHostJUnit4DeviceTest#testAppZygoteSyscalls on git_master on cf_x86_64_phone-userdebug
Reverted Changes:
Ide6811297:tombstoned: switch from goto to RAII.
I8d285c4b4:tombstoned: make it easier to add more types of ou...
Id0f0fa285:tombstoned: support for protobuf fds.
I6be6082ab:Let crash_dump read /proc/$PID.
Id812ca390:Make protobuf vendor_ramdisk_available.
Ieeece6e6d:libdebuggerd: add protobuf implementation.
Change-Id: I0c4f3a17e8b06d6c65255388c571ebf11d371dbb
* changes:
libdebuggerd: add protobuf implementation.
tombstoned: support for protobuf fds.
tombstoned: make it easier to add more types of outputs.
tombstoned: switch from goto to RAII.
Sadly, it looks like we do still really use libcutils for some of the
socket functions.
Test: treehugger
Change-Id: Ic71f97507c89b10d2f3b7a2971064a9e6b1d349d
We're missing useful crashes, especially on hwasan builds.
Bug: http://b/140580637
Test: run crasher
Change-Id: Ib5d8d3bd3fc4d7fec77d0b10302e5595f97a3515
Catch as many early-boot crashes as we can by starting tombstoned
immediately after /data is mounted.
Bug: http://b/139864948
Test: adb shell su 0 dmesg | grep "starting service"
Change-Id: I7f8821102191a445e87020f3efa59a2e0620d9db
Previously, if an intercept ends before we ask for a file descriptor
when doing a backtrace, we'll create a tombstone file instead.
Bug: http://b/114139908
Bug: http://b/115349586
Test: debuggerd_test32
Change-Id: I23c7bb8ae5a982a4374a862d0a4f17bee03eb1d9
We can't actually link an unlinked file back onto disk if it wasn't
opened with O_TMPFILE. Switch to using a temporary filename instead.
Bug: http://b/77729983
Test: agampe
Change-Id: I1970497114f0056065a1ba65f6358f08b51ec551
Instead of creating tombstone FDs in place and passing them out to
crash_dump directly, create them as O_TMPFILEs and link them into place
when crash_dump reports success, to avoid creating empty tombstones
in cases like an aborting thread racing with another thread that
manages to cleanly exit_group before the dump finishes.
Bug: http://b/77729983
Test: debuggerd_test
Test: adb shell 'for x in `seq 0 50`; do crasher; done'
Change-Id: I31ce4fd4a524abf8bde57152450209483d9d0ba9
Apply the same fix from c2e98f63 to intercept_manager.cpp.
Bug: http://b/64543673
Test: debuggerd_test
Change-Id: Ibfb919e059fa62f8336cfc1426d03ef015590136
The order of arguments is wrong - we're passing flags=static_cast<unsigned>(-1)
and backlog=LEV_OPT_CLOSE_ON_FREE (which is 2).
On versions of libevent prior to 2.1.8, this ends up accidentally setting
OPT_LEAVE_SOCKETS_BLOCKING, OPT_CLOSE_ON_EXEC, OPT_REUSABLE and OPT_THREADSAFE
and limiting our backlog to two. These unintentional changes are relatively
benign; we never make our sockets block, we never exec, we never reuse
sockets and the additional locking overhead should be negligible. The
backlog of two might be a problem in theory, but there haven't been any
reports of issues caused by it.
Things get worse on 2.1.8 - that version introduces several new flags,
one of which is OPT_DISABLED. This disables the new listener by default,
which means that our event loop returns early because it has no active listeners
for any of its events.
Bug: 64543673
Test: Manual.
Change-Id: I9954bc7fe1af761de1a950d935dd2e6ce7e2c5f5
For java traces, log the kind of dump as well as the PID of the
completed dump. This makes it easier to correlate dump requests with the
actual file they're written to.
Sample log statement:
E /system/bin/tombstoned: Traces for pid 4737 written to: /data/anr/trace_00
The message for native traces / tombstones remains unchanged because
several tools parse it.
Test: manual
Bug: 32064548
Change-Id: I7b3792dd5ae312ee0bc055c22ec3f7c747152072
The only case where tombstoned creates files for java traces is
when the process is signalled "by hand" using "shell kill -3", or
by the program itself. Such traces do not correspond to an ANR, so
name those files "trace_XX".
When dumpstate / system_server want to dump java traces, they set up
a tombstoned intercept and manage the lifetime of any associated file
that themselves.
Bug: 32064548
Test: manual, debuggerd_test
Change-Id: I97006ec7c0cd35de4b9564f535e77af846cc3891
Make it easy to find out where a specific crash's tombstone was written
to by adding a log.
Bug: http://b/62268830
Test: crasher
Change-Id: I1961dfb19f76a42a8448ebafd4be153b73cb6800
The SELinux changes that this depends on have now landed.
This change also adds a few lower level unit tests of intercept
functionality.
Test: make; debuggerd_test
Change-Id: I0be5e85e7097e26b71db269c9ed92d9b438bfb28
All intercept requests and crash dump requests must now specify a
dump_type, which can be one of kDebuggerdNativeBacktrace,
kDebuggerdTombstone or kDebuggerdJavaBacktrace. Each process can have
only one outstanding intercept registered at a time.
There's only one non-trivial change in this changeset; and that is
to crash_dump. We now pass the type of dump via a command line
argument instead of inferring it from the (resent) signal, this allows
us to connect to tombstoned before we wait for the signal as the
protocol requires.
Test: debuggerd_test
Change-Id: I189b215acfecd08ac52ab29117e3465da00e3a37
.. for ART and the frameworks to link against. In the new stack dumping
scheme (see related bug), the Java runtime will communicate with
tombstoned in order to obtain a FD to which it can write its traces.
Also move things around to separate headers that are private
implementation details from headers that constitute the public debuggerd
API. There are currently only three such headers :
- tombstoned/tombstoned.h
- debuggerd/client.h
- debuggerd/handler.h
Bug: 32064548
Test: make
Change-Id: If1b8578550e373d84828b180bbe585f1088d1aa3
The changes here involve :
- Creating and opening a new socket to receive trace dump requests on. Having
different sockets allows us to install different sets of access control rules.
- A minor refactor to allow us to share common pieces of implementation
between the java and native dumping code. This will also allow us to
add a unit test for all file / directory related logic.
There are two java trace specific additions here :
- We use SO_PEERCRED instead of trusting the PID written to the seocket
because requests come in from untrusted processes.
- Java trace dumps are not interceptible.
kJavaTraceDumpsEnabled is set to false for now but the value of the flag
will be flipped in a future change.
Bug: 32064548
Test: Manual; Currently working on a unit_test for CrashType.
Change-Id: I1d62cc7a7035fd500c3e2b831704a2934d725e35
Previously, there was no way to detect when tombstoned processed an
intercept request packet, making it possible for a intercept request
followed by a crash_dump being processed in the wrong order.
Add a response to intercept registration, to eliminate this race.
Test: debuggerd_test
Change-Id: If38c6d14081ebc86ff1ed0edd7afaeafc40a8381
Also make it loop, so that upon failing to start a dequeued crash
request, we continue to the next one.
Bug: http://b/36685795
Test: debuggerd_test
Change-Id: I94889125f16f4681c6fa0fa9cac456302602ce01
Previously, we would increment num_concurrent_dumps and fail to
decrement it if we failed to start the request. Change this to
only increment after we've successfully started the dump.
Bug: http://b/36685795
Test: debuggerd_test
Change-Id: I66169ed56ed44271e1d8fe1298d95260be7a32a3
Don't try to connect to ourselves in a signal handler (e.g. if someone
does `killall -ABRT tombstoned`).
Test: killall -ABRT tombstoned
Change-Id: Ib69a206f741acb523c9f2883d474c940b6ebfab2
Make tombstones group readable to allow them to be picked up by the
dropbox service.
Bug: http://b/35979630
Test: killall -ABRT rild; dumpsys dropbox
Change-Id: If57cc17563c80d5b5c4887b0937905bffef6b231
