Feature outlined here is only available on userdebug or eng builds.
Blocked for security reasons because requires ptrace capabilities.
Add ro.llk.stack to list a set of symbols that should rarely happen
but if persistent in multiple checks, indicates a live lock condition.
At ro.llk.stack.timeout_ms the process is sent a kill, if it remains,
then panic the kernel.
There is no ABA detection in the paths, the condition for the
stack symbol being present instantaneously must be its rarity of
being caught. If a livelock occurs in the path of the symbol, then
it is possible more than one path could be stuck in the state, but
the best candidate symbols are found underneath a lock resulting in
only one process being the culprit, and the best aim. There may be
processes that induce a look of persistence, if so the symbol is not
a candidate for checking.
Add ro.llk.blacklist.process.stack to list process names we want
to skip checking. This configuration parameter is also used to
prevent sepolicy noise when trying to acquire stacks from non
ptrace'able services.
Test: gTest llkd_unit_tests
Bug: 33808187
Bug: 111910505
Bug: 80502612
Change-Id: Ie71221e371b189bbdda2a1155d47826997842dcc
Introduce a standalone live-lock daemon (llkd), to catch kernel
or native user space deadlocks and take mitigating actions. Will
also configure [khungtaskd] to fortify the actions.
If a thread is in D or Z state with no forward progress for longer
than ro.llk.timeout_ms, or ro.llk.[D|Z].timeout_ms, kill the process
or parent process respectively. If another scan shows the same
process continues to exist, then have a confirmed live-lock condition
and need to panic. Panic the kernel in a manner to provide the
greatest bugreporting details as to the condition. Add a alarm self
watchdog should llkd ever get locked up that is double the expected
time to flow through the mainloop. Sampling is every
ro.llk_sample_ms.
Default will not monitor init, or [kthreadd] and all that [kthreadd]
spawns. This reduces the effectiveness of llkd by limiting its
coverage. If in the future, if value in covering kthreadd spawned
threads, the requirement will be to code drivers so that they do not
remain in a persistent 'D' state, or that they have mechanisms to
recover the thread should it be killed externally. Then the
blacklists can be adjusted accordingly if these conditions are met.
An accompanying gTest set have been added, and will setup a persistent
D or Z process, with and without forward progress, but not in a
live-lock state because that would require a buggy kernel, or a module
or kernel modification to stimulate.
Android Properties llkd respond to (*_ms parms are in milliseconds):
- ro.config.low_ram default false, if true do not sysrq t (dump
all threads).
- ro.llk.enable default false, allow live-lock daemon to be enabled.
- ro.khungtask.enable default false, allow [khungtaskd] to be enabled.
- ro.llk.mlockall default true, allow mlock'd live-lock daemon.
- ro.khungtask.timeout default 12 minutes.
- ro.llk.timeout_ms default 10 minutes, D or Z maximum timelimit,
double this value and it sets the alarm watchdog for llkd.
- ro.llk.D.timeout_ms default ro.llk.timeout_ms, D maximum timelimit.
- ro.llk.Z.timeout_ms default ro.llk.timeout_ms, Z maximum timelimit.
- ro.llk.check_ms default 2 minutes sampling interval
(ro.llk.timeout_ms / 5) for threads in D or Z state.
- ro.llk.blacklist.process default 0,1,2 (kernel, init and
[kthreadd]), and process names (/comm or /cmdline) init,[kthreadd],
lmkd,lmkd.llkd,llkd,[khungtaskd],watchdogd,[watchdogd],
[watchdogd/0] ...
- ro.llk.blacklist.parent default 0,2 (kernel and [kthreadd]) and
"[kthreadd]". A comma separated lists of process ids, /comm names
or /cmdline names.
- ro.llk.blacklist.uid default <empty>, comma separated list of
uid numbers or names from getpwuid/getpwnam.
Test: llkd_unit_test
Bug: 33808187
Bug: 72838192
Change-Id: I32e8aa78aef10834e093265d0f3ed5b4199807c6
