llkd: add live-lock daemon
Introduce a standalone live-lock daemon (llkd), to catch kernel
or native user space deadlocks and take mitigating actions. Will
also configure [khungtaskd] to fortify the actions.
If a thread is in D or Z state with no forward progress for longer
than ro.llk.timeout_ms, or ro.llk.[D|Z].timeout_ms, kill the process
or parent process respectively. If another scan shows the same
process continues to exist, then have a confirmed live-lock condition
and need to panic. Panic the kernel in a manner to provide the
greatest bugreporting details as to the condition. Add a alarm self
watchdog should llkd ever get locked up that is double the expected
time to flow through the mainloop. Sampling is every
ro.llk_sample_ms.
Default will not monitor init, or [kthreadd] and all that [kthreadd]
spawns. This reduces the effectiveness of llkd by limiting its
coverage. If in the future, if value in covering kthreadd spawned
threads, the requirement will be to code drivers so that they do not
remain in a persistent 'D' state, or that they have mechanisms to
recover the thread should it be killed externally. Then the
blacklists can be adjusted accordingly if these conditions are met.
An accompanying gTest set have been added, and will setup a persistent
D or Z process, with and without forward progress, but not in a
live-lock state because that would require a buggy kernel, or a module
or kernel modification to stimulate.
Android Properties llkd respond to (*_ms parms are in milliseconds):
- ro.config.low_ram default false, if true do not sysrq t (dump
all threads).
- ro.llk.enable default false, allow live-lock daemon to be enabled.
- ro.khungtask.enable default false, allow [khungtaskd] to be enabled.
- ro.llk.mlockall default true, allow mlock'd live-lock daemon.
- ro.khungtask.timeout default 12 minutes.
- ro.llk.timeout_ms default 10 minutes, D or Z maximum timelimit,
double this value and it sets the alarm watchdog for llkd.
- ro.llk.D.timeout_ms default ro.llk.timeout_ms, D maximum timelimit.
- ro.llk.Z.timeout_ms default ro.llk.timeout_ms, Z maximum timelimit.
- ro.llk.check_ms default 2 minutes sampling interval
(ro.llk.timeout_ms / 5) for threads in D or Z state.
- ro.llk.blacklist.process default 0,1,2 (kernel, init and
[kthreadd]), and process names (/comm or /cmdline) init,[kthreadd],
lmkd,lmkd.llkd,llkd,[khungtaskd],watchdogd,[watchdogd],
[watchdogd/0] ...
- ro.llk.blacklist.parent default 0,2 (kernel and [kthreadd]) and
"[kthreadd]". A comma separated lists of process ids, /comm names
or /cmdline names.
- ro.llk.blacklist.uid default <empty>, comma separated list of
uid numbers or names from getpwuid/getpwnam.
Test: llkd_unit_test
Bug: 33808187
Bug: 72838192
Change-Id: I32e8aa78aef10834e093265d0f3ed5b4199807c6
2018-02-20 19:47:40 +01:00
|
|
|
/*
|
|
|
|
* Copyright (C) 2018 The Android Open Source Project
|
|
|
|
*
|
|
|
|
* Licensed under the Apache License, Version 2.0 (the "License");
|
|
|
|
* you may not use this file except in compliance with the License.
|
|
|
|
* You may obtain a copy of the License at
|
|
|
|
*
|
|
|
|
* http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
*
|
|
|
|
* Unless required by applicable law or agreed to in writing, software
|
|
|
|
* distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
* See the License for the specific language governing permissions and
|
|
|
|
* limitations under the License.
|
|
|
|
*/
|
|
|
|
|
|
|
|
#ifndef _LLKD_H_
|
|
|
|
#define _LLKD_H_
|
|
|
|
|
|
|
|
#ifndef LOG_TAG
|
|
|
|
#define LOG_TAG "livelock"
|
|
|
|
#endif
|
|
|
|
|
|
|
|
#include <stdbool.h>
|
|
|
|
#include <sys/cdefs.h>
|
|
|
|
|
|
|
|
__BEGIN_DECLS
|
|
|
|
|
|
|
|
bool llkInit(const char* threadname); /* threadname NULL, not spawned */
|
|
|
|
unsigned llkCheckMilliseconds(void);
|
|
|
|
|
|
|
|
/* clang-format off */
|
2020-05-13 15:25:20 +02:00
|
|
|
#define LLK_ENABLE_WRITEABLE_PROPERTY "llk.enable"
|
|
|
|
#define LLK_ENABLE_PROPERTY "ro." LLK_ENABLE_WRITEABLE_PROPERTY
|
|
|
|
#define LLK_ENABLE_DEFAULT false /* "eng" and userdebug true */
|
|
|
|
#define KHT_ENABLE_WRITEABLE_PROPERTY "khungtask.enable"
|
|
|
|
#define KHT_ENABLE_PROPERTY "ro." KHT_ENABLE_WRITEABLE_PROPERTY
|
|
|
|
#define LLK_ENABLE_SYSRQ_T_PROPERTY "ro.llk.sysrq_t"
|
|
|
|
#define LLK_ENABLE_SYSRQ_T_DEFAULT true
|
|
|
|
#define LLK_MLOCKALL_PROPERTY "ro.llk.mlockall"
|
|
|
|
#define LLK_MLOCKALL_DEFAULT true
|
|
|
|
#define LLK_KILLTEST_PROPERTY "ro.llk.killtest"
|
|
|
|
#define LLK_KILLTEST_DEFAULT true
|
|
|
|
#define LLK_TIMEOUT_MS_PROPERTY "ro.llk.timeout_ms"
|
|
|
|
#define KHT_TIMEOUT_PROPERTY "ro.khungtask.timeout"
|
|
|
|
#define LLK_D_TIMEOUT_MS_PROPERTY "ro.llk.D.timeout_ms"
|
|
|
|
#define LLK_Z_TIMEOUT_MS_PROPERTY "ro.llk.Z.timeout_ms"
|
|
|
|
#define LLK_STACK_TIMEOUT_MS_PROPERTY "ro.llk.stack.timeout_ms"
|
|
|
|
#define LLK_CHECK_MS_PROPERTY "ro.llk.check_ms"
|
llkd: add live-lock daemon
Introduce a standalone live-lock daemon (llkd), to catch kernel
or native user space deadlocks and take mitigating actions. Will
also configure [khungtaskd] to fortify the actions.
If a thread is in D or Z state with no forward progress for longer
than ro.llk.timeout_ms, or ro.llk.[D|Z].timeout_ms, kill the process
or parent process respectively. If another scan shows the same
process continues to exist, then have a confirmed live-lock condition
and need to panic. Panic the kernel in a manner to provide the
greatest bugreporting details as to the condition. Add a alarm self
watchdog should llkd ever get locked up that is double the expected
time to flow through the mainloop. Sampling is every
ro.llk_sample_ms.
Default will not monitor init, or [kthreadd] and all that [kthreadd]
spawns. This reduces the effectiveness of llkd by limiting its
coverage. If in the future, if value in covering kthreadd spawned
threads, the requirement will be to code drivers so that they do not
remain in a persistent 'D' state, or that they have mechanisms to
recover the thread should it be killed externally. Then the
blacklists can be adjusted accordingly if these conditions are met.
An accompanying gTest set have been added, and will setup a persistent
D or Z process, with and without forward progress, but not in a
live-lock state because that would require a buggy kernel, or a module
or kernel modification to stimulate.
Android Properties llkd respond to (*_ms parms are in milliseconds):
- ro.config.low_ram default false, if true do not sysrq t (dump
all threads).
- ro.llk.enable default false, allow live-lock daemon to be enabled.
- ro.khungtask.enable default false, allow [khungtaskd] to be enabled.
- ro.llk.mlockall default true, allow mlock'd live-lock daemon.
- ro.khungtask.timeout default 12 minutes.
- ro.llk.timeout_ms default 10 minutes, D or Z maximum timelimit,
double this value and it sets the alarm watchdog for llkd.
- ro.llk.D.timeout_ms default ro.llk.timeout_ms, D maximum timelimit.
- ro.llk.Z.timeout_ms default ro.llk.timeout_ms, Z maximum timelimit.
- ro.llk.check_ms default 2 minutes sampling interval
(ro.llk.timeout_ms / 5) for threads in D or Z state.
- ro.llk.blacklist.process default 0,1,2 (kernel, init and
[kthreadd]), and process names (/comm or /cmdline) init,[kthreadd],
lmkd,lmkd.llkd,llkd,[khungtaskd],watchdogd,[watchdogd],
[watchdogd/0] ...
- ro.llk.blacklist.parent default 0,2 (kernel and [kthreadd]) and
"[kthreadd]". A comma separated lists of process ids, /comm names
or /cmdline names.
- ro.llk.blacklist.uid default <empty>, comma separated list of
uid numbers or names from getpwuid/getpwnam.
Test: llkd_unit_test
Bug: 33808187
Bug: 72838192
Change-Id: I32e8aa78aef10834e093265d0f3ed5b4199807c6
2018-02-20 19:47:40 +01:00
|
|
|
/* LLK_CHECK_MS_DEFAULT = actual timeout_ms / LLK_CHECKS_PER_TIMEOUT_DEFAULT */
|
2020-05-13 15:25:20 +02:00
|
|
|
#define LLK_CHECKS_PER_TIMEOUT_DEFAULT 5
|
|
|
|
#define LLK_CHECK_STACK_PROPERTY "ro.llk.stack"
|
|
|
|
#define LLK_CHECK_STACK_DEFAULT \
|
2018-12-12 18:19:49 +01:00
|
|
|
"cma_alloc,__get_user_pages,bit_wait_io,wait_on_page_bit_killable"
|
2020-05-13 15:25:20 +02:00
|
|
|
#define LLK_IGNORELIST_PROCESS_PROPERTY "ro.llk.ignorelist.process"
|
|
|
|
#define LLK_IGNORELIST_PROCESS_DEFAULT \
|
2018-10-23 00:52:32 +02:00
|
|
|
"0,1,2,init,[kthreadd],[khungtaskd],lmkd,llkd,watchdogd,[watchdogd],[watchdogd/0]"
|
2020-05-13 15:25:20 +02:00
|
|
|
#define LLK_IGNORELIST_PARENT_PROPERTY "ro.llk.ignorelist.parent"
|
|
|
|
#define LLK_IGNORELIST_PARENT_DEFAULT "0,2,[kthreadd],adbd&[setsid]"
|
|
|
|
#define LLK_IGNORELIST_UID_PROPERTY "ro.llk.ignorelist.uid"
|
|
|
|
#define LLK_IGNORELIST_UID_DEFAULT ""
|
|
|
|
#define LLK_IGNORELIST_STACK_PROPERTY "ro.llk.ignorelist.process.stack"
|
2021-03-15 17:17:58 +01:00
|
|
|
#define LLK_IGNORELIST_STACK_DEFAULT "init,lmkd.llkd,llkd,keystore,keystore2,ueventd,apexd"
|
llkd: add live-lock daemon
Introduce a standalone live-lock daemon (llkd), to catch kernel
or native user space deadlocks and take mitigating actions. Will
also configure [khungtaskd] to fortify the actions.
If a thread is in D or Z state with no forward progress for longer
than ro.llk.timeout_ms, or ro.llk.[D|Z].timeout_ms, kill the process
or parent process respectively. If another scan shows the same
process continues to exist, then have a confirmed live-lock condition
and need to panic. Panic the kernel in a manner to provide the
greatest bugreporting details as to the condition. Add a alarm self
watchdog should llkd ever get locked up that is double the expected
time to flow through the mainloop. Sampling is every
ro.llk_sample_ms.
Default will not monitor init, or [kthreadd] and all that [kthreadd]
spawns. This reduces the effectiveness of llkd by limiting its
coverage. If in the future, if value in covering kthreadd spawned
threads, the requirement will be to code drivers so that they do not
remain in a persistent 'D' state, or that they have mechanisms to
recover the thread should it be killed externally. Then the
blacklists can be adjusted accordingly if these conditions are met.
An accompanying gTest set have been added, and will setup a persistent
D or Z process, with and without forward progress, but not in a
live-lock state because that would require a buggy kernel, or a module
or kernel modification to stimulate.
Android Properties llkd respond to (*_ms parms are in milliseconds):
- ro.config.low_ram default false, if true do not sysrq t (dump
all threads).
- ro.llk.enable default false, allow live-lock daemon to be enabled.
- ro.khungtask.enable default false, allow [khungtaskd] to be enabled.
- ro.llk.mlockall default true, allow mlock'd live-lock daemon.
- ro.khungtask.timeout default 12 minutes.
- ro.llk.timeout_ms default 10 minutes, D or Z maximum timelimit,
double this value and it sets the alarm watchdog for llkd.
- ro.llk.D.timeout_ms default ro.llk.timeout_ms, D maximum timelimit.
- ro.llk.Z.timeout_ms default ro.llk.timeout_ms, Z maximum timelimit.
- ro.llk.check_ms default 2 minutes sampling interval
(ro.llk.timeout_ms / 5) for threads in D or Z state.
- ro.llk.blacklist.process default 0,1,2 (kernel, init and
[kthreadd]), and process names (/comm or /cmdline) init,[kthreadd],
lmkd,lmkd.llkd,llkd,[khungtaskd],watchdogd,[watchdogd],
[watchdogd/0] ...
- ro.llk.blacklist.parent default 0,2 (kernel and [kthreadd]) and
"[kthreadd]". A comma separated lists of process ids, /comm names
or /cmdline names.
- ro.llk.blacklist.uid default <empty>, comma separated list of
uid numbers or names from getpwuid/getpwnam.
Test: llkd_unit_test
Bug: 33808187
Bug: 72838192
Change-Id: I32e8aa78aef10834e093265d0f3ed5b4199807c6
2018-02-20 19:47:40 +01:00
|
|
|
/* clang-format on */
|
|
|
|
|
|
|
|
__END_DECLS
|
|
|
|
|
|
|
|
#ifdef __cplusplus
|
|
|
|
extern "C++" { /* In case this included wrapped with __BEGIN_DECLS */
|
|
|
|
|
|
|
|
#include <chrono>
|
|
|
|
|
|
|
|
__BEGIN_DECLS
|
|
|
|
/* C++ code allowed to not specify threadname argument for this C linkage */
|
|
|
|
bool llkInit(const char* threadname = nullptr);
|
|
|
|
__END_DECLS
|
|
|
|
std::chrono::milliseconds llkCheck(bool checkRunning = false);
|
|
|
|
|
|
|
|
/* clang-format off */
|
|
|
|
#define LLK_TIMEOUT_MS_DEFAULT std::chrono::duration_cast<milliseconds>(std::chrono::minutes(10))
|
|
|
|
#define LLK_TIMEOUT_MS_MINIMUM std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::seconds(10))
|
|
|
|
#define LLK_CHECK_MS_MINIMUM std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::seconds(1))
|
|
|
|
/* clang-format on */
|
|
|
|
|
|
|
|
} /* extern "C++" */
|
|
|
|
#endif /* __cplusplus */
|
|
|
|
|
|
|
|
#endif /* _LLKD_H_ */
|