Chatty logs would distort the average log size by elevating the
elements, but not the size. Add statistical collection for the
number of elements that report chatty, and subtract that from
the number of elements to improve the pruning estimate. Pick
minElements as 1% rather than 10% of the total with this more
accurate number of elements, to a minumum of 4.
Bug: 24511000
Change-Id: I3f36558138aa0b2a50e4fac6440c3a8505d95276
- switch to coalesce instead of merge in naming of functions
and variables. Confusing since we also to merge-sorts and
other activities in the logger.
- define maxPrune rather than using a number in the code path.
Bug: 24511000
- If doing a clear, skip accounting
- Ensure for busy checking, behind a region lock for instance, only
break out if there was something to do. Basically move the filter
actions first, and defer checking the region lock to the ends of
the loops.
Bug: 23711431
Change-Id: Icc984f406880633516fb17dda84188a30d092e01
- Propagate to caller the clearing errors, busy blocked by reader.
- For clear, perform retries within logd with a one second lul each,
telling readers to skip, but on final retry to kill all readers if
problem still persists due to block reader (or high volume logspammer).
Bug: 23711431
Change-Id: Ie4c46bc9480a7f49b96a81fae25a95c603270c33
- The reader region level indicates the location to protect, but once
it has been passed to the calling reader, then allow us to go
one beyond so that a clear or prune thread can remove the entry.
Bug: 23711431
Change-Id: I0f2389858dd8c83366c034360f67d7c363625b56
Vote three times in /proc/pid/status to look for AID_LOG group
If not, we may default to the callers UID, and the net result is
to perform the task related to that UID. For adb logcat and
shell logcat, the UID is AID_SHELL which typically has no logs,
leaving no net action taken.
Bug: 23711431
Change-Id: I2b5900a2d37173bd995eb308ee9ecafa20602b62
Simplify table generation by placing the line and header formatting
into each type's (UID, PID, TID) object. Switch to const return
values for the ownership passing functions (*ToName() functions
and methods). Use longer variable names to reduce confusion.
Switch from LINES To NUM for pruned column as that more accurately
reflects what is dropped since one entry can contain several lines.
Bug: 22855208
Change-Id: Ib110dce98a68cf5f844eb30f8a192a1f691eeba2
Discovered that we had a few libc hotspots. Adjust code to generally
reduce or nullify the number of calls to malloc, free, strlen,
strcmp, strncmp, memcmp & strncasecmp. Total gain looks to be about
3% of logd's processing time. malloc still contributes to 3%, but all
others are now total 0.5%.
Bug: 23685592
Change-Id: Ife721121667969260cdb8b055524ae90f5911278
* commit '900362c36eb531c88d37d8fa57b66c45ed165122':
logd: klogd deal with nuls in dmesg
logd: log_strtok_r deal with nuls
liblog: logprint: printable nul
logd: klogd: sniff for time correction on Mediatek
* commit '1b2fb587eb7db6f9de1dda8663b33d87a2f5a27e':
logd: klogd deal with nuls in dmesg
logd: log_strtok_r deal with nuls
liblog: logprint: printable nul
logd: klogd: sniff for time correction on Mediatek
Switch to using string and length in all transactions, treating
trailing nuls the same as spaces.
ToDo: change dumpstate (bugreport) to use logcat -b printable _regardless_
Bug: 23517551
Change-Id: I42162365e6bf8ed79d356e7b689a673902116fdb
Rename to log_strntok_r and change from dealing with strings
to dealing with a string and an associated length.
Bug: 23517551
Change-Id: Ia72f1305a53f55eeef9861ac378fb8205fd2378e
Need some more flexibility when parsing kernel messages
cluttered with extra fluff. This is the minimal relaxation
we can do to the rules to ensure that we work on all
possible devices and kernels when sniffing for time
correction information.
We want to minimize any future maintenance, keep in mind
klogd is a "userdebug" or "eng" feature and is disabled
in "user" builds. Manage expectations.
Bug: 23517551
Change-Id: I026d074e14fb2550e728683e85a973bd87e78a9c
(cherry pick from commit 46d159d462)
- sniff for PID in kernel log messages if available
- properly deal with klogd watermark in face of modified output
- deal more stringently with priority tag, must have [ following
- suppress process-name stutter in tag that can happen
- do not use : to demark tag if within [ ]
Mediatek-special change that adds <printk_state>(<cpu>)[<pid>:<comm>]
as a prefix to the printk messages. Along the lines of (simplified
for entertainment purposes, YMMV):
char tbuf[50]; /* printk prefix */
int this_cpu = smp_processor_id();
char state = __raw_get_cpu_var(printk_state);
unsigned tlen = snprintf(tbuf, sizeof(tbuf), "%c(%x)[%d:%s]",
state, this_cpu, current->pid, current->comm);
Bug: 23517551
Change-Id: I568e25c5aa6d8474835454a0e83b19c2921b7985
(cherry pick from commit 5bb2972dce)
A regression that resulted in increased memory consumption for some
logging patterns because we rarely did merge or leading checks, and
age-out checking. On the last prune cycle, we reset for a full scan.
Add some comments describing the pruning processes.
Bug: 23327476
Bug: 23681639
Bug: 23685592
Change-Id: I22b0f339c9269b006831fda9cefe295a263ebb92
A regression that resulted in increased memory consumption for some
logging patterns because we rarely did merge or leading checks, and
age-out checking. On the last prune cycle, we reset for a full scan.
Add some comments describing the pruning processes.
Bug: 23327476
Bug: 23681639
Bug: 23685592
Change-Id: I22b0f339c9269b006831fda9cefe295a263ebb92
(cherry pick from commit 831aa29730)
With part deux we caused an apparent regression by not checking for
stale recorded iterators. This checking was on-purpose bypassesed
when leading prune entries were to be deleted without touching the
statistics engine due to an in-place merge.
Part deux had us leaving iterators we were not focussed on untouched
which in turn because they were left behind, had a much higher
likelihood of being deleted without touching the statistics engine.
Perform the check every delete.
Bug: 23789348
Bug: 23490267
Change-Id: Idc6cc23d1f9e3b6cd9a083139a0de59479fbfe08
With part deux we caused an apparent regression by not checking for
stale recorded iterators. This checking was on-purpose bypassesed
when leading prune entries were to be deleted without touching the
statistics engine due to an in-place merge.
Part deux had us leaving iterators we were not focussed on untouched
which in turn because they were left behind, had a much higher
likelihood of being deleted without touching the statistics engine.
Perform the check every delete.
Bug: 23789348
Change-Id: Idc6cc23d1f9e3b6cd9a083139a0de59479fbfe08
(cherry pick from commit ccfe8446a1)
Regression that cause records to be preserved for more than a day.
Bug: 23681639
Bug: 23685592
Change-Id: I5e4393c8e3ed935790994c77ec51dc6512a6daa6
(cherry pick from commit 49afe0d00f)
Only record watermark if not known, or represents the worst UID
currently under focus. This has resulted in a halving of the average
prune time in the face of heavy spam because we get less processing
spikes.
Bug: 23327476
Bug: 23681639
Bug: 23685592
Change-Id: I19f297042b9fc2c98d902695c1c36df1bf5cd6f6
(cherry pick from commit acb1ddf56c)
Aid monotonic to realtime logging synchronization correction in
the Android ecosystem by providing a periodic notification. We
now have the following messages in the kernel logs:
- PM: suspend entry %Y-%m-%d %H:%M:%S.%09q UTC
- PM: suspend exit %Y-%m-%d %H:%M:%S.%09q UTC
- Suspended for %s.%03q seconds
- healthd: battery l=100 ... %Y-%m-%d %H:%M:%S.%09q UTC
Alter klogd to resynchronize on healthd messages as well.
NB: Time using strftime format, %q is a reference to fractional
second as introduced into log_time strptime method.
Bug: 21868540
Change-Id: I854afc0a07dff9c7f26d2b2f68990e52bf90e300
(cherry pick from commit 618d0dec50)
- Heuristics associated with translation of kernel messages to
Android user space logs.
- Limit is_prio to 4 characters, we got false positives on hex
values like <register contents> with no alpha chars.
- x11 and other register definitions are not valid tags, en0 is
- fix some Android coding standard issues
Change-Id: Idc3dcc53a2cb75ac38628c8ef7a5d5b53f12587a
- sniff for PID in kernel log messages if available
- properly deal with klogd watermark in face of modified output
- deal more stringently with priority tag, must have [ following
- suppress process-name stutter in tag that can happen
- do not use : to demark tag if within [ ]
Mediatek-special change that adds <printk_state>(<cpu>)[<pid>:<comm>]
as a prefix to the printk messages. Along the lines of (simplified
for entertainment purposes, YMMV):
char tbuf[50]; /* printk prefix */
int this_cpu = smp_processor_id();
char state = __raw_get_cpu_var(printk_state);
unsigned tlen = snprintf(tbuf, sizeof(tbuf), "%c(%x)[%d:%s]",
state, this_cpu, current->pid, current->comm);
Bug: 23517551
Change-Id: I568e25c5aa6d8474835454a0e83b19c2921b7985
Only record watermark if not known, or represents the worst UID
currently under focus. This has resulted in a halving of the average
prune time in the face of heavy spam because we get less processing
spikes.
Bug: 23327476
Change-Id: I19f297042b9fc2c98d902695c1c36df1bf5cd6f6
* commit '44a7da74bd841eee680870433c1baf82325fb51e':
logd: log buffer switch to std::list
logd: white and black switch to std::list
logd: logtimes switch to std::list
logd: prune 10% or 256 entries max
(cherry pick from commit c892ea3fa8)
Hold on to last worst uid watermark and bypass a spike to O(n*n*x)
(n=samples, x=number of spammers) wrt chatty trimming.
Bug: 23327476
Change-Id: I9f21ce95e969b67e576417a760f75c4d86acf364
Hold on to last worst uid watermark and bypass a spike to O(n*n*x)
(n=samples, x=number of spammers) wrt chatty trimming.
Bug: 23327476
Change-Id: I9f21ce95e969b67e576417a760f75c4d86acf364
Aid monotonic to realtime logging synchronization correction in
the Android ecosystem by providing a periodic notification. We
now have the following messages in the kernel logs:
- PM: suspend entry %Y-%m-%d %H:%M:%S.%09q UTC
- PM: suspend exit %Y-%m-%d %H:%M:%S.%09q UTC
- Suspended for %s.%03q seconds
- healthd: battery l=100 ... %Y-%m-%d %H:%M:%S.%09q UTC
Alter klogd to resynchronize on healthd messages as well.
NB: Time using strftime format, %q is a reference to fractional
second as introduced into log_time strptime method.
Bug: 21868540
Change-Id: I854afc0a07dff9c7f26d2b2f68990e52bf90e300
Required by logd on devices with USE_CPUSETS defined.
Make /dev/cpuset/background, /dev/cpuset/foreground and
/dev/cpuset/task writeable by system gid. Add logd to system
group for writing to cpuset files and to root group to avoid
regressions. When dropping privs, also drop supplementary groups.
Bug: 22699101
Change-Id: Icc01769b18b5e1f1649623da8325a8bfabc3a3f0
- Heuristics associated with translation of kernel messages to
Android user space logs.
- Limit is_prio to 4 characters, we got false positives on hex
values like <register contents> with no alpha chars.
- x11 and other register definitions are not valid tags, en0 is
- fix some Android coding standard issues
Change-Id: Idc3dcc53a2cb75ac38628c8ef7a5d5b53f12587a
(cherry pick from commit ed777e9eec)
Quick low-risk to resolve possible hash table corruption.
Resolved an unlikely path memory leak.
ToDo: replace lock with nested lock so no lock
helpers are required.
Bug: 22068332
Change-Id: I303ab06608502c7d61d42f111a9c43366f184d0c
Quick low-risk to resolve possible hash table corruption.
Resolved an unlikely path memory leak.
ToDo: replace lock with nested lock so no lock
helpers are required.
Bug: 22068332
Change-Id: I303ab06608502c7d61d42f111a9c43366f184d0c
(cherry pick from commit ee49c6a670)
- regression in log_strtok_r (part deux) In commit
'logd: fix kernel logline stutter'
2c3b300fd8 we introduced log_strtok_r.
as a replacement for strtok_r that dealt with a problem with
some kernel log messages. Fix is to refine definition of
is_timestamp to not match on patterns like [0], requiring
a single period. Another fix is to refine definition of
is_prio to properly escape non-digit content.
- Missing content because SYSLOG_ACTION_SIZE_BUFFER with added logging
is too short for full read of SYSLOG_ACTION_READ_ALL dropping
initial content. Add a margin for additional 1024 bytes.
- Absolute _first_ log entry has sequence number of 1, which is
specifically dropped, start sequence count at 1 rather than 0.
- Remove trailing space for efficiency.
- If tag exists but no content, trick into kernel logging.
Bug: 21851884
Change-Id: I0867a555a3bca09bbf18d18e75e41dffffe57a23
- regression in log_strtok_r (part deux) In commit
'logd: fix kernel logline stutter'
2c3b300fd8 we introduced log_strtok_r.
as a replacement for strtok_r that dealt with a problem with
some kernel log messages. Fix is to refine definition of
is_timestamp to not match on patterns like [0], requiring
a single period. Another fix is to refine definition of
is_prio to properly escape non-digit content.
- Missing content because SYSLOG_ACTION_SIZE_BUFFER with added logging
is too short for full read of SYSLOG_ACTION_READ_ALL dropping
initial content. Add a margin for additional 1024 bytes.
- Absolute _first_ log entry has sequence number of 1, which is
specifically dropped, start sequence count at 1 rather than 0.
- Remove trailing space for efficiency.
- If tag exists but no content, trick into kernel logging.
Bug: 21851884
Change-Id: I0867a555a3bca09bbf18d18e75e41dffffe57a23
(cherry pick from commit e59c469fa8)
- Default level when not specified is ANDROID_LOG_VERBOSE
which is inert.
Bug: 20416721
Bug: 19544788
Bug: 17760225
Change-Id: Icc098e53dc47ceaaeb24ec42eb6f61d6430ec2f6
- Default level when not specified is ANDROID_LOG_VERBOSE
which is inert.
Bug: 20416721
Bug: 19544788
Bug: 17760225
Change-Id: Icc098e53dc47ceaaeb24ec42eb6f61d6430ec2f6
(cherry pick from commit d98f4e8af5)
sched_batch implies only a penalty to latency in scheduling, but
does not imply that the process will be given less cpu time. Increase
the nice level to 10 to prioritize it below ui threads.
Bug: 21696721
Change-Id: I075af059dc755402f7df9b0d7a66cca921ff04b2
sched_batch implies only a penalty to latency in scheduling, but
does not imply that the process will be given less cpu time. Increase
the nice level to 10 to prioritize it below ui threads.
Bug: 21696721
Change-Id: I075af059dc755402f7df9b0d7a66cca921ff04b2
(cherry pick from commit 2c3b300fd8)
- look for cases where one log line contains two without a newline.
- rare condition, occurs when a printk does not have
a terminating newline under certain race conditions.
- the newline may be performed broken up as a second call
- the timestamps can be reversed (showing the race effects).
- driver(s) should really have the newline in their log messages.
Change-Id: Ibfb56b32047da3d6513db059ca6edad0f0105168
- look for cases where one log line contains two without a newline.
- rare condition, occurs when a printk does not have
a terminating newline under certain race conditions.
- the newline may be performed broken up as a second call
- the timestamps can be reversed (showing the race effects).
- driver(s) should really have the newline in there log messages.
Change-Id: Ibfb56b32047da3d6513db059ca6edad0f0105168
(cherry pick from commit 047cc0729f)
- Cleanup resulting from experience and feedback
- When filtering inside logd, drop any leading expire messages, they
are cluttering up leading edge of tombstones (which filter by pid)
- Increase and introduce EXPIRE_RATELIMIT from 1 to 10 seconds
- Increase EXPIRE_THRESHOLD from 4 to 10 count
- Improve the expire messages from:
logd : uid=1000(system) too chatty comm=com.google.android.phone,
expire 2800 lines
change tag to be more descriptive, and reduce accusatory tone to:
chatty : uid=1000(system) com.google.android.phone expire 2800
lines
- if the UID name forms a prefix for comm name, then drop UID name
Change-Id: Ied7cc04c0ab3ae02167649a0b97378e44ef7b588
- Cleanup resulting from experience and feedback
- When filtering inside logd, drop any leading expire messages, they
are cluttering up leading edge of tombstones (which filter by pid)
- Increase and introduce EXPIRE_RATELIMIT from 1 to 10 seconds
- Increase EXPIRE_THRESHOLD from 4 to 10 count
- Improve the expire messages from:
logd : uid=1000(system) too chatty comm=com.google.android.phone,
expire 2800 lines
change tag to be more descriptive, and reduce accusatory tone to:
chatty : uid=1000(system) com.google.android.phone expire 2800
lines
- if the UID name forms a prefix for comm name, then drop UID name
Change-Id: Ied7cc04c0ab3ae02167649a0b97378e44ef7b588
(charry pick from commit 511338dd57)
BasicHashtable is relatively untested, move over to
a C++ template library that has more bake time.
Bug: 20419786
Bug: 21590652
Bug: 20500228
Change-Id: I926aaecdc8345eca75c08fdd561b0473504c5d95
BasicHashtable is relatively untested, move over to
a C++ template library that has more bake time.
Bug: 20419786
Bug: 21590652
Bug: 20500228
Change-Id: I926aaecdc8345eca75c08fdd561b0473504c5d95
(cherry pick from commit 100658c303)
- logpersistd is defined as a thread or process in the context of the
logd domain. Here we define logpersistd as logcat -f in logd domain
and call it logcatd to represent its service mechanics.
- Use logcatd to manage content in /data/misc/logd/ directory.
- Only turn on for persist.logd.logpersistd = logcatd.
- Add logpersist.start, logpersist.stop and logpersist.cat debug
class executables, thus only in the eng and userdebug builds.
ToDo: Wish to add Developer Options menu to turn this feature on or
off, complicated by the fact that user builds have no tools with
access rights to /data/misc/logd.
Bug: 19608716
Change-Id: I57ad757f121c473d04f9fabe9d4820a0eca06f31
- logpersistd is defined as a thread or process in the context of the
logd domain. Here we define logpersistd as logcat -f in logd domain
and call it logcatd to represent its service mechanics.
- Use logcatd to manage content in /data/misc/logd/ directory.
- Only turn on for persist.logd.logpersistd = logcatd.
- Add logpersist.start, logpersist.stop and logpersist.cat debug
class executables, thus only in the eng and userdebug builds.
ToDo: Wish to add Developer Options menu to turn this feature on or
off, complicated by the fact that user builds have no tools with
access rights to /data/misc/logd.
Bug: 19608716
Change-Id: I57ad757f121c473d04f9fabe9d4820a0eca06f31
(cherry pick from commit 5921276a16)
Code in 833a9b1e38 was borken,
simpler approach is to simply check last entry (to save a
syscall) minus EXPIRE_HOUR_THRESHOLD. This does make longer logs
less likely to call upon the spam detection than the algorithm
being replaced, but sadly we ended up with a log entry in the
future at the beginning of the logs confounding the previous
algorithm.
Bug: 21555259
Change-Id: I04fad67e95c8496521dbabb73b5f32c19d6a16c2
Code in 833a9b1e38 was borken,
simpler approach is to simply check last entry (to save a
syscall) minus EXPIRE_HOUR_THRESHOLD. This does make longer logs
less likely to call upon the spam detection than the algorithm
being replaced, but sadly we ended up with a log entry in the
future at the beginning of the logs confounding the previous
algorithm.
Bug: 21555259
Change-Id: I04fad67e95c8496521dbabb73b5f32c19d6a16c2
(cherry pick from commit 833a9b1e38)
Do not invoke worst-UID pruning in the face of other
UIDs logs that are more than a day old, switch to
pruning oldest only.
Change-Id: Icf988b8d5458400a660d0f8e9d2df3f9d9a4c2d9
Do not invoke worst-UID pruning in the face of other
UIDs logs that are more than a day old, switch to
pruning oldest only.
Change-Id: Icf988b8d5458400a660d0f8e9d2df3f9d9a4c2d9
(cherry pick from commit 7718778793)
- Android Coding Standard for Constructors
- Side effects NONE
Change-Id: I2cda9dd73f3ac3ab58f394015cb810820093d47b
(cherry pick from commit ae4d928d81)
- Add a klogd to collect the kernel logs and place them into a
new kernel log buffer
- Parse priority, tag and message from the kernel log messages.
- Turn off pruning for worst UID for the kernel log buffer
- Sniff for 'PM: suspend exit', 'PM: suspend enter' and
'Suspended for' messages and correct the internal definition
time correction against monotonic dynamically.
- Discern if we have monotonic or real time (delineation 1980) in
audit messages.
- perform appropriate math to correct the timestamp to be real time
- filter out any external sources of kernel logging
Change-Id: I8d4c7c5ac19f1f3218079ee3a05a50e2ca55f60d
- Add a klogd to collect the kernel logs and place them into a
new kernel log buffer
- Parse priority, tag and message from the kernel log messages.
- Turn off pruning for worst UID for the kernel log buffer
- Sniff for 'PM: suspend exit', 'PM: suspend enter' and
'Suspended for' messages and correct the internal definition
time correction against monotonic dynamically.
- Discern if we have monotonic or real time (delineation 1980) in
audit messages.
- perform appropriate math to correct the timestamp to be real time
- filter out any external sources of kernel logging
Change-Id: I8d4c7c5ac19f1f3218079ee3a05a50e2ca55f60d
(cherry pick from commit 344bff4391)
- Optional class of statistics for events Tags
- export tagToName from LogBuffer (located in main.cp to address
https://android-review.googlesource.com/#/c/110204)
- Can not handle dropped because getTag() can not work,
will need to be fixed if we start filtering.
Bug: 19608965
Change-Id: I7b90607ca588bf37bab71f19b1570a290e772776
- Optional class of statistics for events Tags
- export tagToName from LogBuffer (located in main.cp to address
https://android-review.googlesource.com/#/c/110204)
- Can not handle dropped because getTag() can not work,
will need to be fixed if we start filtering.
Bug: 19608965
Change-Id: I7b90607ca588bf37bab71f19b1570a290e772776
(cherry pick from commit 023f51f360)
- When searching for uidToName from /data/system/packages.list
one must search for the uid % AID_USER;
- If uid % AID_USER is greater than AID_SHARED_GID_START - AID_APP,
then subract it and try again to pick up uidToName.
Change-Id: I24df42957ff7e746fffa5d39484637b1b726b9c0
The shamu instabilities continued even after throttling SELinux denials
to 5/second. 5 denials per second is too low when doing device bringup,
and there have been some complaints about lost SELinux denials. See,
for example, http://comments.gmane.org/gmane.comp.security.selinux/21941
Bring the limit back up to 20/second to prevent dropping too many
denials on the floor.
This reverts commit a15db51bbf.
(cherrypick of commit 9667a66019)
Change-Id: I05e85cce0a792d05aa557fcc614c0fc019c15014
The shamu instabilities continued even after throttling SELinux denials
to 5/second. 5 denials per second is too low when doing device bringup,
and there have been some complaints about lost SELinux denials. See,
for example, http://comments.gmane.org/gmane.comp.security.selinux/21941
Bring the limit back up to 20/second to prevent dropping too many
denials on the floor.
This reverts commit a15db51bbf.
Change-Id: I05e85cce0a792d05aa557fcc614c0fc019c15014
(Cherry pick from commit 81b3eabc49)
Add EntryBase and EntryBaseDropped base classes for statistical
entries to inherit from. Abstract add(), subtract() and drop()
methods to common LogBufferElement in each for entry policy decisions.
Some move of details out of LogStatistics.cpp and place them into
LogStatistics.h. Add statistical add(), subtract() and
drop() methods to hash table to call entries for policy.
Bug: 19608965
Change-Id: Ib8a33a8fe28871ef165d1632c6546a5c606231e8
- When searching for uidToName from /data/system/packages.list
one must search for the uid % AID_USER;
- If uid % AID_USER is greater than AID_SHARED_GID_START - AID_APP,
then subract it and try again to pick up uidToName.
Change-Id: I24df42957ff7e746fffa5d39484637b1b726b9c0
(Cherry pick from commit 21fb7e0b75)
- Report applications UID, TID/PID by name.
- change wording to have an accurate connotation
- drop privilege check since filtered upstream
Bug: 19608965
Bug: 20334069
Bug: 20370119
Change-Id: I2b1c26580b4c2de293874214ff5ae745546f3cca
- do not time out at 1 second if drop has count of less than 4
(Cherry picked from commit 35173a9ac5)
Bug: 20334069
Bug: 20370119
Change-Id: I787cb553dfab5ed71abd6ed72b63de675f834e0c
- Report applications UID, TID/PID by name.
- change wording to have an accurate connotation
- drop privilege check since filtered upstream
Bug: 19608965
Bug: 20334069
Bug: 20370119
Change-Id: I2b1c26580b4c2de293874214ff5ae745546f3cca
Per-UID quota has a threshold of 12.5% of the total log size.
If less than that space is taken by the UID, then we
will not engage the pruning based on worst UID.
Change-Id: I9f15c9a26938f1115eb75e9c28ddb073e7680e06
Add EntryBase and EntryBaseDropped base classes for statistical
entries to inherit from. Abstract add(), subtract() and drop()
methods to common LogBufferElement in each for entry policy decisions.
Some move of details out of LogStatistics.cpp and place them into
LogStatistics.h. Add statistical add(), subtract() and
drop() methods to hash table to call entries for policy.
Bug: 19608965
Change-Id: Ib8a33a8fe28871ef165d1632c6546a5c606231e8
- Former algorithm anlo coalesced adjacent records
- New algorithm maintains a hash list of all drop
records and coalesces them all.
Bug: 20334069
Bug: 20370119
Change-Id: Idc15ce31fc1087c2cfa39da60c62feade8b88761
Add a return value for the ::log() methods, this allows
us to optimize the wakeup for the readers to only occur
when the log message is actually placed.
This is for a future where we may dedupe identical log
messages, filter out log messages, and certainly if we
filter the messages out with an internal logd check of
__android_log_is_loggable().
Change-Id: I763b2a7c29502ab7fa0a5d5022c7b60244fcfde4
There is some usage statistics that would be hurt by pruning by UID,
since _all_ usage statistics come from system_server. In other words
we expect it to be chatty. Until we formulate and evaluate a better
(eg: per-tag?) filtration mechanism, lets hold off pruning by UID.
Bug: 19608965
Change-Id: Iddd45a671e13bdcf3394c20919ad1f2e4ef36616
With "logd: inject first UID by log buffer message" we can
remove the FUD introduced by having the UID spam filter on
This reverts commit 4141cb2391
Bug: 19608965
Bug: 14469172
Change-Id: Ifdc20b099e0e426546525b11c1dfe8cc0f830a02
- internal dropped entries are associated by prune by worst UID
and are applied by UID and by PID
- track dropped entries by rewriting them in place
- merge similar dropped entries together for same UID(implied),
PID and TID so that blame can more clearly be placed
- allow aging of dropped entries by the general backgound pruning
- report individual dropped entries formatted to reader
- add statistics to track dropped entries by UID, the combination
of statistics and dropped logging can track over-the-top Chattiest
clients.
Bug: 19608965
Change-Id: Ibc68480df0c69c55703270cd70c6b26aea165853
- read packages.list to acquire package names
- hijack the reinit handler as a privileged worker
Bug: 19608965
Change-Id: If7becb34354d6415e5c387ecea7d4109a15259c9
C (but not C++) has a concept of a flexible array member, which
is documented at https://gcc.gnu.org/onlinedocs/gcc/Zero-Length.html .
Using a flexible array member indicates that the structure is
really a header for a variable length object.
In logd's case, the variable length structure android_event_string_t
was embedded within another structure called
android_log_event_string_t. This makes gcc's __builtin_object_size()
function really confused. When compiling with C++,
__builtin_object_size(android_log_event_string_t.payload.data, 1)
would return 0, whereas if you compiled the code with C, the same
call would (properly) return -1.
Code which does automatic bounds checking, such as the proposed
patch at https://android-review.googlesource.com/145411 , will
cause problems for logd if this syntax is used.
Don't try to embed a variable length structure within another
structure. This doesn't appear to be valid C nor C++, and
while it's worked, it seems problematic.
Instead, inline the structure so it's one big happy structure.
Change-Id: I8ac02b7142a4f6560f5f80df2effcf720f9896fc
LogCommand.cpp gets string.h inherited from
private/android_filesystem_config.h it should
not rely on this in the future. The intent is
to move fs_config function into libcutils and
thus deprecate any need for string.h in this
include file.
Bug: 19908228
Change-Id: Iaf3a77298b56efd8004300b17e9a1faafad5b08b
watchdog is triggering on shamu. This may be due to an excessive
number of SELinux denials. Drop the limit from 20/sec to 5/sec.
Bug: 19950451
Bug: 19949988
Change-Id: I979f11e17c241ff2ebda4dec9694ef441dc5d0ed
- Go back to basic requirements
- Simplify
- use hash tables to minimize memory impact
Bug: 19608965
Change-Id: If7becb34354d6415e5c387ecea7d4109a15259c8
- switch to simpler and faster internal sequence number, drops
a syscall overhead on 32-bit platforms.
- add ability to break-out of flushTo loop with filter return -1
allowing in reduction in reader overhead.
Change-Id: Ic5cb2b9afa4d9470153971fc9197b07279e2b79d
This forward port reverts
commit e457b74ce6
No longer as necessary once we add
liblog: Instrument logging of logd write drops
Although this provided an indication of how close statistically we
were to overloading logd it is simpler to understand fails thus to
hunt and peck a corrected value for /proc/sys/net/unix/max_dgram_qlen
Change-Id: I2b30e0fc30625a48fd11a12c2d2cc6a41f26226f
logd suffers major performance degradation when persistent (blocking)
client reader connects to it (e.g. logcat). The root cause of the
degradation is that each time when reader is notified of the arrival
of new log entries, the reader commences its search for the new entries
from the beginning of the linked list (oldest entries first).
This commit alters the search to start from the end of the linked list
and work backwards. This dramatically decreases logd CPU consumption
when blocking reader is connected, and increases the maximum logging
throughput (before the logs start getting lost) by a factor ~ 20.
Change-Id: Ib60955ce05544e52a8b24acc3dcf5863e1e39c5c
- create a structure to depict the private header
expected at logd end of socket.
- utilize this new structure instead of unscalable
byte stream technique used to unpack in logd.
Change-Id: I2d0e5c3531c279f2dc1fbd74807210ff8d804de0
In a scenario in which an on-line (blocking) client is running and
a clean is attempted (logcat -c), the following can be observed:
1) the on-line logger seems to freeze
2) any other clear attempt will have no effect
What is actually happening:
In this case prune function will "instruct" the oldest timeEntry
to skip a huge number (very close to ULONG_MAX) of messages, this
being the cause of 1.
Since the consumer thread will skip all the log entries, mStart
updating will also be skipped. So a new cleaning attempt will have
the same oldest entry, nothing will be done.
Fix description:
a. keep a separated skipAhead count for individual log buffers (log_id_t)
LogTimeEntry::LogTimeEntry
LogTimeEntry::FilterSecondPass
LogTimeEntry::skipAhead
LogTimeEntry::riggerSkip_Locked
b. update LogTimeEntry::mStart even if the current message is skipped
LogTimeEntry::FilterSecondPass
c. while pruning, only take into account the LogTimeEntrys that are monitoring
the log_id in question, and provide a public method of checking this.
LogTimeEntry::isWatching
LogTimeEntry::FilterFirstPass
LogTimeEntry::FilterSecondPass
d. Reset the skip cont befor the client thtread starts to sleep, at this point
we should be up to date.
LogTimeEntry::cleanSkip_Locked
LogTimeEntry::threadStart
Change-Id: I1b369dc5b02476e633e52578266a644e37e188a5
Signed-off-by: TraianX Schiau <traianx.schiau@intel.com>
- logDmesg method consumes considerable memory
resources (typically 128KB depending on kernel)
- In the future (eg: klogd, syslogd) there may be need to
feed multiple logs or threads with the retrieved data.
- By moving the actions of logDmesg into the mainline that
instantiates the thread objects, we can leverage a single
allocation of the the kernel log allocation.
- logDmesg (private) is replaced with log (public) which
has a more useful and descriptive purpose for the class.
Change-Id: Ie2dd0370661493c1e596a7e486904a0e8caab9ff
Impose a limit of 20 selinux denials per second. Denials beyond
that point don't add any value, and have the potential to cause
crashes or denial of service attacks.
Do some other misc cleanup while I'm here.
Bug: 18341932
Change-Id: I6125d629ae4d6ae131d2e53bfa41e1f50277d402
- Recognize pid=0 as special case (kernel or pre-init sourced)
and refrain from treating it in the general case.
Bug: 17526159
Change-Id: I74796043ac34753c6dd10018719ebc0bcd94e012
- simplify access and control exposure to class list
- indent
- compile warning
- Follow standard naming convention for variables and methods
- merge common fragments
- Side Effects: none
Change-Id: I74796043ac34753c6dd10018719ebc0bcd94e010
- sched_yield and lock synchronization in reader thread
startup to give writer thread a chance to catch up
Bug: 17512203
Change-Id: I43cf0b4e2829b22b3ab4e537fa95ce13c76a869c
Added parentheses as it was suggested by compiler.
error: suggest parentheses around operand of '!' or change '&' to '&&' or '!' to '~' [-Werror=parentheses]
Change-Id: Ic3746d9804488411d10d460ddfda11f1fad8efef
Signed-off-by: Arseniy Antonov <arseniy.antonov@intel.com>
- Correct default tense for kernel or user logger
- harden code that parses the benchmark data
- give more meaning to the spam filter test
Change-Id: I97699037ca9d56718f53f08d22be79092fb431de
- auditd spawn log copy to events and main
- logcat delete events as one of the default logs
- debuggerd do not collect events.
- squish multiple spaces
- switch from strcpy to memmove for overlapping buffers
BUG: 14626551
Change-Id: I89b30273ce931ed2b25a53ea9be48e77f4c1bbf4
- auditd switch to recording logs to events log id
- logcat add events as one of the default logs
- debuggerd collect events log as well.
ToDo: debuggerd & bugreport collect intermixed logs.
BUG: 14626551
Change-Id: I958f0e729b7596748be57488a38824db5645be7b
- AID_SYSTEM can set persist.logd.size
- AID_SYSTEM can issue command to /dev/socket/logd to
change the runtime global log sizes.
- Add support for ro.logd.size.* as populated by BoardConfig.mk
- Limit size to maximum ~3% of physical memory.
Bug: 14563261
Bug: 14627052
Change-Id: I606420db2e9d9d032296e71149e4a5b20cbd1137
- permits easier determination of logd thread at
fault in a stack trace from debuggerd.
Bug: 14275676
Change-Id: Iac2c523147e2bcce34ab7ddcecd02582c5fa7cc0
- Drop in-place sorting.
- Add fast-track for uid sizes and elements.
- Add sort() for per-uid and per-pid statistics.
Change-Id: Ib8655d4cc0b3bd4e87534522987bcc7fc7a5251a
- if network read/write broken up, reassemble the pieces.
- Use a 20ms poll to check if a new fragment has been
sent by the other side.
- fixup logd-unit-tests to take a (simplified) fragment
from the liblog changes.
Bug: 14164765
Change-Id: I98ff87888c119e1e8349717646d0f733e8971bc8
* Test statistics output, all four main log id types
are checked to be present, and output is properly
formatted with header (ascii new-line terminated
byte count) and trailer (newline formfeed) frames.
* On dev build test dgram_qlen, circumvent if
liblog benchmarks executed first though.
* Test if kernel and user space loggers are
present, and if any content has leaked to both.
* test benchmark and use it to verify worst UID pruning
Change-Id: I890c89f64825956f40108de806ed750b0170ac6c
Fixes a leak and use-after-free if a client passes in an
already-allocated target buffer (currently none do)
Change-Id: Ie20c30bd5ba31e1b6db73885bd9d2ebd7e40dc27
Signed-off-by: Greg Hackmann <ghackmann@google.com>
Initial commit for an audit daemon that writes kernel audit
messages to the Android logger. The daemon searches dmesg
for all lines that contain "audit" and writes them. Then
receiving the messages from the netlink socket.
It also formats the messages so they are compatable with
ausearch (type=<t> <m> format)
Modified: Mark Salyzyn <salyzyn@google.com>
- do not start auditd
- merge into logd, stripping unnecessary file logging.
- Convert headers and code to support C++
- Fix bugs in libaudit
- squash timestamp (replace with 0.0) due to duplication
- squash pid due to duplication
- squash comm due to duplication
Change-Id: I421bcf33e7e670d596628b1b5c7c25536ce2d3fe
* Only enabled for dev builds because halves performance.
- Used to establish if /proc/sys/net/unix/max_dgram_qlen
is appropriate for the platform
(see logd/LogStatistics.cpp comments)
- enabled if logd.dgram_qlen.statistics is not zero/empty
Change-Id: Ib8ecfeb75b6f0f954c099a6b1032b8afb5efcbd4
- logcat improperly squashes out count
- logcat test enhanced to catch failure
- logd places entries in ascending sorted order
Change-Id: If39d8e2c40852b520c98e7840034ab63c2538e5d
- liblog android_logger_get_log_size and android_logger_get_readable_size
adjusted to return long instead of int because of -G flag extending range
NB: ifdef'd only for userdebug and eng builds
- liblog Add android_logger_[sg]et_prune_list and android_logger_set_log_size
- logcat Add -P, -p and -G flags
- logd Add LogWhiteBlackList and configurable log size
(cherry picked from commit 18a5432158)
Change-Id: I1572338c1b34bd968ad7867857ef708156ec3b6a
- structure packing
- move towards log_time from struct timespec
- extend log_time to cover differences between
log_time and struct timespec
Change-Id: I106ed0b609917306d170044054b5b32645f2a295
- init: set /proc/sys/net/unix/max_dgram_qlen to 300
- libsysutils: Add listen backlog argument to startListener
- logd: set listen backlog to 300
Change-Id: Id6d37d6c937ba2d221e76258d89c9516619caeec
* Modify liblog to send all messages to the new syslog user
space daemon.
Original-Change-Id: I0ce439738cd921efb2db4c1d6a289a96bdbc8bc2
Original-Change-Id: If4eb0d09409f7e9be3eb4bb7017073dc7e931ab4
Signed-off-by: Nick Kralevich <nnk@google.com>
* Add a TARGET_USES_LOGD make flag for BoardConfig.mk to manage
whether logd is enabled for use or not.
* rename syslog to logd to avert confusion with bionic syslog
* Add fake log support back in
* prefilter for logging messages from logd
* Fill in timestamps at logging source
* update abstract log reader
* switch from using suffix for id to v3 format
* log a message when creating devices that a deprecated interface
is being utilized.
Signed-off-by: Mark Salyzyn <salyzyn@google.com>
(cherry pick from commit 099e2c1f6f706a8600c1cef74cce9066fc315480)
Change-Id: I47929a5432977a1d7235267a435cec0a7d6bd440
