Hopefully allowed to touch base here and keep this baby rolling, looks like one person answered and did a fantastic job, I'm not well versed in all things Red Hat but, while rsync is ideal for file translation, interruptions, integrity via checksums, compression, and just by typing rsync of course. It's main purpose, correct me if I'm wrong is delta transfers, maybe I shouldn't say the main purpose, but it's efficiency. Which in this case is less relevant for rotated logs, which are typically new files. Rsync can be resource heavy especially for 4GiB files, if that's a reachable size for your ever so clever app titled 'app'. Offloading the logs to a remote host is a reactive measure that addresses the symptom, but not the underlying problem of why the log is growing rapidly. So now we have continued I/O strain, potential remote issues, potential missed diagnostics, and performance overhead. Especially if the network to 'remote_host' is slow or unreliable, could add to the I/O wait during postrotate phase. SSH dependency - Access and auth, if either fail the postrotate script will fail, with potential logrotate and the system in an inconsistent state. Just to stir things up even more, the wildcard is brave, although it's a questionare and the environment seems pretty assumable. 0 error handling, which again, controlled environment, just messing around, who needs it. Verbose output might generate more than what you want to chew up for logrotate. Ultimately just adding more I/O calls to the system.
Excessive writes, causing I/O wait during writes, even if rotated frequently.
Not sure the entire background for our lovely remote host, but hopefully it's an audit and purge factory. If not, welp, we got another issue.
Offloading the log brushes any application error, misconfig, attack, etc. off to the next or nomansland - void.
The overhead of frequent rotations is viable in high load environments, which hey, I'm assuming everythings game.
Let's first off start by checking out what ol app has so much to talk about and why it wont be quite.
No code block so I'll pretend Markdown exists. Nevermind that's awful we will go for bold.
head -n 100 /var/log/app.log
tail -n 100 /var/log/app.log
Maybe if we dig a little more we can systemd and journal it.
systemd status app.service
journalctl -xeu app --since "1 hour ago" | wc -l
journalctl -xeu app.service -b
Nerd it up a little bit and quantify the log rate. A thousand per minute is definitely a sign of overlogging.
grep "$(date '+%Y-%m-%d %H:%M')" /var/log/app.log | wc -l
Hopefully slide on by with adjusting the log level of the application. Or filtering in rsyslog.
Guess if not, we are in for a bit of a game.
grep -i "error\|exception\|failed" /var/log/app.log | sort | uniq -c | sort -nr | head -n 10
Cross your fingers it's an error loop, pumping messages / stack traces. Evaluate the logs and determine the next best approach. Hopefully a simple configuration error or permissions issue.
We can review system resources at this point,
top -b -n 1
free -h (or -m if you really want it all)
Check for rsyslog usage.
lsof /var/log/app.log
cat /etc/rsyslog.conf /etc/rsyslog.d/app.conf
Temporarily suppress repetitive logs assuming syslog, which is probably harder than log levels, none the less.
# /etc/rsyslog.d/app.conf
if $programname == 'app' and $msg contains "error_string_not_creative_tonight" then stop
Or even better, property filtering, which I think is standard. Maybe not this mess, but standalone.
# /etc/rsyslog.d/app.conf
:programname, isequal, "app" {
:msg, contains, "error_string_still_not_creative" stop
:msg, regex, "DEBUG|TRACE" stop
/var/log/app.log
}
& stop
Could be a lot prettier if we had code blocks or Markdown enabled.
Don't let me forget SELinux or restart rsyslog - shhhhh
restorecon -v /etc/rsyslog.d/app.conf
chcon -t var_log_t /var/log/app.log
ausearch -m avc -ts recent | grep "rsyslog" (or app perhaps)
systemctl restart rsyslog
If app writes directly to app.log we can potentially modify the app's log config or use a pipe to redirect the logs to rsyslog. No need to do so, just being a nerd, so we can skip this step.
Check for high %iowait / latency
df -T /var/log
findmnt /var/log
ioping -c 10 /var/log
iostat -x 1 10
~~ Optimize filesystem mount options - you get the idea I assume. Can provide more details upon request. ~~
Guess we continue resource checks depending on the output of the log, if we didn't already fix the issue via permissions, fs issues, connection, configuration, etc.
lsof -p 1993
free -h (or -m)
ss | ip | netstat -tuln - (Don't pipe those together, netstat is oldschool, ss is modern, ip route is ip route)
ss -tuln sport = :1993
ss -tunp | grep 1993
awk '{print $1}' /var/log/app.log | sort | uniq -c | sort -nr | head -n 10
tcpdump -i eth0 port 1993 -c 100
Maybe a game to be played after all? High traffic, sad face. Potential attack, happy face.
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="0.0.0.0/0" port port="1993" protocol="tcp" limit value="100/m" reject'
firewall-cmd --reload
semanage fcontext -a -t firewalld_var_run_t "/var/run/firewalld.*"
restorecon -R -v /var/run/firewalld
Might as show you ipv6 too.
firewall-cmd --permanent --add-rich-rule='rule family="ipv6" source address="::/0" port port="1993" protocol="tcp" limit value="100/m" reject'
firewall-cmd --reload
semanage fcontext -a -t firewalld_var_run_t "/var/run/firewalld.*"
restorecon -R -v /var/run/firewalld
Could do more error handling, but I've spent enough time on this lol.
firewall-cmd --runtime-to-permanent
firewall-cmd --reload
semanage fcontext -a -t firewalld_var_run_t "/var/run/firewalld.*"
ausearch -m avc -ts recent | grep firewalld
Can also do some error handling, if interested let me know.
Traffic taken care of, this moves on to just maybe we get to play a game. Planning and preparation phase, establishing objectives, scope, and rules of engagement.
To be continued...
Welp, no fun here, we probably fixed the issue by now or know what the problem is, not sure if we are developing the app, but if we are, here we come source code. If not consultations at its best. Good luck.
Determine user and group.
systemctl cat app.service
ps -auxZ | grep app
"[Service]
User=app
Group=app"
The entire post really spawned from this, my bad.
# /etc/logrotate.d/app
maxsize 4G
rotate 4
compress
delaycompress
missingok
notifempty
create 0640 USER GROUP
postrotate
/usr/bin/systemctl reload app.service > /var/log/logrotate_app.log 2>&1 || {
echo "systemctl reload failed for app.service at $(date)" >> /var/log/logrotate_errors.log
/usr/sbin/sendmail -s "Logrotate Reload Failure" aferguson@ordl.org < / /var/log/logrotate_errors.log
exit 1
}
endscript
}
restorecon -v /etc/logrotate.d/app
chcon -t var_log_t /var/log/logrotate_app.log /var/log/logrotate_errors.log
ausearch -m avc -ts recent | grep logrotate | audit2allow -M logrotate_app
semodule -i logrotate_app.pp
Yikes it looked halfway decent before the bold, I copy and pasted from blank page and some of the line spacing also got skewed. We definitely need a code block or ability to write in markdown.
Open Research and Development Laboratories
Enterprise Systems Architect
Kernel Engineer
|RH|FOSS|
Concurrency | Dynamics | Mutants | Memory Space
