Skip to main content

SYSLOG Check Failed

Comments

8 comments

  • cPRex Jurassic Moderator
    Hey there! Do you see anything like "messages lost due to rate-limiting" in /var/log/messages? If so, this could be relevant:
    0
  • coursevector
    I did see it in the service status journal logs. But looking in /messages specifically I see it as well: [CODE=bash][root@host ~]# grep "limiting" /var/log/messages Nov 30 00:51:11 host rsyslogd: imjournal: 3181 messages lost due to rate-limiting Nov 30 01:01:12 host rsyslogd: imjournal: 2765 messages lost due to rate-limiting Nov 30 01:11:13 host rsyslogd: imjournal: 2490 messages lost due to rate-limiting Nov 30 01:21:14 host rsyslogd: imjournal: 2535 messages lost due to rate-limiting Nov 30 01:31:16 host rsyslogd: imjournal: 536 messages lost due to rate-limiting
    I looked over that link you sent a little bit, but I was curious how cPanel handles it. Does cPanel adjust the defaults in anyway? If not, is it safe to assume it's using the default of 200 messages in 5 seconds? If so, is it bad to increase this to say 1000 messages?
    0
  • cPRex Jurassic Moderator
    Great - so that's what happening then. cPanel doesn't do anything with that on our side, but this just means there were too many messages to print at once due to the rate limit settings. Instead of bumping that, since you seem to be frequently hitting 2500+, it might be best to review the log to see what is writing so much data so see if it could be resolved at the source.
    0
  • coursevector
    I see a lot of these, is this caused by backups? Not familiar with that message. That continues for each of the accounts (112) for about 30min. With about 20 a second by the looks of it, i could see how it could quickly hit that limit. I did some research and found this Logs flooded with systemd messages: Created slice, Starting Session - Red Hat Customer Portal, would you say that's a good solution for this situation? Not overly familiar with this logging so I don't want jump the gun here. [CODE=bash]Nov 30 01:01:38 host systemd: Removed slice User Slice of . Nov 30 01:01:38 host systemd: Created slice User Slice of . Nov 30 01:01:38 host systemd: Started Session c167544 of user . Nov 30 01:01:38 host systemd: Removed slice User Slice of . Nov 30 01:01:38 host systemd: Created slice User Slice of . Nov 30 01:01:38 host systemd: Started Session c167545 of user . Nov 30 01:01:38 host systemd: Removed slice User Slice of . Nov 30 01:01:38 host systemd: Created slice User Slice of . Nov 30 01:01:38 host systemd: Started Session c167546 of user . Nov 30 01:01:38 host systemd: Removed slice User Slice of . Nov 30 01:01:38 host systemd: Created slice User Slice of . Nov 30 01:01:38 host systemd: Started Session c167547 of user . Nov 30 01:01:38 host systemd: Removed slice User Slice of . Nov 30 01:01:38 host systemd: Created slice User Slice of . Nov 30 01:01:38 host systemd: Started Session c167548 of user . Nov 30 01:01:38 host systemd: Removed slice User Slice of . Nov 30 01:01:38 host systemd: Created slice User Slice of . Nov 30 01:01:38 host systemd: Started Session c167549 of user . Nov 30 01:01:38 host systemd: Removed slice User Slice of . Nov 30 01:01:38 host systemd: Created slice User Slice of . Nov 30 01:01:38 host systemd: Started Session c167550 of user .
    0
  • cPRex Jurassic Moderator
    I'd try it - it's normal behavior, and not related to cPanel, so trying that filter how they recommend would also be my best recommendation.
    0
  • coursevector
    For future reference, I added this to one of my servers to test. We'll see if this fixes the issue. [CODE=bash]echo 'if $programname == "systemd" and ($msg contains "Starting Session" or $msg contains "Started Session" or $msg contains "Created slice" or $msg contains "Starting user-" or $msg contains "Starting User Slice of" or $msg contains "Removed session" or $msg contains "Removed slice User Slice of" or $msg contains "Stopping User Slice of" or $msg contains "Removed slice" or $msg contains "Stopping user-") then stop' >/etc/rsyslog.d/ignore-systemd-session-slice.conf systemctl restart rsyslog
    0
  • coursevector
    I think it helped but it didn't fix the issue. I still got the SYSLOG notice this morning. I looked at the looked at the service log and see this: [CODE=bash]Dec 02 01:10:47 host.example.com rsyslogd[697298]: imjournal: begin to drop messages due to rate-limiting Dec 02 01:11:32 host.example.com rsyslogd[697298]: imjournal: 1569 messages lost due to rate-limiting Dec 02 01:21:24 host.example.com rsyslogd[697298]: imjournal: begin to drop messages due to rate-limiting Dec 02 01:21:33 host.example.com rsyslogd[697298]: imjournal: 316 messages lost due to rate-limiting Dec 02 01:30:04 host.example.com rsyslogd[697298]: imjournal: begin to drop messages due to rate-limiting Dec 02 01:31:37 host.example.com rsyslogd[697298]: imjournal: 559 messages lost due to rate-limiting
    [CODE=bash][root@host ~]# grep "limiting" /var/log/messages Dec 2 00:51:29 host rsyslogd: imjournal: 3525 messages lost due to rate-limiting Dec 2 01:01:30 host rsyslogd: imjournal: 2200 messages lost due to rate-limiting Dec 2 01:11:32 host rsyslogd: imjournal: 1569 messages lost due to rate-limiting Dec 2 01:21:33 host rsyslogd: imjournal: 316 messages lost due to rate-limiting Dec 2 01:31:37 host rsyslogd: imjournal: 559 messages lost due to rate-limiting
    I checked /var/log/messages again and counted the entries between 01:10 and 01:11 and there was only 388. I checked the conf files again and don't see anything explicitly set. But I also can't figure out what the limit actual is. Can you help there? [CODE=bash] [root@host ~]# grep -i rate /etc/systemd/journald.conf #RateLimitInterval=30s #RateLimitBurst=1000 [root@host ~]# grep -i rate /etc/rsyslog.conf
    0
  • cPRex Jurassic Moderator
    I would also expect those values to be controlled as part of the /etc/systemd/journald.conf file with the two values you provided in that last reply. Setting those to 0 would disable the values, so it might be best to uncomment those if you wanted to adjust the values.
    0

Please sign in to leave a comment.