rsyslogd fails to start due to too many journal files

Symptoms

You may see rsyslogd fail with error similar to the following.

Jul 16 09:26:03 host.domain.tld rsyslogd[1619937]: imjournal: open on state file `/var/lib/rsyslog/imjournal.state' failed [v8.24.0-57.el7_9.1 try http://www.rsyslog.com/e/2013 ]

Description 

This is due to the number of journal files reaching the limit (default 1024). This can be checked with the following command.

journalctl --header | grep -c 'File Path:'

Workaround

Clean-up old journal files:

1) Use journalctl to clean up older journal files. The example below deletes journal files older than two weeks.

journalctl --vacuum-time='2 weeks'

2) Verify the number of journal files decreased:

journalctl --header | grep -c 'File Path:'

3) Restart the rsyslog service:

systemctl restart rsyslog

4) Consider adjusting the configuration of systemd-journald in /etc/systemd/journald.conf to use fewer, but larger journal files and/or lower total storage space. See man journald.conf for details.

Alternatively, you can configure rsyslogd to allow more open files, if needed. The default maximum is 4096 open files. For even more open files, the LimitNOFILE limit needs to be adjusted in the [Service] section of the rsyslog.service systemd unit file.

1) Edit /etc/rsyslogd.conf and add the following statement to the GLOBAL DIRECTIVES section:

$MaxOpenFiles 4096

2) Restart the service:

systemctl daemon-reload
systemctl restart rsyslog