How to filter data from large logs such as /var/log/messages in order to more easily identify problems

Introduction

Sometimes there can be a massive amount of noise in some logs that can prevent a systems administrator from being able to diagnose an issue or identify a problem.

Some common examples would be ftp and firewall logs found in /var/log/messages. While ftp and firewall logs are very important for troubleshooting those systems, they can be so numerous that they make it impossible to review other issues in the system log.

You can use the following procedure to filter out irrelevant logs and get to the core of the important information in large noisy logs.

Procedure

This proceedure will use the /var/log/messages log as the working example, though you can adapt this same technique to any log.

1. Login to the server via Terminal or SSH as the root user

2. Start by reviewing the raw log with the less command:

less /var/log/messages

3. While reviewing the raw log, identify patterns for logs that are getting in your way. A common example would be firewall and ftp logs. Firewall logs have a unique but consistent pattern that we can make use of. That is the fact that they all contain the word "Firewall" with a capital F. The ftp logs also have a unique but consistent pattern where they all contain the word "ftpd".

4. To take advantage of those patterns to filter out those logs, exit less by pressing the q key, and then use the grep -v flag to remove all logs that match those unique patterns, and then pipe the remaining logs into the less command:

grep -v "ftpd\|Firewall" /var/log/messages | less

Notice that there is a \| separating the two patterns. This tells grep to remove any log lines that contain "ftpd" OR "Firewall". The pipe character (|) means OR. Because it needs to be interpreted as s special character, you need to escape it with the backslash: \ . You can string together as many patterns in this manner as you would like.

5. Now that you have removed those log lines and you are viewing the remaining logs, start to look for your desired information. If you find other logs that are getting in your way, repeat steps 3 and 4 until you have pared down the information to only what you need to see.