How to create a custom access log for NGINX that shows the cache status for each request

Introduction

The following guide can be used to enable logging of the $upstream_cache_status variable in NGINX.

The $upstream_cache_status variable contains one of the following based upon how NGINX handled caching for the request:

MISS – The response was not found in the cache and so was fetched from an origin server. The response might then have been cached.
BYPASS – The response was fetched from the origin server instead of served from the cache because the request matched a proxy_cache_bypass directive. The response might then have been cached.
EXPIRED – The entry in the cache has expired. The response contains fresh content from the origin server.
STALE – The content is stale because the origin server is not responding correctly, and proxy_cache_use_stale was configured.
UPDATING – The content is stale because the entry is currently being updated in response to a previous request, and proxy_cache_use_stale updating is configured.
REVALIDATED – The proxy_cache_revalidate directive was enabled and NGINX verified that the current cached content was still valid (If-Modified-Since or If-None-Match).
HIT – The response contains valid, fresh content direct from the cache.

Procedure

1. Login to the server via SSH or Terminal as the root user
2. Create a configuration file to store your custom log format under the /etc/nginx/conf.d/ directory. It can be named anything as long as it ends with .conf.

   touch /etc/nginx/conf.d/custLogFormat.conf

3. Open the file with the text editor of your choice and add the following contents to the file.
   Take note that the new log will be located at: /var/log/nginx/accessWithCacheStatus.log
   

   log_format cacheStatus '$host $server_name $server_port $remote_addr $upstream_cache_status $remote_user [$time_local] " $request " '
                          '$status $body_bytes_sent "$http_referer" '
                          '"$http_user_agent" "$http_x_forwarded_for"';
   
   access_log /var/log/nginx/accessWithCacheStatus.log cacheStatus;

4. The access_log directive above will only log requests that are not directed to a specific servername (vhost). In order to enable this same custom log for all of the domains on the server, you may run the following loop via SSH:

   awk -F"[ :=]" '{print $1,$3}' /etc/userdatadomains | while read -r domain user;do mkdir -vp /etc/nginx/conf.d/users/$user/$domain/; echo "access_log /var/log/nginx/accessWithCacheStatus.log cacheStatus;" > /etc/nginx/conf.d/users/$user/$domain/customCacheAccessLog.conf;echo "Created /etc/nginx/conf.d/users/$user/$domain/customCacheAccessLog.conf";done

5. Once you have all of the configurations in place, you may reload NGINX with the following command:

   /scripts/ea-nginx reload

6. Then you may tail the new log with the following command while you visit some of your sites in browser to test the log:

   tail -fn0 /var/log/nginx/accessWithCacheStatus.log

IMPORTANT: Please note that this newly created log may not have any log rotation enabled for it. You may either manually rotate the log or configure log rotation for it if you need to leave it enabled for any extended period of time. If left unattended, the size of the log could grow large enough to use all of the disk space on the server and cause a crash.

Log Format Note: You'll notice that the domain is included twice in the resulting log which may appear redundant at first. The reason for this is that the first domain is populated by the $host variable, whereas the second domain is populated by the $server_name variable. While these contain the same domain for the majority of requests, there are times when they may differ. Being able to identify requests where these are different may help to understand why a certain behavior is occurring.

Removing the Custom Access Log Configuration

You may remove the custom access log configuration outlined in this article with the following commands:

mkdir -p /root/trashbin
mv -v /etc/nginx/conf.d/custLogFormat.conf /root/trashbin
find /etc/nginx/conf.d/users -name "customCacheAccessLog.conf" -exec mv -v {} /root/trashbin \;

The above commands move the configuration files to /root/trashbin. You may then examine the files that were moved to the trashbin directory to confirm that all of them should be permanently deleted.

Once you are confident that the correct configuration files were removed, you may reload the NGINX configuration with the following command:

/scripts/ea-nginx reload

To permanently delete the files in the trashbin, you may use the rm command. Instructions for using the rm command can be found in the manual page:

Linux Manual Page - rm

Please note that incorrect usage of the rm command could permanently damage the server. If you are unfamiliar with the use of the rm command, please reach out to a systems administrator with the skills, training, and experience required to assist you with this task.