Introduction
The auditd utility can be a valuable tool for monitoring what's happening on your Linux server. It integrates with the kernel to monitor system calls that the system makes.
In this guide, we will learn how to check if auditd is installed, install it if it is not, check to make sure the daemon is running, create a simple audit rule, and check the logs to see what our example rule detected.
Note: Auditd requires access to the kernel, which is not available in containers such as Virtuozzo.
You can install the package but attempts to start it will fail due to not having direct kernel access:
root@server [~] # systemctl start auditd.service
Job for auditd.service failed because the control process exited with error code. See "systemctl status auditd.service" and "journalctl -xe" for details.
The systemd journal will show:
Oct 17 08:37:07 <HOSTNAME> auditd[4819]: Cannot change priority (Operation not permitted)
Oct 17 08:37:07 <HOSTNAME> auditd[4819]: The audit daemon is exiting.
Oct 17 08:37:07 <HOSTNAME> systemd[1]: auditd.service: control process exited, code=exited status=1
Oct 17 08:37:07 <HOSTNAME> systemd[1]: Failed to start Security Auditing Service.
Checking for, and installing, auditd
Your cPanel server should already have a couple audit-libs* packages installed, but it will not have the actual audit package that provides the daemon that we need.
Note: If using Ubuntu, the below examples should replace instances of 'yum' with 'apt' and 'rpm -qa' should be replaced with 'dpkg-query -l'
A default installation should look like this:
root@server [~] # rpm -qa | grep audit
audit-libs-2.8.1-3.el7_5.1.x86_64
audit-libs-python-2.8.1-3.el7_5.1.x86_64
Since it is not installed on this server, let's install it:
root@server [~] # yum install audit
Loaded plugins: fastestmirror, universal-hooks
Loading mirror speeds from cached hostfile
* EA4: 208.74.121.35
* cpanel-addons-production-feed: 208.74.121.35
* epel: fedora-epel.mirror.lstn.net
* remi-safe: repo1.dal.innoscale.net
Resolving Dependencies
--> Running transaction check
---> Package audit.x86_64 0:2.8.1-3.el7_5.1 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
=============================================================================================================================================================================
Package Arch Version Repository Size
=============================================================================================================================================================================
Installing:
audit x86_64 2.8.1-3.el7_5.1 updates 247 k
Transaction Summary
=============================================================================================================================================================================
Install 1 Package
Total download size: 247 k
Installed size: 621 k
Is this ok [y/d/N]: y
Downloading packages:
audit-2.8.1-3.el7_5.1.x86_64.rpm | 247 kB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : audit-2.8.1-3.el7_5.1.x86_64 1/1
Verifying : audit-2.8.1-3.el7_5.1.x86_64 1/1
Installed:
audit.x86_64 0:2.8.1-3.el7_5.1
Complete!
Now we can see the actual audit package:
root@server [~] # rpm -qa | grep audit
audit-libs-2.8.1-3.el7_5.1.x86_64
audit-2.8.1-3.el7_5.1.x86_64
audit-libs-python-2.8.1-3.el7_5.1.x86_64
This will install the daemon process and it will be enabled to start on server reboot, but it will not be running:
root@server [~] # systemctl status auditd.service
● auditd.service - Security Auditing Service
Loaded: loaded (/usr/lib/systemd/system/auditd.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
So let's start it so it will be running when we create our rules:
systemctl start auditd.service
root@server [~] # systemctl status auditd.service
● auditd.service - Security Auditing Service
Loaded: loaded (/usr/lib/systemd/system/auditd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2018-10-17 08:34:59 CDT; 1s ago
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
Process: 27739 ExecStartPost=/sbin/augenrules --load (code=exited, status=0/SUCCESS)
Process: 27729 ExecStart=/sbin/auditd (code=exited, status=0/SUCCESS)
Main PID: 27731 (auditd)
CGroup: /system.slice/auditd.service
└─27731 /sbin/auditd
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: lost 0
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: backlog 0
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: enabled 1
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: failure 1
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: pid 27731
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: rate_limit 0
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: backlog_limit 8192
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: lost 0
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: backlog 1
Oct 17 08:34:59 <HOSTNAME> systemd[1]: Started Security Auditing Service.
Adding rules
Now that we have started the service, you can use the 'auditctl' command to list/create/remove rules:
root@server [~] # auditctl -l
No rules
So let's create our first rule, which takes the format of:
auditctl -w PathToFile -p Permissions -k SearchKey
Let's add a rule to watch /etc/passwd for writes(w) or attribute changes(a) and set the search key to 'passwd' to make it easy to find the entries regarding the /etc/passwd file being changed:
auditctl -w /etc/passwd -p wa -k passwd
Now we can see the rule with auditctl and within the audit.rules file:
root@server [~] # auditctl -l
-w /etc/passwd -p wa -k passwd
Note: This rule will exist until the server gets rebooted. If you wish to make the rule permanent, you will need to add it to the /etc/audit/rules.d/audit.rules file. This is how it will look by default:
root@server [~] # cat /etc/audit/rules.d/audit.rules
## First rule - delete all
-D
## Increase the buffers to survive stress events.
## Make this bigger for busy systems
-b 8192
## Set failure mode to syslog
-f 1
To add our example rule, simply add it to the bottom of the file in the exact same way that it appears when running 'auditctl -l':
root@server [~] # cat /etc/audit/rules.d/audit.rules
## First rule - delete all
-D
## Increase the buffers to survive stress events.
## Make this bigger for busy systems
-b 8192
## Set failure mode to syslog
-f 1
-w /etc/passwd -p wa -k passwd
Searching audit logs
The 'ausearch' command is used to search the audit logs, so we'll use the following to search for the "passwd" search key:
root@server [~] # ausearch -k passwd
----
time->Wed Oct 17 08:45:23 2018
type=CONFIG_CHANGE msg=audit(1539783923.295:77): auid=0 ses=51297 op=add_rule key="passwd" list=4 res=1
That entry shows the rule was added, and it is currently the only entry since /etc/passwd hasn't been modified since we created this rule.
Now let's add a cPanel account to the server so we can see how that operation modifies /etc/passwd:
root@server [~] # ausearch -i -k passwd
----
type=CONFIG_CHANGE msg=audit(10/17/2018 08:45:23.295:77) : auid=root ses=51297 op=add_rule key=passwd list=exit res=yes
----
type=PROCTITLE msg=audit(10/17/2018 09:04:15.572:155) : proctitle=whostmgr5 - wwwacct
type=PATH msg=audit(10/17/2018 09:04:15.572:155) : item=1 name=/etc/passwd inode=1308128 dev=fd:01 mode=file,644 ouid=root ogid=root rdev=00:00 objtype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=PATH msg=audit(10/17/2018 09:04:15.572:155) : item=0 name=/etc/ inode=68 dev=fd:01 mode=dir,755 ouid=root ogid=root rdev=00:00 objtype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=CWD msg=audit(10/17/2018 09:04:15.572:155) : cwd=/usr/local/cpanel/whostmgr/docroot
type=SYSCALL msg=audit(10/17/2018 09:04:15.572:155) : arch=x86_64 syscall=open success=yes exit=10 a0=0x4ffa230 a1=O_RDWR|O_CREAT a2=0644 a3=0x7f9899e7cedd items=2 ppid=32113 pid=32209 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=whostmgr5 - www exe=/usr/local/cpanel/whostmgr/bin/whostmgr5 key=passwd
----
type=CONFIG_CHANGE msg=audit(10/17/2018 09:04:15.573:156) : auid=unset ses=unset op=updated_rules path=/etc/passwd key=passwd list=exit res=yes
----
type=PROCTITLE msg=audit(10/17/2018 09:04:15.573:157) : proctitle=whostmgr5 - wwwacct
type=PATH msg=audit(10/17/2018 09:04:15.573:157) : item=4 name=/etc/passwd inode=1308152 dev=fd:01 mode=file,644 ouid=root ogid=root rdev=00:00 objtype=CREATE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=PATH msg=audit(10/17/2018 09:04:15.573:157) : item=3 name=/etc/passwd inode=1308128 dev=fd:01 mode=file,644 ouid=root ogid=root rdev=00:00 objtype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=PATH msg=audit(10/17/2018 09:04:15.573:157) : item=2 name=/etc/passwd.work.d21f6ce1 inode=1308152 dev=fd:01 mode=file,644 ouid=root ogid=root rdev=00:00 objtype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=PATH msg=audit(10/17/2018 09:04:15.573:157) : item=1 name=/etc/ inode=68 dev=fd:01 mode=dir,755 ouid=root ogid=root rdev=00:00 objtype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=PATH msg=audit(10/17/2018 09:04:15.573:157) : item=0 name=/etc/ inode=68 dev=fd:01 mode=dir,755 ouid=root ogid=root rdev=00:00 objtype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=CWD msg=audit(10/17/2018 09:04:15.573:157) : cwd=/usr/local/cpanel/whostmgr/docroot
type=SYSCALL msg=audit(10/17/2018 09:04:15.573:157) : arch=x86_64 syscall=rename success=yes exit=0 a0=0x580a770 a1=0x6a583f a2=0x7f989a106980 a3=0x3 items=5 ppid=32113 pid=32209 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=whostmgr5 - www exe=/usr/local/cpanel/whostmgr/bin/whostmgr5 key=passwd
The "-i" will print the actual named system call, rather than the number, making it far easier to know what system calls are taking place. The entries we're most concerned with here are the second and last entries. Within those, specifically we're looking mostly at the "syscall" and "exe" values.
The second entry shows that /etc/passwd was opened using the "open" system call ("syscall=open") to the "whostmgr5" executable file ("exe=/usr/local/cpanel/whostmgr/bin/whostmgr5")
A system call is how a program interacts with the kernel, so these are important to look at because they tell us whether an application is requesting to, for example, open a file, write to a file, read from a file, etc.
The last entry then shows that there was a "rename" call also by the same WHM process and, looking a little above that, you can see that a temporary file was created "name=/etc/passwd.work.d21f6ce1" then it was renamed to /etc/passwd, thus replacing the old file with the new one.
Knowing that this change was done by the WHM process, one could then check cPanel's access log (/usr/local/cpanel/logs/access_log) to find the IP address and username that made the request that caused this file to be written to:
10.7.65.81 - root [10/17/2018:14:03:52 -0000] "POST /cpsess3914673904/scripts5/wwwacct HTTP/1.1" 200 0 "https://<HOSTNAME>:2087/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36" "s" "-" 2087
(The time difference comes from the access_log being timestamped in UTC timezone and the audit logs using the system's timezone which is set to US/Central time, so there's a 5-hour difference)
So we now know that on the morning of October 17th, someone at 10.7.65.81 logged into WHM as root and made a POST request to wwwacct, which modified the /etc/passwd file. In this specific instance, one could then reference the /var/cpanel/accounting.log file and see the account that was created:
Wed Oct 17 09:04:16 2018:CREATE:root:root:auditdexample.tld:172.16.0.93:auditdexample
Deleting rules
Lastly, once you've gathered whatever information you're after with your audit rule, you should delete this watch rule using the same format that was used to add it, except replacing -w with -W:
auditctl -W /etc/passwd -p wa -k passwd
root@server [~] # auditctl -l
No rules
The above command will remove the rule from the current running audit session. You will also need to remove the entry from /etc/audit/rules.d/audit.rules if you added it there.
It is important to delete the rules once they've served their purpose, as audit logs can potentially grow very large in size if they're left to run for extended periods.