Skip to main content

You may have noticed our Login page looks different. We're currently improving our systems to provide you with a new more unified login experience.

How to investigate unintended file modifications

Austin Lowery
  • Updated

Introduction

 

Please keep in mind that the investigation of file modification is a task that is best suited for a systems administrator. Although this is not a cPanel related issue, we hope that the following guide will serve as a helpful resource. 

cPanel support cannot assist with implementing the techniques below. Instead please reach out to a systems administrator for assistance.

 

Procedure

First and foremost, collect the timestamp information of the file with the following command:

stat /root/testfile.txt
 File: ‘/root/testfile.txt’
 Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: fd01h/64769d Inode: 2275144 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2020-08-19 11:16:55.596959274 -0500
Modify: 2020-08-19 11:16:55.596959274 -0500
Change: 2020-08-19 11:16:55.596959274 -0500
 Birth: -

 

Each timestamp has a different use:

  • Access - the last time the file was read
  • Modify - the last time the file was modified (content has been modified)
  • Change - the last time meta data of the file was changed (e.g. permissions)

Now that you have the timestamp information for the file start by checking related logs. This can take some outside of the box thinking based on what you know about the file and what kind of services may be interacting with it. A good first log to check is the cron log at:

  • /var/log/cron

 

If, after reviewing related logs you come up short, you can monitor the file to see when it is modified. Below are basic instructions for modifying a file with Audit.

There are also alternative methods such as the use of SystemTap, or the entr utility which are not covered here.

 

Monitor a File with Audit Rules (auditctl / ausearch)

Most cPanel servers will come with Audit installed and ready to use by default. If Audit is not working for you, it is possible that you'll need to review the documentation to ensure that it is installed and enabled: Audit Documentation

 

The short version of the installation is the following commands:

  • yum install audit audit-libs
  • systemctl start auditd
  • systemctl enable auditd

 

To start monitoring a file for reads, writes, appends, and executes, issue the following command via SSH as the root user:

auditctl -w /root/testfile.txt -p rxwa -k testingaudit

 

The -k option is where you specify some sort of search term that you can use to find the audit logs for the rule that you create.

 

The  -p option lets you determine what events you would like to monitor:

  • r - Read operations
  • w - Write operations
  • x - Execute operations
  • a - Attribute change operations

 

Once you have enabled the rule, you can verify that it has been enabled by searching with your search term using the ausearch command:

ausearch -k testingaudit
----
time->Wed Aug 19 11:37:23 2020
type=CONFIG_CHANGE msg=audit(1597855043.337:83): auid=0 ses=26499 op=add_rule key="testingaudit" list=4 res=1

 

Using the same example file I made a simple edit with the vi utility:

# vi /root/testfile.txt

 

Then using the same ausearch command from above, we can see all of the actions that were performed on the file during the simple edit. If you are only looking for specific kind of action, it is ideal to only include the minimally required flags for the -p option to reduce on noise:

[ root@172.16.1.9 [] ~]# ausearch -k testingaudit
----
time->Wed Aug 19 11:37:23 2020
type=CONFIG_CHANGE msg=audit(1597855043.337:83): auid=0 ses=26499 op=add_rule key="testingaudit" list=4 res=1
----
time->Wed Aug 19 11:38:44 2020
type=PROCTITLE msg=audit(1597855124.169:84): proctitle=7669002F726F6F742F7465737466696C652E747874
type=PATH msg=audit(1597855124.169:84): item=0 name="/root/testfile.txt" inode=2275144 dev=fd:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1597855124.169:84): cwd="/root"
type=SYSCALL msg=audit(1597855124.169:84): arch=c000003e syscall=2 success=yes exit=3 a0=bf2750 a1=0 a2=0 a3=7ffe9cf41720 items=1 ppid=661522 pid=669516 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=26499 comm="vi" exe="/usr/bin/vi" key="testingaudit"
----
time->Wed Aug 19 11:38:44 2020
type=PROCTITLE msg=audit(1597855124.169:85): proctitle=7669002F726F6F742F7465737466696C652E747874
type=PATH msg=audit(1597855124.169:85): item=0 name="/root/testfile.txt" inode=2275144 dev=fd:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1597855124.169:85): cwd="/root"
type=SYSCALL msg=audit(1597855124.169:85): arch=c000003e syscall=89 success=no exit=-22 a0=7ffe9cf3fd80 a1=7ffe9cf40dc0 a2=fff a3=7ffe9cf3f160 items=1 ppid=661522 pid=669516 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=26499 comm="vi" exe="/usr/bin/vi" key="testingaudit"
----
time->Wed Aug 19 11:38:48 2020
type=PROCTITLE msg=audit(1597855128.940:86): proctitle=7669002F726F6F742F7465737466696C652E747874
type=PATH msg=audit(1597855128.940:86): item=0 name="testfile.txt" inode=2275144 dev=fd:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1597855128.940:86): cwd="/root"
type=SYSCALL msg=audit(1597855128.940:86): arch=c000003e syscall=191 success=no exit=-61 a0=bf2750 a1=7f35253c1e2f a2=7ffe9cf41d10 a3=84 items=1 ppid=661522 pid=669516 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=26499 comm="vi" exe="/usr/bin/vi" key="testingaudit"
----
time->Wed Aug 19 11:38:48 2020
type=CONFIG_CHANGE msg=audit(1597855128.941:87): auid=0 ses=26499 op=updated_rules path="/root/testfile.txt" key="testingaudit" list=4 res=1
----
time->Wed Aug 19 11:38:48 2020
type=PROCTITLE msg=audit(1597855128.941:88): proctitle=7669002F726F6F742F7465737466696C652E747874
type=PATH msg=audit(1597855128.941:88): item=3 name="testfile.txt~" inode=2275144 dev=fd:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 objtype=CREATE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1597855128.941:88): item=2 name="testfile.txt" inode=2275144 dev=fd:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 objtype=DELETE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1597855128.941:88): item=1 name="/root" inode=2097217 dev=fd:01 mode=040550 ouid=0 ogid=0 rdev=00:00 objtype=PARENT cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1597855128.941:88): item=0 name="/root" inode=2097217 dev=fd:01 mode=040550 ouid=0 ogid=0 rdev=00:00 objtype=PARENT cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1597855128.941:88): cwd="/root"
type=SYSCALL msg=audit(1597855128.941:88): arch=c000003e syscall=82 success=yes exit=0 a0=bf2750 a1=c1f250 a2=fffffffffffffe80 a3=7ffe9cf400e0 items=4 ppid=661522 pid=669516 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=26499 comm="vi" exe="/usr/bin/vi" key="testingaudit"
----
time->Wed Aug 19 11:38:48 2020
type=CONFIG_CHANGE msg=audit(1597855128.941:89): auid=0 ses=26499 op=updated_rules path="/root/testfile.txt" key="testingaudit" list=4 res=1
----
time->Wed Aug 19 11:38:48 2020
type=PROCTITLE msg=audit(1597855128.941:90): proctitle=7669002F726F6F742F7465737466696C652E747874
type=PATH msg=audit(1597855128.941:90): item=1 name="testfile.txt" inode=2275153 dev=fd:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 objtype=CREATE cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=PATH msg=audit(1597855128.941:90): item=0 name="/root" inode=2097217 dev=fd:01 mode=040550 ouid=0 ogid=0 rdev=00:00 objtype=PARENT cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1597855128.941:90): cwd="/root"
type=SYSCALL msg=audit(1597855128.941:90): arch=c000003e syscall=2 success=yes exit=3 a0=bf2750 a1=241 a2=1a4 a3=0 items=2 ppid=661522 pid=669516 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=26499 comm="vi" exe="/usr/bin/vi" key="testingaudit"
----
time->Wed Aug 19 11:38:48 2020
type=PROCTITLE msg=audit(1597855128.945:91): proctitle=7669002F726F6F742F7465737466696C652E747874
type=PATH msg=audit(1597855128.945:91): item=0 name="testfile.txt" inode=2275153 dev=fd:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1597855128.945:91): cwd="/root"
type=SYSCALL msg=audit(1597855128.945:91): arch=c000003e syscall=90 success=yes exit=0 a0=bf2750 a1=81a4 a2=0 a3=7ffe9cf41260 items=1 ppid=661522 pid=669516 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=26499 comm="vi" exe="/usr/bin/vi" key="testingaudit"
----
time->Wed Aug 19 11:38:48 2020
type=PROCTITLE msg=audit(1597855128.945:92): proctitle=7669002F726F6F742F7465737466696C652E747874
type=PATH msg=audit(1597855128.945:92): item=0 name="testfile.txt" inode=2275153 dev=fd:01 mode=0100644 ouid=0 ogid=0 rdev=00:00 objtype=NORMAL cap_fp=0000000000000000 cap_fi=0000000000000000 cap_fe=0 cap_fver=0
type=CWD msg=audit(1597855128.945:92): cwd="/root"
type=SYSCALL msg=audit(1597855128.945:92): arch=c000003e syscall=188 success=yes exit=0 a0=bf2750 a1=7f35253c1e2f a2=c1de20 a3=1c items=1 ppid=661522 pid=669516 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=26499 comm="vi" exe="/usr/bin/vi" key="testingaudit"

 

Once you are done with your audit, you remove the rule with nearly the same command that you used to create it. Except you switch the lower case -w flag with an upper case one: -W

auditctl -W /root/testfile.txt -p rxwa -k testingaudit

Related articles