Basic guide for using strace to diagnose software problems on Linux

Introduction

The strace utility captures data on the system calls that a process makes. The trace output of the system calls of a process provides a wealth of information about what a process does and the errors encountered during the runtime of the process. You would usually be interested in making use of this technique as a last resort due to the complexity and labor-intensive nature of the process. You should always explore every human-readable log and online resource before turning to this diagnostic tool unless you know exactly what you're doing.

Although this technique is not related to cPanel or the basic configuration of cPanel, we offer this guide as a courtesy for individuals who are interested in learning how to better diagnose problems on Linux.

Please keep in mind that the process of using the strace utility to diagnose problems is a task that is best suited for a System Administrator with the skills, training, and expertise required to do so for you. If this guide does not provide enough information for you to properly trace a process and diagnose your problem, you will need to reach out to a systems administrator for further assistance.

Procedure

If the process causing the issue is already running, such as a service daemon, you'll start by identifying the process ID.

1. Access the server via SSH as the root user
2. Run the following command to get a full list of processes and their associated process IDs:

   ps auxf | less

3. Once you have identified the process you want to trace, make a note of its process ID (PID) for use later
4. Run the strace command with the following arguments. Make sure that $PID is replaced by the PID gathered in step 3. 
   Please note that strace collects a large amount of data very rapidly. We recommend running this command using as little time as necessary when reproducing issues to limit unrelated data as much as possible:

   strace -ttvvff -s 1000 -o output.trace -p $PID

5. Take whatever steps you need to trigger the error that you are tracing to find. When the error has been triggered, use Ctrl+C to stop the strace.
6. Review the output.trace file created to see what happened during the trace period:

   less output.trace

If you are looking to trace a command run via the command line, you can do so by supplying the strace arguments with the command string you want to run instead of the PID:

strace -ttvvff -s 1000 -o output.trace $command-string

This will run the command string as if it was run directly from the command line and output all strace data into the output.trace file. When the command is complete, any output will display in the terminal as usual, and the output.trace file will be there for review.

Reviewing the output of strace can be very difficult. It can often help to start at the end of the log and look back to find when the error you are tracking was first recorded and review in detail from that spot farther back. A server makes a large number of low-level system calls for any human-readable command, which can make the contents of the files extremely difficult to parse. It can help to use tools to search for the actual error you receive in the output and look around from there. If you are not very familiar with what you are looking at or looking for, you may consider enlisting a System Administrator to help you track down the issue.

Additional Information

Review the following manual page and external source to learn more about its options, use, and other information:

Manual Pages - strace(1)

Strace IO

The following is a more targeted guide for stracing cPanel and WHM processes:

How to trace cPanel or WHM processes