High iowait

launch

• March 20, 2015 01:01

My server has been playing up lately. Some websites on it are taking 20 seconds to load, while others are taking .5ms. The weird thing is its not consistent. Here is the output from iostat. Is the util column abnormal?
-bash-3.2# iostat -x 1 10 avg-cpu: %user %nice %system %iowait %steal %idle 2.16 0.62 1.03 14.79 0.85 80.56 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util xvda1 149.83 64.69 249.07 45.12 7259.20 891.51 27.71 3.20 10.89 2.12 62.25 xvda2 0.00 0.00 0.00 0.00 0.04 0.00 8.56 0.00 1.48 1.48 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 1.60 0.00 1.20 22.60 5.60 69.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util xvda1 56.44 0.00 707.92 0.00 24792.08 0.00 35.02 7.93 11.21 1.38 97.43 xvda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 3.21 0.00 0.20 25.05 2.81 68.74 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util xvda1 55.00 0.00 570.00 0.00 15200.00 0.00 26.67 7.62 13.34 1.76 100.40 xvda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 0.40 0.00 0.99 20.83 1.79 75.99 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util xvda1 132.00 0.00 571.00 0.00 18632.00 0.00 32.63 7.00 12.21 1.75 100.00 xvda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 1.99 0.00 0.99 21.27 1.99 73.76 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util xvda1 70.30 0.00 274.26 0.00 8221.78 0.00 29.98 3.73 14.15 3.57 97.82 xvda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.00 21.00 0.83 78.17 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util xvda1 54.00 191.00 514.00 43.00 16640.00 1872.00 33.24 6.14 11.06 1.74 96.80 xvda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 0.00 0.00 0.19 22.12 0.19 77.50 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util xvda1 20.00 0.00 240.00 0.00 3680.00 0.00 15.33 2.14 8.80 4.10 98.40 xvda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 0.38 0.00 1.34 18.13 1.15 79.01 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util xvda1 190.29 0.00 525.24 0.00 17188.35 0.00 32.72 4.92 9.44 1.84 96.70 xvda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 0.80 0.00 2.41 20.72 2.41 73.64 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util xvda1 190.00 0.00 680.00 0.00 19904.00 0.00 29.27 6.53 9.52 1.39 94.80 xvda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 avg-cpu: %user %nice %system %iowait %steal %idle 0.40 0.00 0.40 21.69 1.41 76.10 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util xvda1 84.16 0.00 535.64 0.00 10225.74 0.00 19.09 6.14 11.45 1.83 98.22 xvda2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
Any help on how to debug this would be appreciated!

• 

  cPanelMichael

  • March 20, 2015 14:26

  Hello, I do see the iowait values are elevated, but it's difficult to determine why that is. The following thread is a good place to get started when troubleshooting this type of issue: Troubleshooting High Loads On Linux Systems Thank you.

  0
• 

  launch

  • March 20, 2015 22:10

  Thanks, I've been reading through that but I'm not sure what values are abnormal. Do these look ok? sar -p
  - CPU %user %nice %system %iowait %steal %idle Average: all 1.78 0.32 0.91 14.12 0.55 82.31
  sar -r
  - kbmemfree kbmemused %memused kbbuffers kbcached kbswpfree kbswpused %swpused kbswpcad Average: 1295779 1849949 58.81 3489 570074 0 0 0.00 0
  sar -d says "Requested activities not available in file"

  0
• 

  launch

  • March 22, 2015 08:12

  I've managed to solve this. The server had a cron job that was running too often. That link really did help. Thanks!

  0
• 

  cPanelMichael

  • March 23, 2015 16:21

  I am happy to see you were able to determine the cause of the issue. Thank you for updating us with the outcome.

  0

Please sign in to leave a comment.