CPU wait while everything else is optimal
Would appreciate some direction or input. CentOS 7.2 x64, 32GB RAM, E5-2609 Dual Processors 12-cores total, 2 x 500GB SSD Disks RAID-1, MariaDB 10.0, Apache 2.4 + PHP suPHP Prefork.
The load you'll see below with 650 active users who are mostly using a chat room with MySQL backend. What is concerning me is the CPU wait is 7%-10% which end up slowing down the application. Everything else is fine, even the CPU doesn't seem to be fully utilized.
How can I find out what is causing the higher Wait ?
iostat: looks very good, no load on disks
vmstat: no memory issues, no blocks
top - 19:47:50 up 10:03, 1 user, load average: 2.17, 2.40, 2.41
Tasks: 943 total, 7 running, 935 sleeping, 0 stopped, 1 zombie
%Cpu0 : 12.0 us, 10.7 sy, 0.0 ni, 67.2 id, 9.7 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu1 : 9.7 us, 8.7 sy, 0.0 ni, 72.6 id, 9.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 8.7 us, 7.7 sy, 0.0 ni, 74.8 id, 8.4 wa, 0.0 hi, 0.3 si, 0.0 st
%Cpu3 : 9.7 us, 8.7 sy, 0.0 ni, 72.6 id, 8.4 wa, 0.0 hi, 0.7 si, 0.0 st
%Cpu4 : 10.1 us, 12.5 sy, 0.0 ni, 69.0 id, 7.7 wa, 0.0 hi, 0.7 si, 0.0 st
%Cpu5 : 12.0 us, 12.3 sy, 0.0 ni, 67.0 id, 8.0 wa, 0.0 hi, 0.7 si, 0.0 st
%Cpu6 : 9.1 us, 6.0 sy, 0.0 ni, 80.9 id, 4.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu7 : 12.3 us, 8.6 sy, 0.0 ni, 74.4 id, 4.7 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu8 : 13.4 us, 5.0 sy, 0.0 ni, 78.5 id, 3.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu9 : 6.3 us, 9.0 sy, 0.0 ni, 80.0 id, 4.7 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu10 : 12.0 us, 8.0 sy, 0.0 ni, 76.6 id, 3.3 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu11 : 10.2 us, 6.9 sy, 0.0 ni, 78.2 id, 4.6 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 32672512 total, 28286664 free, 2184524 used, 2201324 buff/cache
KiB Swap: 7812092 total, 7812092 free, 0 used. 30028120 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3596 mysql 20 0 1181580 319288 9280 S 78.2 1.0 601:18.10 mysqld
21505 nobody 20 0 317164 16012 8220 S 2.0 0.0 0:04.96 httpd
21753 nobody 20 0 318200 16092 7572 S 2.0 0.0 0:03.36 httpd
22713 nobody 20 0 318156 15404 6864 S 2.0 0.0 0:02.92 httpd
22881 nobody 20 0 317172 14488 6636 S 2.0 0.0 0:02.44 httpd
16573 bestfan+ 20 0 317944 16484 7864 D 1.7 0.1 0:12.52 httpd
20502 nobody 20 0 317272 15508 7896 S 1.7 0.0 0:06.63 httpd
21529 nobody 20 0 317932 15096 6488 S 1.7 0.0 0:04.35 httpd
21683 nobody 20 0 317208 13948 6340 S 1.7 0.0 0:05.12 httpd
iostat: looks very good, no load on disks
avg-cpu: %user %nice %system %iowait %steal %idle
7.50 0.01 5.06 0.54 0.00 86.88
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.03 1.64 1.23 29.21 53.26 169.46 14.63 0.02 0.76 5.48 0.56 0.19 0.58
sdb 0.00 0.00 0.01 0.00 0.13 0.07 37.09 0.00 6.06 6.81 0.00 5.82 0.01
avg-cpu: %user %nice %system %iowait %steal %idle
10.34 0.00 6.90 3.66 0.00 79.10
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
vmstat: no memory issues, no blocks
vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
6 0 0 28145148 1052 2202556 0 0 4 14 95 19 8 5 87 1 0
3 0 0 28147484 1052 2202548 0 0 0 0 30490 34875 9 9 75 8 0
1 0 0 28147436 1052 2202544 0 0 0 0 24018 30270 8 6 82 3 0
5 0 0 28143656 1052 2202436 0 0 0 1288 30721 37232 11 8 77 5 0
4 1 0 28146204 1052 2202420 0 0 0 0 28151 33762 10 8 78 4 0
-
The load you'll see below with 650 active users who are mostly using a chat room with MySQL backend. What is concerning me is the CPU wait is 7%-10% which end up slowing down the application. Everything else is fine, even the CPU doesn't seem to be fully utilized.
Hello, Have you ran a MySQL tuning application to determine if any changes to the MySQL configuration can help? There's a thread here that should work for MariaDB as well:0 -
You may also find this StackOverflow thread helpful: How to find out which process is consuming "wait CPU" (i.e. I/O blocked) Thank you. 0
Please sign in to leave a comment.
Comments
3 comments