Kernel Issue Possibly?

At 08:40 every morning this occurs it seems, and disk IO then goes sky high:
Jul 24 08:44:36 lin01 kernel: [ 7433.979199] INFO: task httpd:3184 blocked for more than 120 seconds.
Jul 24 08:44:36 lin01 kernel: [ 7433.979230]       Tainted: P           ---------------    2.6.32-531.1.2.lve1.2.54.el6.x86_64 #1
Jul 24 08:44:36 lin01 kernel: [ 7433.979253] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 24 08:44:36 lin01 kernel: [ 7433.979276] httpd         D ffff880160a34d90     0  3184   2684    0 0x00000080
Jul 24 08:44:36 lin01 kernel: [ 7433.979280]  ffff880160a37d08 0000000000000086 0000000000000000 0000000000000000
Jul 24 08:44:36 lin01 kernel: [ 7433.979284]  ffff880160a37c70 0000000000000000 0000000000000000 8000000112759065
Jul 24 08:44:36 lin01 kernel: [ 7433.979286]  0000000000000000 ffff880160a35358 0000000100000000 000000000001ec80
Jul 24 08:44:36 lin01 kernel: [ 7433.979288] Call Trace:
Jul 24 08:44:36 lin01 kernel: [ 7433.979295]  [] ? sock_aio_write+0x0/0x180
Jul 24 08:44:36 lin01 kernel: [ 7433.979300]  [] __mutex_lock_slowpath+0x13e/0x180
Jul 24 08:44:36 lin01 kernel: [ 7433.979302]  [] mutex_lock+0x2b/0x50
Jul 24 08:44:36 lin01 kernel: [ 7433.979306]  [] do_filp_open+0x36c/0xd10
Jul 24 08:44:36 lin01 kernel: [ 7433.979310]  [] ? strncpy_from_user+0x4a/0x90
Jul 24 08:44:36 lin01 kernel: [ 7433.979313]  [] ? alloc_fd+0x92/0x160
Jul 24 08:44:36 lin01 kernel: [ 7433.979316]  [] do_sys_open+0x69/0x140
Jul 24 08:44:36 lin01 kernel: [ 7433.979318]  [] sys_open+0x20/0x30
Jul 24 08:44:36 lin01 kernel: [ 7433.979322]  [] system_call_fastpath+0x16/0x1b
Jul 24 08:44:36 lin01 kernel: [ 7433.979325] INFO: task httpd:3200 blocked for more than 120 seconds.
Jul 24 08:44:36 lin01 kernel: [ 7433.979343]       Tainted: P           ---------------    2.6.32-531.1.2.lve1.2.54.el6.x86_64 #1
Jul 24 08:44:36 lin01 kernel: [ 7433.979366] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 24 08:44:36 lin01 kernel: [ 7433.979401] httpd         D ffff880160aa8f90     0  3200   2684    0 0x00000080
Jul 24 08:44:36 lin01 kernel: [ 7433.979404]  ffff880160aabd08 0000000000000086 ffff880160aabc88 0000000000000000
Jul 24 08:44:36 lin01 kernel: [ 7433.979406]  ffff880160aabc70 0000000000000000 0000000000000000 80000000a42f5045
Jul 24 08:44:36 lin01 kernel: [ 7433.979409]  ffff880160aa8f90 ffff880160aa9558 ffff880100000000 000000000001ec80
Jul 24 08:44:36 lin01 kernel: [ 7433.979411] Call Trace:
Jul 24 08:44:36 lin01 kernel: [ 7433.979413]  [] ? __sb_start_write+0xd5/0x1b0
Jul 24 08:44:36 lin01 kernel: [ 7433.979415]  [] ? sock_aio_write+0x0/0x180
Jul 24 08:44:36 lin01 kernel: [ 7433.979418]  [] __mutex_lock_slowpath+0x13e/0x180
Jul 24 08:44:36 lin01 kernel: [ 7433.979420]  [] mutex_lock+0x2b/0x50
Jul 24 08:44:36 lin01 kernel: [ 7433.979422]  [] do_filp_open+0x36c/0xd10
Jul 24 08:44:36 lin01 kernel: [ 7433.979424]  [] ? strncpy_from_user+0x4a/0x90
Jul 24 08:44:36 lin01 kernel: [ 7433.979427]  [] ? alloc_fd+0x92/0x160
Jul 24 08:44:36 lin01 kernel: [ 7433.979429]  [] do_sys_open+0x69/0x140
Jul 24 08:44:36 lin01 kernel: [ 7433.979431]  [] sys_open+0x20/0x30
Jul 24 08:44:36 lin01 kernel: [ 7433.979433]  [] system_call_fastpath+0x16/0x1b
Jul 24 08:44:36 lin01 kernel: [ 7433.979435] INFO: task httpd:3208 blocked for more than 120 seconds.
Jul 24 08:44:36 lin01 kernel: [ 7433.979453]       Tainted: P           ---------------    2.6.32-531.1.2.lve1.2.54.el6.x86_64 #1
Jul 24 08:44:36 lin01 kernel: [ 7433.979477] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 24 08:44:36 lin01 kernel: [ 7433.979500] httpd         D ffff880160ac1090     0  3208   2684    0 0x00000080
Jul 24 08:44:36 lin01 kernel: [ 7433.979502]  ffff880160ac3d08 0000000000000086 ffff880160ac3c88 0000000000000000
Jul 24 08:44:36 lin01 kernel: [ 7433.979504]  ffff880160ac3c70 0000000000000000 0000000000000000 8000000164e5c065
Jul 24 08:44:36 lin01 kernel: [ 7433.979506]  ffff880160ac1090 ffff880160ac1658 ffff880100000000 000000000001ec80
Jul 24 08:44:36 lin01 kernel: [ 7433.979509] Call Trace:
Jul 24 08:44:36 lin01 kernel: [ 7433.979510]  [] ? __sb_start_write+0xd5/0x1b0
Jul 24 08:44:36 lin01 kernel: [ 7433.979513]  [] ? sock_aio_write+0x0/0x180
Jul 24 08:44:36 lin01 kernel: [ 7433.979515]  [] __mutex_lock_slowpath+0x13e/0x180
Jul 24 08:44:36 lin01 kernel: [ 7433.979517]  [] mutex_lock+0x2b/0x50
Jul 24 08:44:36 lin01 kernel: [ 7433.979519]  [] do_filp_open+0x36c/0xd10
Jul 24 08:44:36 lin01 kernel: [ 7433.979522]  [] ? strncpy_from_user+0x4a/0x90
Jul 24 08:44:36 lin01 kernel: [ 7433.979524]  [] ? alloc_fd+0x92/0x160
Jul 24 08:44:36 lin01 kernel: [ 7433.979526]  [] do_sys_open+0x69/0x140
Jul 24 08:44:36 lin01 kernel: [ 7433.979528]  [] sys_open+0x20/0x30
Jul 24 08:44:36 lin01 kernel: [ 7433.979530]  [] system_call_fastpath+0x16/0x1b
Jul 24 08:44:36 lin01 kernel: [ 7433.979533] INFO: task httpd:3220 blocked for more than 120 seconds.
Jul 24 08:44:36 lin01 kernel: [ 7433.979552]       Tainted: P           ---------------    2.6.32-531.1.2.lve1.2.54.el6.x86_64 #1
Jul 24 08:44:36 lin01 kernel: [ 7433.979575] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 24 08:44:36 lin01 kernel: [ 7433.979599] httpd         D ffff880160ae26c0     0  3220   2684    0 0x00000080
Jul 24 08:44:36 lin01 kernel: [ 7433.979602]  ffff880160ae9d08 0000000000000086 ffff880160ae9d08 0000000000000000
Jul 24 08:44:36 lin01 kernel: [ 7433.979604]  ffff880160ae9c70 0000000000000000 0000000000000000 ffff880160ae9e58
Jul 24 08:44:36 lin01 kernel: [ 7433.979606]  ffff880160ae26c0 ffff880160ae2c88 ffff880100000000 000000000001ec80
Jul 24 08:44:36 lin01 kernel: [ 7433.979609] Call Trace:
Jul 24 08:44:36 lin01 kernel: [ 7433.979610]  [] ? __sb_start_write+0xd5/0x1b0
Jul 24 08:44:36 lin01 kernel: [ 7433.979612]  [] ? sock_aio_write+0x0/0x180
Jul 24 08:44:36 lin01 kernel: [ 7433.979615]  [] __mutex_lock_slowpath+0x13e/0x180
Jul 24 08:44:36 lin01 kernel: [ 7433.979617]  [] mutex_lock+0x2b/0x50
Jul 24 08:44:36 lin01 kernel: [ 7433.979619]  [] do_filp_open+0x36c/0xd10
Jul 24 08:44:36 lin01 kernel: [ 7433.979622]  [] ? do_sync_read+0xfa/0x140
Jul 24 08:44:36 lin01 kernel: [ 7433.979624]  [] ? strncpy_from_user+0x4a/0x90
Jul 24 08:44:36 lin01 kernel: [ 7433.979626]  [] ? alloc_fd+0x92/0x160
Jul 24 08:44:36 lin01 kernel: [ 7433.979628]  [] do_sys_open+0x69/0x140
Jul 24 08:44:36 lin01 kernel: [ 7433.979630]  [] sys_open+0x20/0x30
Jul 24 08:44:36 lin01 kernel: [ 7433.979633]  [] system_call_fastpath+0x16/0x1b
Jul 24 08:44:36 lin01 kernel: [ 7433.979635] INFO: task httpd:3224 blocked for more than 120 seconds.
Jul 24 08:44:36 lin01 kernel: [ 7433.979651]       Tainted: P           ---------------    2.6.32-531.1.2.lve1.2.54.el6.x86_64 #1
Jul 24 08:44:36 lin01 kernel: [ 7433.979674] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 24 08:44:36 lin01 kernel: [ 7433.979696] httpd         D ffff880160aee740     0  3224   2684    0 0x00000080
Jul 24 08:44:36 lin01 kernel: [ 7433.979698]  ffff880160af5d08 0000000000000086 ffff880162ffc638 0000000000000000
Jul 24 08:44:36 lin01 kernel: [ 7433.979701]  ffff880160af5c70 0000000000000000 0000000000000000 8000000151932065
Jul 24 08:44:36 lin01 kernel: [ 7433.979703]  ffff880160aee740 ffff880160aeed08 ffff880100000000 000000000001ec80
Jul 24 08:44:36 lin01 kernel: [ 7433.979705] Call Trace:
Jul 24 08:44:36 lin01 kernel: [ 7433.979707]  [] ? __sb_start_write+0xd5/0x1b0
Jul 24 08:44:36 lin01 kernel: [ 7433.979709]  [] ? sock_aio_write+0x0/0x180
Jul 24 08:44:36 lin01 kernel: [ 7433.979712]  [] __mutex_lock_slowpath+0x13e/0x180
Jul 24 08:44:36 lin01 kernel: [ 7433.979714]  [] mutex_lock+0x2b/0x50
Jul 24 08:44:36 lin01 kernel: [ 7433.979716]  [] do_filp_open+0x36c/0xd10
Jul 24 08:44:36 lin01 kernel: [ 7433.979718]  [] ? d_free+0x58/0x60
Jul 24 08:44:36 lin01 kernel: [ 7433.979720]  [] ? d_kill+0x54/0x70
Jul 24 08:44:36 lin01 kernel: [ 7433.979722]  [] ? strncpy_from_user+0x4a/0x90
Jul 24 08:44:36 lin01 kernel: [ 7433.979725]  [] ? alloc_fd+0x92/0x160
Jul 24 08:44:36 lin01 kernel: [ 7433.979727]  [] do_sys_open+0x69/0x140
Jul 24 08:44:36 lin01 kernel: [ 7433.979728]  [] sys_open+0x20/0x30
Jul 24 08:44:36 lin01 kernel: [ 7433.979731]  [] system_call_fastpath+0x16/0x1b
Jul 24 08:44:36 lin01 kernel: [ 7433.979733] INFO: task httpd:3226 blocked for more than 120 seconds.
Jul 24 08:44:36 lin01 kernel: [ 7433.979758]       Tainted: P           ---------------    2.6.32-531.1.2.lve1.2.54.el6.x86_64 #1
Jul 24 08:44:36 lin01 kernel: [ 7433.979781] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 24 08:44:36 lin01 kernel: [ 7433.979803] httpd         D ffff880160afb2d0     0  3226   2684    0 0x00000080
Jul 24 08:44:36 lin01 kernel: [ 7433.979805]  ffff880160afdd08 0000000000000086 ffff880160afdc88 0000000000000000
Jul 24 08:44:36 lin01 kernel: [ 7433.979807]  ffff880160afdc70 0000000000000000 0000000000000000 8000000085167065
Jul 24 08:44:36 lin01 kernel: [ 7433.979809]  ffff880160afb2d0 ffff880160afb898 ffff880100000000 000000000001ec80
Jul 24 08:44:36 lin01 kernel: [ 7433.979812] Call Trace:
Jul 24 08:44:36 lin01 kernel: [ 7433.979814]  [] ? __sb_start_write+0xd5/0x1b0
Jul 24 08:44:36 lin01 kernel: [ 7433.979816]  [] ? sock_aio_write+0x0/0x180
Jul 24 08:44:36 lin01 kernel: [ 7433.979818]  [] __mutex_lock_slowpath+0x13e/0x180
Jul 24 08:44:36 lin01 kernel: [ 7433.979821]  [] mutex_lock+0x2b/0x50
Jul 24 08:44:36 lin01 kernel: [ 7433.979823]  [] do_filp_open+0x36c/0xd10
Jul 24 08:44:36 lin01 kernel: [ 7433.979825]  [] ? strncpy_from_user+0x4a/0x90
Jul 24 08:44:36 lin01 kernel: [ 7433.979827]  [] ? alloc_fd+0x92/0x160
Jul 24 08:44:36 lin01 kernel: [ 7433.979830]  [] do_sys_open+0x69/0x140
Jul 24 08:44:36 lin01 kernel: [ 7433.979832]  [] sys_open+0x20/0x30
Jul 24 08:44:36 lin01 kernel: [ 7433.979834]  [] system_call_fastpath+0x16/0x1b
Jul 24 08:44:36 lin01 kernel: [ 7433.979836] INFO: task httpd:3227 blocked for more than 120 seconds.
Jul 24 08:44:36 lin01 kernel: [ 7433.979854]       Tainted: P           ---------------    2.6.32-531.1.2.lve1.2.54.el6.x86_64 #1
Jul 24 08:44:36 lin01 kernel: [ 7433.979875] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 24 08:44:36 lin01 kernel: [ 7433.979896] httpd         D ffff880160afa7c0     0  3227   2684    0 0x00000080
Jul 24 08:44:36 lin01 kernel: [ 7433.979898]  ffff880160affd08 0000000000000086 ffff880160affc88 ffff8800bc1773c0
Jul 24 08:44:36 lin01 kernel: [ 7433.979900]  ffff8800a3be4880 0000000000000000 0000000000000000 80000001376da045
Jul 24 08:44:36 lin01 kernel: [ 7433.979903]  ffff880160afa7c0 ffff880160afad88 ffff880100000000 000000000001ec80
Jul 24 08:44:36 lin01 kernel: [ 7433.979905] Call Trace:
Jul 24 08:44:36 lin01 kernel: [ 7433.979907]  [] ? __sb_start_write+0xd5/0x1b0
Jul 24 08:44:36 lin01 kernel: [ 7433.979909]  [] __mutex_lock_slowpath+0x13e/0x180
Jul 24 08:44:36 lin01 kernel: [ 7433.979912]  [] mutex_lock+0x2b/0x50
Jul 24 08:44:36 lin01 kernel: [ 7433.979914]  [] do_filp_open+0x36c/0xd10
Jul 24 08:44:36 lin01 kernel: [ 7433.979916]  [] ? strncpy_from_user+0x4a/0x90
Jul 24 08:44:36 lin01 kernel: [ 7433.979918]  [] ? alloc_fd+0x92/0x160
Jul 24 08:44:36 lin01 kernel: [ 7433.979920]  [] do_sys_open+0x69/0x140
Jul 24 08:44:36 lin01 kernel: [ 7433.979922]  [] sys_open+0x20/0x30
Jul 24 08:44:36 lin01 kernel: [ 7433.979925]  [] system_call_fastpath+0x16/0x1b
Jul 24 08:44:36 lin01 kernel: [ 7433.979927] INFO: task httpd:3233 blocked for more than 120 seconds.
Jul 24 08:44:36 lin01 kernel: [ 7433.979945]       Tainted: P           ---------------    2.6.32-531.1.2.lve1.2.54.el6.x86_64 #1
Jul 24 08:44:36 lin01 kernel: [ 7433.979966] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 24 08:44:36 lin01 kernel: [ 7433.979980] httpd         D ffff880160b31390     0  3233   2684    0 0x00000080
Jul 24 08:44:36 lin01 kernel: [ 7433.979982]  ffff880160b33d08 0000000000000086 0000000000000000 0000000000000000
Jul 24 08:44:36 lin01 kernel: [ 7433.979985]  ffff880160b33c70 0000000000000000 ffffea0000000000 ffff880160b33e58
Jul 24 08:44:36 lin01 kernel: [ 7433.979987]  0000000000000000 ffff880160b31958 0000000100000000 000000000001ec80
Jul 24 08:44:36 lin01 kernel: [ 7433.979989] Call Trace:
Jul 24 08:44:36 lin01 kernel: [ 7433.979992]  [] __mutex_lock_slowpath+0x13e/0x180
Jul 24 08:44:36 lin01 kernel: [ 7433.979994]  [] mutex_lock+0x2b/0x50
Jul 24 08:44:36 lin01 kernel: [ 7433.979996]  [] do_filp_open+0x36c/0xd10
Jul 24 08:44:36 lin01 kernel: [ 7433.980000]  [] ? cpumask_any_but+0x31/0x50
Jul 24 08:44:36 lin01 kernel: [ 7433.980003]  [] ? strncpy_from_user+0x4a/0x90
Jul 24 08:44:36 lin01 kernel: [ 7433.980007]  [] ? alloc_fd+0x92/0x160
Jul 24 08:44:36 lin01 kernel: [ 7433.980010]  [] do_sys_open+0x69/0x140
Jul 24 08:44:36 lin01 kernel: [ 7433.980013]  [] sys_open+0x20/0x30
Jul 24 08:44:36 lin01 kernel: [ 7433.980015]  [] system_call_fastpath+0x16/0x1b
Jul 24 08:44:36 lin01 kernel: [ 7433.980017] INFO: task httpd:3236 blocked for more than 120 seconds.
Jul 24 08:44:36 lin01 kernel: [ 7433.980037]       Tainted: P           ---------------    2.6.32-531.1.2.lve1.2.54.el6.x86_64 #1
Jul 24 08:44:36 lin01 kernel: [ 7433.980062] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 24 08:44:36 lin01 kernel: [ 7433.980076] httpd         D ffff880160b30880     0  3236   2684    0 0x00000080
Jul 24 08:44:36 lin01 kernel: [ 7433.980079]  ffff880160b3bd08 0000000000000086 ffff880160b3bc88 0000000000000000
Jul 24 08:44:36 lin01 kernel: [ 7433.980081]  ffff880160b3bc70 0000000000000000 0000000000000000 80000000f551d065
Jul 24 08:44:36 lin01 kernel: [ 7433.980091]  ffff880160b30880 ffff880160b30e48 ffff880100000000 000000000001ec80
Jul 24 08:44:36 lin01 kernel: [ 7433.980093] Call Trace:
Jul 24 08:44:36 lin01 kernel: [ 7433.980095]  [] ? __sb_start_write+0xd5/0x1b0
Jul 24 08:44:36 lin01 kernel: [ 7433.980097]  [] ? sock_aio_write+0x0/0x180
Jul 24 08:44:36 lin01 kernel: [ 7433.980100]  [] __mutex_lock_slowpath+0x13e/0x180
Jul 24 08:44:36 lin01 kernel: [ 7433.980102]  [] mutex_lock+0x2b/0x50
Jul 24 08:44:36 lin01 kernel: [ 7433.980104]  [] do_filp_open+0x36c/0xd10
Jul 24 08:44:36 lin01 kernel: [ 7433.980106]  [] ? strncpy_from_user+0x4a/0x90
Jul 24 08:44:36 lin01 kernel: [ 7433.980109]  [] ? alloc_fd+0x92/0x160
Jul 24 08:44:36 lin01 kernel: [ 7433.980110]  [] do_sys_open+0x69/0x140
Jul 24 08:44:36 lin01 kernel: [ 7433.980112]  [] sys_open+0x20/0x30
Jul 24 08:44:36 lin01 kernel: [ 7433.980115]  [] system_call_fastpath+0x16/0x1b
Jul 24 08:44:36 lin01 kernel: [ 7433.980117] INFO: task httpd:3534 blocked for more than 120 seconds.
Jul 24 08:44:36 lin01 kernel: [ 7433.980131]       Tainted: P           ---------------    2.6.32-531.1.2.lve1.2.54.el6.x86_64 #1
Jul 24 08:44:36 lin01 kernel: [ 7433.980149] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jul 24 08:44:36 lin01 kernel: [ 7433.980177] httpd         D ffff880160adf190     0  3534   2684    0 0x00000080
Jul 24 08:44:36 lin01 kernel: [ 7433.980181]  ffff8801e5d1dd08 0000000000000082 ffff8801e5d1dd08 0000000000000000
Jul 24 08:44:36 lin01 kernel: [ 7433.980185]  ffff8801e5d1dc70 0000000000000000 0000000000000000 8000000183387065
Jul 24 08:44:36 lin01 kernel: [ 7433.980189]  ffff880160adf190 ffff880160adf758 ffff880100000000 000000000001ec80
Jul 24 08:44:36 lin01 kernel: [ 7433.980193] Call Trace:
Jul 24 08:44:36 lin01 kernel: [ 7433.980197]  [] ? __sb_start_write+0xd5/0x1b0
Jul 24 08:44:36 lin01 kernel: [ 7433.980200]  [] ? sock_aio_write+0x0/0x180
Jul 24 08:44:36 lin01 kernel: [ 7433.980203]  [] __mutex_lock_slowpath+0x13e/0x180
Jul 24 08:44:36 lin01 kernel: [ 7433.980207]  [] mutex_lock+0x2b/0x50
Jul 24 08:44:36 lin01 kernel: [ 7433.980210]  [] do_filp_open+0x36c/0xd10
Jul 24 08:44:36 lin01 kernel: [ 7433.980214]  [] ? strncpy_from_user+0x4a/0x90
Jul 24 08:44:36 lin01 kernel: [ 7433.980217]  [] ? alloc_fd+0x92/0x160
Jul 24 08:44:36 lin01 kernel: [ 7433.980219]  [] do_sys_open+0x69/0x140
Jul 24 08:44:36 lin01 kernel: [ 7433.980222]  [] sys_open+0x20/0x30
any ideas?
cPanelMichael
July 24, 2014 14:22
Hello :) You may want to provide that output to your data center or system administrator to have them check this issue on your behalf. Have you checked to see if there are any other causes of the high load? This thread might be helpful: Troubleshooting High Loads On Linux Systems Thank you.
Kernel Issue Possibly?

Comments

Didn't find what you were looking for?
