High server load / out of memory
Hi all, I'm hoping someone can point me in the right direction with this as I'm no expert when it comes to servers!
My server has been down/unresponsive twice in the past 2 days, this has just started happening and I've made no changes to the server. After it's occurred I'm left with SSH access and a server that has pretty much no processes running on it.
Looking into it as far as I'm able, I'm seeing a lot of these entries in var/log/messages, anything between 5 and 20 of these per second before the issue occurs:
Then at some point the following occurs and the server starts shutting processes down until almost nothing is running, the log file then ends until a reboot is done:
I'm trying to figure out where to go from here. Is it a DDos, does some firewall config need changing or is there some other process causing this somewhere. In short, I'm lost and any advice to diagnose this further would be greatly appreciated. I've included a bigger snippet of that log and also the sar log from the time of the issue where you can see the server load go way high:
Many thanks!
May 7 17:48:13 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=111.250.188.187 DST=69.10.138.117 LEN=67 TOS=0x00 PREC=0x00 TTL=238 ID=47804 DF PROTO=UDP SPT=39601 DPT=53 LEN=47
Then at some point the following occurs and the server starts shutting processes down until almost nothing is running, the log file then ends until a reboot is done:
May 7 17:48:22 server kernel: lfd - (child) c invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
I'm trying to figure out where to go from here. Is it a DDos, does some firewall config need changing or is there some other process causing this somewhere. In short, I'm lost and any advice to diagnose this further would be greatly appreciated. I've included a bigger snippet of that log and also the sar log from the time of the issue where you can see the server load go way high:
03:30:01 PM 1 185 0.00 0.11 0.15
03:40:01 PM 1 186 0.04 0.03 0.08
03:50:01 PM 1 198 0.85 0.39 0.20
04:00:01 PM 1 207 0.31 0.52 0.35
04:10:01 PM 0 200 0.26 0.42 0.48
04:20:01 PM 1 192 0.02 0.21 0.35
04:30:01 PM 6 226 0.27 0.44 0.37
04:40:02 PM 0 207 1.23 2.28 1.44
04:50:01 PM 26 267 5.44 3.55 2.33
05:00:01 PM 0 276 0.13 1.54 2.18
05:10:01 PM 0 204 0.82 1.24 1.72
05:20:01 PM 1 214 0.14 0.56 1.17
05:30:02 PM 1 235 2.80 2.04 1.63
05:40:05 PM 25 265 17.26 8.98 4.55
05:50:17 PM 1 382 79.55 34.77 15.53
Average: 1 194 1.16 0.70 0.48
06:23:46 PM LINUX RESTART
06:30:01 PM runq-sz plist-sz ldavg-1 ldavg-5 ldavg-15
06:40:01 PM 1 190 1.00 1.00 0.82
May 7 17:48:13 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=111.250.188.187 DST=69.10.138.117 LEN=67 TOS=0x00 PREC=0x00 TTL=238 ID=47804 DF PROTO=UDP SPT=39601 DPT=53 LEN=47
May 7 17:48:13 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=37.119.206.219 DST=69.10.138.117 LEN=71 TOS=0x00 PREC=0x00 TTL=244 ID=56014 DF PROTO=UDP SPT=6091 DPT=53 LEN=51
May 7 17:48:13 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=94.115.87.90 DST=69.10.138.116 LEN=69 TOS=0x00 PREC=0x00 TTL=241 ID=22871 DF PROTO=UDP SPT=4674 DPT=53 LEN=49
May 7 17:48:13 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=64.106.94.250 DST=69.10.138.117 LEN=61 TOS=0x00 PREC=0x00 TTL=241 ID=63838 DF PROTO=UDP SPT=56782 DPT=53 LEN=41
May 7 17:48:13 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=98.33.204.243 DST=69.10.138.116 LEN=69 TOS=0x00 PREC=0x00 TTL=239 ID=62156 DF PROTO=UDP SPT=36663 DPT=53 LEN=49
May 7 17:48:14 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=8.11.53.149 DST=69.10.138.116 LEN=61 TOS=0x00 PREC=0x00 TTL=245 ID=37941 DF PROTO=UDP SPT=17323 DPT=53 LEN=41
May 7 17:48:14 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=58.17.253.7 DST=69.10.138.117 LEN=73 TOS=0x00 PREC=0x00 TTL=243 ID=1789 DF PROTO=UDP SPT=28459 DPT=53 LEN=53
May 7 17:48:14 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=63.123.136.49 DST=69.10.138.116 LEN=73 TOS=0x00 PREC=0x00 TTL=244 ID=12424 DF PROTO=UDP SPT=1321 DPT=53 LEN=53
May 7 17:48:14 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=62.91.116.38 DST=69.10.138.116 LEN=73 TOS=0x00 PREC=0x00 TTL=243 ID=9588 DF PROTO=UDP SPT=61148 DPT=53 LEN=53
May 7 17:48:14 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=67.105.254.185 DST=69.10.138.117 LEN=65 TOS=0x00 PREC=0x00 TTL=245 ID=47358 DF PROTO=UDP SPT=33677 DPT=53 LEN=45
May 7 17:48:16 server kernel: Firewall: *UDP_IN Blocked* IN=eth0 OUT= MAC=00:50:56:9d:2b:e4:00:11:5d:01:8c:00:08:00 SRC=110.180.50.152 DST=69.10.138.117 LEN=63 TOS=0x00 PREC=0x00 TTL=244 ID=38706 DF PROTO=UDP SPT=3586 DPT=53 LEN=43
May 7 17:48:22 server kernel: lfd - (child) c invoked oom-killer: gfp_mask=0x200da, order=0, oom_adj=0, oom_score_adj=0
May 7 17:48:22 server kernel: lfd - (child) c cpuset=/ mems_allowed=0
May 7 17:48:22 server kernel: Pid: 53710, comm: lfd - (child) c Not tainted 2.6.32-358.14.1.el6.x86_64 #1
May 7 17:48:22 server kernel: Call Trace:
May 7 17:48:22 server kernel: [] ? cpuset_print_task_mems_allowed+0x91/0xb0
May 7 17:48:22 server kernel: [] ? dump_header+0x90/0x1b0
May 7 17:48:22 server kernel: [] ? __delayacct_freepages_end+0x2e/0x30
May 7 17:48:22 server kernel: [] ? security_real_capable_noaudit+0x3c/0x70
May 7 17:48:22 server kernel: [] ? oom_kill_process+0x82/0x2a0
May 7 17:48:22 server kernel: [] ? select_bad_process+0xe1/0x120
May 7 17:48:22 server kernel: [] ? out_of_memory+0x220/0x3c0
May 7 17:48:22 server kernel: [] ? __alloc_pages_nodemask+0x8ac/0x8d0
May 7 17:48:22 server kernel: [] ? alloc_pages_vma+0x9a/0x150
May 7 17:48:22 server kernel: [] ? do_wp_page+0xfd/0x920
May 7 17:48:22 server kernel: [] ? swap_info_get+0x63/0xe0
May 7 17:48:22 server kernel: [] ? handle_pte_fault+0x2cd/0xb50
May 7 17:48:22 server kernel: [] ? handle_mm_fault+0x23a/0x310
May 7 17:48:22 server kernel: [] ? __do_page_fault+0x139/0x480
May 7 17:48:22 server kernel: [] ? __switch_to+0x1ac/0x320
May 7 17:48:22 server kernel: [] ? thread_return+0x4e/0x76e
May 7 17:48:22 server kernel: [] ? do_sigaction+0x91/0x1d0
May 7 17:48:22 server kernel: [] ? set_task_comm+0x53/0x70
May 7 17:48:22 server kernel: [] ? do_page_fault+0x3e/0xa0
May 7 17:48:22 server kernel: [] ? page_fault+0x25/0x30
May 7 17:48:22 server kernel: Mem-Info:
May 7 17:48:22 server kernel: Node 0 DMA per-cpu:
May 7 17:48:22 server kernel: CPU 0: hi: 0, btch: 1 usd: 0
May 7 17:48:22 server kernel: Node 0 DMA32 per-cpu:
May 7 17:48:22 server kernel: CPU 0: hi: 186, btch: 31 usd: 30
May 7 17:48:22 server kernel: active_anon:97981 inactive_anon:97705 isolated_anon:864
May 7 17:48:22 server kernel: active_file:171 inactive_file:317 isolated_file:0
May 7 17:48:22 server kernel: unevictable:0 dirty:18 writeback:167 unstable:0
May 7 17:48:22 server kernel: free:12232 slab_reclaimable:4130 slab_unreclaimable:13454
May 7 17:48:22 server kernel: mapped:232 shmem:17 pagetables:15743 bounce:0
May 7 17:48:22 server kernel: Node 0 DMA free:4644kB min:668kB low:832kB high:1000kB active_anon:5200kB inactive_anon:5144kB active_file:12kB inactive_file:40kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15368kB mlocked:0kB dirty:0kB writeback:0kB mapped:24kB shmem:0kB slab_reclaimable:48kB slab_unreclaimable:12kB kernel_stack:16kB pagetables:296kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:29 all_unreclaimable? no
May 7 17:48:22 server kernel: lowmem_reserve[]: 0 993 993 993
May 7 17:48:22 server kernel: Node 0 DMA32 free:44284kB min:44384kB low:55480kB high:66576kB active_anon:386724kB inactive_anon:385676kB active_file:672kB inactive_file:1228kB unevictable:0kB isolated(anon):3456kB isolated(file):0kB present:1017504kB mlocked:0kB dirty:72kB writeback:668kB mapped:904kB shmem:68kB slab_reclaimable:16472kB slab_unreclaimable:53804kB kernel_stack:3272kB pagetables:62676kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:704 all_unreclaimable? no
May 7 17:48:22 server kernel: lowmem_reserve[]: 0 0 0 0
May 7 17:48:22 server kernel: Node 0 DMA: 15*4kB 13*8kB 20*16kB 14*32kB 10*64kB 0*128kB 2*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 4644kB
May 7 17:48:22 server kernel: Node 0 DMA32: 1175*4kB 488*8kB 396*16kB 633*32kB 70*64kB 6*128kB 1*256kB 1*512kB 1*1024kB 1*2048kB 0*4096kB = 44284kB
May 7 17:48:22 server kernel: 11419 total pagecache pages
May 7 17:48:22 server kernel: 10747 pages in swap cache
May 7 17:48:22 server kernel: Swap cache stats: add 3451250, delete 3440503, find 819493753/820108390
May 7 17:48:22 server kernel: Free swap = 0kB
May 7 17:48:22 server kernel: Total swap = 524280kB
May 7 17:48:22 server kernel: 262143 pages RAM
May 7 17:48:22 server kernel: 9171 pages reserved
May 7 17:48:22 server kernel: 204471 pages shared
May 7 17:48:22 server kernel: 226961 pages non-shared
Many thanks!
-
Hello, The "c invoked oom-killer" message indicates your server is running out of memory. The following thread is a good place to get started: Troubleshooting High Loads On Linux Systems Thank you. 0
Please sign in to leave a comment.
Comments
1 comment