doveadm was terminated because the system is low on memory

Hi everyone, Suddenly our mail server stopped working and we keep receiving Cpanel email notifications saying: Out of memory: The process "doveadm" was terminated because the system is low on memory. In order to avoid a system crash due to low memory, the kernel terminated the process named "doveadm" with thePID "22260".

For addtional details, see the attached dmesg log dump.
[80119.994247] [22673] 0 22673 77061 35855 109 0 0 lmtp
[80119.994386] [22674] 0 22674 62295 29350 89 0 0 lmtp
[80119.994522] [22676] 0 22676 77061 34434 100 0 0 lmtp
[80119.994671] [22677] 0 22677 62724 30888 93 0 0 lmtp
[80119.994875] [22678] 0 22678 77061 33572 102 0 0 lmtp
[80119.995036] [22680] 0 22680 77061 33274 100 0 0 lmtp
[80119.995216] [22681] 0 22681 62460 29904 90 0 0 lmtp
[80119.995384] [22723] 0 22723 11034 745 26 0 0 /usr/local/cpan
[80119.995565] Out of memory: Kill process 22260 (doveadm) score 74 or sacrifice child
[80119.995712] Killed process 22260 (doveadm) total-vm:713376kB, anon-rss:237012kB, file-rss:0kB

Dovecot's logs showing the following:

Oct 22 11:33:34 vps122722 dovecot: lmtp(swat@example.net): Fatal: block_alloc(268435456): Out of memory

Oct 22 11:33:34 vps122722 dovecot: lmtp(swat@example.net): Error: Raw backtrace: /usr/lib64/dovecot/libdovecot.so.0(+0x8bd1a) [0x7f7ffca41d1a] -> /usr/lib64/dovecot/libdovecot.so.0(+0x8bd86) [0x7f7ffca41d86] -> /usr/lib64/dovecot/libdovecot.so.0(i_fatal+0) [0x7f7ffc9e1d2d] -> /usr/lib64/dovecot/libdovecot.so.0(+0xa4381) [0x7f7ffca5a381] -> /usr/lib64/dovecot/libdovecot.so.0(+0xa43bc) [0x7f7ffca5a3bc] -> /usr/lib64/dovecot/libdovecot-storage.so.0(maildir_uidlist_refresh+0x392) [0x7f7ffcd34d62] -> /usr/lib64/dovecot/libdovecot-storage.so.0(maildir_uidlist_get_mailbox_guid+0x6a) [0x7f7ffcd35eaa] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x6a9e0) [0x7f7ffcd2d9e0] -> /usr/lib64/dovecot/libdovecot-storage.so.0(+0x9f367) [0x7f7ffcd62367] -> /usr/lib64/dovecot/libdovecot-storage.so.0(mailbox_get_metadata+0x6e) [0x7f7ffcd0737e] -> /usr/lib64/dovecot/libdovecot-lda.so.0(mail_deliver_deduplicate_guid_if_needed+0x80) [0x7f7ffcff88c0] -> /usr/lib64/dovecot/libdovecot-lda.so.0(mail_deliver_save+0x196) [0x7f7ffcff91e6] -> /usr/lib64/dovecot/libdovecot-lda.so.0(mail_deliver+0x11b) [0x7f7ffcff95ab] -> dovecot/lmtp() [0x40681d] -> dovecot/lmtp(cmd_data+0x1ed) [0x406eed] -> dovecot/lmtp(client_input_handle+0x20b) [0x40559b] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_call_io+0x5b) [0x7f7ffca5463b] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run_internal+0xb7) [0x7f7ffca55bf7] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_handler_run+0x25) [0x7f7ffca546e5] -> /usr/lib64/dovecot/libdovecot.so.0(io_loop_run+0x38) [0x7f7ffca54888] -> /usr/lib64/dovecot/libdovecot.so.0(master_service_run+0x13) [0x7f7ffc9e7b33] -> dovecot/lmtp(main+0x169) [0x404e49] -> /lib64/libc.so.6(__libc_start_main+0xfd) [0x7f7ffc640d1d] -> dovecot/lmtp() [0x404c19]

Oct 22 11:33:34 vps122722 dovecot: lmtp(swat@example.net): Fatal: master: service(lmtp): child 24838 returned error 83 (Out of memory (service lmtp { vsz_limit=512 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)

Oct 22 11:33:34 vps122722 dovecot: lmtp(swat@example.net): Fatal: master: service(lmtp): child 24835 returned error 83 (Out of memory (service lmtp { vsz_limit=512 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)

We've tried deleting the mail queue from WHM and restarting Dovecot but in vain... Can anyone please help? Thank you!

cPanelNick

October 22, 2016 19:52

Please try increasing the Process Memory Limit for Mail (MB) under Mailserver Configuration in WHM.

soufian

October 22, 2016 22:08

Thanks CpanelNick for your quick response, it seems like increasing the Memory limit doesn't fix the problem, the error evolved to this :

Fatal: master: service(lmtp): child 30376 returned error 83 (Out of memory (service lmtp { vsz_limit=1024 MB }, you may need to increase it) - set CORE_OUTOFMEM=1 environment to get core dump)

My guess is that it's not a pure memory problem as nothing has changed before and after the bug in term of load and usage.

SysSachin

October 23, 2016 02:04

Please try to increase vsz_limit in the exim configartion using below steps : Login to WHM >> Service Configuration >> Mail Server Configuration. Once changed we can confirm the modifications by executing the command


doveconf | grep vsz_limit

cPanelMichael

October 25, 2016 16:33

Hello @soufian, Does the issue persist if you up the value to 2048? If so, would you mind opening a support ticket so that we can take a closer look? You can post the ticket number here so we can update this thread with the outcome. Thank you.

doveadm was terminated because the system is low on memory

Comments

Didn't find what you were looking for?