Skip to main content

[CPANEL-26002] LMTP errors after upgrading to cPanel & WHM Version 78

Adam Reece | WebBox
Moderator Notice: Please see

Comments

34 comments

Date Votes
  • keat63
    looking for the smtp defer message, I found this, which may or may not be of help. It mentions being blacklisted. Might be worth a quick blacklist check
    0
  • Adam Reece | WebBox
    Looks like it's actually a quota issue due to the server being a virtual machine. Andrea at cPanel found this log entry in "/var/log/maillog": [QUOTE]Feb 22 13:49:57 demeter dovecot: lmtp(128010): Error: mailbox_get_status(INBOX, STATUS_CHECK_OVER_QUOTA) failed: Mailbox INBOX: quota: Failed to get quota resource STORAGE_BYTES for INBOX: quota-fs: quotactl(Q_GETQUOTA, /dev/simfs) failed: No such device
    What I'm not sure of is why it only impacted this VM. We've got 15 of them and it only happened to this one today with the v78 update. Edit: Support have got back to me to say the quota setup is fine.
    0
  • kernow
    We also had this problem on all our servers after upgrading to 78.0.11. We have configservers MailScanner installed and if you do too, the fix is to close MailScanner, check and kill any MailScanner processes still running, then restart MailScanner.
    0
  • cPanelMichael
    We also had this problem on all our servers after upgrading to 78.0.11. We have configservers MailScanner installed and if you do too, the fix is to close MailScanner, check and kill any MailScanner processes still running, then restart MailScanner.

    Hello @kernow, We confirmed that MailScanner was not installed on the affected system as part of investigation in the support ticket that was opened by the original poster. Can you provide more information about the specific error message you encountered prior to restarting MailScanner, along with the steps you used to reproduce that error? Also, please post the output from the following commands: cat /var/cpanel/envtype mount|grep simfs
    Thank you.
    0
  • kernow
    Can't remember the exact error prior to restarting MailScanner, we had this same problem about a year ago after a cpanel update so we knew what to check. Checking the status of MailScanner said something like, "MailScanner: Could not use Custom Function code /usr/mailscanner/usr/share/MailScanner/perl/custom/MailControl.pm" It also said "SpamAssassin is not installed. This problem affected CloudLinux 6 & 7 servers. cat /var/cpanel/envtype standardroot@servername
    mount|grep simfs
    Returns nothing
    0
  • cPanelMichael
    Hello @kernow, This doesn't appear to relate to the issue described by the original poster. I recommend reporting this behavior to the developers of the MailScanner plugin you are using to verify if it's an issue they can address. Thank you.
    0
  • Adam Reece | WebBox
    For us the issue was that there was leftover Virtuozzo packages installed after the VM had been migrated to KVM. I don't know why it only started causing a problem now, but when cPanel's support removed these, and instructed me to do a reboot, the Exim queue delivered to Dovecot via LMTP successfully. [QUOTE]Hi there, With the assistance of my colleague, we were able to determine the issue. Was this KVM instance migrated from a VZ instance? I ask because there are remnants of VZ configuration files on this server: ----- [14:31:29 demeter root@11496321 ~]cPs# stat /etc/rc.d/init.d/vzquota File: `/etc/rc.d/init.d/vzquota' Size: 919 Blocks: 8 IO Block: 4096 regular file Device: fd01h/64769d Inode: 12046 Links: 1 Access: (0755/-rwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2019-02-21 19:14:03.460000071 +0000 Modify: 2017-09-30 00:38:28.000000000 +0100 Change: 2017-12-11 12:29:20.327019231 +0000 ----- ----- [14:32:03 demeter root@11496321 ~]cPs# rpm -qf /etc/rc.d/init.d/vzquota file /etc/rc.d/init.d/vzquota is not owned by any package ----- ----- [14:32:06 demeter root@11496321 ~]cPs# cat /etc/rc.d/init.d/vzquota #!/bin/sh # chkconfig: 2345 10 90 # description: prepare container to use OpenVZ 2nd-level disk quotas ### BEGIN INIT INFO # Provides: vzquota # Required-Start: $local_fs $time $syslog # Required-Stop: $local_fs # Default-Start: 2 3 4 5 # Default-Stop: 0 1 6 # Short-Description: Start vzquota at the end of boot # Description: Configure OpenVZ disk quota for a container. ### END INIT INFO start() { if [ ! -L /etc/mtab ]; then rm -f /etc/mtab >/dev/null 2>&1 ln -sf /proc/mounts /etc/mtab fi dev=$(awk '($2 == "/") && ($4 ~ /usrquota/) && ($4 ~ /grpquota/) {print $1}' /etc/mtab) if test -z "$dev"; then dev="/dev/simfs" rm -f /etc/mtab >/dev/null 2>&1 echo "/dev/simfs / reiserfs rw,usrquota,grpquota 0 0" > /etc/mtab grep -v " / " /proc/mounts >> /etc/mtab 2>/dev/null chmod 644 /etc/mtab fi [ -e "$dev" ] || mknod $dev b 0 22 quotaon -aug } case "$1" in start) start ;; *) exit esac ----- I have moved this file out of the way. I would recommend that you reboot the server so that KVM configurations are used. You may want to speak with your data center regarding the remnant files. If you have anymore questions, please feel free to ask. Thank you,
    After rebooting I ran `exim -qff -v` and everything delivered fine. Hope this helps someone else. :)
    0
  • cPAusaf
    This seems to be a permissions/ownership issue. If another user experiences this, then I would recommend running: /scripts/mailperm --verbose $username If you happen to see 'Unable to fix ownership', then please open a support ticket so that we may take a closer look.
    0
  • Niclas Hedfors
    Hello, Our LIVE production server was updated to 78.0.12 last night. After that most of our clients wont receive any email. In "track Delivery" we se the following error. LMTP error after RCPT TO:: 451 4.3.0 Temporary internal error
    I have submitted a ticket to support but as this is affecting a lots of clients i want it resolved as soon as possible and trying to get help wherever i can.
    0
  • keat63
    Could it be related to this
    0
  • Niclas Hedfors
    Tried to edit my last post but apparently it was spam-related? It's quota related. As I change the quota for the cPanel-account to "unlimited" it starts to work properly. For now, i've changed the quota on all accounts to "unlimited", so its not that urgent any more, but i still want a solution to this.
    0
  • Niclas Hedfors
    Could it be related to this

    I've looked at that but I'm not sure, as it seems to be quota related in my case and the OP in that thread got i confirmed from the support it wasn't.
    0
  • keat63
    If you could update this thread with your ticket number, one of the mods will update us with the outcome. It may help someone in the future or coming days. Seems odd to get two very similar errors in such close proximity.
    0
  • Niclas Hedfors
    My support ticket number: 11530727 According to the maillog, there is some problem with an NFS mount on our server that not report quota at all. What we can see in the maillog this didn't work before the update either, but it seems like it is handled a bit different after the update. Got this from cPanel Support: [QUOTE]It does appear that an upstream change to dovecot now treats these LMTP errors as fatal. Reviewing the changelogs, I don't see anything specific in dovecot 2.3 about LMTP errors, but I do see there was lots of changes made to LMTP.
    Locating it to a quota problem made me solve it temporary by set all customers to "unlimited" as i wrote earlier. All customers are up and running. For now, since quota never worked properly, having customers on unlimited quota practically doesn't make much of a differens. I will try to resolve the NFS part, with my colleagues who know more about the NFS connection, to make the quota work properly.
    0
  • cPanelMichael
    Hello Everyone, We published the following Dovecot security update in cPanel & WHM version /etc/mtab uses /dev/simfs, whereas /proc/mounts uses /dev/ploop$$. The difference exists because the vzquota init script incorrectly changes "ploop" to "simfs" in the /etc/mtab file at boot time time (/etc/mtab should instead exist as a symbolic link to /proc/mounts on these systems). If the output does not look like this on your system, then it's likely there's a different underlying cause of the quota errors on your system. In such cases, feel free to respond to this thread with the error message you see in /var/log/maillog and we'll be happy to help you diagnose the problem. Upon confirming the system is affected by this particular issue, you can run the following commands to fix this: chkconfig --level 234 vzquota off mv -v /etc/mtab{,.old} ln -s /proc/mounts /etc/mtab reboot
    These commands disable vzquota at boot time, setup /etc/mtab is a symbolic link to /proc/mounts, and reboot the server. Let us know if you have any questions. Thank you. Edit Log: 03/06/2019: Updated case number to CPANEL-26002. 03/01/2019: Updated the workaround instructions to note the commands should be ran on the individual server.
    0
  • deddy
    To gain some time I did the following: vi 20-lmtp.conf # Verify quota before replying to RCPT TO. This adds a small overhead. # lmtp_rcpt_check_quota = no changed to # Verify quota before replying to RCPT TO. This adds a small overhead. lmtp_rcpt_check_quota = no service dovecot restart
    0
  • cPanelMichael
    # Verify quota before replying to RCPT TO. This adds a small overhead. # lmtp_rcpt_check_quota = no changed to # Verify quota before replying to RCPT TO. This adds a small overhead. lmtp_rcpt_check_quota = no

    Hello @deddy, While this may act as a temporary workaround, keep in mind this is an untested change and can potentially lead to additional delivery issues if a cPanel account or individual email account is at it's quota limit. Thank you.
    0
  • shaun
    We've seen this issue too recently on a number of customer servers. The cause so far looks to be permissions related, mainly ownership. Here's a the fix we've been using... Run the following command /scripts/mailperm --verbose username |grep "ownership" | grep "should be"
    If you see output that looks simliar to the following Unable to fix ownership on /home/username/mail/domain.tld/.Sent/cur/1484153449.M120071P468622.server1.domain.tld,S=64721,W=65681:2,S: currently (556:556), should be (618:618)
    Then you need to correct the permissions on that account. The important part of the output is the "should be" part with the numbers in the parenthesis. In my example thats 556:556 and 618:618. You can then do the following cd /home/username/mail find ./ -user 556 -exec chown 616 {} \; find ./ -group 556 -exec chgrp 618 {} \;
    Now you should run /scripts/mailperm --verbose username two more times. The first time you may see output showing the script fixing permissions. The second run you should not see any output. If you do, there are more issues that need to be fixed. Hope that helps
    0
  • cPanelMichael
    Hi @shaun, Can you access one of the affected servers and look at the /etc/fstab file to see if the root filesystem (e.g. "/") is associated with a device that doesn't exist? For instance, many of the affected systems we've seen have an entry like this in their /etc/fstab file despite /dev/simfs not existing as a device: /dev/simfs / reiserfs rw,usrquota,grpquota 0 0
    Here's a command you can use to help identify if an invalid device name is utilized: diff -u /etc/mtab /proc/mounts
    Thank you.
    0
  • shaun
    @cPanelMichael I've not seen anything from simfs. We've mainly been seeing this issue on servers were it looks like somebody had attempted to repair a users mail dir with incorrect ownership and permissions. The issue did not present itself until recently.
    0
  • PRKGeoff
    I was receiving this same error on accounts created back when I was running cPanel v74. Once the server updated to v78 various accounts could no longer receive emails however forwarders to external email accounts worked as normal. If I created a new test email account in v78 it worked as expected. I tried the fix permissions scripts and confirmed quotas were enabled. Quotacheck didn't return any errors. I'm using Ext4. I also ran the remove_dovecot_index_files script and restarted exim and dovecot but it made no difference. I ended up comparing the permissions on files between failing and working accounts and found that the /home/*user*/mail/*domain.com*/*address*/maildirsize files had different permissions - 0440 vs 0640. Once I updated the permissions on the file in the failing account to 640 it started to receive mail successfully again. It appears Exim now expects to be able to write to the maildirsize file of each email account as *user* now.
    0
  • cPanelMichael
    I was receiving this same error on accounts created back when I was running cPanel v74. Once the server updated to v78 various accounts could no longer receive emails however forwarders to external email accounts worked as normal. If I created a new test email account in v78 it worked as expected.

    Hi @PRKGeoff, Can you confirm the specific error messages you noticed before correcting the permissions on the maildirsize files? Was there any specific output to /var/log/maillog at the time of the error? Thank you.
    0
  • PRKGeoff
    Hi @PRKGeoff, Can you confirm the specific error messages you noticed before correcting the permissions on the maildirsize files? Was there any specific output to /var/log/maillog at the time of the error? Thank you.

    Hi @cPanelMichael, It looks like it was actuallly Dovecot that needed to update the maildirsize file. The messages in /var/log/maillog for failure and success were: Apr 4 16:25:04 web1 dovecot: lmtp(13165): Error: Mailbox INBOX: quota: Failed to get quota resource MESSAGE for INBOX: quota-maildir: Failed to get MESSAGE: open(/home/user/mail/domain.com/info/maildirsize) failed: Permission denied Apr 4 16:25:04 web1 dovecot: lmtp(13165): Error: mailbox_get_status(INBOX, STATUS_CHECK_OVER_QUOTA) failed: Mailbox INBOX: quota: Failed to get quota resource MESSAGE for INBOX: quota-maildir: Failed to get MESSAGE: open(/home/user/mail/domain.com/info/maildirsize) failed: Permission denied Apr 5 00:13:38 web1 dovecot: lmtp(info@domain.com)<76761>: msgid=<2426755953_20190405_124832_CSP_user@domain2.com>: saved mail to INBOX
    /var/log/exim_mainlog would contain something like: 2019-04-01 08:40:16 1hAj7a-009S6O-S7 == info@domain.com R=virtual_user T=dovecot_virtual_delivery defer (-44): LMTP error after RCPT TO:: 451 4.3.0 Temporary internal error 2019-04-01 08:40:16 1hAj7a-009S6O-S7 ** info@domain.com: retry timeout exceeded 2019-04-01 08:40:16 1hAj7g-009S9X-O5 <= <> R=1hAj7a-009S6O-S7 U=mailnull P=local S=75634 T="Mail delivery failed: returning message to sender" for bounce-mc.us13_57844637.701597-info=domain.com@domain2.com
    The NDR received when attempting to send was: This message was created automatically by mail delivery software. A message that you sent could not be delivered to one or more of its recipients. This is a permanent error. The following address(es) failed: info@domain.com LMTP error after RCPT TO:: 451 4.3.0 Temporary internal error: retry timeout exceeded
    0
  • cPanelMichael
    Hello @PRKGeoff, Thanks for taking the time to share the log output with us. I've added this information to the internal case (CPANEL-26002). Thank you.
    0
  • attiliok
    Hi, I have some problem, quite urgent. Here is a part of exim log: Apr 11 12:09:10 server2 dovecot: lmtp(15728): Connect from local Apr 11 12:09:10 server2 dovecot: lmtp(15728): Error: setmntent(/etc/mtab) failed: Permission denied Apr 11 12:09:10 server2 dovecot: lmtp(15728): Error: Mailbox INBOX: quota: Failed to get quota resource STORAGE_BYTES for INBOX: quota-fs: Mount point unknown Apr 11 12:09:10 server2 dovecot: lmtp(15728): Error: mailbox_get_status(INBOX, STATUS_CHECK_OVER_QUOTA) failed: Mailbox INBOX: quota: Failed to get quota resource STORAGE_BYTES for INBOX: quota-fs: Mount point unknown Apr 11 12:09:10 server2 dovecot: lmtp(15728): Disconnect from local: Client has quit the connection (state=READY) Thanks in advance
    0
  • Infopro
    Hi, I have some problem, quite urgent.

    Have you seen this post?
    0
  • attiliok
    Yes. The output of the commands suggested does'nt resamble the one described in the post. That is why I posted my maillog file As a workaround I can change the quota of the affected account to UNLIMITED from cpanel, and the problem seems solved. But I don't know how many accounts are affected. Thanks
    0
  • cPanelMichael
    Hello Everyone, Case CPANEL-26800 is now open to specifically address LMTP delivery failures occurring as a result of invalid permissions on an email account's maildirsize file. CPANEL-26002 remains open to address LMTP delivery failures occurring when the configured disk is different from the one currently mounted for the partition where LMTP delivers mail. A solution developed to prevent the delivery failures is now undergoing internal testing. I'll monitor both cases and update this thread with new information on the case statuses as it becomes available. Thank you.
    0
  • attiliok
    Hi, Is there some new info about the problem? Or some workaround to avoid setting the disk space to UNLIMITED? Thanks
    0
  • cPanelMichael
    Is there some new info about the problem? Or some workaround to avoid setting the disk space to UNLIMITED?

    Hello @attiliok, Can you verify the specific error output you see in /var/log/maillog when this happens on the affected server? Thank you.
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post