Outgoing Email Abuse from localhost
I recently noticed that some accounts on one of our servers, were sending out spam mail.
However, the strange part is that the logs were showing that the mail was coming from our IPv6 loopback address i.e. ::1 Assuming that webmail maybe compromised, I investigated further. But I noticed that most of these accounts did not have any email address or forwarders on our server - they were using Google Apps. Hence there was no way of authenticating to send out mail. All the emails are going out from info@ accounts only. The maillog shows:
Other info: [LIST]SMTP Authentication is enabled on the server.
Nobody user is prevented from sending out mail.
CSF is enabled. CSF did not send any mail alerts.
IPv6 is enabled in Exim. This is happening only from the ::1 IPv6 Address.
This is happening to about 7 - 10 domains on the server.
This is happening only to the INFO@ email address for all those domains.
Some of the accounts are using Google Apps for mail and don't even have an email address configured on my server.
One of the accounts was suspended on 1-Aug and still remains suspended, still mails went out from it.
The Firewall logs show multiple unsuccessful brute force attempts for all those accounts a day prior to the mail spamming. Besides that nothing suspicious.
Running CLOUDLINUX 6.8 x86_64 with WHM 58.0 (build 13)
Any ideas on how this could be happening?
1 (compromiseddomain.com)[::1] info@compromiseddomain.com alt2.gmail-smtp-in.l.google.com[173.194.213.26] jii93@gmail.com
1 (compromiseddomain.com)[::1] info@compromiseddomain.com alt4.gmail-smtp-in.l.google.com[209.85.203.27] jignre@gmail.com
1 (compromiseddomain.com)[::1] info@compromiseddomain.com gmail-smtp-in.l.google.com[74.125.129.27] jien@gmail.com
1 (compromiseddomain.com)[::1] info@compromiseddomain.com gmail-smtp-in.l.google.com[74.125.129.27] jibsney@gmail.com
1 (compromiseddomain.com)[::1] info@compromiseddomain.com gmail-smtp-in.l.google.com[74.125.129.27] jig78@gmail.com
However, the strange part is that the logs were showing that the mail was coming from our IPv6 loopback address i.e. ::1 Assuming that webmail maybe compromised, I investigated further. But I noticed that most of these accounts did not have any email address or forwarders on our server - they were using Google Apps. Hence there was no way of authenticating to send out mail. All the emails are going out from info@ accounts only. The maillog shows:
2016-08-10 07:31:18 1bXIpa-003UXt-QG <= info@compromiseddomain H=(compromiseddomain) [::1]:48468 P=esmtp S=1617 id=2025813250.1694955.1470794479702@compromiseddomain T="howre you?" for pashl23@orange.fr
2016-08-10 07:31:19 1bXIpa-003UXt-QG From: header (rewritten was: [info@compromiseddomain], actual sender is not the same system user) original=[info@compromiseddomain] actual_sender=[nobody@myserver.com]
2016-08-10 07:31:19 SMTP connection from (compromiseddomain) [::1]:48468 lost
Other info: [LIST]
-
Hello, I recommend running the following command to rule out the possibility that SPAM is coming from a script uploaded to an account: awk '/cwd=\/home\// {print $3}' /var/log/exim_mainlog|sort|uniq -c|sort -n
Look for directories with a large send count and review the files in those directories to see if any scripts are able to send email. Thank you.0 -
My problem is similar. Below is exim_mainlog extract. 2016-08-17 01:39:05 1bZiKO-0007WK-GW H=(userdomain.com) [::1]:56891 Warning: Message has been scanned: no virus or other harmful content was found 2016-08-17 01:39:05 1bZiKO-0007WK-GW <= info@userdomain.com H=(userdomain.com) [::1]:56891 P=esmtp S=2256 id=2128087589.1190747.1471369143281@userdomain.com T="Fw: Hi there." for recipient@aol.com 2016-08-17 01:39:05 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1bZiKO-0007WK-GW 2016-08-17 01:39:05 1bZiKO-0007WK-GW SMTP connection identification H= A=::1 P=56891 M=1bZiKO-0007WK-GW U=nobody ID=99 S=nobody B=authenticated_local_user 2016-08-17 01:39:05 1bZiKO-0007WK-GW From: header (rewritten was: [info@userdomain.com], actual sender is not the same system user) original=[info@userdomain.com] actual_sender=[nobody@myserver.com] 2016-08-17 01:39:10 1bZiKO-0007WK-GW => recipient@aol.com R=lookuphost T=remote_smtp H=mailin-02.mx.aol.com [152.163.0.99] X=TLSv1:DHE-RSA-AES256-SHA:256 CV=yes C="250 2.0.0 Ok: queued as 05C7B700000BF" 2016-08-17 01:39:10 1bZiKO-0007WK-GW Completed
Prevent "nobody" from sending mail is "ON". How is the server still allowing emails to be sent by nobody to remote address? "cwd" from the log above shows it is not from a script. Please help. Thank you.0 -
Great. You have the exact same issue that I did.... It was solved by enabling mod_ruid2. This helped us determine the user that was sending out the mails and also the script. The cause was a compromised WordPress plugin i.e. Revolution Slider. It was using the cPanel password to authenticate the mail going out. This was one of the latest versions of Revolution Slider which has no known vulnerability. Removing the entire folder was the only solution, as every sub-folder in the templates folder was infected. Resetting the cPanel password stopped the mails from authenticating. 0 -
Now coming to the point about the Tweak Setting to prevent the nobody user from sending out mail. When I reported this to cPanel, this was their response: "... The tweak setting 'Prevent "nobody" from sending mail' is a restriction that only applies to emails sent with /usr/sbin/sendmail and does not restrict emails sent as SMTP through a local TCP port. If you wanted to disallow sending SMTP emails through localhost, you would need to enable the 'SMTP Restrictions' feature in the Security Center in WHM. This would add firewall rules to block connecting to the SMTP ports locally, forcing PHP scripts to use sendmail (and honor the tweak setting mentioned earlier). It is worth noting that if you use CSF to configure your firewall, the 'SMTP Restrictions' setting in WHM will not work. You would need to enable the following features in the CSF firewall settings to achieve the same effect: SMTP_BLOCK = ON, SMTP_ALLOWLOCAL = OFF. ..." Also, when they investigated the issue on the server, the Support Rep's response was: "... Thanks again for the thorough follow up. To clarify though, in my previous testing, while accessing the "nobody" user, it didn't seem to matter whether I connected to the server on the IPv4 or IPv6 addresses, the messages were blocked if I didn't first authenticate with another user, which seems to be a deliberate choice to prevent servers using DSO or CGI from blocking every source of mail. Reviewing this a bit further, I found a much older case where we found users having similar problems while on DSO were encountering conflicts with the following options in Exim: ---------------------------------------------------- "Query Apache server status to determine the sender of email sent from processes running as nobody" "Trust X-PHP-Script headers to determine the sender of email sent from processes running as nobody" ---------------------------------------------------- Both are enabled by default, but in the past, have allowed some user's scripts to send mail after having been identified by Exim as not actually being sent as nobody. Though this server shouldn't encounter this problem again with the most recent changes to its settings, if you would like to go back to the DSO handler, I'd recommend disabling both of these options as well and expect things will work more as you'd expect. ..." 0 -
I have no idea how the emails can be sent if what cPanel support said is working. My server is not using CSF. SMTP Restrictions is enabled. Prevent "nobody" from sending mail is ON. I might just switch to mod_ruid2 like what you did or force reset the cPanel password. Thank you very much for the update, ruzbehraja. 0 -
I have no idea how the emails can be sent if what cPanel support said is working.
The emails are being authenticated with a username and password. In my case it was a cPanel username and password. Mails were going out from a script which was in a WordPress plugin folder. To find out which user was being used to authenticate the mails, after you install mod_ruid2 grep the logs again. I think what cPanel really needs to highlight in the Tweak Settings option explanation is that "The tweak setting 'Prevent "nobody" from sending mail' is a restriction that only applies to emails sent with /usr/sbin/sendmail and does not restrict emails sent as SMTP through a local TCP port." If you still can't find out, open a support ticket with cPanel and do post back here if you bump into something interesting.0 -
Thank you for the explanation, ruzbehraja. I was looking at the response given by cPanel saying that enabling SMTP Restrictions will disallow sending SMTP emails through localhost when coupled with 'Prevent "nobody" from sending mail'. I looked at the description in WHM -> Server Configuration -> Tweak Settings ... Enabling this feature will redirect outgoing SMTP connections to the local mail server. root, exim, and mailman are still allowed to make direct connections.
and WHM -> Security Center -> SMTP Restrictions This feature prevents users from bypassing the mail server to send mail, a common practice used by spammers. It will allow only the MTA, mailman, and root to connect to remote SMTP servers.
Do the descriptions mean that the SMTP Restrictions feature is only to disallow connections to remote SMTP servers? If yes, does it mean that the cPanel response is wrong in saying that it forces PHP scripts to use sendmail? My log shows it is using /usr/sbin/exim.2016-08-17 01:39:05 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1bZiKO-0007WK-GW
I am kind of confused.0 -
Do the descriptions mean that the SMTP Restrictions feature is only to disallow connections to remote SMTP servers? If yes, does it mean that the cPanel response is wrong in saying that it forces PHP scripts to use sendmail?
Block outgoing SMTP (except for root, exim and mailman) forces scripts/users to use the exim/sendmail binary instead of sockets access i.e. directly to a remote servers port 25 When they are forced to use sendmail, they have to send out as the nobody user. Which is when the nobody "Tweak Setting" will take effect or be triggered. If they are not forced to use sendmail, they can connect to your localhost random port and send through exim as if they are an authenticated local user, as what happened. i.e. they connected to the ::1 loopback and as your log says:U=nobody ID=99 S=nobody B=authenticated_local_user
They authenticated as a local user but sent the mail as 'nobody'. I am also not 100% clear on how this can be prevented and whether a similar tweak should be made applicable to prevent this in future. I am still a bit skeptical on whether or not it is related to the IPv6 address, because till date I have never faced such an issue with the IPv4 loopback, having the same settings. Also, when I went to the hostslists loopback Advanced Settings in Exim, and removed ::1, the spam stopped going out, however, webmail's connection also broke, so I couldn't test it for long.0 -
Thank you for clearing up on SMTP Restrictions. They authenticated as a local user but sent the mail as 'nobody'.
All the spam is going through ::1 using an authenticated user and nobody@myserver.com as the actual sender.2016-08-17 01:39:05 1bZiKO-0007WK-GW SMTP connection identification H= A=::1 P=56891 M=1bZiKO-0007WK-GW U=nobody ID=99 S=nobody B=authenticated_local_user
Would it be possible to write a custom filter to drop such emails? Anyone? cPanel has so many users, but it seems that this thread is only you and me. LOL0 -
cPanel has so many users, but it seems that this thread is only you and me. LOL
The modus operandi of this attack is to send out such few mails frequently that they don't trigger an alarm. Just about 10-20 mails are sent out every hour, from different domains from the info@ email address (which was obviously spoofed). Surprisingly all the email addresses must be valid, because I didn't notice anything (bounces) in the mail queue - all got delivered perfectly. This may have been happening for a couple of days, but only when I manually saw the mail statistics did I notice the bunch of ::1 loopback addresses all together. Hence I am pretty sure it is happening to others, but they have not realized it yet. Also, people maybe under a false impression that the nobody tweak setting should be blocking this. Which even I was under and I guess so were you. :)Would it be possible to write a custom filter to drop such emails? Anyone?
I don't think you need a filter for this because the mails were authenticated with a stolen password. Only thing is that it was spoofing the nobody user as the sender.0 -
2016-08-17 01:39:05 cwd=/var/spool/exim 3 args: /usr/sbin/exim -Mc 1bZiKO-0007WK-GW 2016-08-17 01:39:05 1bZiKO-0007WK-GW SMTP connection identification H= A=::1 P=56891 M=1bZiKO-0007WK-GW U=nobody ID=99 S=nobody B=authenticated_local_user 2016-08-17 01:39:05 1bZiKO-0007WK-GW From: header (rewritten was: [info@userdomain.com], actual sender is not the same system user) original=[info@userdomain.com] actual_sender=[nobody@myserver.com]
Is this considered enabled? The only thing I have not done is to use Mod_Ruid2 or suPHP or DSO with MPM ITK to force the scripts to run as the user in order to identify. Is it the only way? Is there any log for such kind of authentication?0 -
Is this considered enabled?
Yes, it does look enabled based on that output.The only thing I have not done is to use Mod_Ruid2 or suPHP or DSO with MPM ITK to force the scripts to run as the user in order to identify.
It's not required, but it will make figuring out the source of the SPAM leaving your system easier. Thank you.0
Please sign in to leave a comment.
Comments
14 comments