Skip to main content

Tracking IMAP bandwidth use

LoadFactor
I have a user that blew our their bandwidth allocation overnight. Over 3GB in the span of a few minutes. There's one IMAP account with just 24MB of mail, so a massive new device resync isn't the issue. I've gone through the mail delivery reports and there's two 5MB messages and a normal level of sapm that didn't get past greylisting. Grepped /var/log/maillog matching the domain for anything unusual, nothing obvious came up. LFD reports nothing in terms of some brute force attempt. I'm stumped. Where else should I be looking?

Comments

12 comments

Date Votes
  • LoadFactor
    I also just looked at munin... there's no corresponding spike in network traffic, no unusual change in system load. Nothing. Meanwhile the cPanel bandwidth report shows two narrow spikes, one over 300 MB/min.
    0
  • cPanelLauren
    Hi @LoadFactor Where are you looking for bandwidth usage for this account specifically? Within WHM or cPanel? Depending on where you're seeing the usage the following documentation might be helpful: Apparent Discrepancies in Bandwidth Usage Statistics - Version 78 Documentation - cPanel Documentation View Bandwidth Usage - Version 78 Documentation - cPanel Documentation Bandwidth - Version 78 Documentation - cPanel Documentation
    0
  • LoadFactor
    cPanel (3.23 GB) WHM (3.23 GB) It's not a question of the statistic, it's tracking down what it is in IMAP that used it. cPanel is reporting these spikes in traffic but munin's network traffic is normal. With no spikes in system load or reports from LFD, it suggests that it wasn't a DOS attack. And there's nothing even close to 3GB in the mail delivery logs. Total mail volume on that account looks like it's under 25 MB... I would have been getting mail queue size alerts anyway. 57897 57901
    0
  • cPanelLauren
    Hello, The following threads go over how to identify the usage: Specifically what to look for in /var/log/maillog
    0
  • LoadFactor
    I did mention I had grepped maillog. This is why I'm stumped. Bytes in: 37,862, bytes out: 1,074,256 Mar 31 03:56:11 dovecot: imap()<28277>: Logged out in=572, out=1623, bytes=572/1623 Mar 31 03:56:11 dovecot: imap()<28278>: Logged out in=417, out=1440, bytes=417/1440 Mar 31 05:03:38 dovecot: imap()<3649>: Logged out in=417, out=1443, bytes=417/1443 Mar 31 05:03:38 dovecot: imap()<3650>: Logged out in=572, out=1626, bytes=572/1626 Mar 31 05:59:25 dovecot: imap()<6376>: Logged out in=417, out=1440, bytes=417/1440 Mar 31 05:59:25 dovecot: imap()<6375>: Logged out in=572, out=1631, bytes=572/1631 Mar 31 06:59:36 dovecot: imap()<12317>: Logged out in=572, out=1631, bytes=572/1631 Mar 31 06:59:36 dovecot: imap()<12315><5u4VzGGFOcFBXE5Y>: Logged out in=417, out=1440, bytes=417/1440 Mar 31 07:58:33 dovecot: imap()<18411>: Logged out in=551, out=1555, bytes=551/1555 Mar 31 07:58:33 dovecot: imap()<18414>: Logged out in=135, out=950, bytes=135/950 Mar 31 07:58:38 dovecot: imap()<18487>: Logged out in=417, out=1432, bytes=417/1432 Mar 31 08:31:06 dovecot: imap()<8446>: Logged out in=587, out=2212, bytes=587/2212 Mar 31 08:31:06 dovecot: imap()<8457>: Logged out in=262, out=1149, bytes=262/1149 Mar 31 08:31:07 dovecot: imap()<8463>: Logged out in=307, out=1241, bytes=307/1241 Mar 31 08:31:08 dovecot: imap()<8473><+5EKFGOFRMFBXE5Y>: Logged out in=308, out=1225, bytes=308/1225 Mar 31 09:14:55 dovecot: imap()<3764>: Logged out in=183, out=6707, bytes=183/6707 Mar 31 09:14:56 dovecot: imap()<3765><36CqsGOFXcFBXE5Y>: Logged out in=277, out=2019, bytes=277/2019 Mar 31 09:15:00 dovecot: imap()<3799>: Logged out in=114, out=582, bytes=114/582 Mar 31 09:20:04 dovecot: imap()<3757>: Logged out in=872, out=66405, bytes=872/66405 Mar 31 09:20:04 dovecot: imap()<3747>: Logged out in=508, out=2265, bytes=508/2265 Mar 31 17:36:33 dovecot: imap()<19814>: Logged out in=535, out=2043, bytes=535/2043 Mar 31 17:36:33 dovecot: imap()<19819><1i6ksmqFtsFBXE5Y>: Logged out in=307, out=1241, bytes=307/1241 Mar 31 17:36:34 dovecot: imap()<19831>: Logged out in=267, out=1147, bytes=267/1147 Mar 31 17:36:37 dovecot: imap()<19873>: Logged out in=114, out=582, bytes=114/582 Mar 31 17:46:03 dovecot: imap()<27101>: Logged out in=114, out=582, bytes=114/582 Mar 31 17:47:12 dovecot: imap()<27030>: Logged out in=598, out=2486, bytes=598/2486 Mar 31 17:47:12 dovecot: imap()<27032>: Logged out in=536, out=1607, bytes=536/1607 Mar 31 17:47:12 dovecot: imap()<27034>: Logged out in=380, out=2940, bytes=380/2940 Mar 31 19:50:59 dovecot: imap()<11919>: Logged out in=271, out=7407, bytes=271/7407 Mar 31 19:51:00 dovecot: imap()<11936>: Logged out in=409, out=1431, bytes=409/1431 Mar 31 19:51:00 dovecot: imap()<11938>: Logged out in=180, out=985, bytes=180/985 Mar 31 19:51:01 dovecot: imap()<11943>: Logged out in=308, out=1217, bytes=308/1217 Mar 31 19:51:01 dovecot: imap()<11942>: Logged out in=178, out=993, bytes=178/993 Mar 31 19:51:04 dovecot: imap()<11982>: Logged out in=114, out=582, bytes=114/582 Apr 1 07:42:34 dovecot: imap()<28346>: Logged out in=271, out=6257, bytes=271/6257 Apr 1 07:42:36 dovecot: imap()<28378>: Logged out in=582, out=2135, bytes=582/2135 Apr 1 07:42:36 dovecot: imap()<28452><73ZWhHaFIMJBXE5Y>: Logged out in=262, out=1149, bytes=262/1149 Apr 1 07:42:39 dovecot: imap()<28487>: Logged out in=114, out=582, bytes=114/582 Apr 1 07:42:39 dovecot: imap()<28467>: Logged out in=307, out=1241, bytes=307/1241 Apr 1 07:42:40 dovecot: imap()<28503>: Logged out in=308, out=1225, bytes=308/1225 Apr 1 07:46:42 dovecot: imap()<28347>: Logged out in=1264, out=26999, bytes=1264/26999 Apr 1 07:46:42 dovecot: imap()<28355>: Logged out in=874, out=3412, bytes=874/3412 Apr 1 18:57:51 dovecot: imap()<533>: Logged out in=243, out=6804, bytes=243/6804 Apr 1 18:57:51 dovecot: imap()<536>: Logged out in=171, out=31495, bytes=171/31495 Apr 1 18:57:52 dovecot: imap()<539>: Logged out in=173, out=19850, bytes=173/19850 Apr 1 18:57:52 dovecot: imap()<544>: Logged out in=183, out=6025, bytes=183/6025 Apr 1 18:57:52 dovecot: imap()<543>: Logged out in=304, out=1223, bytes=304/1223 Apr 1 18:57:53 dovecot: imap()<556>: Logged out in=262, out=1141, bytes=262/1141 Apr 1 18:57:53 dovecot: imap()<570><9i1S83+FWsJBXE5Y>: Logged out in=307, out=1241, bytes=307/1241 Apr 1 18:57:53 dovecot: imap()<574><1ttY83+FW8JBXE5Y>: Logged out in=308, out=1225, bytes=308/1225 Apr 1 18:57:56 dovecot: imap()<598>: Logged out in=114, out=582, bytes=114/582 Apr 1 18:59:20 dovecot: imap()<532><4/Iy83+FUcJBXE5Y>: Logged out in=2267, out=413637, bytes=2267/413637 Apr 1 18:59:20 dovecot: imap()<537>: Logged out in=292, out=2081, bytes=292/2081 Apr 1 18:59:20 dovecot: imap()<540>: Logged out in=417, out=1460, bytes=417/1460 Apr 1 19:59:20 dovecot: imap()<11500>: Logged out in=114, out=582, bytes=114/582 Apr 1 20:00:40 dovecot: imap()<11434><8J/FzoCFccJBXE5Y>: Logged out in=1018, out=3200, bytes=1018/3200 Apr 1 20:00:40 dovecot: imap()<11433>: Logged out in=956, out=4071, bytes=956/4071 Apr 2 15:09:26 dovecot: imap()<12672>: Logged out in=273, out=8594, bytes=273/8594 Apr 2 15:09:27 dovecot: imap()<12683>: Logged out in=183, out=8527, bytes=183/8527 Apr 2 15:09:27 dovecot: imap()<12678><0Xwv4JCFmsJBXE5Y>: Logged out in=717, out=6323, bytes=717/6323 Apr 2 15:09:28 dovecot: imap()<12687>: Logged out in=262, out=1149, bytes=262/1149 Apr 2 15:09:28 dovecot: imap()<12690><++FI4JCFnsJBXE5Y>: Logged out in=307, out=1241, bytes=307/1241 Apr 2 15:09:28 dovecot: imap()<12691><+kBO4JCFn8JBXE5Y>: Logged out in=308, out=1225, bytes=308/1225 Apr 2 15:09:31 dovecot: imap()<12734>

    : Logged out in=114, out=582, bytes=114/582 Apr 2 15:10:42 dovecot: imap()<12675>: Logged out in=1470, out=138647, bytes=1470/138647 Apr 2 15:10:42 dovecot: imap()<12684>: Logged out in=417, out=1443, bytes=417/1443 Apr 2 17:28:26 dovecot: imap()<17270>: Logged out in=271, out=9034, bytes=271/9034 Apr 2 17:28:27 dovecot: imap()<17286><4WhT0ZKFssJBXE5Y>: Logged out in=372, out=1356, bytes=372/1356 Apr 2 17:28:27 dovecot: imap()<17295>: Logged out in=262, out=1141, bytes=262/1141 Apr 2 17:28:28 dovecot: imap()<17301>: Logged out in=307, out=1233, bytes=307/1233 Apr 2 17:28:28 dovecot: imap()<17304>: Logged out in=267, out=1139, bytes=267/1139 Apr 2 17:28:31 dovecot: imap()<17361>: Logged out in=114, out=582, bytes=114/582 Apr 2 19:11:51 dovecot: imap()<18274>: Logged out in=114, out=582, bytes=114/582 Apr 2 19:14:48 dovecot: imap()<18243>: Logged out in=1071, out=2656, bytes=1071/2656 Apr 2 19:14:48 dovecot: imap()<18242>: Logged out in=927, out=4013, bytes=927/4013 Apr 2 19:32:50 dovecot: imap()<7128>: Logged out in=330, out=1269, bytes=330/1269 Apr 2 19:32:50 dovecot: imap()<7132>: Logged out in=180, out=993, bytes=180/993 Apr 2 19:32:51 dovecot: imap()<7144>: Logged out in=225, out=1085, bytes=225/1085 Apr 2 19:32:51 dovecot: imap()<7149><9Rg6jpSF78JBXE5Y>: Logged out in=226, out=1069, bytes=226/1069 Apr 2 19:32:51 dovecot: imap()<7148>: Logged out in=179, out=1007, bytes=179/1007 Apr 2 19:32:51 dovecot: imap()<7134><5CwvjpSF7MJBXE5Y>: Logged out in=242, out=1477, bytes=242/1477 Apr 2 19:34:03 dovecot: imap()<7114>: Logged out in=595, out=2220, bytes=595/2220 Apr 2 19:34:03 dovecot: imap()<7120>: Logged out in=417, out=1443, bytes=417/1443 Apr 2 19:34:03 dovecot: imap()<7145><9g80jpSF7sJBXE5Y>: Logged out in=178, out=993, bytes=178/993 Apr 3 07:56:19 dovecot: imap()<4007>: Logged out in=114, out=582, bytes=114/582 Apr 3 07:56:20 dovecot: imap()<4016>: Logged out in=171, out=39835, bytes=171/39835 Apr 3 07:56:21 dovecot: imap()<4031>: Logged out in=414, out=1436, bytes=414/1436 Apr 3 07:56:21 dovecot: imap()<4039>: Logged out in=262, out=1149, bytes=262/1149 Apr 3 07:56:21 dovecot: imap()<4014><8IQe8Z6FKsNBXE5Y>: Logged out in=564, out=3301, bytes=564/3301 Apr 3 07:56:22 dovecot: imap()<4046>: Logged out in=308, out=1225, bytes=308/1225 Apr 3 07:57:29 dovecot: imap(

    0
  • cPanelLauren
    I did mention I had grepped maillog

    You did but you didn't include what you looked for specifically. Feel free to open a ticket so that our analysts can track down the source of the discrepancy and possibly shed some light on the issue for you. You can open the ticket using the link in my signature and once open please update here with the ticket ID so we can update this thread with the outcome. Thanks!
    0
  • LoadFactor
    Done. The Support Request ID is: 11848183
    0
  • cPanelLauren
    Hi @LoadFactor Great, I'm watching that ticket and will update here with the findings as soon as available. Thanks!
    0
  • LoadFactor
    With the caveat that I still need to work with the user to figure out why this is happening, it seems that the secret is in the grep! My initial grep was on the user's email and the volume of data returned made it hard to find the issue. The linked grep only reported a subset of bandwidth consuming commands. In particular just one UID SEARCH seems to have used a gigabyte! A better command sees to be: cat /var/log/maillog | grep "account@domain.com" | grep "bytes=" I've also posted this on the thread.
    0
  • LoadFactor
    I have finally identified what happened here. The end user's Android phone updated and for some reason the mail client on their phone started making multiple UID SEARCH requests to IMAP. For some reason, dovecot saw fit to return anything from the usual <2 KB though to responses over 1 GB, often in the order of 40 MB. This on a mailbox with only 21 MB of mail. The problem continued overnight, so IMAP traffic ran up to about 40GB total. I do not want to see that end user's phone bill! We took a scorched earth approach to this: wiped out everything in the mail account, deleted the mail app, wiped data, and reinstalled. Hopefully this puts an end to it.
    0
  • cPanelLauren
    Hi @LoadFactor Wow, that's a pretty interesting case though, I'm glad you found the cause of the issue and I believe it will be good to have this as a reference in the event someone else runs into this issue as well.
    0
  • Metro2
    After noticing that I have a customer who in recent months has been averaging over 1GB per day in just IMAP bandwidth despite the fact that he's really only doing about 50MB a day of data transfer, I came here to the trusty forum just now to see if anyone else ever encountered such a thing and found this thread right away. (Thank you to the OP and others who responded!) So using grep commands and sifting through his /var/log/maillog entries, I discovered at least a clue to the source of the problem and I'll be following up with the customer today to see what we can determine what's really happening, but so far here's what I discovered and might be helpful to others who end up finding this thread as well... When he logs in to his mail from his normal devices through his ISP connection, from what I can tell, *looks like* his IMAP mail transactions are normal / expected sizes like this - "Logged out in=1462, out=19339, bytes=1462/19339" (so basically typical 5kb to 50kb emails) But then I noticed a pattern which *looks like* may be coming from an Outlook.com or Exchange IP, since there are also steady connections coming from a prod.outlook.com IP address IP address - every 30 minutes an IMAP connection to his account occurs, and shows a 54MB transaction, almost all like this - "Logged out in=2336, out=54564254, bytes=2336/54564254" One thing I can see for sure is this - every 30 minutes an IMAP connection occurs with a 54MB outbound transaction (yet there are NO signs of anything even close to that large in Mailscanner > MailControl search of his email address - largest one showing in there during the past 3 days is only 5MB). Now, I'm a bit sleep-deprived at the moment so I'm not 100% sure if it's actually the outlook.com login connection or his regular ISP login connections that are generating the out = 54MB transactions, but now at least I have a starting point based on finding this thread (and other linked within) and more closely inspecting the maillog. When I do finally figure out which connection is causing it and manage to resolve the issue, I'll come back here and post an update, hopefully more helpful than the info above so far. Big thanks to everyone here on the forums who has posted about this! First time I've ever encountered this "IMAP using tons of bandwidth for no obvious reason" issue.
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post