[CPANEL-31965] Default email account ghost disk usage
Hello! I have this weird problem with one customer, his default email account not in use to catch all is showing almost 5gig disk usage.
-04-06 a la(s) 11.53.18.png">64373
If i look into the account webmail its empty. If i look with file manager its empty. But, if i look the disk usage viewer i noticed a 5gb excess inside /mail and is nothing here. Where i need to look up?
-04-06 a la(s) 12.02.38.png">64377
Thanks!
-
What is reported when you go to /home/$user/mail
and runs something like the following:du -h --max-depth=10 -
Same issue here. It shows my default email account to have 346MB, but when I go in the webmail of the default mail account is empty. Moreover when I look at disk quota it shows the parent mail folder filled with 1.2GB, but the children folders are only 850MB, see screenshot: 0 -
Same response to you @Gino Viroli What is reported when you go to
/home/$user/mail
and runs something like the following:du -h --max-depth=1
0 -
Hello @cPanelLauren here is my result, looks fine from console. [root@alfonsina ~]# cd /home/username/mail/ [root@alfonsina mail]# du -h --max-depth=1 24K ./.Drafts 4.0K ./new 4.0K ./cur 24K ./.Sent 4.0K ./tmp 28K ./.Archive 24K ./.Trash 28K ./.spam 2.4G ./username.com 24K ./.Junk 2.4G . [root@alfonsina mail]#0 -
Looks fine here too: [CODE=bash]root@xx [/home/***.it/mail]# du -h --max-depth=1 20K ./cur 24K ./.Drafts 869M ./***.it 24K ./.mailbox_format.cpanel 24K ./.Junk 24K ./.Sent 4.0K ./tmp 24K ./.spam 24K ./.Archive 4.0K ./new 24K ./.Trash 869M . 0 -
@cPanelLauren it appears to be a glitch in cPanel, find below the result from the mother folder that shows "/mail" folder to be only 869MB and not 1.2GB as shown by cPanel root@xx [/home/***]# du -h --max-depth=1 28K ./.htpasswds 20K ./.subaccounts 600K ./.cpanel 8.0K ./.pki 866M ./mail 3.1M ./.cphorde ... 4.0K ./win_shared 2.9G . root@xx [/home/***]#0 -
Hello! Checking today the problem persist, is not a matter of caching or someting. Adding more info to the matter, the statics on the sidebar show less disk usage in total than the default email account by itself. 0 -
Yea, I'm aware of an issue that presented similarly to this which is why I wanted to confirm the Disk space was not actually being used @Gino Viroli The last case associated with this was CPANEL-26448 and was resolved in v82 of cPanel & WHM if you're running v86 can you please open a ticket and then update this thread with the ticket ID. 0 -
@benito you can open the ticket in "WHM >> Support >> Create Support Ticket" 0 -
Yea, I'm aware of an issue that presented similarly to this which is why I wanted to confirm the Disk space was not actually being used @Gino Viroli The last case associated with this was CPANEL-26448 and was resolved in v82 of cPanel & WHM if you're running v86 can you please open a ticket and then update this thread with the ticket ID.
Done, new ticket id is 934701450 -
Hey guys, I hate to say it but it looks like this issue has sprouted a new case CPANEL-31965 in which users are experiencing this behavior once again in v86 - I must have missed it when it was opened so I wasn't aware of it making a resurgence. @benito thank you for opening the ticket, the case is currently marked as Needs Reproduction and I've reached out to the product owner of that team to see if he needs anything from your server to reproduce this. We'll let you know what they come back with. Right now the only workaround for this is to run the following: /scripts/generate_maildirsize --confirm $USER
But this is only temporary as if you access the default email account the issue will reoccur.0 -
Hello! No problem, its nice to see its a bug and you are working on it. Thanks 0 -
Hello, After some discussion with the Technical Support team here, If anyone here who is experiencing this issue could open a ticket we would very much appreciate it. Our Senior Technical Analysts need a system where the issue is currently occurring in order to replicate the issue as we've been unable to do so internally. Anyone who does open a ticket please update this thread with the ticket ID so I can inform the Senior Tech Analyst. Thanks! 0 -
Support Ticket ID is: 93476363 0 -
Hi @Metro2 I see one of our Senior Tech Analysts has picked this p and is working on attempting to replicate the issue. Thank you for that. If anyone else who is experiencing this issue can please open a ticket it would be helpful in identifying a resolution to this. 0 -
Hi @Metro2 I see one of our Senior Tech Analysts has picked this p and is working on attempting to replicate the issue. Thank you for that. If anyone else who is experiencing this issue can please open a ticket it would be helpful in identifying a resolution to this.
Hi, Has this issue been resolved yet? I'm having the same problem.0 -
This issue is not currently resolved. We are still looking to be able to replicate this and if you're experiencing this issue please open a ticket. Once it's open please provide the Ticket ID in response to this thread so I can notify the Senior Analyst working on this case. 0 -
I have Support ID is: 93504578 Hope you can help me resolve this problem. 0 -
Hello, Thank you for opening a ticket it would appear that your issue was related to uncompressed files. [QUOTE] Presently there are 43227 files that are compressed. Dovecot uses what the space would be when uncompressed and is why the mailbox discrepancy is seen. 0 -
Any update on this issue? I'm having the same problem. 0 -
@cPanelLauren We are having the same problems reported in these topics. Does version on WHM 90.0.5 correct these issues? 0 -
Support Ticket ID is: 93687839 root@mx.domain.net [~]# cd /home/dotsmedia/mail/ root@mx.domain.net [mail]# du -h --max-depth=1 8.0K ./new 28K ./.Archive 88K ./cur 20G ./dotsmedia.agency 24K ./.Drafts 24K ./.spam 4.0K ./tmp 24K ./.Junk 24K ./.Sent 24K ./.Trash 20G . @cPanelLauren 0 -
I checked in on this case and it is not resolved. The workaround being provided is listed in this thread but I've quoted it below: Hey guys, I hate to say it but it looks like this issue has sprouted a new case CPANEL-31965 in which users are experiencing this behavior once again in v86 - I must have missed it when it was opened so I wasn't aware of it making a resurgence. @benito thank you for opening the ticket, the case is currently marked as Needs Reproduction and I've reached out to the product owner of that team to see if he needs anything from your server to reproduce this. We'll let you know what they come back with. Right now the only workaround for this is to run the following:
/scripts/generate_maildirsize --confirm $USER
But this is only temporary as if you access the default email account the issue will reoccur.
Our Senior Technical Analysts do still need reproduction on this and I've pointed them in the direction of your ticket. They should be getting back to you soon!0 -
This issue is still happening on all our servers and is identical to the previous case that was marked as resolved. The default account may be using no space at all but often shows as several GB in size. This not only happens on all our existing servers but also on new servers where accounts have been copied over from older servers. We though that perhaps the transfer process would "reset" this issue but sadly not. I wonder if it is related to compression. If it helps, we're using a compression level of 7 rather than the default 6. To add further comment for discussion, there does seem to be strange output to the user for the mailbox sizes. For example, in the Disk usage tool, the uncompressed totals are displayed. Why? The file size of a compressed file is what it is. If you upload a zip file, it's file size is calculated based on the size on disk -not the size it would be if it was unpacked. So why provide such confusing information to the end user? Makes no sense. Below, the user has a 10GB quota and it shows they are using 16GB in their Email accounts. But they are not using that much space - the files are compressed - so they take up less space. What's the point in showing the file size of what the files would be if they were not compressed? It serves no purpose at all! This is a classic example of developers being developers and not taking into account that the end user doesn't understand, care, or need to know about compression. All you are doing is presenting confusing figures that ultimately result in a poor experience for the end user and more support tickets for us to deal with. 16,499.26 MB * Contained in the mail directory. 18,398.74 MB total disk space used. 10,240.00 MB quota limit (9,160.47 MB used). 0 -
From what I understand it appears that the issue is only affecting accounts added prior to the fix for the issue that was implemented in v82 - meaning that the fix is only resolving the issue on accounts added after the fix rather than resolving it for existing ones. Running: /usr/local/cpanel/scripts/generate_maildirsize --confirm $user
Has been proven to resolve the issue for the accounts in this situation. I'm confirming right now with our senior analysts that this includes accounts that have been transferred. I'll let you know what I find out.0 -
[QUOTE] From what I understand it appears that the issue is only affecting accounts added prior to the fix for the issue that was implemented in v82 - meaning that the fix is only resolving the issue on accounts added after the fix rather than resolving it for existing ones.
So you are saying that the issue of the default account total being incorrect is only happening on accounts added before the v82 fix and this problem no longer happens on new accounts. Meaning that the fix was put in place to stop it happening going forward but it won't resolve any existing accounts with this problem. Therefore, running /usr/local/cpanel/scripts/generate_maildirsize --confirm $user should resolve the incorrect display of totals on the old accounts and the v82 fix should prevent it from recurring? However, many have reported running that script, only to find the totals go wonky again soon after. So what we need of course is a permanent fix for the issue on the pre v82 accounts. Lauren, could you possibly get a comment from your higher ups about the decision to show the uncompressed sizes for display instead of the actual disk usage? It's pretty clear that this is causing a great deal of confusion for customers. There appears to be no benefit to anyone in doing this. I'd like to hear some justification for it if possible.0 -
Right now, you've got it. Up until the last few weeks we actually couldn't replicate the issue, until it was found that it was only occurring on accounts that got added before v82. The case is still open, that's not the official fix it's a workaround until the issue is resolved. I agree that you need a permanent resolution. We're hoping now that we have a reliable method of replication that development will be able to identify a suitable solution. As far as the compression issue, I can open an inquiry with development to find out why it was done this way, I don't disagree that it is confusing 0 -
Hello all, just to add to this, the only way I can get the email usage displayed correctly is to run "/usr/local/cpanel/scripts/generate_maildirsize --confirm $user" as mentioned above, but .cpanel/email_accounts.json" also needs to be removed to show the correct usage. all this has to be done daily to get correct usage displayed. I am currently on v90.0.10 but al accounts have been made on v80 or so 0 -
@4u123 I did some digging on the compression discrepancy and found the following very detailed explanation: [QUOTE] This makes sense for the purpose of disk quotas, as filesystem blocks cannot be shared by multiple files, so all of this space must be accounted to the system user which owns the file, even if the programs associated with the data aren't actually using that leftover space. This is a form of so-called "internal" fragmentation. If blocks could be shared by multiple files, and if these files could be owned by multiple system users, this would change the way the system would calculate disk quotas. On the other hand, Dovecot cannot make assumptions about the underlying storage mechanism. Under Maildir or mbox, Dovecot on Linux could in theory ask the system about the number of filesystem blocks actually in use by particular files; however, the Dovecot software is not specific to Linux, so it does not make sense to make assumptions that its storage will only be allocated in blocks of a particular size. Moreover, this would add extreme amounts of complexity for little benefit. Therefore, it is much easier and makes more sense for it to account for usage in terms of the actual size of messages, regardless of how they are stored. For this reason, it is exceedingly unlikely that the usage reported by the system will exactly match the usage reported by Dovecot. That said, this is not the only source of difference between the two figures at work here. Compression does not affect the size of messages, which never changes, even though it does change the amount of storage Dovecot needs for a given message. Additionally, compression at the point of storage is not a native capability of Dovecot and is instead implemented as a plugin. Changing Dovecot to be able to account for compression when accounting for usage against quotas likely would require additional complexity without sufficient benefit. Both of these figures are correct, because they measure two quantities which are entirely different but related. The use of compression wildly exacerbates the difference, but even without use of compression, differences could be significant due to internal fragmentation. It just so happens that cPanel chooses to use one figure in certain places and the other figure in others. 0 -
@4u123 I did some digging on the compression discrepancy and found the following very detailed explanation:
Wow - who knew it could be so complicated? Sadly that leaves the end user in a very confusing situation. Like I said before - they don't care about the different ways that nerds can calculate usage - they just want to know how much disk space they are using. Surely for the end user, all that's required is to calculate the file usage the same way as everything else. Why can't you simply calculate the usage in the mail folders according to the quotas - why wouldn't that be enough? To say "both totals are correct" is not right. This isn't a science convention, it's a commercial product. Both totals are not correct. There can only be one total! The message sizes are irrelevant because the messages are compressed. The reason you display the totals in cpanel is solely to let the client know how much usage they are using. They don't need to know that dovecot provides a different total based on the actual message size as it would be when not compressed - they just don't need to know this info. In my example above it says... 18,398.74 MB total disk space used. No - that's not true. If that much space had actually been used, it couldn't be allocated for anything else and would therefore be the actual total. But it isn't. 10,240.00 MB quota limit (9,160.47 MB used). 9,160.47 MB is the total disk space used as far as the end user is concerned. That's how much space on disk the client is using. You don't need the other figure at all. Why even display it? As I said before and didn't receive an answer... I would really like to hear some justification for this. It's clearly nuts.0
Please sign in to leave a comment.
Comments
39 comments