Could not create new files inside "/", COMET is to blame ?

nyjimbo

• September 08, 2013 15:39

We just has a situation on one of our servers were everything seemed to be fine but we could no longer create NEW files on the "/" mount point. Disk space used was only about 40% but nothing NEW could be written. Lots of stuff in WHM were failing but the server seemed fine, SMART was ok, nothing weird in logs. After banging my head against the wall I found that "/root/.cpanel/comet/channels" had almost a gigabyte of files in there that appear to be email related. After deleting some of these files suddenly we could write new files again.. What is "comet" and why is it doing this ? Was this about using up all file handles and therefore not being able to write new files ? Can we just delete all of these files or will it break something ? What can we do to avoid this in the future ? Thank you.

• 

  quietFinn

  • September 08, 2013 21:19

  I found a thread regarding that:

  0
• 

  nyjimbo

  • September 08, 2013 21:35

  Thank you. It is inodes, after deleting a lot of stuff it shows: root@xxxxx[~]# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda6 262144 226897 35247 87% / tmpfs 487769 1 487768 1% /dev/shm /dev/sdb1 18317312 1725052 16592260 10% /home2 /dev/sda1 52416 46 52370 1% /boot /dev/sdc1 30531584 363 30531221 1% /backup /dev/sda8 5218304 47717 5170587 1% /home /dev/sda7 131072 194 130878 1% /tmp /dev/sda2 1638400 219199 1419201 14% /usr /dev/sda3 1638400 34844 1603556 3% /var root@xxxxx [~]#
  I am going to delete more so we don't see this again for a while.

  0
• 

  cPanelKenneth

  • September 09, 2013 15:02

  Comet ([url=http://en.wikipedia.org/wiki/Comet_%28programming%29]Comet (programming) - Wikipedia, the free encyclopedia) is a way of streaming information from the web server (in this case the cPanel & WHM daemon cpsrvd) and the browser. In cPanel & WHM its primary use is with the Email Queue Manager, and Email Delivery Reports. We have an internal case (59137) regarding the problem you describe. I'll speak to some of our developers about prioritizing a fix for this. In the meantime I recommend having something monitor /root/.cpanel/comet, clearing out files once it reaches a certain threshold.

  0
• 

  R2B2

  • February 01, 2014 02:43

  Thanks. Follow-up question -- I'm on a server that's used over 1.7M Inodes inside of /root/.comet. :( Running /usr/local/cpanel/bin/purge_dead_comet_files only deleted about 140K files. Is it ok to delete everything inside of /root/.cpanel/comet/channels, and subscriptions?

  0
• 

  cPanelMichael

  • February 04, 2014 17:37

  [quote="R2B2, post: 1561491">Thanks. Follow-up question -- I'm on a server that's used over 1.7M Inodes inside of /root/.comet. :( Running /usr/local/cpanel/bin/purge_dead_comet_files only deleted about 140K files. Is it ok to delete everything inside of /root/.cpanel/comet/channels, and subscriptions?
  There is an additional case open (#88721) to further improve the handling of comet datastore files. You can monitor the cPanel 11.42 change log for this case number: cPanel 11.42 - Change Log Manually deleting the files is not recommended unless you are at or close to 100% inode usage and need the additional space. Thank you.

  0

Please sign in to leave a comment.