Replica corruption

Hi All, We have multiple server set up: Centos 7 Cpanel running on vsphere ESXi 6.5 (one C6 cloudlinux) Multiple accounts/shared hosting on each Local mysql/email on each VMTools installed on all Aswell as R1soft backups, we run replication with Nakivo, which uses VMWare's storage CDT to back up incremental snapshots. Everything works nicely, Nakivo does what it should do and replicates to another server for failover, however we have a problem with Cpanel/Linux corrupting on the replicas. Even with quiescing on and tested for a month with a pre/post thaw (ffreeze / ffthaw) script running, still we get file corruption. fsck takes ages to run. The corruption seems to be "incorrect file type was 1, should be 2) on alot of files, mail, php etc, also unconnected/unattached directory inode, connect to lost/found yes, bad header/extent, invalid magic etc etc. Once fsck was unable to repair, so we had to start a new replica. This issue has been ongoing since we noticed it last year and we still can't find a way of reliable replicating a live cpanel server. I'm told that pre/post thaw scripts work for most servers. How is everyone else out there achieving this? How are you replicating your servers for a fast recovery in the event of failure? These days we can't rely on nightly cpanel backups which may take hours to restore, even after setting up the whole linux/cpanel install again... noone would tolerate that length of downtime. I'd like to open the discussion for this as I'm sure there are others who have these problems. Appreciate any feedback!

cPanelLauren

May 11, 2018 15:46

This isn't a feature that cPanel natively supports but you might want to check out the following: Active-Active Redundancy or High-availability

uk01

May 11, 2018 17:44

Thanks I"ve seen those, people have wanted that for years and ha is essential in today"s hosting industry. Virtualisation is the only other solution at the moment so I posted here in the hope it may raise a discussion about the ways people achieve that without corruption on a busy server. If not, I"ll post on wht or another forum

May 11, 2018 18:47

Hi @uk01 I agree and I hope it does start some discussion and someone else in the community has seen the issue before. I did take a look through other threads to see if anyone else had posted related items but it doesn't appear that many have posted about it outside of MySQL corruption. Here is one of the threads I found for you:

Replica corruption

Comments

Didn't find what you were looking for?