Possible memory leak in cpanel backup service
I've got two servers. Both almalinux 8 cpanel, running 114 latest. 1 has 6 core, 8gb ram and 160gb ssd. Other has 8 cores, 16gb ram and 160gb ssd.
About 2 weeks ago, the lower spec'd one started failing backups, and when the backup failed so did many other services, upon inspection RAM and swap were both at 100% utilization. Rebooted server and everything came back online. It would do one or 2 more backups in the coming days then the same thing would happen. I thought perhaps it was just a misconfiguration and I haven't had time to debug, so I just rebooted every once in awhile to keep everything running.
Now, my other server is doing the same thing. Backup starts... Then queueprocd fails first, followed by sshd, mysqld and many others.
RAM and swap 100% utilized.
There aren't many common elements between the two servers other than cpanel. One runs a dotnet webapp, the other runs a few php webapps.
Need help troubleshooting this please.
Thanks!
-
Hey there! I'm guessing because it is a backup process that it is running on a non-convenient time for you to watch the server. There's a few options to make this easier: -you could adjust the backup cron in WHM >> Configure Cron Jobs so this runs at a normal time, allowing you to watch the server load while the backups happen or -you can manually execute a backup by running "/usr/local/cpanel/bin/backup --force" and then watching what happens on the system. I will say, there aren't any known issues with memory usage and cPanel backups that I am aware of at this time, so it's possible that you're in uncharted territory, but hopefully watching things in real-time while the backups run will shed some light on the issue. 0 -
Okay, I will give that a test as soon as i'm able. Hopefully I am wrong and there's something else going on :) 0 -
I am also running AlmaLinux (v8.9 with 12GB RAM) and WHM Backup definitely uses 100% RAM. It's reproducible every day from when it starts at 3:00AM to around 3:18AM. For me that does not result in any service failures, and I know my backups are fairly large (16GB), but still 100% seems excessive?
It also takes 14 hours for RAM usage to drop back down to its baseline of around 65%.
Unless this isn't considered abnormal (?), is there any setting to limit how much RAM is used during the WHM Backup process?
02:50:01 AM kbmemfree kbavail kbmemused %memused kbbuffers kbcached kbcommit %commit kbactive kbinact kbdirty
03:00:01 AM 4.2G 5.3G 7.8G 65.4% 0.0k 985.1M 0.0k 0.0% 7.1G 480.2M 380.0k
03:10:01 AM 296.4M 5.2G 11.7G 97.6% 0.0k 4.7G 0.0k 0.0% 10.7G 713.4M 131.7M
03:20:02 AM 36.0k 5.2G 12.0G 100.0% 0.0k 5.0G 0.0k 0.0% 11.1G 644.4M 140.0k
03:30:02 AM 124.0k 5.2G 12.0G 100.0% 0.0k 5.0G 0.0k 0.0% 11.1G 645.8M 276.0k
03:40:01 AM 126.0M 5.3G 11.9G 99.0% 0.0k 4.9G 0.0k 0.0% 10.9G 724.2M 384.0k
03:50:00 AM 76.9M 5.3G 11.9G 99.4% 0.0k 5.0G 0.0k 0.0% 10.9G 787.3M 160.0k
04:00:05 AM 30.8M 5.2G 12.0G 99.7% 0.0k 5.0G 0.0k 0.0% 10.9G 779.5M 672.0k
04:10:01 AM 151.8M 5.3G 11.9G 98.8% 0.0k 4.9G 0.0k 0.0% 10.9G 721.3M 384.0k
04:20:00 AM 253.8M 5.3G 11.8G 97.9% 0.0k 4.8G 0.0k 0.0% 10.8G 709.8M 164.0k
04:30:00 AM 486.1M 5.3G 11.5G 96.0% 0.0k 4.6G 0.0k 0.0% 10.5G 716.5M 292.0k
04:40:01 AM 621.9M 5.3G 11.4G 94.9% 0.0k 4.5G 0.0k 0.0% 10.5G 655.4M 300.0k
04:50:01 AM 621.9M 5.3G 11.4G 94.9% 0.0k 4.5G 0.0k 0.0% 10.5G 659.0M 264.0k
05:00:01 AM 814.6M 5.3G 11.2G 93.4% 0.0k 4.3G 0.0k 0.0% 10.3G 629.0M 248.0k
05:10:01 AM 844.5M 5.2G 11.2G 93.1% 0.0k 4.2G 0.0k 0.0% 10.3G 623.4M 500.0k
05:20:01 AM 989.4M 5.3G 11.0G 91.9% 0.0k 4.1G 0.0k 0.0% 10.2G 604.1M 376.0k
05:30:02 AM 1.1G 5.3G 10.9G 91.2% 0.0k 4.0G 0.0k 0.0% 10.1G 591.4M 700.0k
05:40:01 AM 1.1G 5.3G 10.9G 91.1% 0.0k 4.0G 0.0k 0.0% 10.1G 591.4M 740.0k
05:50:01 AM 948.9M 5.3G 11.1G 92.3% 0.0k 4.1G 0.0k 0.0% 10.1G 717.9M 408.0k
06:00:01 AM 995.1M 5.2G 11.0G 91.9% 0.0k 4.1G 0.0k 0.0% 10.1G 662.8M 24.0k
06:10:02 AM 1.1G 5.2G 10.9G 90.6% 0.0k 3.9G 0.0k 0.0% 9.9G 672.2M 1.1M
06:20:01 AM 1.2G 5.3G 10.8G 90.0% 0.0k 3.9G 0.0k 0.0% 9.9G 633.6M 396.0k
06:30:01 AM 1.2G 5.3G 10.8G 90.1% 0.0k 3.9G 0.0k 0.0% 9.9G 643.6M 704.0k
06:40:01 AM 1.2G 5.3G 10.8G 90.3% 0.0k 3.9G 0.0k 0.0% 9.9G 659.0M 420.0k
06:50:01 AM 1.1G 5.3G 10.9G 90.4% 0.0k 3.9G 0.0k 0.0% 9.9G 663.2M 400.0k
07:00:00 AM 1.1G 5.2G 10.9G 90.5% 0.0k 3.9G 0.0k 0.0% 9.9G 674.4M 192.0k
07:10:00 AM 1.2G 5.3G 10.8G 90.4% 0.0k 3.9G 0.0k 0.0% 9.9G 672.1M 288.0k
07:20:01 AM 1.2G 5.3G 10.8G 90.3% 0.0k 3.9G 0.0k 0.0% 9.9G 673.7M 292.0k
07:30:01 AM 1.2G 5.3G 10.8G 90.2% 0.0k 3.9G 0.0k 0.0% 9.9G 669.1M 496.0k
07:40:01 AM 1.2G 5.2G 10.8G 90.2% 0.0k 3.9G 0.0k 0.0% 9.9G 666.5M 236.0k
07:50:01 AM 1.2G 5.2G 10.8G 90.0% 0.0k 3.9G 0.0k 0.0% 9.9G 662.8M 368.0k
08:00:02 AM 1.2G 5.2G 10.8G 89.8% 0.0k 3.8G 0.0k 0.0% 9.9G 658.4M 444.0k
08:10:01 AM 1.2G 5.2G 10.8G 89.6% 0.0k 3.8G 0.0k 0.0% 9.9G 655.6M 632.0k
08:20:01 AM 1.3G 5.3G 10.7G 88.8% 0.0k 3.8G 0.0k 0.0% 9.8G 643.6M 372.0k
08:30:02 AM 1.4G 5.3G 10.6G 88.7% 0.0k 3.7G 0.0k 0.0% 9.8G 638.0M 580.0k
08:40:01 AM 1.3G 5.2G 10.7G 88.9% 0.0k 3.7G 0.0k 0.0% 9.8G 641.5M 352.0k
08:50:01 AM 1.4G 5.2G 10.6G 88.7% 0.0k 3.7G 0.0k 0.0% 9.8G 635.2M 608.0k
09:00:01 AM 1.4G 5.2G 10.6G 88.1% 0.0k 3.6G 0.0k 0.0% 9.7G 624.4M 196.0k
09:10:01 AM 1.5G 5.2G 10.5G 87.8% 0.0k 3.6G 0.0k 0.0% 9.7G 614.6M 4.0k
09:20:01 AM 1.5G 5.2G 10.5G 87.6% 0.0k 3.6G 0.0k 0.0% 9.7G 608.9M 336.0k
09:30:01 AM 1.5G 5.2G 10.5G 87.3% 0.0k 3.5G 0.0k 0.0% 9.6G 605.6M 332.0k
09:40:01 AM 1.6G 5.2G 10.4G 87.1% 0.0k 3.5G 0.0k 0.0% 9.6G 558.0M 200.0k
09:50:00 AM 1.5G 5.2G 10.5G 87.5% 0.0k 3.5G 0.0k 0.0% 9.7G 562.6M 180.0k
10:00:00 AM 1.4G 5.1G 10.6G 88.1% 0.0k 3.5G 0.0k 0.0% 9.8G 565.8M 308.0k
10:10:01 AM 1.5G 5.2G 10.5G 87.3% 0.0k 3.5G 0.0k 0.0% 9.7G 569.5M 616.0k
10:20:01 AM 1.4G 5.1G 10.6G 88.0% 0.0k 3.5G 0.0k 0.0% 9.7G 572.5M 296.0k
10:30:01 AM 1.4G 5.1G 10.6G 88.2% 0.0k 3.5G 0.0k 0.0% 9.7G 594.7M 320.0k
10:40:00 AM 1.4G 5.1G 10.6G 88.1% 0.0k 3.5G 0.0k 0.0% 9.7G 600.0M 224.0k
10:50:01 AM 1.4G 5.1G 10.6G 88.3% 0.0k 3.5G 0.0k 0.0% 9.8G 600.8M 404.0k
11:00:01 AM 1.4G 5.1G 10.6G 88.3% 0.0k 3.5G 0.0k 0.0% 9.7G 599.3M 348.0k
11:10:01 AM 1.4G 5.1G 10.6G 88.2% 0.0k 3.5G 0.0k 0.0% 9.7G 598.2M 324.0k
11:20:01 AM 1.4G 5.1G 10.6G 88.3% 0.0k 3.5G 0.0k 0.0% 9.8G 595.5M 196.0k
11:30:01 AM 1.4G 5.0G 10.6G 88.3% 0.0k 3.5G 0.0k 0.0% 9.8G 593.8M 196.0k
11:40:01 AM 1.4G 5.0G 10.6G 88.2% 0.0k 3.4G 0.0k 0.0% 9.7G 588.7M 496.0k
11:50:01 AM 1.5G 5.1G 10.5G 87.8% 0.0k 3.4G 0.0k 0.0% 9.7G 585.4M 264.0k
12:00:02 PM 1.4G 5.0G 10.6G 88.5% 0.0k 3.5G 0.0k 0.0% 9.7G 655.8M 364.0k
12:10:01 PM 1.6G 5.2G 10.4G 86.7% 0.0k 3.4G 0.0k 0.0% 9.6G 580.2M 356.0k
12:20:01 PM 1.6G 5.1G 10.4G 86.9% 0.0k 3.4G 0.0k 0.0% 9.6G 603.2M 296.0k
12:30:02 PM 1.6G 5.1G 10.4G 86.6% 0.0k 3.3G 0.0k 0.0% 9.6G 579.2M 464.0k
12:40:00 PM 1.6G 5.1G 10.4G 86.5% 0.0k 3.3G 0.0k 0.0% 9.6G 565.4M 160.0k
12:50:00 PM 1.7G 5.1G 10.3G 85.8% 0.0k 3.2G 0.0k 0.0% 9.5G 549.9M 256.0k
01:00:01 PM 1.8G 5.0G 10.2G 85.0% 0.0k 3.1G 0.0k 0.0% 9.4G 528.8M 300.0k
01:10:01 PM 1.9G 5.1G 10.1G 84.3% 0.0k 3.0G 0.0k 0.0% 9.4G 519.5M 332.0k
01:20:01 PM 1.9G 5.0G 10.1G 84.4% 0.0k 3.0G 0.0k 0.0% 9.4G 516.3M 424.0k
01:30:01 PM 1.9G 5.1G 10.1G 84.1% 0.0k 3.0G 0.0k 0.0% 9.3G 520.9M 300.0k
01:40:01 PM 1.9G 5.0G 10.1G 83.9% 0.0k 3.0G 0.0k 0.0% 9.3G 524.3M 288.0k
01:50:01 PM 1.9G 5.0G 10.1G 84.2% 0.0k 3.0G 0.0k 0.0% 9.3G 530.2M 892.0k
02:00:01 PM 1.9G 5.0G 10.1G 84.0% 0.0k 3.0G 0.0k 0.0% 9.3G 531.6M 164.0k
02:10:01 PM 2.0G 5.1G 10.0G 83.2% 0.0k 3.0G 0.0k 0.0% 9.2G 540.8M 308.0k
02:20:01 PM 3.3G 5.1G 8.7G 72.8% 0.0k 1.7G 0.0k 0.0% 8.0G 544.6M 280.0k
02:30:01 PM 3.3G 5.1G 8.7G 72.2% 0.0k 1.6G 0.0k 0.0% 8.0G 503.8M 656.0k
02:40:01 PM 3.7G 5.1G 8.3G 69.4% 0.0k 1.3G 0.0k 0.0% 7.7G 434.6M 436.0k
02:50:01 PM 3.7G 5.1G 8.3G 69.4% 0.0k 1.3G 0.0k 0.0% 7.7G 415.3M 312.0k
03:00:00 PM 3.7G 5.1G 8.3G 69.3% 0.0k 1.3G 0.0k 0.0% 7.7G 399.8M 236.0k
03:10:01 PM 3.7G 5.1G 8.3G 69.4% 0.0k 1.3G 0.0k 0.0% 7.7G 388.6M 300.0k
03:20:01 PM 3.7G 5.0G 8.3G 69.5% 0.0k 1.3G 0.0k 0.0% 7.7G 381.3M 380.0k
03:30:00 PM 3.7G 5.1G 8.3G 69.3% 0.0k 1.3G 0.0k 0.0% 7.7G 381.1M 152.0k
03:40:01 PM 3.7G 5.1G 8.3G 69.2% 0.0k 1.2G 0.0k 0.0% 7.7G 357.8M 296.0k
03:50:01 PM 3.7G 5.1G 8.3G 69.3% 0.0k 1.2G 0.0k 0.0% 7.7G 365.7M 364.0k
04:00:02 PM 3.7G 5.0G 8.3G 69.3% 0.0k 1.2G 0.0k 0.0% 7.8G 347.4M 256.0k
04:10:01 PM 3.7G 5.0G 8.3G 69.2% 0.0k 1.2G 0.0k 0.0% 7.8G 344.6M 296.0k
04:20:01 PM 3.8G 5.1G 8.2G 68.3% 0.0k 1.2G 0.0k 0.0% 7.7G 324.4M 360.0k
04:30:00 PM 3.9G 5.2G 8.1G 67.9% 0.0k 1.2G 0.0k 0.0% 7.6G 316.4M 196.0k
04:40:01 PM 3.9G 5.2G 8.1G 67.7% 0.0k 1.2G 0.0k 0.0% 7.6G 303.5M 304.0k
04:50:01 PM 3.9G 5.2G 8.1G 67.7% 0.0k 1.2G 0.0k 0.0% 7.6G 303.0M 364.0k
05:00:02 PM 3.9G 5.2G 8.1G 67.6% 0.0k 1.2G 0.0k 0.0% 7.6G 300.0M 612.0k
05:10:01 PM 4.3G 5.2G 7.7G 63.9% 0.0k 735.5M 0.0k 0.0% 7.4G 78.9M 344.0k0 -
Kenric Ashe - there isn't a way to limit the RAM being used, but we'd like to take a look at this if you could create a ticket. We can have a technician monitor the server at 3AM to watch the problem in real-time.
0
Please sign in to leave a comment.
Comments
4 comments