backup large folder

sahafy

• March 17, 2022 02:49

Hi, I want to backup large folder (+20GB) containing images between 10K and 100K, without need to compress, just backup to one file. I use this command: tar cf upload.tar upload
where upload is the folder name the problem is that this takes +6 hours to backup the folder, and sometimes it goes very slowly that the backup file increases only 100MB in 1hour. n.b. no folder contains more than 1000 files any solution to backup the folder fast? thanks

• 

  cPRex Jurassic Moderator

  • March 17, 2022 13:47

  Hey there! Do you happen to know how many files are in the folder? If there are a very large number of small files, it can take a considerable amount of time to back up. We often see this with large email folders where many hundreds of thousands of files are being backed up that are all very small. Six hours does still seem like a lot of time. Is there any load on the server during the time you try and perform this work?

  0
• 

  sahafy

  • March 17, 2022 21:13

  No folder has more than 1K images, each image is less than 100kb, so there are 100Ks of files. No, there is no high load on the server during the backup process, tar appear on top
  but all services are stable. is tar cf upload.tar upload
  better than zip -r -1 upload.zip upload/
  ?

  0
• 

  cPRex Jurassic Moderator

  • March 18, 2022 13:24

  I don't really believe in "better" - just "different" Each of those will work just fine and create a bundle of files. With hundreds of thousands of files I could see it taking several hours to complete.

  0
• 

  hmaddy

  • March 20, 2022 17:57

  Check you io rates.

  0

Please sign in to leave a comment.