Occasionally, I have found the need to store large files in the cloud (dropbox, one drive, google drive, etc). However, many of those cloud storage services have limits on the maximum size of file.
File size limits make sense because companies don’t want to store your movies. It costs them money for the storage, and they make the most money if you pay a lot and utilize the product a little.
Compression
The first step to storing large files should be compression. If you can compress your files, it only makes them slightly more difficult to access, and it uses less space.
When compressing, the first thing you need to decide is what format. In my experience, and based on a few benchmarks that I’ve read (but can’t remember); zip is the most convenient. 7z is a better compression in all circumstances that I have personally tried.
I purchase many premium stl files for 3d printing, and I have found that zip is almost double the size of my 7z files in some cases. Its never even been a fair fight for lowly zip files. However, 7z isn’t built in to many operating systems.
If you value convenience more than space, pick zip. But if you need it to be compressed as small as possible, go with 7z.
For Mac, I prefer to use 7z from the command line. I install it from homebrew
|
1 2 3 4 5 6 7 |
brew install p7zip #create an archive 7z a file.7z input #extract an archive 7z x file.7z |
Splitting Files
If you’re creating a 7z archive, you can add in a parameter for splitting the file to ensure your archive isn’t too large for your cloud storage solution.
|
1 2 |
#7z -v option supports b k m g (bytes, kilobytes, megabytes, gigabytes) 7z a file.7z input -v100m |
But sometimes you just need a quick and dirty, split a file to be a certain size
in Mac, you can split using the terminal command split and then recombine the files by just doing a simple cat command
|
1 |
split -b 5000m file_in file_out |
the split command uses the -b flag for specifying the size of the splits in bytes. From the man page, you can use m or k at the end. but unfortunately it doesn’t seem to support g. so the above command will split into roughly 5gb files. (roughly because of the difference between 1000 and 1024)
When split finishes, the files output will by default end in aa ab ac etc. so when you need to recombine them you just need a simple cat command
|
1 |
cat base_name?? > output |
the ?? will do the matching. you can even test it out to make sure your files are all in order by just running something like echo base_name?? and checking the output