For when you see the $10 per month dropbox charge on the credit card and think… “I can do better”.

If you're like me, you’ve lost data on a broken hard drive. Over time we learn and we start backing up our computers to the cloud. Moreover, with great options like Dropbox and Google Drive it’s easy, but after the free space is used up, do we have to settle for the monthly fees? Or is there a plan B?

The Money

To date, Dropbox charges $10/month for 1TB of space.

Amazon S3 charges $0.023 per GB per month.

Therefore, if you’re going to store more than ~434 GB the more economical option is Dropbox, anything less and you would save money with S3.

434GB*0.023 = $9.98

The Logistics

After deciding to move my data, the next realization was how much I had collected over the years. It wasn’t anything near the 450GB breakeven point but still, 77GB was more than I wanted to download to my local computer only to have to upload and delete again.

The answer was to launch a temporary EC2 instance dedicated to the data transfer. The instance will sync to my Dropbox, downloading all the data, and then push that data to an s3 bucket for long-term storage.

DropBox Account

To estimate how much data you need to move, check to user dropdown on your DropBox account page.

EC2 Instance

The instance was an Ubuntu Server 18.04 LTS on an m5.large machine. I personally had 77GB of data so I opted to add 100 GiB of Instance storage (1GB is 0.931323 GiB)*.

It’s important that this instance is assigned a role that has read/write privileges to the S3 bucket where you will store the data. The quick and dirty method would be to give the role AmazonS3FullAccess. However, the more secure and preferred way is to modify the Resource field within the JSON file to contain only the target bucket.

S3 Bucket

The S3 bucket can be existing or new, whichever makes the most sense for your situation. That being said it is recommended to keep it in the same region as the EC2 instance to reduce latency.

The Setup

After launching all the resources we will need to ssh into the EC2 instance.

ssh -i *YOUR-PEM-FILE-LOCATION* ubuntu@**YOUR-EC2-ADDRESS*

The first step is to modify the inotify.max_users. These entities are used to monitor individual files or directories within the Linux OS. However, in our case, we are going to move so many files that the default number of max users will cause a system error and halt the DropBox sync.

To avoid that, increase the setting to at least 100,000 and reboot the instance.

echo fs.inotify.max_user_watches=100000 | sudo tee -a /etc/sysctl.conf; sudo sy$ sudo reboot

The reboot will terminate the ssh session, so you will need to connect again.

Syncing DropBox

Now that the instance is ready we can sync the Dropbox account to it. Download/extract the official Linux64 tar file and run the preconfigured installation script (dropboxd).

cd ~ && wget -O - " https://www.dropbox.com/download?plat=lnx.x86_64 " | tar xzf - ~/.dropbox-dist/dropboxd

When you run the dropboxd script it will output a link to the DropBox webpage asking you to verify that you are a real person and that you would like to sync the account. Copy-paste this link into a browser and confirm.

After you confirm the account the dropboxd script will need to continue running until the instance is terminated. This means that you will need to ssh into the instance again from another terminal to complete the rest of the tasks.

The alternative is to write the output of the dropboxd script to a text file.

~/.dropbox-dist/dropboxd >> DropBoxSync_output.txt 2>&1

In this case, you will need to read the end of the tail DropBoxSync_output.txt file to obtain the webpage link.

tail ~/DropBoxSync_output.txt

The instance is now synced to DropBox and will start downloading all your files automatically.

Syncing to S3

After our entire DropBox drive has been downloaded to the instance the final step is to transfer the data from the EC2 instance to the S3 bucket.

To start we must install the AWS command line.

sudo apt -y install awscli

And finally, we push the data to S3.

aws s3 sync ~/Dropbox s3://**YOUR-BUCKET-LOCATION**/Dropbox

Finished

That’s it. Depending on how much data you're moving, the process could take between a few minutes and several days. It’s best to monitor the S3 bucket size (bytes) metrics to see how what percentage of the data has been transferred.