Duplicity is a Python utility that backups your stuff, encrypts it and allows you to upload them away using as little bandwidth as possible using the rsync algorithm.

Cost effectiveness and security aren’t things that especially come to mind when thinking about sending your backups to the cloud.

However, as the cost of mass storage and bandwidth goes down, so does the pricing of cloud Object Storage. As far as security is concerned, your backups are encrypted locally by Duplicity before being uploaded. This ensures that no one without your encryption key can read the files.

Duplicity made Openstack Swift a native backend in version 0.6.22 released back a year ago, in august 2013.

In this post, I’ll go over how you can install Duplicity on Ubuntu, back up a directory and send it to an Openstack Swift object storage service.

Installing Duplicity

If you’re using Ubuntu 14.04 (Trusty), Duplicity is provided by the Ubuntu repositories with a recent enough version: 0.6.23.

You can install it directly through apt:

apt-get install duplicity

For older distributions, the version of Duplicity provided by Ubuntu doesn’t yet have the Swift backend in it.

Thankfully, Duplicity has a PPA that provides 0.6.23 packages for Lucid, Precise, Quantal and Saucy.

add-apt-repository ppa:duplicity-team/ppa apt-get update apt-get install duplicity

Generating the encryption key

Duplicity encrypts your backups with the help of an encryption key. This ensures that no one without the key will be able to read your backups.

To generate your key, use the following command:

gpg --gen-key

This still start prompting you for options. Usually the defaults are fine.

Do make sure to set a passphrase to further improve the security of your key. This makes it so if your key is ever obtained, it will not be usable unless the passphrase is also compromised.

gpg (GnuPG) 1.4.16; Copyright (C) 2013 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Please select what kind of key you want: (1) RSA and RSA (default) (2) DSA and Elgamal (3) DSA (sign only) (4) RSA (sign only) Your selection? 1 RSA keys may be between 1024 and 4096 bits long. What keysize do you want? (2048) 4096 Requested keysize is 4096 bits Please specify how long the key should be valid. 0 = key does not expire <n> = key expires in n days <n>w = key expires in n weeks <n>m = key expires in n months <n>y = key expires in n years Key is valid for? (0) 0 Key does not expire at all Is this correct? (y/N) y You need a user ID to identify your key; the software constructs the user ID from the Real Name, Comment and Email Address in this form: "Heinrich Heine (Der Dichter) <heinrichh@duesseldorf.de>" Real name: My Name Email address: email@example.org Comment: You selected this USER-ID: "My Name <email@example.org>" Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o You need a Passphrase to protect your secret key. We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. ......+++++ ...............+++++ .......+++++ .......................+++++ gpg: key 528CAC31 marked as ultimately trusted public and secret key created and signed. gpg: checking the trustdb gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model gpg: depth: 0 valid: 2 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 2u pub 4096R/528CAC31 2014-08-11 Key fingerprint = 0725 7C03 DFDE DCB5 805C 3EFF 503B 7B6A 528C AC31 uid My Name <email@example.org> sub 4096R/DBD5888E 2014-08-11

Getting a Swift Object Storage Service

The Openstack foundation set up a nifty marketplace website where you can find Openstack service providers.

On it, you can:

Filter by project (such as Nova or Swift)

Filter by Location (pick a location near you for best throughput and latency)

See the version of the services provided (Icehouse is the latest release at this time with Juno scheduled for October 2014)

If you get lost amidst the list of providers, feel free to ask me for advice.

Backing up a directory to Swift

Once you’ve installed Duplicity, have your encryption key ready and you have a Swift service, it’s really… really… easy.

Duplicity requires some environment variables to be set. I source this little bash script but feel free to do anything you’d like:

#!/bin/bash # Swift credentials export SWIFT_USERNAME = tenant:username export SWIFT_PASSWORD = password export SWIFT_AUTHURL = https://keystone_auth_host/v2.0 export SWIFT_AUTHVERSION = 2 # Optional, your gpg passphrase passed from Duplicity to gpg # Otherwise Duplicity will request the passphrase from stdin. # export PASSPHRASE=password

A note about Keystone authentication

Keystone authentication requires SWIFT_AUTHVERSION to be ‘2’ and also adds a dependency to python-keystoneclient which is NOT included as a dependency to Duplicity. You’ll need to install it:

apt-get install python-keystoneclient # OR pip install python-keystoneclient

Once your credentials are in the environmental variables, you’re all set to use Duplicity. Here’s the syntax:

duplicity <dir> swift://<container>

A note about storing credentials in environmental variables

I don’t like the general idea of storing credentials in plain text, let alone as environmental variables but this is how Duplicity works apparently. Alternatively, you can also invoke Duplicity with variables inline, like so:

SWIFT_USERNAME=tenant:username [...] duplicity <dir> swift://<container>

Still, be careful about letting these sort of entries in your BASH history. If you have a clever way to send credentials to Duplicity and GPG in a secure way, please comment !

So, let’s pretend I want to backup /root to the root_backup container:

duplicity /root swift://root_backup Local and Remote metadata are synchronized, no sync needed. Last full backup date: none GnuPG passphrase: Retype passphrase to confirm: No signatures found, switching to full backup. --------------[ Backup Statistics ]-------------- StartTime 1407817325.28 (Tue Aug 12 00:22:05 2014) EndTime 1407817326.27 (Tue Aug 12 00:22:06 2014) ElapsedTime 0.99 (0.99 seconds) SourceFiles 244 SourceFileSize 11097577 (10.6 MB) NewFiles 244 NewFileSize 11097577 (10.6 MB) DeletedFiles 0 ChangedFiles 0 ChangedFileSize 0 (0 bytes) ChangedDeltaSize 0 (0 bytes) DeltaEntries 244 RawDeltaSize 10913257 (10.4 MB) TotalDestinationSizeChange 8555954 (8.16 MB) Errors 0 -------------------------------------------------

That’s it ! If I run it a second time, you’ll see that hardly any data is actually transferred:

duplicity /root swift://root_backup Local and Remote metadata are synchronized, no sync needed. Last full backup date: Tue Aug 12 00:21:55 2014 GnuPG passphrase: Retype passphrase to confirm: --------------[ Backup Statistics ]-------------- StartTime 1407817392.57 (Tue Aug 12 00:23:12 2014) EndTime 1407817392.68 (Tue Aug 12 00:23:12 2014) ElapsedTime 0.11 (0.11 seconds) SourceFiles 246 SourceFileSize 11272304 (10.8 MB) NewFiles 10 NewFileSize 198608 (194 KB) DeletedFiles 5 ChangedFiles 4 ChangedFileSize 26672 (26.0 KB) ChangedDeltaSize 0 (0 bytes) DeltaEntries 19 RawDeltaSize 184262 (180 KB) TotalDestinationSizeChange 182580 (178 KB) Errors 0 -------------------------------------------------

Looking at the result

Using Swift Client, I can see the files Duplicity uploaded:

swift list --lh root_backup 175K 2014-08-12 04:22:07 duplicity-full-signatures.20140812T042155Z.sigtar.gpg 196 2014-08-12 04:22:08 duplicity-full.20140812T042155Z.manifest.gpg 8.2M 2014-08-12 04:22:07 duplicity-full.20140812T042155Z.vol1.difftar.gpg 248 2014-08-12 04:23:13 duplicity-inc.20140812T042155Z.to.20140812T042304Z.manifest.gpg 178K 2014-08-12 04:23:12 duplicity-inc.20140812T042155Z.to.20140812T042304Z.vol1.difftar.gpg 5.8K 2014-08-12 04:23:13 duplicity-new-signatures.20140812T042155Z.to.20140812T042304Z.sigtar.gpg 8.5M

Isn’t that just awesome ?

Have any tips or comments about Duplicity with Swift ? Please let me know !