Quick Start¶

This guide will show you how to do a backup - scrub - restore - cleanup cycle with sqlite as metadata backend and a file storage backup target (e.g. NFS).

What you need to know:¶ Backup source A block device or image file (e.g. containing a VM) to be backed up. backy2 can not backup folders or multiple files. The source must not be modified during backup, so either stop all writers or create a snapshot. Backup target A storage (currently supported: filesystem or S3 compatible) to which the backed up data will be saved in 4MB blocks. Backup Metadata A Database containing information on how to reassemble the stored blocks to get the original data back. Currently an SQL database. Version A version of a backup. A version is a backup on a specific time for a specific backup source. A version is identified by its UUID.

Installation¶ Ubuntu 18.04¶ Installation: Get the latest release from https://github.com/wamdam/backy2/releases sudo dpkg -i backy2_*.deb # Install the debian archive sudo apt-get -f install # Install the dependencies # if you're using rbd/rados from ceph, we recommend the newer libraries sudo apt install python3-rbd python3-rados # if you're going to use postgresql sudo apt install python3-psycopg2 Edit backy.cfg: vim /etc/backy.cfg Especially look if these paths are good. Metadata storage path engine: sqlite:////var/lib/backy2/backy.sqlite Data storage path path: /var/lib/backy2/data # This should be the mountpoint of NFS Other values of interest are simultaneous_writes and simultaneous_reads . Depending on your backup source and target you may want to go lower (i.e. disk with very slow seeks) or higher (raid source or target, S3 target, …). Tip For reference, we use about the half the number of disks as value for simultaneous access. So if you have 40 OSDs in ceph/rbd on the backup source and a 20 disk raid 10 backup target (which makes only 10 parallel disks on writes), you’d use simultaneous_reads: 20 simultaneous_writes: 5 Of course your mileage may vary. These settings have great impact on the backup and restore performance. Higher values need a bit more RAM.

Encryption key¶ Important: Set a unique encryption key. backy2 will not work without one. Also make sure it’s a good key. Generate it via: $ openssl rand -hex 32 Set it to the config in the [DEFAULTS] Section: encryption_key: decafbaddecafbaddecafbzddecafbaddecafbaddecafbaddecafbaddecafba Note The above key is intentionally invalid so that nobody copy&pastes this. The resulting key must be 64 hex characters long (which are effectively 32 bytes for the resulting AES key).

backup¶ Initialize the database: $ backy2 initdb INFO: $ /usr/bin/backy2 initdb INFO: Backy complete. Note Initializing a database multiple times does not destroy any data, instead will fail because it finds already-existing tables. Create demo data: For demonstration purpose, create a 40MB test file: $ dd if = /dev/urandom of = TESTFILE bs = 1M count = 40 40 +0 records in 40 +0 records out 41943040 bytes ( 42 MB, 40 MiB ) copied, 0 .175196 s, 239 MB/s Backup the image (works similar with a device): $ backy2 backup file://TESTFILE myfirsttestbackup INFO: $ /usr/bin/backy2 backup file://TESTFILE myfirsttestbackup INFO: Backed up 1 /10 blocks ( 10 .0% ) INFO: Backed up 2 /10 blocks ( 20 .0% ) INFO: Backed up 3 /10 blocks ( 30 .0% ) INFO: Backed up 4 /10 blocks ( 40 .0% ) INFO: Backed up 5 /10 blocks ( 50 .0% ) INFO: Backed up 6 /10 blocks ( 60 .0% ) INFO: Backed up 7 /10 blocks ( 70 .0% ) INFO: Backed up 8 /10 blocks ( 80 .0% ) INFO: Backed up 9 /10 blocks ( 90 .0% ) INFO: Backed up 10 /10 blocks ( 100 .0% ) INFO: New version: 8fd42f1a-2364-11e7-8594-00163e8c0370 ( Tags: [ b_daily,b_weekly,b_monthly ]) INFO: Backy complete. List backups: $ backy2 ls INFO: $ /usr/bin/backy2 ls +---------------------+-------------------+---------------+------+------------+--------------------------------------+-------+-----------+----------------------------+------------+ | date | name | snapshot_name | size | size_bytes | uid | valid | protected | tags | expire | +---------------------+-------------------+---------------+------+------------+--------------------------------------+-------+-----------+----------------------------+------------+ | 2017 -04-17 11 :54:07 | myfirsttestbackup | | 10 | 41943040 | 8fd42f1a-2364-11e7-8594-00163e8c0370 | 1 | 0 | b_daily,b_monthly,b_weekly | 2019 -11-11 | +---------------------+-------------------+---------------+------+------------+--------------------------------------+-------+-----------+----------------------------+------------+ INFO: Backy complete.

scrub¶ Scrubbing reads all the blocks from the backup target (or some of them if you use the -p option) and compares them with the metadata or, if you pass a source option ( -s ), also with the original data. $ backy2 scrub 8fd42f1a-2364-11e7-8594-00163e8c0370 INFO: $ /usr/bin/backy2 scrub 8fd42f1a-2364-11e7-8594-00163e8c0370 INFO: Backy complete. If an error occurs (for example, if backup target blocks couldn’t be read or data has changed), this looks like that: $ backy2 scrub 8fd42f1a-2364-11e7-8594-00163e8c0370 INFO: $ /usr/bin/backy2 scrub 8fd42f1a-2364-11e7-8594-00163e8c0370 ERROR: Blob not found: Block ( uid = '2c0910bef8qnAm54mnyBsonRsPBfTzP' , version_uid = '8fd42f1a-2364-11e7-8594-00163e8c0370' , id = 8 , date = datetime.datetime ( 2017 , 4 , 17 , 11 , 54 , 7 , 639022 ) , checksum = '41c9aa8df42913b3544270a10f1b219cd1b5e1ad9d51700e97acdaeaed3cea843ffaad99590e07de260918ce3847a8b612c9f51f2c945a2d14238956a49ca572' , size = 4194304 , valid = 1 ) INFO: Marked block invalid ( UID 2c0910bef8qnAm54mnyBsonRsPBfTzP, Checksum 41c9aa8df42913b3544270a10f1b219cd1b5e1ad9d51700e97acdaeaed3cea843ffaad99590e07de260918ce3847a8b612c9f51f2c945a2d14238956a49ca572. Affected versions: 8fd42f1a-2364-11e7-8594-00163e8c0370 INFO: Marked version invalid ( UID 8fd42f1a-2364-11e7-8594-00163e8c0370 ) ERROR: Marked version invalid because it has errors: 8fd42f1a-2364-11e7-8594-00163e8c0370 The exit code is then != 0. Which one exactly depends on the kind of error: $ echo $? 20 Also, the version is marked invalid as you can see in: $ backy2 ls INFO: $ /usr/bin/backy2 ls +---------------------+-------------------+---------------+------+------------+--------------------------------------+-------+-----------+----------------------------+ | date | name | snapshot_name | size | size_bytes | uid | valid | protected | tags | +---------------------+-------------------+---------------+------+------------+--------------------------------------+-------+-----------+----------------------------+ | 2017 -04-17 11 :54:07 | myfirsttestbackup | | 10 | 41943040 | 8fd42f1a-2364-11e7-8594-00163e8c0370 | 0 | 0 | b_daily,b_monthly,b_weekly | +---------------------+-------------------+---------------+------+------------+--------------------------------------+-------+-----------+----------------------------+ INFO: Backy complete Just in case you are able to fix the error, just scrub again and backy2 will mark the version valid again.

restore¶ Restore into a file or device: $ backy2 restore 8fd42f1a-2364-11e7-8594-00163e8c0370 file://RESTOREFILE INFO: $ /usr/bin/backy2 restore 8fd42f1a-2364-11e7-8594-00163e8c0370 file://RESTOREFILE INFO: Restored 1 /10 blocks ( 10 .0% ) INFO: Restored 2 /10 blocks ( 20 .0% ) INFO: Restored 3 /10 blocks ( 30 .0% ) INFO: Restored 4 /10 blocks ( 40 .0% ) INFO: Restored 5 /10 blocks ( 50 .0% ) INFO: Restored 6 /10 blocks ( 60 .0% ) INFO: Restored 7 /10 blocks ( 70 .0% ) INFO: Restored 8 /10 blocks ( 80 .0% ) INFO: Restored 9 /10 blocks ( 90 .0% ) INFO: Restored 10 /10 blocks ( 100 .0% ) INFO: Backy complete. backy2 prevents you from restoring into an existing file (or device). So if you try again, it will fail: $ backy2 restore 8fd42f1a-2364-11e7-8594-00163e8c0370 file://RESTOREFILE INFO: $ /usr/bin/backy2 restore 8fd42f1a-2364-11e7-8594-00163e8c0370 file://RESTOREFILE ERROR: Target already exists: file://RESTOREFILE Error opening restore target. You must force the restore. If you want to overwrite data, you must --force . Note For more (and possibly faster) restore methods, please refer to the section Restore.

expire¶ Usually you know when to expire a backup at backup time. So when creating a backup you may also add the expiration date: $ backy2 backup file://tmp/test test -e 2020 -01-24 The expiration date can be shown in backy2 ls and you may also show expired versions only via backy2 ls -e .