Reading Time: 6 minutes

This time I would like to explain how I backup my server with the help of some build in tools and a script written by myself. As I’m running my blogs and my mail server, which is Scalix, on a Linux server I need to make sure, that no data is lost during an outage and that I will be back online in a short period of time. Therefore I have two strategies to backup my stuff.

Using Proxmox, it is very easy to do an online backup of my virtual machine, as described here:

How To: Proxmox Live Backup

The drawback with this method is, that it takes some time to do the backup, it consumes a lot of system resources (I/O) and the backup is very large in size. Using a file based backup from within the virtual machine, backing up only the real important stuff.

The drawback here, you cannot restore the system from this backup. But in combination with method one, it works perfectly.

I will not describe the first method, as this is already described in the mentioned post. I do this kind of full backup once a week and use method two to get the increment data from every day between two full backups. As method two will not work out of the box, you have to make sure the data is consistent, I will cover this method in this post.

Requirements for the Backup

First of all, you need to have an external backup server to make sure the data is protected against loss. I will use an external rsync server which is included in my server subscription. This is still in the same data center, but finger crossed, it is in a different fire compartment. The full backup, produced with Proxmox is copied to my NAS system at home. This will make sure, that at least one of the backups will survive.

In addition to an external storage, you should also have LVM (Logical Volume Manager) installed on your system. This is the only option to make online backups of your system and keep data consistent using LVM snapshots. Instead of LVM, some hardware raid systems support a similar feature.

LVM (Logical Volume Manager)

I will not describe how to configure LVM, this is outside of the scope of this post, but I will explain shortly, what LVM is. LVM is included in the Linux kernel and provides a layer between the file system and the block device. LVM has features like encryption, easy expanding the volume and many more, but we would like to use only the snapshot function in the first place.

To use the snapshot function of LVM you need to make sure, that you still have some free unallocated space on your drive. This space will be used during the snapshot phase to write new changes to the disk. Make sure to leave enough free space on the disk, as you can expand you volume any time. There is no reason to oversize your volume in the first place. Keep in mind, if the space is not enough to hold all the data changes during the snapshot phase, you can be in big trouble.

Consistent LVM Snapshot

Creating a LVM snapshot is very fast but not instantaneous. There is a very little chance that some applications can be in an inconsistent state. MySQL such as could be in the middle of a transaction. Doing a snapshot in this moment will lead to an inconsistent MySQL backup. The same issue can happen with S calix . As those are the only applications which needs spacial handling in my environment, I will only cover them. But keep this in mind and check if you have applications on your system which needs special handling as well.

Scalix

To get a consistent backup of the Scalix message store, you need to suspend Scalix before creating the backup. This could be accomplished using omsuspend like this:

omsuspend -s 120 & sleep 2 sync; sync; sync omsuspend -r

This first line will suspend Scalix and no user interaction with the Scalix system is possible. The last line will resume the Scalix system again and Scalix will run normal.

MySQL

To avoid the situation, the you create the snapshot, when MySQL is in the middle of a transaction, you can lock MySQL. This will somehow pause MySQL. You need to issue this command to the MySQL server:

FLUSH TABLES WITH READ LOCK;

This will write the database to disk and lock the tables. Do this with a permanent session, as the lock is removed as soon as the session is terminated.

To unlock the tables again, after the snapshot is created, use this command:

UNLOCK TABLES;

You should now have a consistent snapshot and you can start the backup process… wait, I didn’t told you, how to create the snapshot.

Create the Snapshot

The snapshot is created very easy and needs only one command, two if you actually would like to get access to the snapshot data. It takes some more effort to get all the information needed, before creating the snapshot.

First of all, you need to know, the name of the volume for which you would like to create the snapshot of . In a very simple environment, you will have only one volume for all data. If you have split your data across multiple volumes, you can check the specific path using this command:

df /var/opt/scalix/sx/ Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/scalix-root 96816868 6806528 85092276 8% /

This will tell me, that the Scalix message store is on the volume group “ scalix ” on the volume “root”. To get the correct path to the volume use this command:

lvscan ACTIVE '/dev/scalix/swap_1' [4.09 GiB] inherit ACTIVE '/dev/scalix/root' [93.80 GiB] inherit

As you can see, there are two volumes, but as the command from above told us to use the volume “root” we have now the complete path to the volume, which is

/dev/scalix/root

You also need a name for the snapshot volume. I chose “backup-snapshot”. Using this two pieces of information we can create the snapshot using this command:

lvcreate -l100%FREE -s -n backup-snapshot /dev/scalix/root

This will use 100% of the free space on the disk and create the snapshot. If you issue lvscan again it should look like this:

lvscan ACTIVE '/dev/scalix/swap_1' [4.09 GiB] inherit ACTIVE Original '/dev/scalix/root' [93.80 GiB] inherit ACTIVE Snapshot '/dev/scalix/backup-snapshot' [1.87 GiB] inherit

The last step would be to mount the snapshot to get access to the files. We can use a simple mount command here:

mount /dev/scalix/backup-snapshot /mnt/backup

Make sure, to create the mount folder before using this command. After backing up everything you need, you can simply unmount and drop the snapshot:

unmount /mnt/backup lvremove -f /dev/scalix/backup-snapshot

Backup

As we now know, how to create the snapshot and remove it, I would like to share my thoughts about the backup itself. I run this kind of backup every day, except the day, when I run the full VM backup. I use rsync and create a backup folder for every day. This works fast and reliable and I can benefit from just backing up the changes.

My backup includes all configuration files within “/etc” and all logs in “/var/log”. To include the Scalix messages store I also backup “/var/opt” and to get the MySQL files I include “/var/lib/ mysql ”. “/var/www” is also included, as I have a Photoblog running, which stores the pictures there.

From my point of view, this includes all files needed to recover from a crash, using the full VM backup plus the file backup.

The Backup Script

To do all the above steps automatically I wrote a script for my usage. Feel free to use it and adapt the script to your own needs.

#!/bin/bash #set -x #is mysql present #0=no #1=yes mysql=1 #is scalix present #0=no #1=yes scalix=1 #mysql user name mysqlusername="mysql" #mysql user file mysql_user_file=/etc/mysql/debian.cnf #Path to the Volume to make the snapshot of lvmpath="/dev/scalix/root" #Logical Volume Name for the snapshot lvmname="backup-snapshot" #Path to the mountfolder mountfolder="/mnt/snapshot" #create path to snapshot volume lvmsnapshotpath=${lvmpath%/*}"/"$lvmname #list of folders to backup, separate with space backupfolder="/etc /var/log /var/opt /var/lib/mysql /var/www" #backup destination. backupdestination="rsync://backup-server/folder/" #get day of week dow=$(date +%a) #clean up if something goes wrong function clean_up { #unlock mysql unlock_mysql #resume scalix activate_scalix #remove the snapshot remove_snapshot #exit the script exit } trap clean_up SIGHUP SIGINT SIGTERM #lock mysql function lock_mysql { if [ $mysql -eq 0 ]; then return fi #create a screen session to lock mysql during snapshot screen -S "mysql" -d -m #pause for 1sec to get the session created sleep 1 #connect to mysql screen -r "mysql" -X stuff "/usr/bin/mysql --defaults-extra-file=$mysql_user_file -u $mysqlusername " #lock mysql screen -r "mysql" -X stuff "FLUSH TABLES WITH READ LOCK; " #wait for 1sec to fully lock mysql sleep 1 } #unlock mysql function unlock_mysql { if [ $mysql -eq 0 ]; then return fi #check if the screen session is still tunning if ! screen -list | grep -q "mysql"; then return fi #unlock mysql again screen -r "mysql" -X stuff "UNLOCK TABLES; " #quit from mysql screen -r "mysql" -X stuff "quit " #end the screen session screen -r "mysql" -X stuff "exit " } #create the snapshot function create_snapshot { #create the snapshot lvcreate -l100%FREE -s -n $lvmname $lvmpath } #mount the snapshot function mount_snapshot { #create the mountfolder mkdir -p $mountfolder #moun the snapshot mount $lvmsnapshotpath $mountfolder } #do the backup function backup_everything { #do some backup rsync -apE --delete --stats $backupfolder $backupdestination$dow } #remove the snapshot function remove_snapshot { #unmount the snapshot folder umount $mountfolder #remove the snapshot lvremove -f $lvmsnapshotpath } #suspend scalix function suspend_scalix { if [ $scalix -eq 0 ]; then return fi /opt/scalix/bin/omsuspend -s 120 & sleep 2 sync; sync; sync } #activate scalix again function activate_scalix { if [ $scalix -eq 0 ]; then return fi /opt/scalix/bin/omsuspend -r } lock_mysql suspend_scalix create_snapshot activate_scalix unlock_mysql mount_snapshot backup_everything remove_snapshot exit

You need to modify some of the variables at the top of the script, to make it work for you environment.

You can run the script as a cron job to run it automatically. I run my script every night at 04:30am.

If you have any questions, regarding this post or if you would like to provide feedback, please use the comment function below.

Share this: Facebook

LinkedIn

Reddit

Twitter

Email

More

Print

Tumblr



Pinterest

Pocket



Telegram

WhatsApp



Skype



Like this: Like Loading...