Rodrigo Rosenfeld Rosas

Fri, 24 Jun 2016 15:31:00 +0000 (Updated at Mon, 04 Jul 2016 16:32:00 +0000)

For some years I have been using rsnapshot to back up our databases and documents using an incremental approach. We create a new back-up every hour and retain the last 24 hours backup, one back-up per day for the past 7 days and one back-up per week for the past 4 weeks.

Rsnapshot is great. It uses hard-links to achieve incremental back-up, saving up a lot of space. It's a combination of "cp -al" and rsync. But we were facing a problem related to free inodes count on our ext4 partition. By the way, NewRelic does not monitor the free inodes count (df -i) so I found this problem the hard way, after the back-up stopped working due to lack of free inodes.

I've created a custom check in our own monitoring system to alert about low free inodes available and then I tried to tweak some ext4 settings to avoid this problem again in the new partition. We have 26GB spread on 2.6 million of individually gzipped documents (they are served directly by nginx) which will create almost 100 million hard-links in that back-up partition. There are hardlinks around the original documents as well as part of a smart strategy to save space when the same document is used in multiple transactions (they are not changed). Otherwise they would take some extra Gigabytes.

Recently, my custom monitoring system sent me an alert that 75% of the inodes were used while about only 30% of disk space was being actually used. So, I decided to investigate a bit more about other filesystems which dealt with inodes dynamically.

The btrfs filesystem

That's how I found btrfs, a modern file-system which not only does not have a limit on inodes but, as I'll describe, has some very interesting features for dealing with incremental back-up in a faster and better way than rsnapshot.

Initially I wasn't thinking about replacing rsnapshot, but after reading about support for subvolumes and snapshots in btrfs I changed my mind and decided to replace rsnapshot with a custom script. I've tried to adapt rsnapshot for several hours to make the workflow I wanted work without success though. Here's an issue related to btrfs support.

Before I talk about how btrfs helps our back-up system, let me explain a few issues I had with rsnapshot.

Rsnapshot issues

I've been living with some issues with rsnapshot in the past years. I want the full back-up procedure to take less than an hour so that we would be able to run it every hour. I had to tweak its settings a few times in order to get the script to finish in less than an hour but in the past days it was taking already almost 40 minutes to complete. A while back, before the tweaks, I had to change the interval to back-up every two hours.

One of the slow parts of rsnapshot is removing the last back-up snapshot when rotating. It doesn't matter if you use "rm -rf" or whatever other method. Removing a big tree of files is slow. An alternative would be to move the latest snapshot to the first one (hourly.0), since this would save the "rm -rf" time and also the "cp -al" time, skipping to the rsync phase. But I wasn't able to figure out how to make that happens with rsnapshot.

Also, some of the procedures could be done in parallel to speed up the process but rsnapshot doesn't provide direct support to specify this and it's hard to write proper shell script to manage those cases.

The goal

After reading about btrfs I figured out that the back-up procedure could be made much faster and be simplified. Then I created a Ruby script, which I'll show in the next section, and integrated it in our automation tools in one day. I've replaced rsnapshot with it in our back-up server, with the new script and it's running pretty well for the last two days taking about 8 minutes to complete the procedure on each run.

So, let me explain the strategy I wanted to implement to help you understanding the script.

As I said, btrfs supports subvolumes. Btrfs implements copy-on-write (CoW), so basically, this allows to both create and delete snapshots from subvolumes instantly (constant time). That means we replace the slow "rm -rf hourly.23" with the instantaneous "btrfs subvolume delete hourly.23" and "cp -al ..." with the instantaneous "btrfs subvolume snapshot ...".

In order for a regular user to delete subvolumes with btrfs, the user_subvol_rm_allowed fs option must be used. Also, deleting a subvolume doesn't work if there are other subvolumes inside it, so they must be removed first. There's no switch or tool in the btrfs-progs package that allows you to delete them recursively. This is important to understand the script.

Our back-up procedure consists of getting a recent dump of two production PostgreSQL databases (the main database and the one used by Redmine) and syncing two directories containing files (the main application files and the files uploaded to Redmine).

The idea is to get them inside a static path as the first step. The main reason for that is that if something goes wrong in the process after syncing the documents (the slowest part), for example, we wouldn't lose the transferred files the next time we try to run the script. So, basically here's how I implemented it (there's a simpler strategy I'll explain next):

/var/backups/latest [regular directory]

/var/backups/latest/postgres [subvolume] - the main db dump is stored here

/var/backups/latest/tickets-db [subvolume] - the tickets db dump is stored here

/var/backups/latest/docmanager [subvolume] - the 2.6 million documents are rsynced here

/var/backups/latest/tickets-files [subvolume] - Redmine files go here

After the procedure is finished to get them in the latest state it creates a tmp directory and create a snapshot for each subvolume inside tmp and once everything works fine the back-ups are rotated and tmp is moved to hourly.0. Removing hourly.23 in the rotation phase has to remove the inner subvolumes first.

After implementing this (it was an iterative process) I realized it could be simplified to use a simpler infra-structure. "latest" would be a subvolume and everything inside it regular files and directories. Than the "tmp" directory wouldn't be used and after rotating a snapshot of "latest" would be used to create "hourly.0". I didn't update the script yet because I'm not sure if it worths changing, since the current layout is more modular, which is useful in case I want to take some snapshot of just part of the back-up for some reason. So the sample back-up script in the next section will use my current tested approach, which is the situation described first above.

The main database has over 500MB in PostgreSQL custom format, and it's much faster to rsync it than using scp. Initially those databases were not stored in the "latest" diretory and I used "scp" to copy them directly to the "tmp" directory, but I changed the strategy to save some time and bandwidth.

The script should exit with a message and non zero exit status code when something fails so that I would be notified if anything goes wrong by Cron (by setting the MAILTO=my@email.com in the beggining of the crontab file). It shouldn't affect the existing valid snapshots either in that case.

It shouldn't run in case the previous procedure hasn't finish, so there's a simple lock mechanism preventing that from happen in case it takes over an hour to complete. The second attempt will fail and I should get an e-mail telling me that happened.

It should also have a dry-run mode (which I call test mode) that will output the commands without running it, which is useful while designing the back-up steps. It should also allow for commands to run concurrently so it uses some indentation to show the order the commands are run.

Finally, it will report in the logs the issued commands and their status (finished or failed) as well as any commands output (STDOUT or STDERR) and the time each command took as well as the total time in the end of the procedure.

Finally, now that you understand what the script is supposed to do, here's the actual implementation.

The script

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 #!/usr/bin/env ruby require ' open3 ' require ' thread ' require ' logger ' require ' time ' class Backup def run (args) @start_time = Time .now @backup_root_path = File .expand_path ' /var/backups ' #@backup_root_path = File.expand_path '~/backups' @log_path = " #{ @backup_root_path } /backup.log " @tmp_path = " #{ @backup_root_path } /tmp " @exiting = false Thread .current[ :indenting_level ] = 0 setup_logger lock_or_exit log ' Starting back-up procedure ' parse_args args.clone run_scripts if @action == ' hourly ' rotate unlock report_completed end private def setup_logger File .write @log_path , ' ' unless File .exist? @log_path logfile = File .open( @log_path , File :: WRONLY | File :: APPEND ) logfile.sync = true @logger = Logger .new logfile @logger .level = Logger :: INFO @logger .datetime_format = ' %Y-%m-%d %H:%M:%S ' @logger_mutex = Mutex .new end def lock_or_exit if File .exist?(pidfile) && run_command( " kill -0 #{ pid = File .read pidfile } " ) abort " There's another backup in progress. Pid: #{ pid } (from #{ pidfile } ). " end File .write pidfile, Process .pid end def unlock File .unlink pidfile end def pidfile @pidfile ||= " #{ @backup_root_path } /backup.pid " end def run_command! (cmd, sucess_in_test_mode = true , abort_on_stderr : false ) run_command cmd, sucess_in_test_mode, abort_on_stderr : abort_on_stderr, abort_on_error : true end def run_command (cmd, sucess_in_test_mode = true , abort_on_stderr : false , abort_on_error : false ) indented_cmd = ' ' * indenting_level + cmd Thread .current[ :indenting_level ] += 1 if @test_mode @logger_mutex .synchronize{ puts indented_cmd} return sucess_in_test_mode end start = Time .now log " started: ' #{ indented_cmd } ' " stdout, stderr, status = Open3 .capture3 cmd stdout = stdout.chomp stderr = stderr.chomp success = status == 0 log stdout unless stdout.empty? log stderr, :warn unless stderr.empty? if (!success && abort_on_error) || (abort_on_stderr && !stderr.empty?) die " ' #{ cmd } ' failed to run with exit status #{ status } , aborting. " end log " finished: ' #{ indented_cmd } ' ( #{ success ? ' successful ' : " failed with #{ status } " } ) " + " [ #{ human_duration Time .now - start } ] " success end def indenting_level Thread .current[ :indenting_level ] end def log (msg, level = :info ) return if @test_mode @logger_mutex .synchronize{ @logger .send level, msg } end VALID_OPTIONS = [ ' hourly ' , ' daily ' , ' weekly ' ].freeze def parse_args (args) args.shift if @test_mode = (args.first == ' test ' ) unless args.size == 1 && VALID_OPTIONS .include?( @action = args.first) abort " Usage: 'backup [test] action', where action can be hourly, daily or weekly. If test is specified the commands won't run but will be shown. " end end def die (message) log message, :fatal was_exiting = @exiting @exiting = true delete_tmp_path_if_exists unless was_exiting unlock abort message end def create_tmp_path delete_tmp_path_if_exists create_subvolume @tmp_path end def create_subvolume (path, skip_if_exists = false ) return if skip_if_exists && File .exist?(path) run_script %Q{ btrfs subvolume create " #{ path } " } end def delete_tmp_path_if_exists delete_subvolume_if_exists @tmp_path , delete_children : true end def delete_subvolume_if_exists (path, delete_children : false ) return unless File .exist?(path) Dir [ " #{ path } /* " ].each{|s| delete_subvolume_if_exists s } if delete_children run_script %Q{ btrfs subvolume delete -c " #{ path } " } end def run_script (script) run_command! script end def run_scripts (scripts = all_scripts) case scripts when Par il = indenting_level last_il = il scripts.map do |s| Thread .start do Thread .current[ :indenting_level ] = il run_scripts s last_il = [ Thread .current[ :indenting_level ], last_il].max end end .each & :join Thread .current[ :indenting_level ] = last_il when Array scripts.each{|s| run_scripts s } when String run_script scripts when Proc scripts[] else die " Invalid script ( #{ scripts.class } ): #{ scripts } " end end Par = Class .new Array def all_scripts [ Par [->{create_tmp_path}, " mkdir -p #{ @backup_root_path } /latest " , dump_main_db_on_d1, dump_tickets_db_on_d1], Par [local_docs_sync, local_tickets_files_sync, local_main_db_sync, local_tickets_db_sync], Par [main_docs_script, tickets_files_script, main_db_script, tickets_db_script], ] end def dump_main_db_on_d1 %q{ ssh backup@backup-server.com "pg_dump -Fc -f /tmp/main_db.dump } + %q{ main_db_production" } end def dump_tickets_db_on_d1 %q{ ssh backup@backup-server.com "pg_dump -Fc -f /tmp/tickets.dump redmine_production" } end def local_docs_sync [ ->{ create_subvolume local_docmanager, true }, " rsync -azHq --delete-excluded --delete --exclude doc --inplace " + " backup@backup-server.com:/var/main-documents/production/docmanager/ " + " #{ local_docmanager } / " , ] end def local_docmanager @local_docmanager ||= " #{ @backup_root_path } /latest/docmanager " end def local_tickets_files_sync [ ->{ create_subvolume local_tickets_files, true }, " rsync -azq --delete --inplace backup@backup-server.com:/var/redmine/files/ " + " #{ local_tickets_files } / " , ] end def local_tickets_files @local_tickets_files ||= " #{ @backup_root_path } /latest/tickets-files " end def local_main_db_sync [ ->{ create_subvolume local_main_db, true }, " rsync -azq --inplace backup@backup-server.com:/tmp/main_db.dump " + " #{ local_main_db } /main_db.dump " , ] end def local_main_db @local_main_db ||= " #{ @backup_root_path } /latest/postgres " end def local_tickets_db_sync [ ->{ create_subvolume local_tickets_db, true }, " rsync -azq --inplace backup@backup-server.com:/tmp/tickets.dump " + " #{ local_tickets_db } /tickets.dump " , ] end def local_tickets_db @local_tickets_db ||= " #{ @backup_root_path } /latest/tickets-db " end def main_docs_script create_snapshot_cmd local_docmanager, " #{ @tmp_path } /docmanager " end def create_snapshot_cmd (from, to) " btrfs subvolume snapshot #{ from } #{ to } " end def main_db_script create_snapshot_cmd local_main_db, " #{ @tmp_path } /postgres " end def tickets_db_script create_snapshot_cmd local_tickets_db, " #{ @tmp_path } /tickets-db " end def tickets_files_script create_snapshot_cmd local_tickets_files, " #{ @tmp_path } /tickets-files " end LAST_DIR_PER_TYPE = { ' hourly ' => 23 , ' daily ' => 6 , ' weekly ' => 3 }.freeze def rotate last = LAST_DIR_PER_TYPE [ @action ] path = ->(n, action = @action ){ " #{ @backup_root_path } / #{ action } . #{ n } " } delete_subvolume_if_exists path[last], delete_children : true n = last while (n -= 1 ) >= 0 run_script " mv #{ path[n] } #{ path[n+ 1 ] } " if File .exist?(path[n]) end dest = path[ 0 ] case @action when ' hourly ' run_script " mv #{ @tmp_path } #{ dest } " when ' daily ' , ' weekly ' die ' last hourly back-up does not exist ' unless File .exist?(hourly0 = path[ 0 , ' hourly ' ]) create_tmp_path Dir [ " #{ hourly0 } /* " ].each do |subvolume| run_script create_snapshot_cmd subvolume, " #{ @tmp_path } / #{ File .basename subvolume } " end run_script " mv #{ @tmp_path } #{ dest } " end end def report_completed log " Backup finished in #{ human_duration Time .now - @start_time } " end def human_duration (total_time_sec) n = total_time_sec.round parts = [] [ 60 , 60 , 24 ].each{|d| n, r = n.divmod d; parts << r; break if n.zero?} parts << n unless n.zero? pairs = parts.reverse.zip( %w( d h m s ) [-parts.size.. -1 ]) pairs.pop if pairs.size > 2 # do not report seconds when irrelevant pairs.flatten.join end end Backup .new.run( ARGV ) if File .expand_path( $PROGRAM_NAME ) == File .expand_path( __FILE__ )

So, this is what I get running the test mode:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 $ ruby backup.rb test hourly btrfs subvolume create "/home/rodrigo/backups/tmp" mkdir -p /home/rodrigo/backups/latest ssh backup@backup-server.com "pg_dump -Fc -f /tmp/main_db.dump main_db_production" ssh backup@backup-server.com "pg_dump -Fc -f /tmp/tickets.dump redmine_production" btrfs subvolume create "/home/rodrigo/backups/latest/docmanager" btrfs subvolume create "/home/rodrigo/backups/latest/tickets-files" btrfs subvolume create "/home/rodrigo/backups/latest/postgres" btrfs subvolume create "/home/rodrigo/backups/latest/tickets-db" rsync -azHq --delete-excluded --delete --exclude doc --inplace backup@backup-server.com:/var/main-documents/production/docmanager/ /home/rodrigo/backups/latest/docmanager/ rsync -azq --delete --inplace backup@backup-server.com:/var/redmine/files/ /home/rodrigo/backups/latest/tickets-files/ rsync -azq --inplace backup@backup-server.com:/tmp/main_db.dump /home/rodrigo/backups/latest/postgres/main_db.dump rsync -azq --inplace backup@backup-server.com:/tmp/tickets.dump /home/rodrigo/backups/latest/tickets-db/tickets.dump btrfs subvolume snapshot /home/rodrigo/backups/latest/tickets-db /home/rodrigo/backups/tmp/tickets-db btrfs subvolume snapshot /home/rodrigo/backups/latest/tickets-files /home/rodrigo/backups/tmp/tickets-files btrfs subvolume snapshot /home/rodrigo/backups/latest/postgres /home/rodrigo/backups/tmp/postgres btrfs subvolume snapshot /home/rodrigo/backups/latest/docmanager /home/rodrigo/backups/tmp/docmanager mv /home/rodrigo/backups/tmp /home/rodrigo/backups/hourly.0

The "all_scripts" method is the one you should adapt for your needs.

Final notes

I hope that script will help you serving as a base for your own back-up script in Ruby in case I was able to convince you to give this strategy a try. Unless you are already using some robust back-up solution such as Bacula or other advanced systems, this strategy is very simple to implement, takes little space and allows for fast incremental backups and might interest you.

Please let me know if you have any questions in the comments section or if you'd suggest any improvements over it. Or if you think you've found a bug I'd love to hear about it.

Good luck dealing with your back-ups. :)