Re: Using S3 as an infinitely large disk device (that never breaks) Posted by: greg13070 Posted on: Apr 18, 2006 7:24 PM in response to: hvrietsc Reply



Also a question for the s3 filesystem gurus:

I understand that s3fs (I think that would be a cool name for this project, don't you all?) has many and versatile uses. I am personally looking for a very easy backup/restore strategy. On my old server I use rsync with hard links to do incremental backups to a separate physical drive, and it's great:



I can just cd over to the backup drive and examine or restore files from any time in the past. Backups can be incremental forever and well.. it's just really awesome.



Anyway, now that s3 exists, I want the backups to be offsite. Today I am using s3sync for this, but it has some limitations and is a hack (especially with my modifications):



Now, I want to stop using s3sync as soon as something more suitable is available. My naive estimation tells me that to use the s3 fuse filesystem as a backup target for rsync would require about 1-to-1 free-space to used-space locally to cache the fuse blocks is that correct? I understand they eventually can be sync'd over to s3 and purged, but in order to perform a backup all the files must be copied. Even an incremental backup must check all the files for changes right? so all the blocks would have to be available locally at run time at least for a little while.



I think I can live with this limitation if correct, but I wanted to ask people here for ideas and/or optimization suggestions.



I know that I could just change strategy and use something more traditional like backup manager (with s3 support thanks to bdixon), but I have become very accustomed to the tremendous benefits associated with rsync backups.



Any feedback appreciated. Sorry for the rambling post =P Is there a chance of eventually getting this project to run without having to rebuild fuse just to compile hello? I am trying to run with debian packages, and it's dependency hell to try and get the system set up to compile fuse (kernel-dev, etc etc etc).Also a question for the s3 filesystem gurus:I understand that s3fs (I think that would be a cool name for this project, don't you all?) has many and versatile uses. I am personally looking for a very easy backup/restore strategy. On my old server I use rsync with hard links to do incremental backups to a separate physical drive, and it's great: http://www.mikerubel.org/computers/rsync_snapshots/ I can just cd over to the backup drive and examine or restore files from any time in the past. Backups can be incremental forever and well.. it's just really awesome.Anyway, now that s3 exists, I want the backups to be offsite. Today I am using s3sync for this, but it has some limitations and is a hack (especially with my modifications): http://developer.amazonwebservices.com/connect/thread.jspa?messageID=37287 Now, I want to stop using s3sync as soon as something more suitable is available. My naive estimation tells me that to use the s3 fuse filesystem as a backup target for rsync would require about 1-to-1 free-space to used-space locally to cache the fuse blocks is that correct? I understand they eventually can be sync'd over to s3 and purged, but in order to perform a backup all the files must be copied. Even an incremental backup must check all the files for changes right? so all the blocks would have to be available locally at run time at least for a little while.I think I can live with this limitation if correct, but I wanted to ask people here for ideas and/or optimization suggestions.I know that I could just change strategy and use something more traditional like backup manager (with s3 support thanks to bdixon), but I have become very accustomed to the tremendous benefits associated with rsync backups.Any feedback appreciated. Sorry for the rambling post =P