Writing file systems in Go with FUSE

Motivation

Some time ago, I decided I wanted to solve my own storage needs better, and I realized that I can’t just rely on synchronizing files. I needed a filesystem that combines the best of three worlds: local files, network file systems, and file synchronization. This project is called Bazil, as in bazillion bytes.

To make Bazil possible, I needed to be able to easily write a filesystem in Go. And now you can, too, with bazil.org/fuse.

What we’ll build today is an example Go application that serves a Zip archive as a filesystem:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 $ unzip -v archive.zip Archive: archive.zip Length Method Size Cmpr Date Time CRC-32 Name -------- ------ ------- ---- ---------- ----- -------- ---- 0 Stored 0 0% 2014-12-11 04:03 00000000 buried/ 0 Stored 0 0% 2014-12-11 04:03 00000000 buried/deep/ 5 Stored 5 0% 2014-12-11 04:03 2efcceec buried/deep/loot 13 Stored 13 0% 2014-12-11 04:03 f4247453 greeting -------- ------- --- ------- 18 18 0% 4 files $ zipfs archive.zip mnt & $ tree mnt mnt ├── buried │ └── deep │ └── loot └── greeting 2 directories, 2 files $ cat mnt/greeting hello, world

FUSE

FUSE (Filesystem In Userpace) is a Linux kernel filesystem that sends the incoming requests over a file descriptor to userspace. Historically, these have been served with a C library of the same name, but ultimately FUSE is just a protocol. Since then, the protocol has been implemented for other platforms such as OS X, FreeBSD and OpenBSD.

bazil.org/fuse is a reimplementation of that protocol in pure Go.

Structure of Unix filesystems

Unix filesystems consist of inodes (“index nodes”). These nodes are files, directories, etc. Directories contain directory entries (dirent) that point to child inodes. A directory entry is identified by its name, and carries very little metadata. The inode manages both the metadata (including things like access control) and the content of the file.

Open files are identified in userspace with file descriptors, which are just safe references to kernel objects known as handles.

Go API

Our FUSE library is split into two parts. The low-level protocol is in bazil.org/fuse while the higher-level, optional, state machine keeping track of object lifetimes is bazil.org/fuse/fs .

Each file system has a root entry. The interface fs.FS has a method Root that returns an fs.Node .

To access a file (see its metadata, open it, etc), the kernel looks it up by name by sending a fuse.LookupRequest to the FUSE server, stating the parent directory and basename. This request is served by a Lookup method on the parent fs.Node . The method returns an fs.Node , and the result is cached in the kernel and reference counted. Dropping a cache entry sends a ForgetRequest , and when the reference count reaches zero, Forget gets called.

Files are renamed with Rename , deleted with Remove , and so on.

Kernel file handles are created for example by opening a file. Opening an existing file sends an OpenRequest , you guessed it, served by Open . All methods creating new handles return a Handle . Handles are closed by a combination of Flush and Release .

The default Open action, if the method is not implemented, is to use the fs.Node also as a Handle ; this tends to work well for stateless read-only files.

Reads from a Handle are served by Read , writes with Write , and apart from all the extra data available these look similar to io.ReaderAt and io.WriterAt . Note that file size changes via Setattr , not based on Write , and Attr needs to return the correct Size .

Listing a directory happens by reading an open file handle that is a directory. Instead of file contents, the read returns marshaled directory entries. The ReadDir method implements a slightly higher-level API, where you return a slice of directory entries.

And so on. Learning to write a file system requires a decent understanding of the kernel data structures and their state changes on an abstract level, but the actual Go parts of it are quite simple. So let’s dive into the code.

zipfs

As our example project, we’ll write a filesystem that shows a read-only view of the contents of a Zip archive.

The full source code is available at https://github.com/bazillion/zipfs

Skeleton

Let’s start with a skeleton with a argument parsing:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 package main import ( "archive/zip" "flag" "fmt" "io" "log" "os" "path/filepath" "strings" "bazil.org/fuse" "bazil.org/fuse/fs" ) // We assume the zip file contains entries for directories too. var progName = filepath . Base ( os . Args [ 0 ]) func usage () { fmt . Fprintf ( os . Stderr , "Usage of %s:

" , progName ) fmt . Fprintf ( os . Stderr , " %s ZIP MOUNTPOINT

" , progName ) flag . PrintDefaults () } func main () { log . SetFlags ( 0 ) log . SetPrefix ( progName + ": " ) flag . Usage = usage flag . Parse () if flag . NArg () != 2 { usage () os . Exit ( 2 ) } path := flag . Arg ( 0 ) mountpoint := flag . Arg ( 1 ) if err := mount ( path , mountpoint ); err != nil { log . Fatal ( err ) } }

Mounting is a bit cumbersome due to OSXFUSE behaving very differently from Linux; there are several stages where errors may show up.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 func mount ( path , mountpoint string ) error { archive , err := zip . OpenReader ( path ) if err != nil { return err } defer archive . Close () c , err := fuse . Mount ( mountpoint ) if err != nil { return err } defer c . Close () filesys := & FS { archive : & archive . Reader , } if err := fs . Serve ( c , filesys ); err != nil { return err } // check if the mount process has an error to report <- c . Ready if err := c . MountError ; err != nil { return err } return nil }

Filesystem

On to the actual file system. We just hold a pointer to the zip archive:

1 2 3 type FS struct { archive * zip . Reader }

And we need to provide the Root method:

1 2 3 4 5 6 7 8 var _ fs . FS = ( * FS )( nil ) func ( f * FS ) Root () ( fs . Node , fuse . Error ) { n := & Dir { archive : f . archive , } return n , nil }

Directories

Zip files contain a list of files, but typical zip archivers include entries for the directories, with a name ending in a slash. We rely on this behavior later.

Let’s define our Dir type, and implement the mandatory Attr method. We use the *zip.File to serve directory metadata.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 type Dir struct { archive * zip . Reader // nil for the root directory, which has no entry in the zip file * zip . File } var _ fs . Node = ( * Dir )( nil ) func zipAttr ( f * zip . File ) fuse . Attr { return fuse . Attr { Size : f . UncompressedSize64 , Mode : f . Mode (), Mtime : f . ModTime (), Ctime : f . ModTime (), Crtime : f . ModTime (), } } func ( d * Dir ) Attr () fuse . Attr { if d . file == nil { // root directory return fuse . Attr { Mode : os . ModeDir | 0755 } } return zipAttr ( d . file ) }

Directory entry lookup

For our filesystem to contain anything useful, we need to be able to find entries by name. We just iterate over the zip entries, matching paths:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 var _ = fs . NodeRequestLookuper ( & Dir {}) func ( d * Dir ) Lookup ( req * fuse . LookupRequest , resp * fuse . LookupResponse , intr fs . Intr ) ( fs . Node , fuse . Error ) { path := req . Name if d . file != nil { path = d . file . Name + path } for _ , f := range d . archive . File { switch { case f . Name == path : child := & File { file : f , } return child , nil case f . Name [: len ( f . Name ) - 1 ] == path && f . Name [ len ( f . Name ) - 1 ] == '/' : child := & Dir { archive : d . archive , file : f , } return child , nil } } return nil , fuse . ENOENT }

Files

Our Lookup above returned File types when the matched entry did not end in a slash. Let’s define type File , using the same zipAttr helper as for directories:

1 2 3 4 5 6 7 8 9 type File struct { file * zip . File } var _ fs . Node = ( * File )( nil ) func ( f * File ) Attr () fuse . Attr { return zipAttr ( f . file ) }

Files are not very useful unless you can open them:

1 2 3 4 5 6 7 8 9 10 11 var _ = fs . NodeOpener ( & File {}) func ( f * File ) Open ( req * fuse . OpenRequest , resp * fuse . OpenResponse , intr fs . Intr ) ( fs . Handle , fuse . Error ) { r , err := f . file . Open () if err != nil { return nil , err } // individual entries inside a zip file are not seekable resp . Flags |= fuse . OpenNonSeekable return & FileHandle { r : r }, nil }

Handles

1 2 3 4 5 type FileHandle struct { r io . ReadCloser } var _ fs . Handle = ( * FileHandle )( nil )

We hold an “open file” inside our handle. In this case, it’s just a helper type in archive/zip , but in another filesystem this might be a *os.File , a network connection, or such. We should be careful to close them:

1 2 3 4 5 var _ fs . HandleReleaser = ( * FileHandle )( nil ) func ( fh * FileHandle ) Release ( req * fuse . ReleaseRequest , intr fs . Intr ) fuse . Error { return fh . r . Close () }

And then let’s handle actual Read operations:

1 2 3 4 5 6 7 8 9 10 11 12 var _ = fs . HandleReader ( & FileHandle {}) func ( fh * FileHandle ) Read ( req * fuse . ReadRequest , resp * fuse . ReadResponse , intr fs . Intr ) fuse . Error { // We don't actually enforce Offset to match where previous read // ended. Maybe we should, but that would mean'd we need to track // it. The kernel *should* do it for us, based on the // fuse.OpenNonSeekable flag. buf := make ([] byte , req . Size ) n , err := fh . r . Read ( buf ) resp . Data = buf [: n ] return err }

Readdir

At this point, our files are accessible by cat and such, but you need to know their names. Let’s add support for ReadDir :

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 var _ = fs . HandleReadDirer ( & Dir {}) func ( d * Dir ) ReadDir ( intr fs . Intr ) ([] fuse . Dirent , fuse . Error ) { prefix := "" if d . file != nil { prefix = d . file . Name } var res [] fuse . Dirent for _ , f := range d . archive . File { if ! strings . HasPrefix ( f . Name , prefix ) { continue } name := f . Name [ len ( prefix ):] if name == "" { // the dir itself, not a child continue } if strings . ContainsRune ( name [: len ( name ) - 1 ], '/' ) { // contains slash in the middle -> is in a deeper subdir continue } var de fuse . Dirent if name [ len ( name ) - 1 ] == '/' { // directory name = name [: len ( name ) - 1 ] de . Type = fuse . DT_Dir } de . Name = name res = append ( res , de ) } return res , nil }

Testing zipfs

Prepare a zip file:

1 2 3 4 $ mkdir -p data/buried/deep $ echo hello, world >data/greeting $ echo gold >data/buried/deep/loot $ ( cd data && zip -r -q ../archive.zip . )

Mount it:

1 2 $ mkdir mnt $ zipfs archive.zip mnt &

Lookup directory entries:

1 2 3 4 $ ls -ld mnt/greeting -rw-r--r-- 1 root root 13 Dec 11 2014 mnt/greeting $ ls -ld mnt/buried drwxr-xr-x 1 root root 0 Dec 11 2014 mnt/buried

Read file contents:

1 2 3 4 $ cat mnt/greeting hello, world $ cat mnt/buried/deep/loot gold

Readdir (the “total 0” is not correct, but that doesn’t matter):

1 2 3 4 5 6 7 $ ls -l mnt total 0 drwxr-xr-x 1 root root 0 Dec 11 2014 buried -rw-r--r-- 1 root root 13 Dec 11 2014 greeting $ ls -l mnt/buried total 0 drwxr-xr-x 1 root root 0 Dec 11 2014 deep

Unmount (for OS X, use umount mnt ):

1 $ fusermount -u mnt

That’s it! For a longer and more featureful examples to read, see https://github.com/bazillion/bolt-mount (screencast of a code walkthrough) and all of the projects importing fuse.

Resources