tech-kern archive

Patches for journalling support

To : tech-kern%NetBSD.org@localhost

: Subject : Patches for journalling support

: From : Simon Burge <simonb%wasabisystems.com@localhost>

: Date: Sun, 2 Mar 2008 23:05:23 +1100

Hi folks, Wasabi Systems Inc is pleased to make our journalling code available to the NetBSD community. This code is known as WAPBL - Write Ahead Physical Block Logging, and has been used to provide meta-data journalling in production environments for over 4 years. WAPBL journals meta-data only - not file data, and has been used on filesystems ranging from from 16MB to multiple terabytes in size. WAPBL was originally written by Darrin B. Jewell while at Wasabi. Many thanks to Antti Kantee, Andy Doran and Greg Oster for their work in adapting our journalling code to NetBSD -current (our contribution was based on the netbsd-4 branch) and testing. WAPBL is enabled in the kernel with: options WAPBL Currently WAPBL uses space after the filesystem (but before the partition's end) for its journal. The suggested method is to use: newfs -s -64m <other options> /dev/<foo> to leave 64MB of space for the journal. Obviously this will destroy any existing data in the filesystem on /dev/<foo> and you should, of course, back up any data on this filesystem first. The journal size itself is still the subject of some research. Solaris uses "10% of filesystem size to a maximum of 64MB", and at Wasabi we've always used a 64MB journal on larger filesystems. As well as the kernel, the following binaries need to be updated: /sbin/fsck /sbin/fsck_ffs /sbin/mount /sbin/mount_ffs The easiest way to do this is to just rebuild using build.sh and reinstall either base.tgz or the above binaries. There are three known issues: - "mount -u -o log" doesn't work yet, so you can't have logging enabled on the root filesystem. "mount -u -o log" is disabled in this patch (see around line 222 of sys/kern/vfs_syscalls.c). - Under some very heavy workloads there seems to be some sort of bottleneck. 20 parallel extracts of a src.tar.gz (a tar of the NetBSD source tree) on a quad-core machine has triggered this problem, but we can't reproduce it on a dual-core machine. Using "options WAPBL_DEBUG_SERIALIZE" seems to be a workaround. - The log itself must currently be at the end of the partition (see the newfs command above). We have plans to implement a log that lives in the filesystem itself and is allocated at mount time. The patches are available at: http://www.NetBSD.org/~simonb/wapbl-current-20080302.diff Note that even though the basis of these patches have been in production use for a few years, these patches should still be considered experimental as they are against NetBSD -current and not the same source code base we use at Wasabi. Pending serious problems, I'd like to integrate these changes in two or three weeks time, depending on feedback. On behalf of Wasabi Systems, Simon. -- Simon Burge <simonb%wasabisystems.com@localhost> NetBSD Development, Support and Service: http://www.wasabisystems.com/ This email and any attached documents may be confidential and property of Wasabi Systems. If you are not the intended recipient, you may not disclose, copy, distribute, or act in reliance on the information in this email.