I have a ZFS based iSCSI SAN that serves ZVOLs to a bunch of VM servers. Today a network glitch caused all the iSCSI volumes mounted on the clients to go RO. The only way out of that is to shut them all down and restart, often running fsck to get the iSCSI volumes back on-line. Well, fsck decided to thoroughly destroy one of the volumes. So, it does not look like I'm going to be able to fix the mess fsck made.

I've read quite a bit about recovering files on ZFS, however, in this case I'm dealing with ZVOLs, which in a sense is a lot simpler, but I haven't seen anything dealing with trying to roll back the contents of a block device. Any suggestions?

-TIA-

Some dataset details:

Dataset zpool1/vm3 [ZVOL], ID 59, cr_txg 12078, 44.6G, 2 objects, rootbp DVA[0]=<6:6c2c4b1e00:200> DVA[1]=<7:487aa4b200:200> [L0 DMU objset] fletcher4 lz4 LE contiguous unique double size=800L/200P birth=7736596L/7736596P fill=2 cksum=4c78779ec:2049fb2de6c:6f2f6c4a44e9:1042484aee3ded Deadlist: 1K (512/512 comp) mintxg 0 -> obj 48 mintxg 1 -> obj 4157 Object lvl iblk dblk dsize lsize %full type 0 7 16K 16K 7.00K 16K 6.25 DMU dnode dnode flags: USED_BYTES dnode maxblkid: 0 Object lvl iblk dblk dsize lsize %full type 1 5 16K 8K 44.6G 200G 36.45 zvol object dnode flags: USED_BYTES dnode maxblkid: 26214399 Object lvl iblk dblk dsize lsize %full type 2 1 16K 512 0 512 100.00 zvol prop dnode flags: USED_BYTES dnode maxblkid: 0 microzap: 512 bytes, 1 entries size = 214748364800

The system is CentOS 7.1

Linux san1srvp01 3.10.0-514.6.1.el7.x86_64 #1 SMP Wed Jan 18 13:06:36 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

There isn't a relevant snapshot; I figured that went without saying.

The reason that I'm asking the question, and what I'm after relates to articles like this one from Max Burning which delves into object recovery through forensic techniques. This of course relies on no shortage of knoledge of ZFS internals. Most of what I've seen though deals with walking back over file objects which is very different from the objects that implement raw block storage and I have seen next to nothing on the internals related to ZVOLs.

Even if I can't technically "roll back" the changes that fsck made it would at least be helpful to walk back and find some of the key original blocks. This should be possible given ZFS's COW behavior... and sufficient knoledge which I lack, but I don't usually let that stop me.