I experienced an error in my ZFS machine. This may have happend after I lost power to my server, but I can not tell for sure. My ZFS Pool Gewuerzglas with RAIDZ-1 is no longer willing to import. I always get the error when trying to import:

cannot import 'Gewuerzglas': I/O error Destroy and re-create the pool from a backup source.

I already tried several things. Non seemed to work. Any suggestens how to rescue this pool? As all the drives are still online it seems to me that the data might be still there but some check sums gone bad?

What I have tried so far

root@openmediavault:~# zpool import pool: Gewuerzglas id: 15011586312885837941 state: ONLINE action: The pool can be imported using its name or numeric identifier. config: Gewuerzglas ONLINE raidz1-0 ONLINE sda ONLINE sde ONLINE sdj ONLINE sdc ONLINE sdg ONLINE sdf ONLINE sdi ONLINE sdh ONLINE sdd ONLINE

root@openmediavault:~# zpool import Gewuerzglas cannot import 'Gewuerzglas': I/O error Destroy and re-create the pool from a backup source. root@openmediavault:~# zpool import -f Gewuerzglas cannot import 'Gewuerzglas': I/O error Destroy and re-create the pool from a backup source. root@openmediavault:~# zpool import -f -F Gewuerzglas cannot import 'Gewuerzglas': I/O error Destroy and re-create the pool from a backup source. root@openmediavault:~# zpool import -f -F -X Gewuerzglas cannot import 'Gewuerzglas': I/O error Destroy and re-create the pool from a backup source. root@openmediavault:~# zpool import -m -f -o readonly=on Gewuerzglas cannot import 'Gewuerzglas': I/O error Destroy and re-create the pool from a backup source.

root@openmediavault:~# zpool import -f -F -n Gewuerzglas

Does not show anything

I could also make this screenshot when I connected a monitor to my server

Anyone knows how to repair or rescue data?

System information

Distribution Name | Openmediavault Distribution Version | 3.0.59 Linux Kernel | Linux 3.16.0-4-amd64 Architecture | Debian ZFS Version | version: 0.6.5.8-2~bpo8+1 SPL Version | version: 0.6.5.8-2~bpo8+2

fdisk -l output:

root@openmediavault:~# fdisk -l Disk /dev/sdb: 55,9 GiB, 60021399040 bytes, 117229295 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x3fb31afd Device Boot Start End Sectors Size Id Type /dev/sdb1 * 63 112084897 112084835 53,5G 83 Linux /dev/sdb2 112084898 117212193 5127296 2,5G 5 Extended /dev/sdb5 112084900 113466426 1381527 674,6M 82 Linux swap / Solaris Disk /dev/sda: 1,8 TiB, 2000398934016 bytes, 3907029168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: B9434DE2-9E3A-504E-B242-B789190FA040 Device Start End Sectors Size Type /dev/sda1 2048 3907012607 3907010560 1,8T Solaris /usr & Apple ZFS /dev/sda9 3907012608 3907028991 16384 8M Solaris reserved 1 Disk /dev/sdc: 1,8 TiB, 2000398934016 bytes, 3907029168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: DC494CB2-6208-6B4E-ACB9-190924BCBC70 Device Start End Sectors Size Type /dev/sdc1 2048 3907012607 3907010560 1,8T Solaris /usr & Apple ZFS /dev/sdc9 3907012608 3907028991 16384 8M Solaris reserved 1 Disk /dev/sdd: 1,8 TiB, 2000398934016 bytes, 3907029168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 0A7AAF08-3D15-5747-8188-8121A00A70C9 Device Start End Sectors Size Type /dev/sdd1 2048 3907012607 3907010560 1,8T Solaris /usr & Apple ZFS /dev/sdd9 3907012608 3907028991 16384 8M Solaris reserved 1 Disk /dev/sde: 1,8 TiB, 2000398934016 bytes, 3907029168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: DDF59778-5338-C543-B96C-D5D151C953F4 Device Start End Sectors Size Type /dev/sde1 2048 3907012607 3907010560 1,8T Solaris /usr & Apple ZFS /dev/sde9 3907012608 3907028991 16384 8M Solaris reserved 1 Disk /dev/sdg: 1,8 TiB, 2000398934016 bytes, 3907029168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: F4B63B28-F391-B14E-B9B9-A6CE0F4467E9 Device Start End Sectors Size Type /dev/sdg1 2048 3907012607 3907010560 1,8T Solaris /usr & Apple ZFS /dev/sdg9 3907012608 3907028991 16384 8M Solaris reserved 1 Disk /dev/sdi: 1,8 TiB, 2000398934016 bytes, 3907029168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: E367E0F7-C817-354D-AC4A-B730F9FAAB19 Device Start End Sectors Size Type /dev/sdi1 2048 3907012607 3907010560 1,8T Solaris /usr & Apple ZFS /dev/sdi9 3907012608 3907028991 16384 8M Solaris reserved 1 Disk /dev/sdj: 1,8 TiB, 2000398934016 bytes, 3907029168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 93A033FA-229E-234B-8AF1-C9869F3D05FF Device Start End Sectors Size Type /dev/sdj1 2048 3907012607 3907010560 1,8T Solaris /usr & Apple ZFS /dev/sdj9 3907012608 3907028991 16384 8M Solaris reserved 1 Disk /dev/sdh: 1,8 TiB, 2000398934016 bytes, 3907029168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 18213F37-6CEC-BF48-AC84-E6434144BAFE Device Start End Sectors Size Type /dev/sdh1 2048 3907012607 3907010560 1,8T Solaris /usr & Apple ZFS /dev/sdh9 3907012608 3907028991 16384 8M Solaris reserved 1 Disk /dev/sdf: 1,8 TiB, 2000398934016 bytes, 3907029168 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 03F63180-AF94-E442-8758-66F78148FA82 Device Start End Sectors Size Type /dev/sdf1 2048 3907012607 3907010560 1,8T Solaris /usr & Apple ZFS /dev/sdf9 3907012608 3907028991 16384 8M Solaris reserved 1

lsscsi output

root@openmediavault:~# lsscsi [0:0:0:0] disk ATA HUA722020ALA330 A34H /dev/sda [2:0:0:0] disk ATA OCZ-VERTEX2 1.11 /dev/sdb [6:0:0:0] disk ATA Hitachi HUA72302 A840 /dev/sdc [6:0:1:0] disk ATA Hitachi HUA72302 A840 /dev/sdd [6:0:2:0] disk ATA Hitachi HUA72302 A840 /dev/sde [6:0:3:0] disk ATA SAMSUNG HD204UI 0001 /dev/sdf [6:0:4:0] disk ATA Hitachi HUA72302 A840 /dev/sdg [6:0:5:0] disk ATA ST32000542AS CC34 /dev/sdh [6:0:6:0] disk ATA Hitachi HUA72302 A840 /dev/sdi [6:0:7:0] disk ATA Hitachi HUA72302 A840 /dev/sdj

lsscsi -c output

root@openmediavault:~# lsscsi -c Attached devices: Host: scsi0 Channel: 00 Target: 00 Lun: 00 Vendor: ATA Model: HUA722020ALA330 Rev: A34H Type: Direct-Access ANSI SCSI revision: 05 Host: scsi2 Channel: 00 Target: 00 Lun: 00 Vendor: ATA Model: OCZ-VERTEX2 Rev: 1.11 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi6 Channel: 00 Target: 00 Lun: 00 Vendor: ATA Model: Hitachi HUA72302 Rev: A840 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi6 Channel: 00 Target: 01 Lun: 00 Vendor: ATA Model: Hitachi HUA72302 Rev: A840 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi6 Channel: 00 Target: 02 Lun: 00 Vendor: ATA Model: Hitachi HUA72302 Rev: A840 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi6 Channel: 00 Target: 03 Lun: 00 Vendor: ATA Model: SAMSUNG HD204UI Rev: 0001 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi6 Channel: 00 Target: 04 Lun: 00 Vendor: ATA Model: Hitachi HUA72302 Rev: A840 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi6 Channel: 00 Target: 05 Lun: 00 Vendor: ATA Model: ST32000542AS Rev: CC34 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi6 Channel: 00 Target: 06 Lun: 00 Vendor: ATA Model: Hitachi HUA72302 Rev: A840 Type: Direct-Access ANSI SCSI revision: 05 Host: scsi6 Channel: 00 Target: 07 Lun: 00 Vendor: ATA Model: Hitachi HUA72302 Rev: A840 Type: Direct-Access ANSI SCSI revision: 05

I somehow managed to import the pool. But unfortunaly it is still not working. I get different status from zpool.

root@openmediavault:~# zpool status pool: Gewuerzglas state: UNAVAIL status: One or more devices could not be used because the label is missing or invalid. There are insufficient replicas for the pool to continue functioning. action: Destroy and re-create the pool from a backup source. see: http://zfsonlinux.org/msg/ZFS-8000-5E scan: none requested config: NAME STATE READ WRITE CKSUM Gewuerzglas UNAVAIL 0 0 0 insufficient replic as raidz1-0 UNAVAIL 0 0 0 insufficient replic as 2196863600760350153 FAULTED 0 0 0 was /dev/sda1 sde ONLINE 0 0 0 sdj ONLINE 0 0 0 sdc ONLINE 0 0 0 sdg ONLINE 0 0 0 sdf ONLINE 0 0 0 sdi ONLINE 0 0 0 sdh ONLINE 0 0 0 8554831367157426907 FAULTED 0 0 0 was /dev/sdd1

The otherone I become is

root@openmediavault:~# zpool status pool: Gewuerzglas state: UNAVAIL status: One or more devices are faulted in response to persistent errors. There are insufficient replicas for the pool to continue functioning. action: Destroy and re-create the pool from a backup source. Manually marking the device repaired using 'zpool clear' may allow some data to be recovered. scan: none requested config: NAME STATE READ WRITE CKSUM Gewuerzglas UNAVAIL 0 0 0 insufficient replicas raidz1-0 UNAVAIL 0 0 0 insufficient replicas 2196863600760350153 FAULTED 0 0 0 was /dev/sda1 sde ONLINE 0 0 0 sdj ONLINE 0 0 0 sdc ONLINE 0 0 0 sdg ONLINE 0 0 0 sdf ONLINE 0 0 0 sdi ONLINE 0 0 0 sdh ONLINE 0 0 0 8554831367157426907 FAULTED 0 0 0 was /dev/sdd1 root@openmediavault:~# zpool clear Gewuerzglas cannot clear errors for Gewuerzglas: one or more devices is currently unavailable root@openmediavault:~# blkid /dev/sdb1: UUID="883ce00c-edc5-479f-9fac-8941b0413ef8" TYPE="ext4" PARTUUID="3fb31afd-01" /dev/sdb5: UUID="f670cf30-a948-47bc-98bf-17f0626bd9de" TYPE="swap" PARTUUID="3fb31afd-05" /dev/sdc1: LABEL="Gewuerzglas" UUID="15011586312885837941" UUID_SUB="273389127680630550" TYPE="zfs_member" PARTLABEL="zfs" PARTUUID="5503a0c7-981f-e74d-b5c9-4084eb1df641" /dev/sdd1: LABEL="Gewuerzglas" UUID="15011586312885837941" UUID_SUB="2196863600760350153" TYPE="zfs_member" PARTLABEL="zfs" PARTUUID="97861472-b1ed-954a-831c-119b0d08f45f" /dev/sde1: LABEL="Gewuerzglas" UUID="15011586312885837941" UUID_SUB="737221115472686340" TYPE="zfs_member" PARTLABEL="zfs" PARTUUID="cf5d27f2-22f6-1842-b728-50707ed22be4" /dev/sdg1: LABEL="Gewuerzglas" UUID="15011586312885837941" UUID_SUB="15202153139351954887" TYPE="zfs_member" PARTLABEL="zfs" PARTUUID="20815fac-4447-a04f-8b13-25d0d839ca90" /dev/sdi1: LABEL="Gewuerzglas" UUID="15011586312885837941" UUID_SUB="10997858781946851491" TYPE="zfs_member" PARTLABEL="zfs" PARTUUID="e123a49f-67de-d941-a02e-284ee4dc2670" /dev/sdh1: LABEL="Gewuerzglas" UUID="15011586312885837941" UUID_SUB="7665980383293405333" TYPE="zfs_member" PARTLABEL="zfs" PARTUUID="0cb1d5df-2c3d-db46-a8c5-da8cdeff94b0" /dev/sdj1: LABEL="Gewuerzglas" UUID="15011586312885837941" UUID_SUB="12174557459640099897" TYPE="zfs_member" PARTLABEL="zfs" PARTUUID="a949ef49-83d0-884d-828e-8b7cee1c8545" /dev/sdf1: LABEL="Gewuerzglas" UUID="15011586312885837941" UUID_SUB="5962312056749847786" TYPE="zfs_member" PARTLABEL="zfs" PARTUUID="840ace6e-d262-8e48-a3a5-a50ae7e3472d" /dev/sdk1: LABEL="Gewuerzglas" UUID="15011586312885837941" UUID_SUB="8554831367157426907" TYPE="zfs_member" PARTLABEL="zfs-a70471c39f62b609" PARTUUID="2ccfee22-e90e-8949-8971-f581e13ff243" /dev/sdc9: PARTUUID="b7e6e29d-2862-bf4a-8346-ef5566fca891" /dev/sdd9: PARTUUID="c1ebbcb0-a458-0e47-8f9f-2f9b8bc67165" /dev/sde9: PARTUUID="2e134ff0-1290-8c40-b50d-10b4fb022c64" /dev/sdg9: PARTUUID="6cd1a402-0714-5a4a-acf2-9097314401f3" /dev/sdi9: PARTUUID="739d94f8-8eb2-444c-99ff-65b6fb7e48a7" /dev/sdh9: PARTUUID="d2f5af40-ba13-7947-a46a-23aa1b00ea8e" /dev/sdj9: PARTUUID="840de1b5-f24d-9446-9471-bca8d4aff777" /dev/sdf9: PARTUUID="72b593e7-e390-8540-bba0-15469cd74d96" /dev/sdk9: PARTUUID="a8ba619d-c59e-be41-a1bc-82c81ffc01b5"

I don't get why it is telling me that these drives are currently unavailable as they seem not to. I can see them and I just did long smart check with all the drives.