Introduction

I’ve been doing some work on my homelab that I haven’t documented here.

I recently decommissioned my Raspberry Pi 3 that was running my Unifi controller, Dokuwiki, and Network UPS Tools (NUT). I replaced the RPi3 with an Intel i3 NUC with 12GB RAM and a Crucial 2.5″ SSD. I chose to use Proxmox as the hypervisor on the NUC because it is open source, has a well-proven record, and has a low learning curve. At the time of this writing, I’m running five KVM virtual machines:

mgmt01 – Jump server and Ansible master

unifi01 – Unifi controller

nginx01 – LEMP stack, running Dokuwiki and NUT

vpn01 – Pritunl

log01 – Graylog

However, the one thing I’m missing is a backup server.

3-2-1 backup

The golden rule is to follow the 3-2-1 method for backups:

3 backups

2 different types of media

1 backup offsite

Currently, I’m following the 3-2-1 backup method, except the central location is my desktop PC. This is both inconvenient and a waste of space on my desktop. Ideally, I’d like have all the backups go to a NAS-type device, and have that device push the backups to a cloud provider (I already use Backblaze B2).

Devices

I looked at quite a few options, from single-board devices to full NAS devices, but ruled most of them out.

Device Pro Con Raspberry Pi 3 Model B+ Cheap

Small form-factor

Large community and great aftermarket support CPU/memory is limited

Fastest storage connection is USB2.0

Gigabit ethernet, but bound to USB bus, so max throughput is 300Mbps Any Raspberry Pi clone (e.g., BananaPi, OrangePi, NanoPi, etc…) Cheap

Small form-factor Typically isn’t in mainline kernel Any x86 single-board device (e.g., Udoo, UP board, Minnowboard, etc…) Small form-factor

SATA3 and USB3.0

x86 architecture Small community and limited aftermarket support

Expensive for what the devices offer ODROID C2 Cheap

Small form-factor

Gigabit ethernet Fastest storage connection is USB2.0 ODROID XU4 Cheap

Small form-factor

Gigabit ethernet

USB3.0 No SATA ports ESPRESSObin Cheap

Small form-factor

Gigabit ethernet

SATA3 and USB3.0 Relatively new

Small community and limited aftermarket support Helios4 NAS form-factor

Gigabit ethernet

SATA3 and USB3.0 Relatively new

Small community and limited aftermarket support

For the same price, I could get a NAS from Synology or QNAP GnuBee PC2 NAS form-factor

Gigabit ethernet

SATA3 and USB3.0 CPU/memory is limited

Relatively new

Small community and limited aftermarket support

For the same price, I could get a NAS from Synology or QNAP An actual 2-bay NAS (e.g., Synology, QNAP, etc…) NAS form-factor

Gigabit ethernet

SATA3 and USB3.0

Official support and updates CPU/memory is limited

Expensive

In the end, I decided on the ODROID-HC2. The HC2 is the larger version of the HC1 (the HC1 holds 2.5″ drives, while the HC2 holds 3.5″ drives).

Both the HC2 and HC1 are based on the XU4, but lack the XU4’s HDMI port, USB 3.0 ports, eMMC slot, and GPIO connectors. The HC2 features:

Samsung Exynos 5422 eight-core CPU

2GB DDR3 RAM

USB3.0-based SATA3 port

Gigabit ethernet (Realtek chipset)

USB2.0 port

MicroSD slot

UART for serial console

Stackable aluminum frame that acts as a heatsink and drive cage

I purchased the HC2 from ameriDroid, along with a power cable and clear top cover. I also picked up a WD Red 4TB 5400 RPM hard drive. The Red drives are WD’s NAS drives, and the 5400 RPM speed will keep the unit quiet.

I chose the HC2, which is only a single-drive device, because I’m not looking for a 2-bay NAS with RAID support. The primary purpose of this device will be to backup all my other devices, then ship those backups offsite. If the drive dies, I’ll put a new drive in and the only thing I’ll have lost is old backups (locally, anyways).

Software

The HC2 has wide support for Linux, including an Ubuntu image from ODROID, Armbian, Arch Linux Arm, and OpenMediaVault. From these, I decided to go with OpenMediaVault (OMV). OMV is a NAS solution based on (at the time of this writing) Debian 9 (Stretch). It runs services like SSH, FTP, SFTP, SMB/CIFS, NFS, rsync and many more. It also features a nice web interface, S.M.A.R.T. monitoring, and a plugin system to enhance functionality (e.g., LDAP, iSCSI, etc…).

Currently, OMV3 is being retired, to be replaced with OMV4. However, OMV4 is still being tested for ARM boards. Because of this, you can install OMV4 one of three ways:

Install Armbian first, then install OMV on top of Armbian

Install OMV3, then upgrade to OMV4

Install the current OMV4 test image

For this install, I chose to use the OMV4 test image.

Installation

The installation of OMV4 was pretty straight-forward, and there are plenty of guides for the XU4 that apply to the HC1 and HC2 (e.g., here, here, and here). You can also use any guide for OMV (e.g., here, here, and here) and take just the parts you need. I’m going to document what worked for me and what I enabled, but you can enable/disable whatever services you like.

Writing the MicroSD card

First, download the newest OMV4 image for the XU4/HC1/HC2 from here. You’ll need to extract the .img file from the .xz file, then pipe that through dd to write to your MicroSD card (I’m assuming you’re running Linux here).

sudo unxz OMV_4_Odroid_XU4_HC1_HC2.img.xz sudo dd status=progress bs=4M if=OMV_4_Odroid_XU4_HC1_HC2.img of=/dev/sdX sudo sync

First boot

When you boot the HC2 for the very first time, expect it to take 30 minutes or so. Since this server is headless (i.e., it has no graphics output), you won’t have the option to watch a screen to verify its state. Wait for the server to ping, then head to the web interface. By default, the web interface username/password is admin/openmediavault.

Navigate to System, then Date & Time, set your timezone, and enable NTP.

Navigate to System, then Update Management. Click Check at the top of the page to check for available updates.

Check the box next to each package, then click Upgrade at the top of the page to update the package(s).

Once completed, reboot the HC2 from the menu at the top-right of the page.

Enable HTTPS

Navigate to System, then Certificates, then select the SSL tab, click Add, then click Create. Fill in the information as necessary.

Navigate to System, then General Settings. In the Secure connection section, enable SSL/TLS and select the certificate you created. You will need to open a new tab using HTTPS instead of HTTP.

Change default web admin password

Navigate to System, then General Settings, then select the Web Administrator Password tab to update the password.

Enable SSH

I do most of my updating and management through Ansible, which requires SSH access.

Navigate to Services, then SSH, and enable the SSH service. Also, enable Permit root login.

Change default root password

By default, the root username/password is root/openmediavault.

Now that SSH is enabled, you can SSH to the server as root. Upon login, you should be prompted to change root’s password

ssh root@IP_goes_here

Once you’ve changed the password, logout.

exit

Disable SSH for root

Navigate to Services, then SSH, and disable Permit root login.

Add a secondary user

Navigate to Access Rights Management, then User. Click Add, name your user, and set a password.

Enable S.M.A.R.T

Navigate to Storage, then S.M.A.R.T., then select the Settings tab, and enable S.M.A.R.T. monitoring.

Navigate to the Devices tab, select your device and click Edit at the top of the page, then enable S.M.A.R.T. monitoring.

Drive setup

Physical setup

Navigate to Storage, then Disks. Select your hard drive (e.g., /dev/sda) and click Wipe at the top of the page.

Setup LVM – SKIP THIS STEP

I tried to setup LVM so that I could have one encrypted partition, and one unencrypted partition. I was able to get it working, however, every reboot would break the setup. Specifically, under Filesystems, the filesystem would be marked Missing after a reboot. Checking dmesg, I would see the drive was offline with the error Medium access timeout failure. Offlining disk!.

root@backup01:~# dmesg | grep sda [ 14.588107] sd 0:0:0:0: [sda] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB) [ 14.588119] sd 0:0:0:0: [sda] 4096-byte physical blocks [ 14.589011] sd 0:0:0:0: [sda] Write Protect is off [ 14.589025] sd 0:0:0:0: [sda] Mode Sense: 53 00 00 08 [ 14.589455] sd 0:0:0:0: [sda] Disabling FUA [ 14.589466] sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA [ 14.603406] sd 0:0:0:0: [sda] Attached SCSI disk [ 14.868930] device-mapper: table: 254:0: adding target device sda caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=33553920 [ 14.868939] device-mapper: table: 254:0: adding target device sda caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=33553920 [ 14.870900] device-mapper: table: 254:1: adding target device sda caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=2194476629504 [ 14.870908] device-mapper: table: 254:1: adding target device sda caused an alignment inconsistency: physical_block_size=4096, logical_block_size=512, alignment_offset=0, start=2194476629504 [ 30.572176] sd 0:0:0:0: [sda] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD IN [ 30.572231] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x85 85 08 0e 00 00 00 01 00 00 00 00 00 00 40 ec 00 [ 45.597301] sd 0:0:0:0: [sda] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD IN [ 45.597352] sd 0:0:0:0: [sda] tag#1 CDB: opcode=0x88 88 00 00 00 00 00 ff 78 7f 7f 00 00 00 08 00 00 [ 61.612177] sd 0:0:0:0: [sda] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD IN [ 61.612228] sd 0:0:0:0: [sda] tag#1 CDB: opcode=0x85 85 08 0e 00 00 00 01 00 00 00 00 00 00 40 a1 00 [ 76.637308] sd 0:0:0:0: [sda] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD IN [ 76.637361] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x88 88 00 00 00 00 00 ff 78 7f 7f 00 00 00 08 00 00 [ 76.786240] sd 0:0:0:0: [sda] tag#0 Medium access timeout failure. Offlining disk! [ 76.786522] sd 0:0:0:0: [sda] killing request [ 76.786690] sd 0:0:0:0: [sda] UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=0x00 [ 76.786787] sd 0:0:0:0: [sda] CDB: opcode=0x88 88 00 00 00 00 00 ff 78 7f 7f 00 00 00 08 00 00 [ 76.786834] blk_update_request: I/O error, dev sda, sector 4286087039 [ 76.859645] blk_update_request: I/O error, dev sda, sector 0 [ 1462.638016] blk_update_request: I/O error, dev sda, sector 0 [ 1462.640632] blk_update_request: I/O error, dev sda, sector 0 [ 1462.655726] blk_update_request: I/O error, dev sda, sector 0 [ 1462.674249] blk_update_request: I/O error, dev sda, sector 0 [ 1462.678512] blk_update_request: I/O error, dev sda, sector 0 [ 1462.697718] blk_update_request: I/O error, dev sda, sector 0 [ 2675.814067] blk_update_request: I/O error, dev sda, sector 0 [ 2698.300818] blk_update_request: I/O error, dev sda, sector 0

I could verify this by checking the file below.

cat /sys/block/sda/device/state

The only way to bring the device back online was to echo running to that file. In fact, this is the official “fix” according to RedHat.

echo running > /sys/block/sda/device/state

Upon reboot, the device would go offline again. Because of this, I recommend skipping the LVM setup and moving straight onto creating one large filesystem.

Instead of having shares that are at a fixed size, I’m going to use Logical Volume Manager (LVM) to create smaller volumes that I can extend/shrink as needed. If you want to skip this setup, you can move on to Format and mount volumes.

Navigate to System, then Plugins. From the search box, search for lvm. Select openmediavault-lvm2 and click Install at the top of the page.

For LVM to work, we need to create three types of groups

a physical volume

a volume group

one or more logical volumes

Navigate to Storage, then Logical Volume Management.

Navigate to the Physical volumes tab, click Add, and select your primary drive.

Navigate to the Volume groups tab, click Add, name your volume group, and select the physical volume you just created.

Navigate to the Logical volumes tab, click Add, name your logical volume, select the volume group you just created, and choose a partition size.

Format and mount volumes

Navigate to Storage, then File System, then click Create. Select the logical volume you just created, then choose a name and filesystem type (I’m using XFS).

Once the filesystem is created, select your new filesystem and click Mount at the top of the page.

Create shared folder

Navigate to Access Rights Management, then Shared Folders. Click Add, name your folder, and select the filesystem we created earlier.

Enable SMB/CIFS

I have Windows, Linux, and Mac clients on my network. Because of this, I’m choosing to use Server Message Block (SMB) to share files.

Navigate to Services, then SMB/CIFS. On the Settings tab, enable the SMB service.

Navigate to the Shares tab, click Add, then enable the share and select the shared folder we created earlier.

Grant privileges

Navigate to Access Rights Management, then Shared Folders, select your shared folder, and click Privileges. Here, grant Read/Write privileges to the user you created earlier.

Client setup

I won’t cover how to setup every single type of client. In this case, Google “how to connect to smb from YOUR DEVICE HERE“.

Performance

As you can see below, the performance is close to 1Gbps, which means the HC2 is able to almost saturate the network connection. Because of this, I don’t believe any of the tweaks suggested here or here apply. Specifically, the CPU governor is set to ondemand by default (shown in the command below) and since we’re using XFS, the NTFS tweaks don’t apply.

sudo grep GOVERNOR /etc/default/openmediavault

Write performance

Here, I’m copying a 1GB file from a Windows 10 laptop to OMV4 over wired ethernet.

Here is the same test, but with a 10GB file.

Read performance

Here, I’m copying a 1GB file from a OMV4 to aWindows 10 laptop over wired ethernet.

Here is the same test, but with a 10GB file.