Contributed by jcr on 2010-04-04 from the keep-it-simple dept.

On Tue, 30 Mar 2010 17:13:55 -0500 Ron McDowell wrote: > Hi J.C., anytime anyone has posted to misc@ asking about how to build > a bootable install image on a USB stick, there is much vitriol and > little information, so I'm asking privately. > > I've got a DOS formatted 2GB USB stick seen as sd0, and > a /usr/release directory full of stuff made by 'make release': > > In the past I have used unetbootin from an XP box to build one from > installXX.iso, but others on the list claim it's so easy to do... but > never tell how. > > Frustrating. >

Thanks Ron for giving me permission to put this up on undeadly.

When your goal is to create USB Flash Stick install media, the typical terse response of, "Just do a normal installation," can be real frustrating without further explanation. Part of the frustration is being told to make other install media (floppy/CD) to make the desired install media (usb flash). The trouble is, you're essentially looking to create a problem where none exists, and worse, a detailed explanation like the one below might scare people away rather than show them how simple the suggested way of doing it is. When you combine this with the normal standing against blind "How-To" recipes, we'll be spending time going into the torrid details without gaining much of anything.

Let's start with the suggested way to create a bootable USB flash stick as stated in FAQ.14.

Do a normal installation of OpenBSD to your flash stick.

Copy the install set files to the device.

Copy bsd.rd to the root partition.

The result is you can both run the operating system from the stick and install the operating system from the stick. To do an installation, at the "boot>" prompt type in "bsd.rd" and then do the installation normally. As suggested in FAQ.14, you can copy over the install set files for both i386 and amd64 so you'll be able to install either from the same device.

There is endless debate regarding "wear leveling" issues on flash based storage devices. If you get the wild idea to avoid the "wear leveling" by doing something weird, you will probably do it wrong, and the result will be an unreliable system.

Explaining why the above is the suggested best practice is where things start to get complicated. The short answer is, a default installation process of OpenBSD contains a whole lot of know-how applied in an attempt to get things right. More importantly, if it's not right, you want to find out about problems quickly, and having a full installation of OpenBSD gives you the best chance of figuring out if your hardware is reliable.

Even a very lengthy long answer would be incomplete, so if you want to understand all the details, you are better off reading the source. Below will only cover some of the most basic aspsect of flash storage devices, and provide a less reliable answer to the qeustion of how to make bootable install media with them.

USB flash sticks are the modern equivalent of unreliable floppy diskettes. Though flash devices often work, they are not as reliable as other storage devices like hard disks. Similar can be said for Compact Flash, {mini, micro, pico, nano, femto, whatever}SD, supposedly SmartMedia, and all the rest of the related flash based storage devices, even including some of the Solid State Disks (SSD) currently available.

Part of the reliability problem is caused by the high-volume, low-cost, and poor-quality manufacturing of the devices and their components. They are designed to be inexpensive and compatible with existing systems, but to achieve these goals, significant trade-offs were made. Some would even say the trade-offs resulted in very poor design decisions. The most obvious of these trade-offs is the fact flash devices are treated as "disks" by both the system and the software, when internally, they are very different from disks.

Another issue with flash storage devices is inconsistency. Since flash storage devices are put together in countless different ways, and use countless different kludges to appear as "disks" to the system, having both the system and software treat them as disks is a risky proposition. It doesn't always work, and if you test enough devices, you'll find some combinations of systems, software and devices which simply refuse to work.

When you're talking about hardware so cheap that people give it away as advertisements at trade shows (like 4+ GB flash sticks), the probability of it being good enough to work reliably is so close to zero it's negligible. Many of the flash sticks you own only seem to work correctly when you run fdisk(8) (MBR partition table partitions), disklabel(8) (filesystem partitions), newfs(8), mount(8), copy, move, delete and whatever, but often they fail miserably if you run fsck(8) on them or do any other serious file integrity checks (e.g. cksum(1)). Sadly, this is very common, and you have to be more than a bit strict to catch this kind of shoddy hardware.

The problems with flash storage devices are magnified by the way various systems boot also being horribly inconsistent within a single architecture. Of course, this is magnified yet again due to the differences across multiple architectures.

Here's a classic example. If you take one of the floppy diskette installation images like "floppy47.fs" and mindlessly dd(1) the floppy image onto a USB flash stick, there are at least a few systems which actually can still boot from the resulting monstrosity. The resulting USB flash stick has no fdisk partition table (MBR), but since some rare systems are built to use USB-based floppy diskette drives (e.g. some brands of laptops), they mistake the USB flash stick as being a floppy and continue to boot from it.

In this very messy situation, all possible "best methods" are bad, and suggesting anything is unwise. The FAQ makes the suggestion it does only because it's your best chance at getting things working in a reliable fashion.

If you insist on shooting yourself in the foot through manual attempts to make bootable flash sticks, below is the target practice yours truly did on his own feet.

First you want to read fdisk(8) and FAQ.14, the set up your fdisk partition table and MBR

# fdisk -iy sd0

Writing MBR at offset 0.

Next, you want to read disklabel(8) and read FAQ.14 again. You'll want to delete all the existing disklabel partitions and create a new "a" partition accepting the defaults so it covers the whole disk. By default, the first "track" is skipped (offset: [63]), even though tacks don't actually exist on flash based storage.

# disklabel -E sd0 Label editor (enter '?' for help at any prompt) > p m OpenBSD area: 63-4160835; size: 2031.6M; free: 0.0M # size offset fstype [fsize bsize cpg] c: 2035.0M 0 unused i: 2031.6M 63 MSDOS > d * > a a offset: [63] size: [4160772] FS type: [4.2BSD] > p m OpenBSD area: 63-4160835; size: 2031.6M; free: 0.0M # size offset fstype [fsize bsize cpg] a: 2031.6M 63 4.2BSD 2048 16384 1 c: 2035.0M 0 unused > q Write new label?: [y] y

Next we toss a file system (ffs) on the new partition.

# newfs /dev/rsd0a /dev/rsd0a: 2031.6MB in 4160772 sectors of 512 bytes 11 cylinder groups of 202.47MB, 12958 blocks, 25984 inodes each super-block backups (for fsck -b #) at: 32, 414688, 829344, 1244000, 1658656, 2073312, 2487968, 2902624, 3317280, 3731936, 4146592,

And finally we do a quick test with fsck(8) to make sure things are working.

# fsck -fp /dev/rsd0a /dev/rsd0a: BAD SUPER BLOCK: VALUES IN SUPER BLOCK DISAGREE WITH THOSE IN LAST ALTERNATE /dev/rsd0a: UNEXPECTED INCONSISTENCY; RUN fsck_ffs MANUALLY.

Lovely. It seems we have a piece of junk, so unplug it and throw it away.

"But But But But But I don't want to throw it away!!!!"



Seriously, you are far better off throwing it away and spending your time on more important things like making reliable hardware work for you.

"But But But But But can't fsck fix it????"



The only fair answer is, "maybe," but this seldom works with flash devices, and even if it does seem to work at first, it may silently fail again immediately afterwards.

"But But But But But are you sure????"



Nothing is ever certain with flash storage devices, so if you insist on further target practice on your feet, you can try the following.

One of the tricks you can try is erasing the flash device entirely, but you need to realize the "erased state" for flash is when it is filled with all 1's. People regularly make the mistake of filling flash based storage devices with all zeros (as is typically done with real disks) without every realizing what they are doing.

# tr '\000' '\377' < /dev/zero | dd bs=16384 of=/dev/rsd0c dd: /dev/sd0c: end of device 130241+0 records in 130240+0 records out 2133852160 bytes transferred in 714.616 secs (2986009 bytes/sec)

The above doesn't consider what might be done to the input data of all 1's by the disk controller of the system, the usb controller of the system, or the super double secret flash controller on the device itself. None the less, it's as close as you'll get to overwriting the whole device with 1's to put all the flash in the erase state. From here, repeat the previous steps through doing your fsck test, but if it fails again, this time, just throw it away.

Assuming you were a bit more lucky than me and your device passed fsck, now it's time to make the flash stick device bootable.

# fsck -fp /dev/rsd0a /dev/rsd0a: 1 files, 1 used, 967555 free (19 frags, 120942 blocks, 0.0% fragmentation) # mount /dev/sd0a /mnt # cp /usr/mdec/boot /mnt # cp bsd.rd /mnt/bsd # /usr/mdec/installboot -v /mnt/boot /usr/mdec/biosboot sd0 boot: /mnt/boot proto: /usr/mdec/biosboot device: /dev/rsd0c /mnt/boot is 3 blocks x 16384 bytes fs block shift 2; part offset 63; inode block 24, offset 936 using MBR partition 3: type 0xA6 offset 63 # umount /mnt # fsck -fp /dev/rsd0a /dev/rsd0a: 3 files, 3047 used, 964509 free (13 frags, 120562 blocks, 0.0% fragmentation)

One of the important things to note in the above is the use of installboot(8) to make the device bootable is only correct for some architectures. Other architectures need to use disklabel(8) to properly install the first (and if needed second) stage bootstrap code. Additionally, the second stage bootstrap loader, boot(8), /usr/mdec/boot file in the example above can be arch specific. Although I'm yet to see the i386 boot(8) fail on amd64, mixing and matching other archs will most likely make a real mess.

The minimal install set for i386 (from a snapshot or release) would be:

# l files/ total 500060 drwxr-xr-x 2 root wheel 512 Mar 31 03:04:36 2010 ./ drwxr-xr-x 3 root wheel 512 Mar 30 15:48:38 2010 ../ -rw-r--r-- 1 root wheel 2162 Mar 30 15:48:52 2010 SHA256 -rw-r--r-- 1 root wheel 50551104 Mar 30 15:50:08 2010 base47.tgz -rw-r--r-- 1 root wheel 7592844 Mar 30 15:49:56 2010 bsd -rw-r--r-- 1 root wheel 7616392 Mar 30 15:49:54 2010 bsd.mp -rw-r--r-- 1 root wheel 6304746 Mar 30 15:49:51 2010 bsd.rd -rw-r--r-- 1 root wheel 92648166 Mar 30 15:49:43 2010 comp47.tgz -rw-r--r-- 1 root wheel 514498 Mar 30 15:49:21 2010 etc47.tgz -rw-r--r-- 1 root wheel 2618018 Mar 30 15:49:19 2010 game47.tgz -rw-r--r-- 1 root wheel 9463948 Mar 30 15:49:17 2010 man47.tgz -rw-r--r-- 1 root wheel 364819 Mar 30 15:49:16 2010 misc47.tgz -rw-r--r-- 1 root wheel 15550317 Mar 30 15:49:12 2010 xbase47.tgz -rw-r--r-- 1 root wheel 70053 Mar 30 15:49:10 2010 xetc47.tgz -rw-r--r-- 1 root wheel 39684757 Mar 30 15:49:05 2010 xfont47.tgz -rw-r--r-- 1 root wheel 19879062 Mar 30 15:48:53 2010 xserv47.tgz -rw-r--r-- 1 root wheel 2946170 Mar 30 15:50:14 2010 xshare47.tgz

# du -h files/ 244M files/

Without getting dangerously creative with newfs, a supposed "256 MByte" flash stick will only give you roughly 235 MByte of usable space, so you'll need something larger to hold the minimum install set files. You'll want to put the install sets in the typical directory (e.g. /4.7/i386/) to keep from overwriting the kernel in / as well as make your life easy when installing from the device (e.g. the default location where the installer looks for the files).

# mount /dev/sd0a /mnt # mkdir -p /mnt/`uname -r`/`uname -m`/ # cp files/* /mnt/`uname -r`/`uname -m`/

At this point, you should test the integrity of the files you just wrote to the flash stick as well as run fsck one more time.

# cd /mnt/4.7/i386 # for i in *; do grep -e "($i)" SHA256 >> ../sha ; done; # cksum -c ../sha || echo "FAILED!!!" (SHA256) base47.tgz: OK (SHA256) bsd: OK (SHA256) bsd.mp: OK (SHA256) bsd.rd: OK (SHA256) comp47.tgz: OK (SHA256) etc47.tgz: OK (SHA256) game47.tgz: OK (SHA256) man47.tgz: OK (SHA256) misc47.tgz: OK (SHA256) xbase47.tgz: OK (SHA256) xetc47.tgz: OK (SHA256) xfont47.tgz: OK (SHA256) xserv47.tgz: OK (SHA256) xshare47.tgz: OK # cd / # umount /mnt # fsck -fp /dev/rsd0a /dev/rsd0a: 20 files, 128062 used, 839494 free (14 frags, 104935 blocks, 0.0% fragmentation)

Congratulations! You now have a supposedly bootable install media on a USB flash stick which might actually work for an unknown period of time with some, but not all, systems supposedly capable of booting some unknown percentage of USB flash devices.

Did I put enough weasel-words in the above statement to make you nervous?

The simple method suggested in the FAQ of doing a normal installation and copying over the needed install set is a lot easier, is more reilable, and gives you more functionality. You will still occasionally hit problems where particular USB flash sticks and particular systems are incompatible, but luckily, the flash devices are now cheap enough where you can just try another one.