Background (or Why?)

I'm a homelabber with 24U rack half full of devices. Recently I accuired IBM KVM Console for cheap, and quickly got annoyed by reconnecting cables to each device I need to inteact with.

I had old KVM laying around - it had 4 USB + DVI-I ports. After some digging (and disassembly) I found that those should be really DVI-D as inside there's HDMI switch... So quite useless with VGA monitor and all VGA devices in rack.

After researching local classifieds and auctions on our ebay-eqiv. (Allegro) I stumbled across cheap old HP, Dell and IBM KVM Switches, that reminded me one I seen some time ago on local market. With some research I found them all to be made by Avocent, well-known manufacturer of IP-enabled PDUs, KVMs, etc.

Unfortunately this research led me into two important things: First, IP interface uses Java (obviously...), second (and more worrying) those old KVMs has three passwords: console (serial), remote (ip) and for OSCAR, overlay interface for local console. While there are ways to reset first two, last one requires sending device back to manufacturer. I needed mostly local KVM (IP feature would be only a nice addition), but this looked like an gamble, even for $10-$20 equivalent for single device.

Later I found Dell 1082DS which is the subject of this post. It's still an Avocent product, but it's built on newer platform - local interface was visually identical to remote one, so I suspected it runs some kind of embedded *nix device with web browser to access it locally. And there was a seller with asking price of about $25 for a piece in "fully tested and working condition", so in worst case I'd return it, right? And with the right cable those 2nd generation producs can even emulate USB Mass Storage devices, so I really wanted to check them out.

Those KVM switches uses special cables (called SIP in documentation). Those happend to be very expensive in Poland, but I found seller that offered IBM PS/2 SIPs for $1/piece, so I took a chance. Before I even received KVM, I found some HP SIPs on local flea market. Also for $1/piece so I aquired some of those too.

While docs for each of manufacturers mentions only it's own and Avocent cables as compatibile, considering how much I paid for cables - worst case scenario was that I'll return Dell KVM and hunt for IBM/HP one.

Initial disappointment

1082DS had arrived a couple of days later. A few minutes for powerup, console looks OK, first signs of X11 and GTK widgets are here (as suspected). So it's time to check cables, right? So I plugged IBM one, nothing happend. Second - nothing. HP - ditto.

Digging through options I found that they are visible in Tools -> Diagnostics. In section called suspect devices. Sigh. Can't be that easy.

Quick firmware analysis

As I seen obvious signs of Linux here, I decided to check firmware images. Dell has updates available with .fl extension. HP happens to have almost identical KVM (only visible difference is two PSUs integrated) known as 1X1EX8 KVM IP CONSOLE SWITCH G2, so I grabbed similar file from HP website.

Quick binwalk later I found uboot and linux squashfs embedded inside both files. And some filesystem-digging later, I had even more hopes - firmware stores most of hardware configuration in /config. Well, just look at this excerpt:

(...) #Avocent PDUs USING_AVO_SPC = yes USING_CYCLADES_SPC = yes # must say yes to USING_CYCLADES_SPC if using the following USING_CYCLADES_REBRANDED = no #Third-party PDUs - must use licensing scheme for Avocent branded but not necessarily for OEMs #License requirement is turn off for DELL USING_SERVERTECH_SPC = yes SERVERTECH_LICENSE_REQUIRED = yes SERVERTECH_ACCOUNT_OVERRIDE = no USING_BAYTECH_SPC = no /* Baytech is deprecated and should be removed */ BAYTECH_LICENSE_REQUIRED = no USING_APC_SPC = yes APC_LICENSE_REQUIRED = no USING_DELL_EATON = yes EATON_LICENSE_REQUIRE = no (...) # OEM: 0=AVO 1=CPQ 2=DELL 3=HP 4=IBM 5=FTS 7=BBOX 8=DSR 9=APC 10=LENOVO 11=FCL 12=LCACs # RIP: 1=PS/2 2=Sun 3=USB 7=SRL 20=USB2 21=PS2M 26=MPSRL 27=Pacer 28=Pacer2 29=Pacer2PS2 # 30=Pacer3 31=USBHS 32=USBFS rip.cascade.enabled = yes rip.interop = 0 1 0 2 0 3 0 7 0 20 0 21 0 27 0 28 0 30 0 31 0 32 8 1 8 2 8 3 8 7 8 20 8 21 8 26 8 27 8 28 8 30 8 31 8 32 user.accounts.enabled = yes vmedia.allowed = yes vmedia.enabled = yes vmedia.locked = no vmedia.reserved.allowed = yes vmedia.write.allowed = yes (...)

This is just a part of /config/app.cfg file in latest firmware for Dell. They were so nice to even leave comments there. They were missing in 1.16.0 that I had in both my KVMs, so they got added later. Anyway, what was seen, cannot be unseen :)

Of course I included rip.interop in this excerpt for a reason. This is an obvious indication of thing we want to change. And comparing it with HP and APC config (they left APC in Dell firmware too), the most important question was: how to modify it? It's obviously in squashfs, there's probably some kind of checksum on firmware and so on...

But when there's linux, there's probably serial console somewhere, right? So let's start with this and see if I can find any way to modify settings on running device.

What's inside?

Of course this meant I have to find serial connection. 1082DS includes 4 external serial ports - two to communicate with managed PDUs, one for local serial management and one for modem (out-of-bands). I don't have cables or pinouts for those so I had to open device anyway.

There's 4 pin header that grabbed my attention, as it usually indicates debug UART. Not this time! While it was indeed right connector, this is full-blown RS-232 levels serial communication. Quick probing with osciloscope later - Pin 1 is TX, 2 for GND, 4 for RX. So I dug my PL2032 serial cable instead of usual USB-TTLs, powered the device, and...