I am not the first person to build a super-small ESXi server, and I won’t be the last. But for a few moments in history, this might be as small as you can get for an ESXi server with latest Xeon processor, 16GB of ECC memory, 100% SSD, and 2 server-class Ethernet controllers. Easily able to run 16 active VM’s.

So far (8 weeks now) it works perfectly reliably.

Two servers in my carry-on…

I started this project because I organize cyber competitions and may need to travel by plane. Pre-shipping the servers takes time, money, and foresight… and I’m too lazy and broke to do all that. Plus, baggage-handlers make me nervous, so checking my servers in my regular luggage seems like a bad idea. So the solution for me is to fit two servers in my carry-on bag. Must have room for protective padding as well. Plus the smallest toothbrush I can find.

My national airline’s limit for a carry-on is: 23 cm x 40 cm x 55 cm (9 in x 15.5 in x 21.5 in) and 10 kg (22 lb). (You also get a “personal item”, which is my laptop bag, but could fit a 3rd ESXi server in there also, if I had to. This one fits in my laptop bag’s front pocket.)

The Specs…

After way, way too many hours research, here’s the exact config I chose for this home build. Works for me, running VMware ESXi 5.1 free edition. (Shout-outs below to the many sites that helped make this possible.)

MotherBoard Intel S1200KPR $155 Mini-ITX Board, for Xeon-v2, ECC RAM, 2xGbE CPU Xeon E3-1245v2 $280 Intel Xeon 4-core/8-threads plus graphics Memory 16GB DDR3 1600 ECC $155 2 x 8GB, Unbuffered Kingston KVR16E11/8 SSD 256GB OCZ Vertex 4 $135 256GB Solid State Drive CPU Cooling Dynatron K199 $32 Dynatron K199 80mm 2 Ball CPU Cooler (Blower) Case w/power Mini-Box M350 $102 M350 Case + Pico PSU 150W + power adapter 40mm Fans x 2 $8 Case Fans 40mm (1 is a spare) $2 Mounting bracket for fan. TOTAL: $869 Plus Tax and (sometimes) Shipping

Why this motherboard?

S1200KPR is a true Intel server-class mother board. I wanted a true server for the ECC RAM (stable), Xeon processor (excellent multitasking) and, especially, server-class Ethernet controllers.

It’s a mini-ITX, so it’s smaller than the whole ATX family – even the Micro-ATX. The only drawback I hit with this form-factor is 16GB max for the RAM. I wish it allowed more, but *small* was my goal.

The KPR version of this board supports the Ivy Bridge technology and v2 Xeon processors. (Do not settle for the “S1200KP”.) It has plenty of SATA and USB ports: SATA up to 6Gbps, though USB only up to 2.0. (Currently, I don’t believe ESXi supports USB 3.0 anyway, but it would have been nice for future-proofing.)

The GbE controllers were crucial. Last time I tried to add a NIC to my i7 box, the computer store dude said “This one is popular. It costs $15”. All I could respond was “But I wanted to pay a lot more than that.” We agreed on $95 for an IntelPRO/1000GT with the 82572 chip to add to my i7 desktop.

So for this server I wanted the Ethernet controllers to be at least that powerful. Glad to see the S1200KPR comes with both an 82574L and an 82579LM controller already on-board. (There is a PCIe slot on this motherboard, but no room for add-in cards in this case. Remember… *small*.)

Who cares about server-class Ethernet?

I do. Server-class controllers offload all sorts of work from the CPU – even TCP/UDP checksums. And they support VLAN tags. I know desktop NIC’s don’t bother with VLAN tags because tagging stops at the switch. But in my VMware setup, I need to pass the tags to the vSwitches inside the server, so I really did need server-class Ethernet.

VMware 5.1 does not have a driver for the 82579. To add the driver to your ESXi installation package you’ll find great instructions here. Yes, you will get the error message. Yes, I ignored it (as recommended). I’ve had no problems. And no speed degredation on either Ethernet port (though I haven’t got a rigorous test for that.)

Why those other components?

This Xeon E3-1245-v2 is the latest generation Xeon. 3.4GHz, 4 core, 8 threads. LGA 1155 socket. Ivy Bridge. This CPU (and motherboard) support the fastest DDR3 1333/1600 ECC memory (unbuffered). They’re not as fast as the E5 Xeon’s, but much less expensive, and draws less power. Power generates heat, and heat is one of the problems you need to deal with in a *small* case.

They also have VT-x and Hyperthreading, of course. [Edit: Although this Xeon and the mobo chipset both support vt-d, the mobo BIOS does not. I had this confirmed by Intel, unfortunately. They said “…for the S1200KP and KPR. No luck for VT-d on these boards.”]

The 1225/45/75 Xeon’s have on-board graphics, which we absolutely need in this build, since we have no room for an add-in graphics card. I chose th1245v2 chip as a trade-off between price of lower models and performance of the higher ones. Also considered the 1265L-v2 because the “L” means low power. Could have dropped from 77W down to 45W. Might mean a smaller PSU – which would be nice. But the 1265L uses less power because it runs at only 2.5GHz. (I love saying that “only 2.5Ghz…”)

I researched pretty hard to find the fastest ECC memory possible, to get the most out of this server. Glad to get some good advice (CMTL Labs) and a good price (DirectCanada) on some 1600MHz DDR3 which was sure to work with this mobo. I haven’t done any performance tests on it though, but I’d love to hear from anyone who’s tried.

The storage had to be SSD, in my mind. Cost of SSD is way down. And when I watch for performance bottlenecks in my apps, the disk always tops-out long before the CPU. In my case 256GB is enough so that’s where I settled for the price/size tradeoff.

The CPU cooler was crucial. Have I mentioned “small”? The case I chose has severe height restrictions, so the standard CPU coolers just wouldn’t fit. I saw some low-rise coolers that might have worked but. to be sure, I chose a blower. (Blows hot air out the side of the case.) This is noisier than a regular cooler, so all you Home Theater buffs should stay away. In my home office, I moved the server away from my desk. In a classroom, you can’t even hear it.

Only complication during assembly was the pressure-bars that came with this CPU cooler. They can not be used with this mobo – because they would crush some IC’s. So I trimmed down the bars (with a jigsaw) so I could use only the nut at each end. Then it was easy to screw the CPU cooler in place, through the motherboard, as usual.

That case is really small.

The guys at mini-box.com.au have focused on doing one thing really well, and it was exactly what I needed. (I have absolutely no ties to them.) I especially liked that they can bundle the power supply because I’m not an expert on that component. In addition to getting the size right, I also got help from Zap the Super Moderator at AnandTech forum. Apt moniker.

For assembly, I have to give a huge shout-out to Mini-Box guys for their video on how to assemble a server using the M-350 case. The video features a different mobo, but my assembly went perfectly well following the same steps. You can start their most-excellent video here.

As an homage to their great work, I present photographic evidence of all my nasty boxes gagged and bound – and slightly abused – before being flung dramatically into the recycling bin.

Other small cases…

I was inspired to try for an ultra-small build by these photos from Lime Technologies. Their case is only 3.8L, and looked great. But for my build, I went for the M-350, which is only 2.5L.

Note that for an insanely small ESXi box, though not a true server, the guys at Virtually Ghetto are trying to get ESXi 5.1 running on a Mac Mini. Not clear to me if it’s working smoothly yet, but for a desktop box, the Mac Mini has some awesome features!

I ruled it out as a server though, since it doesn’t have the dual server-class Ethernet ports I need, nor the ECC RAM and Xeon processors I prefer. Also, they chose a 2.6GHz instead of 3.4GHz CPU. And you pay for the name: currently $1,600 for a Mac-Mini config that approaches this home build (16GB memory and 256SSD) but ultimately lacks the features mentioned.

[Edit: Excellent blog series describing VMware on an Intel NUC is here. If you don’t mind using a single NIC and a few speed limitations, those things are really, really small!]

Is it powerful enough?

So far, I’ve got 30 VM’s copied onto the disk. 16 of them are running during my events. (70% Linux, 30% Windows). During my events, as many as 6 or 8 are being exercised really hard at any given time. I watch the performance graphs in VMware, and I’m nowhere near any ceilings yet. Am thinking of shooting for 24 active VM’s, and 12 participants blasting simultaneous Nmap scans at them. If anyone tries this, I’d be interested to hear your feedback. [Edit: Ran 24 VM’s at last event. No problems. Again though, only 6 to 8 really active at any one time.]

Caveats…

Not everyone needs an ultra-small ESXi server. I made some trade-offs while building this bundle. The 16GB memory is a severe limitation if you want a lot of VM’s. And there is no way (with the M-350 case) to add a card to the PCIe slot on the motherboard. Plus, the blower is much louder than a normal CPU fan (though much quieter than any full-size server). And there is no place for an optical drive.

Oh yes, just one glitch: the BIOS won’t find the hard drive during boot-up search, so I have to hit [F10] to have it display the “Boot Device Menu”. Only the SSD shows up on the list, so I don’t know why it doesn’t get selected automatically. (Still. I can boot the server – even with no monitor – by hitting [F10] then [Enter] within 15 seconds of boot-up. Never misses.)

But I really needed an ultra-small ESXi server, so I am satisfied with these trade-offs, and very pleased with the result. Now I just need to find a really small toothbrush.