System One: The Dirt Cheap Data Warehouse



Take a look at the front photo of the bottom portion of the rack. From top to bottom, you'll see the following DCDW components:



1) HP DL585 G7 Server, which is the main database server

2) Two Supermicro SC216 chassis, which are the SSD storage chassis, each configured to hold 28 SSD drives

3) The Digital Loggers Inc. web-based power distribution unit (PDU)



The DL585 G7 is a quad AMD CPU server with 11 PCIe slots**. I have it configured with four AMD Opteron 6174 CPUs for a total of 48 true cores. RAM is 384GB with plenty of empty DIMM slots if I need them. The DL585 G7 is unique in that it has four AMD IO chips whereas most quad servers have only two. Each IO chip provides 42 PCIe2 lanes and the four chips provide, when the optional IO board is added, eleven PCIe slots, four x16 and the rest x8. I have inexpensive LSI 9200 and dual-controller 9202 host bus adapters installed into ten of the PCIe slots to provide a total of 14 SAS controllers and 112 disk channels. I have one x16 PCIe slot left over for future expansion. The DL server also has a battery-backed HP P410 RAID card, which I use along with eight internal 1TB laptop drives to provide bulk storage for database backups.



The Supermicro SC216 chassis are the non-expander JBOD versions stripped out and reconfiged with Supermicro JBOD power boards. Each chassis has 24 drive slots at the front and four at the rear, the four rear slots provided by an add-on mobile dock from StarTech. Each JBOD chassis presents seven SFF-8088 ports that connect to the HP sever with 1 meter SFF-8088 SAS cables - you can see the giant bundle of cables in the photo.



The disks used in these JBOD chassis are Samsung 830 and 840 Pro drives. An earlier version used OCZ Vertex3 drives, which worked perfectly and were by far the best deal at the time. When the Samsung 830s dropped below $100, however, they became the new go-to drive until recently when the 840 pros got reasonable. All of the drives are 128GB, which is the sweet spot for throughput per dollar. For almost any other use case I would have used drives with supercaps, but these are not required with Oracle.



All told, the DCDW has around 3 million IOPS and 6.9TB worth of SSD storage, formatted to 5.4TB with the remaining 1.5TB as overprovisioning. Yes, that's 1.5TB of "overprovisioning"! I use Oracle ASM, which is a form of software RAID1E, so I have 2.7TB worth of actual usable space.



The Digital Loggers box gives me web-based remote control of power and is very helpful. I plan on writing up an STH article on this little gem some time soon.



You'd be surprised how little money I have invested in this machine. The DL585 server was bought from eBay for a fraction of its price when new, even though it still has two years of on-site warranty service left. The HBAs cost me between $40 and $200 each - the more expensive being the 9202-16e cards with 16 ports each. If you do the math, I only utilize half of the SAS ports per card. Each LSI card can handle almost six SSD drives before bottlenecking, but SAS chassis present disks in sets of four. With the availability of low-cost LSI HBAs and the relatively high cost of disk chassis, it has been cheaper to deploy units of four disks rather than five or six.



The database throughput for this beast is impressive. With indexes turned off, I can issue a single query and get 16,800MB/Second of actual Oracle database throughput - with compression that's around a quarter billion fact table rows per second!



**HP also sells the DL580 G7, which is an Intel-based version of the same box. In several ways it is even better than the DL585: it has 64 DIMM slots and can take Xeon E7 CPUs with 10 true and 20 HyperThreading cores each. It has, however, one big flaw: The PCIe bandwidth is awful. While it has eleven PCIe2 slots like the DL585, the DL580 has only two Intel IO chips and as a result you get six x8 slots and five x4 slots, which is far too few for that much CPU horsepower. The DL585 showers you with four x16 slots and seven x8.