What? Why? What is the goal?

"Building ARM cluster" – that is a catchy title, isn’t it?

A while ago I have watched a video made by David Guill where he presented his Raspberry Pi cluster. Inspired by that invention I’ve decided to design my own cluster, but in a more customized (to my needs) way.

Before I start describing the whole process of building it I’d like to answer to the question "Why do I need that?"

The reasons might be the following:

Learning purposes – if you operate on more than one machine, then for sure you’ll encounter problems which you have never had before. Starting from distributed computing, finishing at networking/administration.

– if you operate on more than one machine, then for sure you’ll encounter problems which you have never had before. Starting from distributed computing, finishing at networking/administration. More memory/cpu's – more power makes your environment strong

– more power makes your environment strong Running service – you might be able to run your own service! Instead of buying VPS’es create your own backend and pay less (it’s ARM!)

– you might be able to run your own service! Instead of buying VPS’es create your own backend and pay less (it’s ARM!) Ultimate fun – that is what I love most about it. You’ll see how many different areas of IT I’ll go through. Creating smth from the ground and making it working is undoubtedly awesome feeling!

The next unanswered question is "What I’m trying to accomplish?"

The goal is quite simple – run a few services. In general a website where you’ll be able to process a photo. Let say face detection should be fine.

Now you probably think "Easy/I’ve read thousands of such articles etc.". You’re partly right, the service by itself won’t be a fresh idea, but yet I’ll do it on ARM machines in automated manner. You’ll be surprised (or not if you’re a PRO :p ) how much time you may save by applying some tricks (i.e auto imaging or remote power switch).

A bit about services

Whenever you create a service, the three components might be required: database, web-server, computation program. Really? Lets take example service described in previous section (face detection):

Web-server "serves" data to client and interact with him. So, uploading images and serving php/html content will be handled by it

We need to store the results somewhere? For instance we may save the coords of the detected face in database.

Computation program must be able to detect a face in picture. As we know the program is going to use use most of the resources such as memory, cpu time.

According to the above, we see that it would be nice to have 3 different servers for different types of operations. There is no point to run all on a single machine (especially on embedded ARM devices). It will slow us down and make a service unusable.

Let us think about types of machines we may distinguish:

DB will serve serve files for whole service (e.g via iSCSI) and will provide databases. Fast HDD required.

will serve serve files for whole service (e.g via iSCSI) and will provide databases. Fast HDD required. Frontend machines will have web-server running with PHP services running in background

machines will have web-server running with PHP services running in background Computation cluster should be able to perform many operations at the time. Many cores would be much appreciated.

At this point we know what types of machines we are looking for exactly. Let’s find appropriate devices!

Devices

At the time of buying I found this set of devices the best by the terms of ratio between "required type", price and speed. You may choose different, but I’ll continue with what I bought.

The cluster (or as you call set of clusters, because machines are not the same type):

Odroid U3 x4 serves me as frontend device, it has flash memory connected via emmc which makes it ultra fast. 1.7GHz Exynos4412 Prime Cortex-A9 Quad-core processor 10/100Mbps Ethernet 2GB RAM 8GB emmc flash memory



Parallella 16 microserver x2 is a prefect machine for computation purposes: Xilinx Zynq®-7010 All Programmable SoC with Dual Core ARM A9 CPU Epiphany III (16-core CPU Accelerator) 10/100/1000Mbps Ethernet 1GB DDR3 SDRAM



Wandboard Quad x2. Because it has SATA port, it suits me as DB machine: Freescale i.MX6 Quad (Cortex-A9 Quad core) 10/100/1000Mbps Ethernet 2GB DDR3



Beaglebone Black x1 is an additional machine which role is to "control" the others. I’ll explain what that means in further articles. But the most important is the number of GPIO ports. AM335x 1GHz ARM® Cortex-A8 10/100Mbps Ethernet 512MB DDR3 RAM



The price is much higher than raspberry PI, but yet parameters are different.

The next step is to connect and power cluster, in order to do that we need some more equipment:

ROUTER. OpenWrt compatible TP-LINK TL-WDR4300

SWITCH. PLANET FSD-1600. 16 available 10/100Mbps Ethernet ports.

UART-USB converters to provide serial connection

ATX 350W power supply

CAT5 cables + board specific power cables

External HDD (Sata) – I have bought two 120GB SSD Sata2 disks

Relay module x1 – used to turn on/off the electricity to ATX. SRD-05VDC-SL-C module, documentation here

Relay module x8 – used to turn on/off the electricity to boards. SRD-05VDC-SL-C module, documentation here

USB hubs (as many as you need) I am using 13 port USB hub

USB-UART converters x9

Fans (you might need them for parallella)

One note to USB-UART converters based on PL2303 chip. They might not work properly with some types of boards. For instance in ODROID U3 voltage is not sufficient for PL2303HX chip, so I had to make it work with 5v/3.3v levels converters.

Photo below presents the first version of the cluster before I’ve added relay modules.

Power supply

The main power source is ATX. The idea is to take out the 5v cables and connect them to the common DIN rail, the same we do with GND cables. We’ll use 24 pin Molex motherboard connection.

As on the first image (source wikipedia.org) we take all 5v and connect them to DIN rail (as presented below).

The GND cables connect to the second DIN rail.

I would recommend to make 24 molex male/female port, to easily pull in/out the cables to/from ATX. Soldier cables an put them inside the shell as in the picture in the right bottom corner.



For each board you must do the DC Plug cable. It is pretty much manual task, but one thing to remember is that usually the center must be positive and the outer is negative (at least with those boards). At the end you should have 8x DC plugs (as presented below, the diameter is different for boards):

At this point you have 2 DIN rails and 8x ready to connect DC plugs. You may connect plugs to rail, it’ll work.

But whenever you turn on ATX all the boards will start.

Hey! What if I want to re/start only one machine?

With that design you’re not able to do that. So, we’ll improve by adding relay module (one for each board) and connecting it to Beaglebone’s GPIO port. Recall:

8x realay module is used to control boards

1x relay module is used to control ATX

Additionally I recommend to buy as terminal blocks, they will be useful for further wiring.

In order to connect boards via relay modules you must:

Connect GND Din rail to terminal blocks

Din rail to terminal blocks Connect 5V cables from DIN rail to relay module (to each one separately)

cables from DIN rail to relay module (to each one separately) Then output cable (from relay module) connect to terminal blocks

Connect DC Plug cables to free slots (on different side) in terminal blocks

5V input on relay module connect to DIN 5V rail, we” power relay module with the power from ATX

input on relay module connect to DIN rail, we” power relay module with the power from ATX Take off COM GND jumper from relay module and connect realy’s module GND to GND DIN rail

General overview of wiring is described in the picture below.

Connecting to BeagleBone

Ufff, so we have managed to wire it all 🙂 The worst part is behind us. Now it is a time to "rule them all".

Let start with ATX power control using single relay module.

Connect Beaglebone to 5v power supply (not via usb)

power supply (not via usb) Connect 5v input on relay module to 5v output on P9 (BeagleBone GPIO -> look here)

input on relay module to output on P9 (BeagleBone GPIO -> look here) Connect GND on relay module to GND on P9

on relay module to on P9 Choose on free GPIO pin on Beagle ie. P9_12 and connect it to control pin on relay module

Connect +5V standby from molex 24pin to relay module

Great! Now you’re able to power on/off ATX using only BeagleBone!

The next step is to control the boards. In that case we do the same, which is connecting each BB GPIO pin to control pins on relay module. Because some ports on BB are used by default I have chosen those which are free to avoid unexpected behavior after rebooting BeagleBone.

I’m using the following:

P8_16, P9_15

P9_23 ,P9_27

P8_11, P8_12

P8_14, P8_15

P9_12

The following bash script is using sysfs for controlling GPIO pins in Beagle, so make sure that those locations are available to you.

#!/bin/bash declare -A ports ports=( ["atx"]=60 ["odroid0"]=46 ["odroid1"]=47 ["odroid2"]=26 ["odroid3"]=44 ["parallella0"]=45 ["parallella1"]=115 ["wandboard0"]=49 ["wandboard1"]=48) function exportPin { pin=$1 path="/sys/class/gpio/gpio$pin/value" if [ ! -f $path ]; then echo $pin > /sys/class/gpio/export echo "out" > /sys/class/gpio/gpio$pin/direction fi } function parseMach { mach=$1 NO=${mach//[^0-9]/} MACH=${mach/[0-9]/} } function checkATX { pin=${ports["atx"]} exportPin $pin pinVal=$(cat /sys/class/gpio/gpio$pin/value) if [ $pinVal -eq 1 ]; then echo 0 > /sys/class/gpio/gpio$pin/value sleep 0.6 fi } function powerOn { if [ "$1" != "atx" ]; then checkATX fi pin=${ports[$1]} exportPin $pin pinVal=$(cat /sys/class/gpio/gpio$pin/value) if [ $pinVal -eq 0 ]; then echo 1 > /sys/class/gpio/gpio$pin/value fi } function powerOff { turnoffState=0 isOn=1 if [ "$1" == "atx" ]; then turnoffState=1 isOn=0 fi pin=${ports[$1]} exportPin $pin pinVal=$(cat /sys/class/gpio/gpio$pin/value) if [ $pinVal -eq $isOn ]; then echo $turnoffState > /sys/class/gpio/gpio$pin/value fi } function switchPower { if [ "$1" != "atx" ]; then checkATX fi pin=${ports[$1]} exportPin $pin pinVal=$(cat /sys/class/gpio/gpio$pin/value) if [ $pinVal -eq 0 ]; then echo 1 > /sys/class/gpio/gpio$pin/value else echo 0 > /sys/class/gpio/gpio$pin/value fi } function iterate { funcName=$1 target=$2 i=0 parseMach $target if [ "$NO" == "" -a "$target" != "atx" ]; then while true; do key="$MACH$i" if [[ ${ports[$key]} ]]; then echo "$funcName: $key"; $funcName $key i=$((i+1)) sleep 0.6 else break fi done else echo "$funcName: $target" $funcName $target sleep 0.6 fi } ACTION=$1 IFS="," for v in $2; do if [ "$ACTION" == "on" ]; then iterate "powerOn" $v elif [ "$ACTION" == "off" ]; then iterate "powerOff" $v elif [ "$ACTION" == "switch" ]; then iterate "switchPower" $v else echo "Unrecognized action" fi done

The following command will turn on all the odroid machines:

cpower on odroid

The following command will turn off wandboard0 and wandboard2

cpower off wandboard0, wandboard2

USB hub

USB-UART converters are required to access serial console on the board. It is a very important for us beacause this is the only way we may enter the u-boot console etc. or see what is going on with the board whenever it doesn’t boot.

Connect converters to each machine as the following and then connect them to USB hub (connected to BB):

Summary

At the end you should be able to access /dev/ttyUSB[0-7] at your BB and turn off/on ATX and boards by cpower shell script.

Here I present some photos, of my cluster: