Lots of ‘data’ orchestration!

A friend of mine recently introduced me to the game Factorio. In order to continue stroking my addiction, I’m going to use Factorio as a tool to continue teaching about scaling technology.

“Yes honey, I swear it’s work!” — me

I’ve previously written about data pipelines and message routing. In one post I used a fun example with a Gorilla Hairstylist in order to demonstrate routing messages through Kafka, a dumb (intentionally) message bus. One reason we use a tool like Kafka is to enable scaling of applications. Scaling means we increase the available capacity of a system in order to handle more data.

Let’s talk about the game for a bit. Factorio is an open world-building game. You begin as an astronaut stranded on an unknown and hostile planet. You have a pick axe and a few resources. In the beginning, you must manually mine ore, construct items and factories. Over time, you grow the technology and automation capabilities of your base, all while defending it from alien attacks. They don’t like the pollution you’re generating and they want you to cease and desist. Over time, you continually automate processes, tying them together to build out a series of complex manufacturing capabilities all the while focusing heavily on automation over manual intervention.

After playing the game c̶o̶n̶t̶i̶n̶u̶a̶l̶l̶y̶ occasionally over a few weeks, I realized that there are so many similarities between automating in the game and automating software processes. We build applications that have inputs and outputs. They feed other applications. Some aggregate, some are faster than others, but in the end, there’s no human involved. Let’s dig into scaling software using Factorio as a reference!

In order to process more data, historically, we would buy bigger servers; more RAM, CPU, and disk. That’s called vertical scaling. Going bigger by buying a bigger piece of gear.

With the advent of virtual machines, containers and the cloud we can scale ‘horizontally’. Horizontally scaling changes how we process more data. Instead of buying a bigger server, we just add more processing units in the form of containers, etc.

Let me explain some of the items below so you’ll relate my analogy.

This Burner Mining Drill is our source of data. It could be a customer on a web site or an action within a piece of enterprise software. In the game, this pulls raw Iron Ore (in these GIFs) to be used to create other items.

Iron Ore is being used to represent the initial raw data. This could be the actual request from a customer in the form of HTTP parameters. In the game, iron ore is the base element used to make iron-based items such as metal (in the form of iron plates, below) or machinery.

Let’s call this one custom code. Whether it’s an API call to a downstream system or a piece of code that pulls an item out of Kafka, you can ignore it somewhat. In the game, this is a piece of machinery that moves an item from one place to another.

The Stone Furnace takes raw iron ore and turns it into iron plate. This is our data enhancement or persistence layer. This application can take a user request and persist it in a database or make subsequent API calls. In the game, this is how we convert a piece of raw ore to refined metal.

Iron plates represent our finished product or our enhanced data. This is the result of our internal processes. In the game, Iron plate is the refined or processed iron ore. Heavy Metal, baby!

Time to walk through the history of computing!

Now that we have some common lingo to relate the game to computing, I want to dive into the history and capabilities of scaling applications.

The Dark Ages

There were no computers. We manually performed actions and math on an abacus. Much like my poor engineer here who is digging his Iron Ore by hand. Ugh, so manual, back breaking!

Oh, the manual back aches!

Then Adam and/or Eve built the first computer. We were able to automate our punch cards!

That’s a fancy automated digger!

Then computers became more complex. We create an “AND” gate (iron ore) on the left and then sneaker-net it to the next factory. This scales with people.

Note: People don’t scale.

Note: Sneaker-nets work best when it’s built on a Nike backbone.

Note: This is not an endorsement of Nike but I do accept endorsement opportunities.

Out the left, in the right!

What’s an “AND” gate?

“The AND gate is a basic digital logic gate that implements logical conjunction — it behaves according to the truth table to the right. A HIGH output (1) results only if all the inputs to the AND gate are HIGH (1). If none or not all inputs to the AND gate are HIGH, a LOW output results. The function can be extended to any number of inputs.” — Wikipedia

This is great, but thanks to DARPA and Al Gore, we can tie them together with the Internet (of things?). We were now able to take our basic “AND” gate (Iron Ore) in a more complex setup to create a Pentium microchip (or an Iron Plate in this example.)

It’s powered! — — and connected

We have a data bus! Those conveyors are handling the transport of our raw material to the final processing factory. This is excellent. If this were software, we’d have a nice trickle of users to our web site. They’re perusing our content and loving life. Then they tell their friend. Who tells their next friend. Then we go viral!

Viral!

Oh, crap. We’re Viral.

We went viral and we can’t handle the load. Our requests are stacking up! In this (fantastic) comparison, we can’t scale up. We can’t buy a bigger furnace. Our furnace can only handle one request a second (one piece of iron ore being converted to an iron plate). Since our popularity grew and we’re trending, we’re getting 3TPS. Unfortunately, our customers are going to leave since we can’t handle the load. It’s like a Tesla Model 3 pre-order. 2 million people ordered, but they could only make 1k a month. At that rate, tons of customers are going to leave and you’re gonna have a bad day.

seriously bad day — giphy — https://giphy.com/gifs/day-w9yIu38Qa6Vnq

If we can’t vertically scale, how do we horizontally scale? Let’s buy more servers (or furnaces)!

THREE CUSTOMERS!

We’ve done it! We’re handling three times the data with three instances of a smaller processor. If this were code, you’d have your customers on the bottom producing data/requests to your servers. Those requests come in and get picked up by the available processors. If the processor is busy, then the next processor picks it up!

Technical note, this is a naive way of load balancing work. The analogy starts to fall apart. There’s no parallelism. It’s really a serial work flow…. but I’m forcing a game as an analogy to a highly technical concept. It worked well enough for me!