Learn to forget

2016-07-20 by Brujo Benavides

I have many roles/positions inside Erlang Solutions. I’m Inaka’s CTO, I’m one of Erlang Solution’s Tech Leads. I’m also an Erlang Trainer sometimes. But what I truly am is an Erlang Developer. And as such, I want to share with you an experience I had when I was learning Erlang a couple of years ago, which is incidentally something I also have seen (this time from a different perspective) as an Erlang Trainer.

In a Nutshell

The thing I keep seeing when someone is learning Erlang (and it happened to people learning other languages, like Haskell for instance) is that, at the very beginning…

You keep learning things you have to forget later. You’re probably never going to use them anymore. But you need to know them if you want to understand what you’re doing.

And this concept is so intrinsically tied to the philosophy of the language that one of those things that renders so many other building blocks forgettable is usually seen attached to the language name itslef: Erlang/OTP. Let me show you what I mean…

Sequential Erlang

Let’s leave OTP aside for a second. When you start learning Erlang, you usually start with Sequential Erlang, right? Most Erlang books, courses and tutorials start this way.

Erlang being a functional language, one of the very first things everyone learns is how to write recursive functions. It all starts with a database which is actually a list:

fetch(_Key, []) -> undefined; fetch(Key, [{Key, Value} | _]) -> Value; fetch(Key, [_ | Db]) -> fetch(Key, Db).

But then, somebody introduces you to anonymous functions and, of course, high-order functions. Suddenly, you don’t use recursion anymore. Cool kids use lists:dropwhile/3 instead:

fetch(Key, Db) -> case lists:dropwhile(fun({K, _}) -> K =/= Key end, Db) of [] -> undefined; [{_, Value}|_] -> Value end.

Of course, eventually you realize that someone else must have had that same issue before. More often than not, that’s exactly the case and there is a function that you can just use:

fetch(Key, Db) -> proplists:get_value(Key, Db).

Concurrent Erlang

Ok, so writing Sequential Erlang is easier than expected because a lot of things are already written. That’s good, but what about those things that make people choose Erlang in the first place? If you’ve chosen Erlang to develop your system, you probably have a concurrency problem or rather a problem that’s best solved by using a concurrent language.

To me, at least, it wasn’t so intuitive at first that the situation described for Sequential Erlang could apply to Concurrent Erlang as well. But then again, it all starts with a database:

new(DbValues) -> spawn(?MODULE, init, [DbValues]). init(DbValues) -> loop(DbValues). loop(Db) -> receive {fetch, From, Key} -> From ! {reply, proplists:get_value(Key, Db)} end, loop(Db). fetch(Key, Db) -> Db ! {fetch, self(), Key}, receive {reply, Result} -> Result end.

As you can see, to implement that example you have to learn to spawn processes (using erlang:spawn/3 or other variants) and then send and receive messages. Then again, our database is just a process that receives messages and for each message it receives, using its internal state, builds a response and sends it back to the caller as a message. Yeah, it has a tiny bit of internal logic ( proplists:get_value(Key, Db) ) that’s unique to it. But the rest is pretty generic. Even from the caller side, everything boils down to just sending messages and waiting for responses.

As you might have guessed by now, lots of people faced this same issue before and the condensed experiences helped create no other thing than OTP. When you learn how to work with OTP and its behaviours, you no longer need to spawn processes and send messages explicitly:

new(DbValues) -> gen_server:start(?MODULE, DbValues, []). init(DbValues) -> {ok, DbValues}. handle_call({fetch, Key}, _From, Db) -> {reply, proplists:get_value(Key, Db), Db}. fetch(Key, Db) -> gen_server:call(Db, {fetch, Key}). terminate(_, _) -> ok.

Did you see that? You’re suddenly not using spawn or ! anymore. gen_server will take care of all that for you. Furthermore, gen_server will also take care of tons of other things you haven’t even thought about yet, like servers not running or running in other nodes, timeouts, code upgrades, etc.

From this point on, your job as an Erlang developer is no longer about spawning processes and sending messages between them. It’s all about starting servers or fsms inside supervision trees that end up building applications. And that’s only if you’re building a whole new kind of app. But, if what you’re building is generic enough (for instance a web server, with a REST API or web sockets or even SSE support) there surely is an app for that already and you only need to write the pieces of code that are specific to your system. You no longer have to think about processes, even servers or supervisors or what-not. All that is hidden behind the internal logic of the Erlang applications that you use.

These days, you can certainly write a whole system with multiple endpoints, database connections, interactive documentation and even SSE support without even thinking about OTP at all.

Why should I learn all these?

At this point is where the unavoidable question pops up: If, in the end, I’ll not use any of this, what’s the point in learning it?It’s the same question every math student ever has forever asked about logarithms, right? And the answer is unsurprisingly the same: It shapes the way you think.

Building systems with Erlang is not just about writing functional code, spawning processes or interchanging messages. Building Erlang systems is about thinking in terms of scalability, fault-tolerance, reliability and many other things Francesco Cesarini and Steve Vinoski explain in detail in their book. Understanding the mechanisms behind OTP and the process that lead to its creation will help you gain crucial insight on how to architect your system.

While it’s true that you can write a web server in Erlang with limited functional programming knowledge and an almost total lack of awareness about OTP, by not knowing/caring about the internals of what you’re building, you really don’t know what you’re capable of. More importantly, you don’t know what your system is capable of.

Then, there is also the fact that even when you can totally just use OTP, knowing how linked processes behave will save you countless hours of frantical debugging, googling and stack-overflowing just to find out that you have started your gen_server in an inadequate fashion.

Condensed Knowledge

It’s important to notice something I didn’t fully realize until I started teaching the Erlang/OTP course: OTP was not created orimposed, it emerged. It’s not that at some point someone established the right way for building systems with Erlang. On the contrary, OTP was created from multiple already existing systems that were built in the same way. While working in Ericsson, some smart people realized that they were doing the same thing and writing the same code time and time again, making a mistake here and fixing it there. They saw that and they decided to condense all that knowledge (and the code as well, of course) in a big library (a set of applications, actually) eventually called OTP.

That’s the same thought process that lead the Gang of Four to write their famous book. They described many common patterns they observed in several projects and provided names and examples for them. In the making of OTP, common programming patterns were identified and they ended up being encoded as behaviours. Using them now, you’re actually profiting from the condensed knowledge of years of programming and building systems similar to the one you have at hand.

Open-Source Libraries

The process described above didn’t stop with OTP. Maybe we don’t add that many new things to OTP itself anymore, but thanks to the thriving open-source community behind Erlang, we still keep collecting condensed knowledge in apps, libraries, frameworks, that allow others (and ourselves as well) to forget about certain things that we keep constantly doing. Libraries like worker_pool, SumoREST and many others are the result of the same thought process behind OTP: We’re doing this all the time, people! Let’s put it in a library!

And we have now a central place to share all that knowledge: hex.pm. So, if you have learned to forget a couple of internal components but you find yourself writing the same piece of code again and again, I encourage you to create an open-source library with it and share it on hex.pm. Not only will you and others stop reinventing that particular wheel, using the experiences from others, but we will also turn that simple wooden wheel into an all-star high-performance one that will keeps us rolling faster and faster everyday.

Furthermore

I have been an Erlang developer for a long time, and what I described above comes from my experience as a learner and a teacher as well, and from the experiences of colleagues and friends. Based on this, I’m pretty certain this is how most developers learn Erlang, but i’d love to hear about other learning patterns.

To spice things up, there is a brand new language in Beam town: Elixir. I am really interested in discovering what is the learning process like for it, especially from people who don’t come from an Erlang background. I’m currently trying to gather this info by speaking to the alchemists I know, but if you’ve learnt Elixir recently, please share your insight in the comments below. My plan is to eventually write a post about the Elixir learning experience and see what the differences are. Stay tuned!