I was lucky enough to attend ElixirConf.EU for the third time and in this post I'm sharing some thoughts about the talks I saw, some of my notes and insights on the future of this community in general.

Location

I'd never visited Prague before and the conf was an amazing opportunity to combine business and pleasure. While writing this post though, a week after, I realised I didn't visit most of the landmarks that I was planning to. Looks like I was immersed in the vibe of the city and had a great time there 😀.

View over the Vltava river

The Fun Stuff

In the few nights spent in Prague, I picked out the following places, which I wholeheartedly recommend:

Now for the Czech cuisine, I don't remember much, but the bread dumplings (Knedlíky?) and the guláš were quite tasty. The beer on the other hand didn't match my taste.

The venue

Spacious, clean and only a short commute from my staying.

Conference - Day 1

Keynote - The Road to Broadway - José Valim

video

The talk was about broadway, a new open source project by Plataformatec aiming to streamline data processing pipelines.

The presentation started with a short history of steps towards the goal of making collections from eager -> lazy -> concurrent -> distributed.

Imagine the following:

File . stream ( path ) |> ... |> Stream . async ( ) |> ... |> Stream . async ( ) |> ... |> Stream . run

The issues with it are that it's:

Too manual

Moving data between processes a lot

Hard to reason about fault-tolerance

A better abstraction was needed and thus GenStage was born.

It was inspired by Akka streams and the Akka team was helpful and answered many questions.

Some of the companies using GenStage in production:

Broadway

Broadway takes the burden of defining concurrent GenStage topologies and provide a simple configuration API that automatically defines concurrent producers, concurrent processing, batch handling, and more, leading to both time and cost efficient ingestion and processing of data.

Features:

Back-pressure

Automatic acknowledgements at the end of the pipeline

Batching

Automatic restarts in case of failures

Graceful shutdown

Built-in testing

Partitioning

Coming up:

Multiple processors

Metrics and statistics

Surprisingly in the audience 2 or 3 people raised their hands when José asked who's using Broadway already in production.

Want to contribute? Write a producer.

Available producers:

SQS

RabbitMQ (also see: blogpost)

(soon) Kafka

Q & A

Q. Can I build a producer in another language?

Totally doable as long as there's a API for that external app.

Q. Is there an overlap between Flow and Broadway?

Flow focuses on data aggregation. Broadway is about the individual messages and not the data as a whole.

I'm really looking forward to use broadway in some project, it might be the perfect abstraction for some of the problems, I'm solving over and over from scratch.

Q. Is the goal to replace RabbitMQ or Kafka?

No, it's to work with them and become very good friends.

Rewritting Critical Software in Elixir - Renan Ranelli

The talk was a case study of adopting Elixir at Telnyx. Telnyx aims to be like AWS but for telcos.

They went to rewrite a dialplan service which was originally written in Python to Elixir.

Python is notoriously harder to scale vertically & horizontally than Elixir

Characteristics of the service:

Uses freeswitch XML to route calls

Stateless

Latency sensitive

Low throughput

The migration to the new Elixir service was challenging, since changes kept being made in the Python codebase, making it a race for feature parity.

They approached the problem by routing production traffic to both the existing Python service and the experimental Elixir one, capturing the responses and diffing them, to verify compatibility.

They used the responses to automatically generate regression tests.

For them, the experiment was successful as parallelisation was ridiculously simpler and cheaper with Elixir.

Tortoise Evolved - Martin Gausby

This talk was about Tortoise, an MQTT client for Elixir.

MQTT is a lightweight Publish/Subscribe (PubSub) protocol, designed to operate on high-latency networks or unreliable networks. It's commonly used in IoT.

You publish messages in topics. Topics are namespaced using slashes, for example ocp-tower/2/temperature . A client can subscribe to messages using topic filters.

Example:

Start a connection process:

client_id = " toes " { :ok , pid } = Tortoise.Connection . start_link ( client_id : client_id , server : { Tortoise.Transport.Tcp , host : ' localhost ' , port : 1833 } , handler : { Tortoise.Handler.Default , [ ] } )

Then publish a message:

topic = " ocp-tower/3/temperature " payload = << 21 :: float - 32 >> Tortoise . publish ( client_id , topic , payload , [ qos : 0 ] )

Upcoming Changes

Tortoise (for MQTT .1.1) is ready for production today

MQTT will cause some changes to the API

MQTT 5 support in a branch, kinda works, but needs some time

Tortoise will drift towards a low-level API to support everything

I wanted to ask, if there's plan for a Broadway producer in Tortoise, but I completely forgot about it.

Let there be light - Michał Muskała

This talk tries to explore the necessary steps for an Erlang application to boot.

So..

./bin/erl is a sh script, which calls erlexec

is a sh script, which calls erlexec calls beam-smp (OTP 21 removed single-threaded implementation)

calls beam-smp (OTP 21 removed single-threaded implementation) erl_init.c configures time monitoring for time-warp mode. The BEAM has its own way to measure time starts thread progress services (see: here) sets the stack size - 1 MB for regular schedulers and 320 KB for dirty schedulers starts a bunch of system processes

Some essential modules cannot be reloaded and are bootstrapped as embedded C arrays. See preloaded/src/Makefile

There's Perl in the codebase, see instrs.tab

beam_emu.c Implements the main process loop A huge-function with goto all over the place Generated from pseudo-C by perl scripts into C with macros on top of macros Implements bytecode different from BEAM files and a translation is done by the loader

There's an undocumented flag profile_boot , which profiles the boot operation

, which profiles the boot operation You can do some Ruby-like metaprogramming using $handle_undefined_function , but please don't!

Key Takeaways

Many VM services are implemented in Erlang

Command-line arguments are parsed many times each time an erlang program is started (It's 6 may 7 times, I cannot recall exactly)

Elixir adds layers on top and underneath the OTP boot process

Lot's of command-line arguments of the VM are not well documented

It looks like there's a module for everything in OTP

For me this was the most insightful and true talk of the conference. No marketing talk, not another success story. Digging into the core of the system, trying to see how it works. I love code exploration talks. You can learn so much from them.

By the way the brewery (link), where the above photo was taken is awesome.

An Adventure in Distributed Programming - Wiebe-Marten Wijnja

slides

This talk was about Planga a chat application and the challenges they faced building it. The app is open-source.

There were references to:

Planga's working on an Ecto adapter for Riak, see here.

Remarks

Distributed applications are hard

Elixir makes it reasonably bearable

Tooling can be (and is being) improved

Projects to Check out

Q & A

Q. CRDTs. How do you use them? Any patterns?

It depends what you're trying to abstract. We use Riak CRDTs which I believe use delta-CRDTs.

Q. Why did you stop using Mnesia?

The Ecto adapter was far from perfect and also we wanted a DB which doesn't live in the node.

Building Resilient Systems with Stacking - Chris Keathley

slides video

Alternate Title: “How to boot your apps correctly”

Definitions:

Resilience

An ability to recover from or adjust easily to misfortune or change.

System

A group of interacting, interrelated, or interdependent elements forming a complex whole.

…complex systems run as broken systems. The system continues to function because it contains so many redundancies and because people can make it function, despite the presence of many flaws… System operations are dynamic, with components (organizational, human, technical) failing and being replaced continuously.

See: How Complex Systems Fail

Scaling is a problem of handling failure

This is why Erlang is so good at scale.

You have to treat humans as first-class citizens.

Don't use mix for runtime config.

See: https://github.com/keathley/vapor

defmodule Jenga.Config do use GenServer def start_link ( desired_config ) do GenServer . start_link ( __MODULE__ , desired_config , name : __MODULE__ ) end def init ( desired ) do :jenga_config = :ets . new ( :jenga_config , [ :set , :protected , :named_table ] ) case load_config ( :jenga_config , desired ) do :ok -> { :ok , %{ table : :jenga_config , desired : desired } } :error -> { :stop , :could_not_load_config } end end defp load_config ( table , config , retry_count \\ 0 ) defp load_config ( _table , [ ] , _ ) , do : :ok defp load_config ( _table , _ , 10 ) , do : :error defp load_config ( table , [ { k , v } | tail ] , retry_count ) do case System . get_env ( v ) do nil -> load_config ( table , [ { k , v } | tail ] , retry_count + 1 ) value -> :ets . insert ( table , { k , value } ) load_config ( table , tail , retry_count ) end end end

To watch for the health of your system's components, you can use alarms.

Example:

defmodule Jenga.Database.Watchdog do use GenServer def init ( :ok ) do schedule_check ( ) { :ok , %{ status : :degraded , passing_checks : 0 } } end def handle_info ( :check_db , state ) do status = Jenga.Database . check_status ( ) state = change_state ( status , state ) schedule_check ( ) { :noreply , state } end defp change_state ( result , %{ status : status , passing_checks : count } ) do case { result , status , count } do { :ok , :connected , count } -> if count == 3 do :alarm_handler . clear_alarm ( @alarm_id ) end %{ status : :connected , passing_checks : count + 1 } { :ok , :degraded , _ } -> %{ status : :connected , passing_checks : 0 } { :error , :connected , _ } -> :alarm_handler . set_alarm ( { @alarm_id , " We cannot connect to the database”}) %{status: :degraded, passing_checks: 0} {:error, :degraded, _} -> %{status: :degraded, passing_checks: 0} end end end

When communicated with other services, you may want to use circuit-breakers. For BEAM projects fuse is the go-to library.

Lessons From our first trillion messages with Flow - John Mertens

John is a principal engineer for https://change.org It's written primarily in Ruby, but they started adopting Elixir in 2018.

This is an example of how they use Flow:

SqsClient . sqs_producers ( ) |> Flow . from_stages ( ) |> Flow . map ( & prep_incoming_message / 1 ) |> Flow . map ( & run_biz_logic / 1 ) |> Flow . map ( & commit_side_effects / 1 ) |> Flow . map ( & log_errors / 1 ) |> Flow . partition ( window : ack_window , stages : 1 ) |> Flow . reduce ( fn -> [ ] end , & ack_accumulator / 2 ) |> Flow . on_trigger ( fn messages -> ack_messages ( messages , queue_name ) { [ ] , [ ] } end )

With the help of pattern-matching they can handle a variety of data:

@spec run_biz_logic ( Message . t ( ) ) :: Message . t ( ) def run_biz_logic ( % Message { status : :ok , message_type : " click " } = msg ) do # functionality for the "click" message type end def run_biz_logic ( % Message { status : :ok , message_type : " sign " } = msg ) do # functionality for the "sign" message type end

A glimpse of Broadway from change.org's codebase:

def start_link ( { input_queue : queue , sqs_worker_count : producers ) do Broadway . start_link ( __MODULE__ , name : __MODULE__ , producers : [ default : [ module : { BroadwaySQS.Producer , queue_name : queue } , stages : producer_count ] ] , processors : [ default : [ stages : 100 ] ] , batchers : [ default : [ batch_size : 10 , batch_timeout : 5_000 ] ] end def handle_message ( _ , % Message { data : sqs_msg } = message , _ ) do new_msg = sqs_msg |> prepare_incoming_message ( ) |> run_biz_logic |> commit_side_effects ( ) |> log_errors ( ) case new_msg . status do :error -> Message . failed ( message , new_msg . status_metadata . error . response _ -> Message . update_data ( message , fn _ -> new_msg end ) end end

Ecto without SQL - Guilherme de Maio

slides

It turns out [Ecto][ecto] is more than a database library for Elixir.

Ecto is a toolkit for data mapping and language integrated query for Elixir

the main modules of Ecto are:

Schema

Changeset

Repo

Query

It's not an ORM.

Defining a type:

defmodule Sample.Phone do @behaviour Ecto.Type defstruct [ :number ] def type , do : :string def cast ( string ) when is_binary ( string ) do case Phone . parse ( string ) do { :ok , phone } -> { :ok , % __MODULE__ { number : format ( phonenumber ) } } _otherwise -> :error end end def cast ( phone = % __MODULE__ { } ) , do : { :ok , phone } def cast ( nil ) , do : { :ok , nil } def cast ( _ ) , do : :error def dump ( % __MODULE__ { number : string } ) , do : { :ok , string } def dump ( nil ) , do : { :ok , nil } def dump ( _ ) , do : :error def load ( nil ) , do : { :ok , nil } def load ( string ) , do : { :ok , % __MODULE__ { number : string } } end

You can use Ecto without SQL. By defining an adapter.

Some examples:

However Ecto.Query is very SQL-centric.

You may use changesets without Repo for validations. Example:

defmodule ApiWeb.Call.AnswerValidator do @moduledoc """ Validate requests to answer command """ @schema %{ client_state : :string } @all_fields Map . keys ( @schema ) use ApiWeb.Validator , schema : @schema alias ApiWeb.Validator.ClientStateValidator def changeset ( changeset , params ) do changeset |> cast ( params , @all_fields ) |> ClientStateValidator . validate_client_state ( ) end end

This is a way to get a consistent error handling strategy for controller actions.

Closing Keynote - Chris McCord

video

The idea is to be able to write interactive, Real-Time apps that are “scriptless”, meaning that you don't have to write any JavaScript.

Chris showed the following demos on stage:

Imagine my surprise and excitement when Chris opened a browser tab with the first demo, which I wrote 🥳 You may read my previous post about that.

He said, they're going to be able to use LiveView to lift information out of the VM and produce the Phoenix Telemetry integration.

For more demos, guides and tutorials about LiveView, check out this list.

So Chris said that LiveView isn't going to replace SPA frameworks. For example one wouldn't write google docs, or google maps using LiveView. However there's many simpler apps which can be written with LiveView, avoiding the rabbit hole of complexity of client-side development.

LiveView is a new paradigm questioning some of the existing techniques for developing interactive apps. Imagine building an interactive thermostat UI like this one. You'd need to code:

Server

Define routes

Create Controller / Channel

Define JSON/payload contracts

Client

Handle events and syncing state

Handle client actions

Handle error recovery

Navigate the JS library labyrinth

That thermostat example, can be coded like:

defmodule DemoWeb.ThermostatView do use Phoenix.LiveView import Calendar.Strftime def render ( assigns ) do ~L """ <div class="thermostat"> <div class="bar <%= @mode %>"> <a phx-click="toggle-mode"><%= @mode %></a> <span><%= strftime!(@time, "%r") %></span> </div> <div class="controls"> <span class="reading"><%= @val %></span> <button phx-click="dec" class="minus">-</button> <button phx-click="inc" class="plus">+</button> </div> </div> """ end def mount ( _session , socket ) do if connected? ( socket ) , do : Process . send_after ( self ( ) , :tick , 1000 ) { :ok , assign ( socket , val : 72 , mode : :cooling , time : :calendar . local_time ( ) ) } end def handle_info ( :tick , socket ) do Process . send_after ( self ( ) , :tick , 1000 ) { :noreply , assign ( socket , time : :calendar . local_time ( ) ) } end def handle_event ( " inc " , _ , socket ) do { :noreply , update ( socket , :val , & ( &1 + 1 ) ) } end def handle_event ( " dec " , _ , socket ) do { :noreply , update ( socket , :val , & ( &1 - 1 ) ) } end def handle_event ( " toggle-mode " , _ , socket ) do { :noreply , update ( socket , :mode , fn :cooling -> :heating :heating -> :cooling end ) } end end

Where the EEx part of it, can be extracted to a separate file.

An important aspect of app development with LiveView is failure isolation. With a JavaScript application, an exception, unless rescued would freeze the UI. However with LiveView a process isolates failure and restarts a view with a known good state.

Optimisations

LiveView uses morphdom to optimally update the DOM with the changes pushed down the web socket. LiveView tries to push only the changes in the state and positions in the markup to be updated.

In the future the ability to throttle keyboard events will be added and there's an open issue for that.

First-Class Testing

With code like the following, you can test-drive your statefull actual running process without a browser.

test " thermostat controls " do { :ok , thermo_live , _html } = mount ( Endpoint , ThermostatLive ) assert render ( thermo_live ) =~ " 70° " assert render_click ( thermo_live , :inc ) =~ " 71° " assert [ clock_live ] = children ( thermo_live ) assert render ( clock_live ) =~ ~r/ ..:.. [A|P]M / :ok = GenServer . call ( clock_live . pid , { :set , " 12:01 PM " } ) assert render ( clock_live ) =~ " 12:01 PM " end

Finally Chris also showed a comparison of library sizes to build an interactive app:

Name Size (minified) LiveView.js + morphdom 29K Vue 2.5.20 88K React 16.6.3 + React DOM 112K Ember 3.0.0.beta.2 468K

Next Steps

Optimised collections with prepend and append operations

pushState support for proper URLs on state change

Enhanced loading states

File uploads

Guides

Initial hex release

Q & A

Q. WebSockets don't work in some cases where there are proxes

You can give it the longpoll option

Q. What about working on an open standard?

It would impede development. We can get to specify it at some point. Too early to say.

Q. Does it work without JS (for SEO)?

If you curl it, responds with the expected HTML, so yes.

Outro

You might also be interested to read:

Martin Gausby's thoughts on this year's conf

Read my previous posts for 2018 and 2017

This list of bookmarks from the conference

Spotted a Mistake?

Please contact me on twitter, or in the comments, or submit a PR for corrections.

Upcoming Elixir Conferences