There’s been many times when I’ve just wanted to add a simple JSON endpoint to an app to, expose a service, or process webhook events, without the overhead of a full framework. Let’s see how easy it is to build a production ready endpoint with Plug, using Erlang’s Cowboy HTTP server.

Plug Is:

A specification for composable modules between web applications Connection adapters for different web servers in the Erlang VM

If you’re coming from Ruby/Rails, think Rack, from Node, think Express, et al. Of course the concepts of these libraries are similar on the surface, they are unique in their own rights.

Cowboy Is:

A small, fast and modern HTTP server for Erlang/OTP

Additionally, it’s a fault tolerant “server for the modern web” supporting HTTP/2, providing a suite of handlers for Websockets and interfaces for long-lived connections. Without going into too much more detail, it’s safe to say, it’s an acceptable choice for production. Consult the docs for more info.

Poison Is:

A JSON library for Elixir focusing on wicked-fast speed without sacrificing simplicity, completeness, or correctness.

In other words, it’s a super fast, reliable JSON parsing library.

Building the Endpoint

With the short definitions out of the way, let’s build an endpoint to process incoming webhook events. Now, we want this to be “production ready”, what does that mean for our use case?

Fault tolerant: Always available. Can never go down (at least not easily :) Easily configurable: Can be deployed to any environment Well tested: Give us confidence in what we’re shipping

We do have a very simple use case for this, it’s a good idea to understand your own requirements before selecting tools and investing time into a solution.

1. Create a new, supervised, Elixir app:

$ mix new webhook_processor --sup $ cd webhook_processor

--sup will create an app suitable for use as an OTP application. OTP and supervision will give us our #1 requirement from above. Our server will be supervised and restarted automatically in the event of a crash, while the server may crash the Erlang VM should not (at least not easily :).

2. Add Plug, Cowboy, and Poison as dependencies

# ./mix.exs defmodule WebhookProcessor . MixProject do use Mix . Project def project do [ app: :webhook_processor , version: "0.1.0" , elixir: "~> 1.7" , start_permanent: Mix . env () == :prod , deps: deps () ] end # Run "mix help compile.app" to learn about applications. def application do [ # Add :plug_cowboy to extra_applications extra_applications: [ :logger , :plug_cowboy ], mod: { WebhookProcessor . Application , []} ] end # Run "mix help deps" to learn about dependencies. defp deps do [ { :plug_cowboy , "~> 2.0" }, # This will pull in Plug AND Cowboy { :poison , "~> 3.1" } # Latest version as of this writing ] end end

Couple notes here, we added plug_cowboy (in deps ) as a single dependency for Plug AND Cowboy. We need to add :plug_cowboy to the extra_applications list (in application ) as well.

3. Mix deps.get

$ mix deps.get

4. Implement application.ex

# ./lib/webhook_processor/application.ex defmodule WebhookProcessor . Application do @moduledoc "OTP Application specification for WebhookProcessor" use Application def start ( _type , _args ) do # List all child processes to be supervised children = [ # Use Plug.Cowboy.child_spec/3 to register our endpoint as a plug Plug . Cowboy . child_spec ( scheme: :http , plug: WebhookProcessor . Endpoint , options: [ port: 4001 ] ) ] # See https://hexdocs.pm/elixir/Supervisor.html # for other strategies and supported options opts = [ strategy: :one_for_one , name: WebhookProcessor . Supervisor ] Supervisor . start_link ( children , opts ) end end

5. Implement WebhookProcessor.Endpoint

# ./lib/webhook_processor/endpoint.ex defmodule WebhookProcessor . Endpoint do @moduledoc """ A Plug responsible for logging request info, parsing request body's as JSON, matching routes, and dispatching responses. """ use Plug . Router # This module is a Plug, that also implements it's own plug pipeline, below: # Using Plug.Logger for logging request information plug ( Plug . Logger ) # responsible for matching routes plug ( :match ) # Using Poison for JSON decoding # Note, order of plugs is important, by placing this _after_ the 'match' plug, # we will only parse the request AFTER there is a route match. plug ( Plug . Parsers , parsers: [ :json ], json_decoder: Poison ) # responsible for dispatching responses plug ( :dispatch ) # A simple route to test that the server is up # Note, all routes must return a connection as per the Plug spec. get "/ping" do send_resp ( conn , 200 , "pong!" ) end # Handle incoming events, if the payload is the right shape, process the # events, otherwise return an error. post "/events" do { status , body } = case conn . body_params do %{ "events" => events } -> { 200 , process_events ( events )} _ -> { 422 , missing_events ()} end send_resp ( conn , status , body ) end defp process_events ( events ) when is_list ( events ) do # Do some processing on a list of events Poison . encode! (%{ response: "Received Events!" }) end defp process_events ( _ ) do # If we can't process anything, let them know :) Poison . encode! (%{ response: "Please Send Some Events!" }) end defp missing_events do Poison . encode! (%{ error: "Expected Payload: { 'events': [...] }" }) end # A catchall route, 'match' will match no matter the request method, # so a response is always returned, even if there is no route to match. match _ do send_resp ( conn , 404 , "oops... Nothing here :(" ) end end

This looks like a lot, but most of this file is just some helpful comments. The gist is, we are using the macros, get and post, from Plug.Router to generate our routes. This module is a plug itself, and it defines its own plug pipeline. Note, match and dispatch are required in order for us to handle requests and dispatch responses. Pipeline is a key concept here, as the order of plugs determines the order of operations. Notice that match is before we define our parser, this means we will not parse anything unless there is a route match. IF the order was reversed, we’d be trying to parse requests regardless of routes matching. Refer to the docs on Plug.Router for more info.

6. Make the Endpoint Configurable

# ./lib/webhook_processor/application.ex defmodule WebhookProcessor . Application do @moduledoc "OTP Application specification for WebhookProcessor" use Application def start ( _type , _args ) do # List all child processes to be supervised children = [ # Use Plug.Cowboy.child_spec/3 to register our endpoint as a plug Plug . Cowboy . child_spec ( scheme: :http , plug: WebhookProcessor . Endpoint , # Set the port per environment, see ./config/MIX_ENV.exs options: [ port: Application . get_env ( :webhook_processor , :port )] ) ] # See https://hexdocs.pm/elixir/Supervisor.html # for other strategies and supported options opts = [ strategy: :one_for_one , name: WebhookProcessor . Supervisor ] Supervisor . start_link ( children , opts ) end end

We’ve swapped out the hard coded port value of the Cowboy options for an environment variable, this will allow us to run the webhook processor in any environment we need to. Finally, create a config file for each MIX_ENV you need:



#./config/config.exs # This file is responsible for configuring your application # and its dependencies with the aid of the Mix.Config module. use Mix . Config import_config " #{ Mix . env () } .exs" ------------------- # ./config/dev.exs use Mix . Config config :webhook_processor , port: 4001 ------------------- # ./config/test.exs use Mix . Config config :webhook_processor , port: 4002 ------------------- # ./config/prod.exs use Mix . Config config :webhook_processor , port: 80

7. Test

# ./test/webhook_processor/endpoint_test.exs defmodule WebhookProcessor . EndpointTest do use ExUnit . Case , async: true use Plug . Test @opts WebhookProcessor . Endpoint . init ([]) test "it returns pong" do # Create a test connection conn = conn ( :get , "/ping" ) # Invoke the plug conn = WebhookProcessor . Endpoint . call ( conn , @opts ) # Assert the response and status assert conn . state == :sent assert conn . status == 200 assert conn . resp_body == "pong!" end test "it returns 200 with a valid payload" do # Create a test connection conn = conn ( :post , "/events" , %{ events: [%{}]}) # Invoke the plug conn = WebhookProcessor . Endpoint . call ( conn , @opts ) # Assert the response assert conn . status == 200 end test "it returns 422 with an invalid payload" do # Create a test connection conn = conn ( :post , "/events" , %{}) # Invoke the plug conn = WebhookProcessor . Endpoint . call ( conn , @opts ) # Assert the response assert conn . status == 422 end test "it returns 404 when no route matches" do # Create a test connection conn = conn ( :get , "/fail" ) # Invoke the plug conn = WebhookProcessor . Endpoint . call ( conn , @opts ) # Assert the response assert conn . status == 404 end end

These tests are pretty simple, but they confirm our server is working as expected. One could argue that the only thing we should care about from these tests are the response codes, not the side effects that happen when events are processed. Always test UP TO the boundaries of your module, never beyond, unless you are writing an integration style test.

Conclusion

With very little effort we’ve built a small but mighty endpoint. Thanks to Cowboy, you should be able to serve more connections from one server than you’ll likely ever need, so let’s add low cost to the list of benefits as well.

What about deployment? Let's walk through building releases and deploying to AWS:

As always, the code is available on GitHub: https://github.com/jonlunsford/webhook_processor