Erlang gen_server Reloaded

2016-03-30 by Zsolt Laky

Writing gen_servers has become a routine. We have dozens of gen_servers in our applications, but when we get to a point of extending the server behaviour, we start scratching our heads. Which of our servers have the lowest response time ? How long does it take for a request to be processed? Implementing these metrics may involve changing the code in all of our servers. With Common Service Interface (CSI) we can manage our server’s behaviour and metrics in one place.

Road to use CSI

Introduction

In an Erlang application, usually we have a number of services communicating to process the incoming requests from the clients. Most of the time these services are implemented by writing gen_server code. Some of them may be processor intensive, others could be I/O intensive. When we face a situation in which our system slows down, it is hard to find the root cause of the performance degradation. We may have a clue about which service (or gen_server) takes the resources causing the bottleneck, but on the other hand if we were able to measure basic metrics for all the services, the quest for the root cause would be much easier.

Implementing the metrics in each and every service takes time and carries the risk of non-conformity as different developers may have different ways and ideas on how to implement a common metric.

If we had a generalised gen_server that all the services comply with, implementing common behaviours for all services would mean creating the code once and then reusing it as many times as necessary.

So instead of writing a gen_server for a service, we can use a generealised gen_server, to handle the request by making calls to our callback module that implements the functionalities.

The Common Service Interface (CSI)

In short, the CSI sits between the service API and the service logic. When a request is made the service API calls the CSI and tells it how to process the request. Then CSI takes the order, makes the call to the service logic (a callback module), and responds to the caller with the value the service logic gives back.

So we have two things to consider:

The way to handle the incoming requests. The behavior for the callback module where the requests are processed.

Handling incoming requests

The call flow for any incoming requests is as follows: The caller executes a servicename:function(Parameters) call in order to have a result from the service. In the servicename.erl file (which is the API for the service) all functions have the same format so that they’re able to call the csi:calltype(Parameters) API of the CSI server. There are a couple of call types available that are described later on.

For a given service, this is all that is needed to set up the API.

Processing the request in the callback module

As the CSI server is the gen_server for the implemented service, ithandles the calls made in the service.erl API in it’s handle_call function, by calling the callback module’s function(Parameters) . The callback module returns the results to the CSI gen_server that forwards it to the original requester. Let’s assume the callback module name is servicename_service.erl. The following figure shows the call flow:

The service API module is my.erl, the business logic is in my_service.erl. The generic part of the service is in csigenserver.erl.

A tiny example.

In order to quickly understand the usage of CSI, we will implement a small service, called em_service that provides three functionalities:

process_foo(Atom). - Returns the atom hello_world. process_too_long(Atom). - Sleeps for a long time to test the timeout kill functionality process_crashing(Atom). - Throws an exception.

There are two main parts of our em_service.erl callback module:

1. Behavioural functions

Implementing a service needs some housekeeping. When the service server is launched through a start() or a start_link() function call, init_service() is called in the callback module. Here, the service initialises its global state and returns an {ok, ServiceState} tuple. This `ServiceState is used to pass the global state when starting to process each and every request.

When a request reaches the server, it calls the callback module’s init(Args, ServiceState) function to initialise the processing thread. The Args parameter is the same as the one used when the service request call was made. It returns {ok, RequestState} . That is used during the processing of a special request. For example, here a DB connection could be taken from a pool and be passed back as part of the RequestState.

When init returns {ok, RequestState} our CSI gen_server calls the function that will process the request.

When the request is processed, our gen_server finally calls terminate(Reason, RequestState) callback function to have a cleanup after processing a single request.

terminate_service() is the counterpart of init_service() . It is called when the service is being terminated, so this is the last place where the cleanups shall be executed before a service shuts down.

2. Service functions

All service functions have two arguments - the parameter passed when the function is called (can be a list or a tuple when more than a single parameter is needed) and the state of the request processing that was initialized during the init call.

The implementation for the service functions goes to em_service.erl:

- module ( em_service ). - behaviour ( csi_server ). %% General state of the service - record ( em_state ,{}). %% Lifecycle State for every requests' - record ( em_session_state ,{}). - export ([ init_service / 1 , init / 2 , terminate / 2 , terminate_service / 2 ]). - export ([ process_foo / 2 , process_too_long / 2 , process_crashing / 2 ]). %% ==================================================================== %% Behavioural functions %% ==================================================================== init_service ( _InitArgs ) -> { ok , # em_state {}}. init ( _Args , _ServiceState ) -> { ok , # em_session_state {}}. terminate ( _Reason , _State ) -> ok . terminate_service ( _Reason , _State ) -> ok . %% ==================================================================== %% Service functions %% ==================================================================== process_foo ( _Args , State ) -> { hello_world , State }. process_too_long ( _Args , State ) -> timer : sleep ( 100000 ), { long_job_fininshed , State }. process_crashing ( Args , State ) -> A = Args - Args , { A , State }.

The service functions return a tuple containing {Result, NewState} .

So far we have the business logic implemented. Let’s see how to create the API for the service.

Service API

The entry points to a service are very similar to a gen_server implementation. There are a couple of housekeeping functions like start, start_link, stop and then the exposed functionality is declared.

When launching a service, its start or start_link function shall be called. As our em service uses the CSI gen server, it needs to tell the CSI server, what name to use for registering the service locally and also which module implements the callback functions along with the functionality for the service. So start and start_link have two parameters, the name of the service and the service module. In our case the latter is em_service.erl as created above.

The business logic is called through the CSI server, by csi:call_p(?SERVICE_NAME, function, Args) . What happens then, is the CSI server calls your service module function to perform the operation. You might have recognized, we used call_p instead of the simple call() . There are several ways a server can process a request. It can do a parallel or a serialised request processing. When we use call_p() the CSI server spawns a separate process for handling the request. In this process, the callback module’s init() , function() and terminate() will be called as described above. call_p() is to be used for concurrent request processing. call_s() is similar, but provides serialised processing. Both ways, the requester is blocked as long as the callback function returns.

There are a number of other ways to call a service, here I would mention just one. If instead of call_p() we use post_p() it will immediately return a tuple containing the Pid and a Reference of the process spawned for the request and later the result will be sent back to the requester in an Erlang message.

For simplicity here we will take a look at call_p() only.

- module ( em ). - define ( SERVICE_NAME , em_service ). - define ( SERVICE_MODULE , em_service ). %% ==================================================================== %% API functions %% ==================================================================== - export ([ start / 0 , start_link / 0 , stop / 0 ]). - export ([ process_foo / 1 , process_too_long / 1 , process_crashing / 1 ]). start () -> csi : start ( ? SERVICE_NAME , ? SERVICE_MODULE ). start_link () -> csi : start_link ( ? SERVICE_NAME , ? SERVICE_MODULE ). stop () -> csi : stop ( ? SERVICE_NAME ). process_foo ( Atom ) -> csi : call_p ( ? SERVICE_NAME , process_foo ,[ Atom ]). process_too_long ( Atom ) -> csi : call_p ( ? SERVICE_NAME , process_too_long ,[ Atom ]). process_crashing ( Atom ) -> csi : call_p ( ? SERVICE_NAME , process_crashing ,[ Atom ]).

For a more complex service, you may want to look at csi.erl for the API of CSI itself, csi_service.erl to get a feel of how a more complex service is implemented and of the csi_server.erl where all the magic happens.

Extending CSI functionality

It is easy to addnew functionalities to service servers if they use the common CSI framework. The only place we need to implement it is the CSI itself and all services using it will have that new functionality.

Quick Start with the example

Clone the repository, go to its directory.

$ cd example $ make run

You will have an erlang shell. Here is how to play with it:

( em_server@127 . 0.0 . 1 ) 1 > em : start (). { ok , < 0.83 . 0 > } ( em_server@127 . 0.0 . 1 ) 2 > em : module_info / 0 module_info / 1 process_crashing / 1 process_foo / 1 process_too_long / 1 start / 0 start_link / 0 stop / 0 ( em_server@127 . 0.0 . 1 ) 2 > em : process_foo ( test ). hello_world ( em_server@127 . 0.0 . 1 ) 3 > em : process_too_long ( test ). { error , timeout_killed } ( em_server@127 . 0.0 . 1 ) 4 > em : process_crashing ( test ). { error , exception } ( em_server@127 . 0.0 . 1 ) 5 > 19 : 36 : 30.489 [ error ] Exception in service when calling em_service : process_crashing ([ test ]). error : badarith . Stacktrace :[{ em_service , process_crashing , 2 ,[{ file , " src/em_service.erl " },{ line , 46 }]},{ csi_server , process_service_request , 8 ,[{ file , " src/csi_server.erl " },{ line , 402 }]},{ proc_lib , init_p_do_apply , 3 ,[{ file , " proc_lib.erl " },{ line , 237 }]}] 19 : 36 : 30.490 [ error ] CRASH REPORT Process < 0.89 . 0 > with 0 neighbours exited with reason : exception in csi_server : process_service_request / 8 line 425 ( em_server@127 . 0.0 . 1 ) 5 > csi : stats_get_all ( em_service ). [{{ response_time , process_foo },{ 1 , 106 , 106.0 , 106 , 106 }}] ( em_server@127 . 0.0 . 1 ) 6 > csi : services services / 0 services_status / 0 ( em_server@127 . 0.0 . 1 ) 6 > csi : services_status (). [{{ registered_name , em_service }, { csi_service_state , em_service , em_service , { em_state }, true , 36888 , 40985 , csi_stats , [ all ], [], [{ response_time ,[{ " last_nth_to_collect " , 10 }, { " normalize_to_nth " , 8 }]}]}}, {{ registered_name , csi_service }, { csi_service_state , csi_service , csi_service , { csi_service_state }, true , 20500 , 24597 , csi_stats , [ all ], [], [{ response_time ,[{ " last_nth_to_collect " , 10 }, { " normalize_to_nth " , 8 }]}]}}] ( em_server@127 . 0.0 . 1 ) 7 > em : stop (). ok ( em_server@127 . 0.0 . 1 ) 8 >

And if you use lager, here is what you find in the console.log after the sequence above:

2015 - 08 - 04 19 : 35 : 35.973 [ info ] < 0.7 . 0 > Application lager started on node 'em_server@127.0.0.1' 2015 - 08 - 04 19 : 35 : 35.980 [ info ] < 0.7 . 0 > Application csi started on node 'em_server@127.0.0.1' 2015 - 08 - 04 19 : 35 : 35.980 [ info ] < 0.7 . 0 > Application em started on node 'em_server@127.0.0.1' 2015 - 08 - 04 19 : 35 : 35.984 [ info ] < 0.7 . 0 > Application runtime_tools started on node 'em_server@127.0.0.1' 2015 - 08 - 04 19 : 36 : 30.489 [ error ] < 0.89 . 0 > @ csi_server : process_service_request : 415 Exception in service when calling em_service : process_crashing ([ test ]). error : badarith . Stacktrace :[{ em_service , process_crashing , 2 ,[{ file , " src/em_service.erl " },{ line , 46 }]},{ csi_server , process_service_request , 8 ,[{ file , " src/csi_server.erl " },{ line , 402 }]},{ proc_lib , init_p_do_apply , 3 ,[{ file , " proc_lib.erl " },{ line , 237 }]}] 2015 - 08 - 04 19 : 36 : 30.490 [ error ] < 0.89 . 0 > CRASH REPORT Process < 0.89 . 0 > with 0 neighbours exited with reason : exception in csi_server : process_service_request / 8 line 425

Where to go from here?

CSI has an additional functionality to measure the service performance. A WombatOAM plugin is also there to visualize the metrics on the function level. For the full description, please visit: GitHub.

Summary

As gen_server generalises the behaviour for the servers, CSI is the next step for generalising gen_server-s to use common patterns for writing services.

Acknowledgement

Thanks to Sqor Sports for making this repository public and contributing to open source community. For code and examples, please visit GitHub.

Sqor Sports will feature on our next webinar on 7 April: ‘Building a mobile chat app’, you can register for it here.