An event system receives and processes events by following rules that are defined inside the system. All processing happens asynchronously. When an event is sent to the system, it is processed at some point in time, but you will not get an immediate response. Asynchronous processing has advantages if you want to build a scalable solution because it frees you from the burden of an immediate response. Instead you can queue them and process them as fast as you can. In this article, I will demonstrate how the AWS IoT service can be used to process events that have nothing to do with IoT. I will also use the new Serverless Application Model (SAM) to deploy the solution. Needless to mention that the solution will be serverless and highly cost effective for workloads up to 1 mio events per day.

How AWS IoT works

AWS IoT can do many things, but here I focus on messages, topics, rules, and actions.

A message is sent to a topic. In this case, a message is an event. AWS IoT can deal with JSON so I choose JSON as my data representation.

A topic has a name like event/buy and as you can see you can add a hierarchy by using up to 7 forward slashes ( / ).

A rule subscribes to a topic and triggers actions when a message is received. As simple rule can subscribe to the topic event/buy . But you can also use wildcards like event/+ or event/* . + is used for exactly one hierarchy while * matches to any number of hierarchies.

AWS IoT comes with many built-in actions. To mention just a few:

Write the message to DynamoDB.

Save the message as a file to S3.

Send the message to SNS.

Invoke a Lambda function to process the message.

So a message is sent to a topic. If a rule matches the topic’s name, it triggers the defined actions. That’s an event system, isn’t it?

Architecture

The event system I design in this article can handle events that are generated at exchanges like buy and sell events. It is important to store all events on durable storage for archival. For some reasons buy events need to be checked for fraud. If a fraud event is detected, external systems must be notified. The following figure shows the architecture of the system.

The diagram was created with Cloudcraft - Visualize your cloud architecture like a pro.

Event Flow

It all starts with a HTTP API (provided by API Gateway) that triggers a Lambda function for every HTTP POST call /event . The Lambda event-api does some input validation. Depending on the payload the event is published on a topic like event/buy , event/sell , … One rule subscribes to all event/+ topics with an action to write the message to DynamoDB. Another rule subscribes only to the event/buy topic and triggers the buy-event Lambda for every message. The buy-event Lambda decides if the event is fraud or not. If it is fraud, it publishes the event to the alert/fraud topic. A rule subscribes to the alert/fraud topic and triggers two actions: Save message to S3 and send event to SNS.

Implementation

I use the brand new Serverless Application Model (SAM) for this example. SAM builds upon CloudFormation, so most of the interesting pieces happen inside file that I will name template.yml .

You can find the full source code on GitHub.

The first file defines Node.js dependencies that are needed in the Lambda functions.

package.json

{

"name": "sam-iot-example",

"version": "1.0.0",

"author": "Michael Wittig",

"license": "MIT",

"dependencies": {

"uuid": "3.0.1"

},

"devDependencies": {

"aws-sdk": "2.6.9"

}

}



REST API

This is how you define a APi Gateway with SAM.

template.yml



AWSTemplateFormatVersion: '2010-09-09'

Transform: 'AWS::Serverless-2016-10-31'

Resources:

EventApiLambda:

Type: 'AWS::Serverless::Function'

Properties:

Handler: 'event-api-handler.create'

Runtime: 'nodejs6.10'

Policies:

- Version: '2012-10-17'

Statement:

- Effect: Allow

Action: 'iot:Publish'

Resource: !Sub 'arn:aws:iot:${AWS::Region}:${AWS::AccountId}:topic/event/*'

- Effect: Allow

Action: 'iot:DescribeEndpoint'

Resource: '*'

Events:

Http:

Type: Api

Properties:

Path: /event

Method: post



And here comes the implementation that will run inside Lambda. The name of the file must match with the Handler from above.

event-api-handler.js

;



const uuid = require('uuid');

const cache = require('./cache.js');



function publish(iotdata, payload) {

return iotdata.publish({

topic: `event/${payload.type}`,

qos: 0,

payload: JSON.stringify(payload),

}).promise();

}



module.exports.create = (event, context, cb) => {

console.log(JSON.stringify(event));

try {

var payload = JSON.parse(event.body);

} catch(err) {

cb(null, {statusCode: 400});

return;

}

if (payload.id === undefined || payload.id === null) {

payload.id = uuid.v4();

}

if (payload.type === undefined || payload.type === null) {

cb(null, {statusCode: 400});

} else {

cache.iotdata

.then((iotdata) => publish(iotdata, payload))

.then(() => cb(null, {statusCode: 204}))

.catch((err) => cb(err));

}

};



Event archival

Now it’s time to create a rule that inserts the events into a DynamoDB table.