[Today’s random Sourcerer profile: https://sourcerer.io/banister]

Building REST services with Serverless framework, in Node.js, AWS Lambda and DynamoDB

Serverless computing is a very popular approach to building server-side applications, and the Serverless framework claims to be the most flexible way to build Serverless applications. Rather than stick with a given particular Serverless implementation from Amazon, Google, IBM, Microsoft or others, the Serverless framework abstracts away the particulars of each.

The Serverless framework lets you target any of the popular Serverless platforms. Well, you can target the platforms the Serverless framework supports. See the list on their website: https://serverless.com/framework/docs/providers/

With the Serverless framework, you create a service description using a serverless.yml file, create any code or other assets, and use the serverless command-line tool to upload your service, and to interact with facilities provided by your chosen platform. Behind the scenes it translates your service description file and other source files into the deployment files required by your chosen platform. For example, during deployment to Amazon Web Services, the serverless CLI tool creates an AWS CloudFormation description of your service, and uses the AWS API to create required services and configuration settings in the AWS services described in your serverless.yml file.

Earlier we used the Amazon AWS GUI user interface to configure a simple REST service using API Gateway and backed by a Node.js Lambda function. It took a lot of clicking around in the AWS GUI to get everything setup. While that’s useful for getting started with AWS, it’s not any kind of software development best practice. Instead it is best to have source code checked into source control system like Git. With the Serverless framework, you write source code on your laptop that is easy to commit to a source repository like Git.

In this article we’ll implement a simple REST API backed up by a DynamoDB table. To give a rationale, the table will store messages similar to a discussion board like Slack or Sourcerer Channels. The API will support querying for messages in a given “room”, or by the “user”, and will sort the messages in reverse chronological order so the latest shows at the top of the list. These queries will require a DynamoDB global secondary index.

DynamoDB is a NoSQL database running on the Amazon AWS cloud. Amazon developed DynamoDB after realizing most SQL databases did not use nor require all the complexities of JOINS or FOREIGN KEYS or server configuration or replication, etc. Most software could get along with simple tables instead. It being an AWS service, we can use DynamoDB without configuring anything.

Setup

The Serverless framework CLI tool is a Node.js application, so you must first install Node.js on your laptop. The Node.js homepage (https://nodejs.org/en/) offers downloadable packages, and it is available through various package managers for specific operating systems (https://nodejs.org/en/download/package-manager). Simply follow the instructions, it’s easy and straightforward.

Once Node.js is installed, your laptop will have a command, “node”, and you can get help using the “node — help” option. There’s a whole universe of things one can create using Node.js. Books like Node.js Web Development can get you started, if you like. For this article we’ll simply use it to run the Serverless framework CLI tool.

Install the Serverless tool using the command: npm install -g serverless

The npm command is installed as part of installing Node.js. It gives access to the hundreds of thousands of Node.js packages that can be browsed at https://npmjs.com. This command installs the “serverless” package globally so that it can be accessed anywhere in your laptop.

Once you have it installed, get the help message by running: serverless help

Initializing the application

Let’s start the application by creating the scaffolding this way:

Take a look at the two files, because that will familiarize you with some capabilities of the Serverless framework. The file handler.js is a Node.js based Lambda function that you can use as a starting point. The file serverless.yml contains the description of the blank service.

We’ll be ignoring most of what’s in these files, but they’re still interesting reading.

Getting started with serverless.yml

For starters, edit the front of serverless.yml to have this:

The tableName attribute derives the name for the DynamoDB table. The ${self:…} construct references other data in the serverless.yml file, so that the tableName attribute is calculated based on the setting of the stage attribute elsewhere. This is just the name of the table, we’ll define the table structure later.

The provider section is where we say this is to be deployed on the AWS platform. Lambda functions will be created using the Node.js 8.10 Lambda container. We will be deploying to Amazon’s us-east-1 hosting facility.

The functions section will be (because we are on AWS) used to define the Lambda functions in the service. On the Microsoft Azure platform, the functions section defines Azure Functions, and so on. Each listed function is associated with a handler method. Because we are using Node.js for Lambda function implementation, the handler notation is moduleName.functionName. In this case handler.hello refers to the function named hello in handler.js.

For documentation, see: https://serverless.com/framework/docs/providers/aws/events

The events tag on each function describes how the Lambda function is triggered. Remember that AWS Lambda functions can be triggered by many different event sources. The event source shown here is an HTTP request for a POST on /messages. It’s possible to declare event triggers from any of the other available event sources.

For documentation, see: https://serverless.com/framework/docs/providers/aws

All event triggers for HTTP requests are turned into AWS API Gateway configuration.

Setting up AWS/IAM credentials

Before we go any further, we’ll be deploying what we have just to kick the tires. First we must ensure we have both the AWS and Serverless CLI tools to have access to our AWS account. Deploying the application to the AWS infrastructure we must have an AWS account, and some IAM credentials.

The Serverless Framework documentation has a guide: https://serverless.com/framework/docs/providers/aws/guide/credentials/

If you do not already have an AWS account, sign up for one: https://aws.amazon.com/

After logging-in to the AWS account, go to the IAM service. You’ll need to create an IAM user and then create some IAM access tokens. Those tokens will be shown to you exactly once in the history of this or any other universe, so be sure to copy the tokens somewhere secure. You can easily revoke the tokens and generate new ones if you lose track of the tokens.

While creating the IAM user you’ll be asked for the permissions to grant that user. The simplest is to grant Administrator access but then anyone with those tokens can do anything to your account. It’s more correct to assign just the necessary permissions, to follow the principle of least privilege.

You will need to install the aws-cli tool, so you can use the aws command line. The online documentation shows what to do: http://docs.aws.amazon.com/cli/latest/userguide/installing.html

Once you have aws-cli installed, run aws configure to set up access to your AWS account. You’ll be asked for the IAM tokens generated earlier.

You can now run the serverless config credentials command for further setup.

Deploying the sample application

At the moment we do not have our desired application, but we can deploy something for instant gratification.

Follow the messages and you see that it packages the service, sets up a CloudFormation file, copies things to an S3 bucket, and then prints out characteristics of the service. Most important is a list of HTTP URL’s at the bottom.

We can easily use a tool like Postman — an excellent application for exercising HTTP services — to create a suitable POST request. It might look like this:

The body of the POST request is a JSON object as is the response. In the generated source code we see that the message field in the response is the text shown here. The input object is initialized from the event object passed to the handler function. This gives us an insight into the data available to our Lambda function. Of most interest is the input.body field, which is:

“body”: “{

\t\”messageId\”: \”foo\”,

\t\”datePosted\”: \”today\”,

\t\”room\”: \”everyone\”,

\t\”userId\”: \”mary\”,

\t\”message\”: \”Hello World!\”

}”,

This directly corresponds to the POST’d JSON object, encoded as a string. Obviously we can use JSON.parse to decode that string and access the data.

What we’ve done is prove the ability to send data into a Lambda function, and receive resulting data. That’s pretty powerful for a few minutes worth of work.

A Chatroom REST API

We’ve made some good progress, but what we have is not useful to our application. Let’s first develop the API for the application, and afterward integrate it with a DynamoDB table.

The concept we’re developing is an online discussion area. It will be divided into discussion room’s containing messages. Each message will be posted by a user, identified by a userId. Each message consists of a text block, posted at a specific date/time, and will have a unique identifier. The application web site will need to display the messages for a given room, and for a given user.

These API methods should take care of those requirements. Add them in serverless.yml.

This is enough to get started. We should add to the backlog additional methods to delete or edit/update messages, and of course there’s a whole set of API methods to manage the list of rooms, and to manage user accounts. If this were a real software project we’d be working on those API methods too. These methods are sufficient for exploring the technology.

Next, in handler.js delete the existing hello method, and add four new methods patterned off this:

This is a place-holder method for now so we can test the API. Create four instances of this function, changing the name to match each of the handler methods declared in serverless.yml. This method simply shows us the relevant portions of the event structure so we can test request delivery from HTTP to the Lambda function.

Going back to Postman, we can request this: https://OBSCURED.execute-api.OBSCURED.amazonaws.com/dev/messages?foo=bar

And receive this response:

Or for this URL: https://OBSCURED.execute-api.OBSCURED.amazonaws.com/dev/messages/room/green?foo=bar

We receive this response:

So far so good. We are able to invoke each of the handler methods, and the received data is structured nicely for our consumption.

Using DynamoDB to store chatroom messages

We have the beginning of an API that could be used by a chatroom application. Clearly we need a database to store those messages. That cues another AWS service, the DynamoDB database. As said at the top, DynamoDB is a NoSQL database, which means we aren’t going to be slinging around any SQL statements. Whew!

In DynamoDB each IAM account can create a number of tables. Unlike other database systems, one does not create a database within which tables are created. Instead one simply creates tables. With the Serverless framework, the table is described in the serverless.yml file as a “resource”.

Add this to the serverless.yml file:

With the Serverless framework we can create several types of Resource items. In this case we’re creating a DynamoDB table, as shown in the Type parameter.

The Properties section describes the table. In DynamoDB we are not required to declare all columns that will end up in the table. As a NoSQL database, DynamoDB stores whatever object fields it was given. The only item attributes we must declare are ones which are used in keys for indexes.

The declaration corresponds to what’s shown in the AWS CreateTable documentation: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_CreateTable.html

The actual structure here comes from CloudFormation, as described in this documentation: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-dynamodb-table.html

The AttributeDefinitions section is where the known attributes are defined. It is an array of AttributeDefinition objects. Each contains an AttributeName, and an AttributeType. The latter is a code symbol for the data type where S means String, B means Binary, and N means Number. See https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_AttributeDefinition.html

The KeySchema section describes the primary key for the table. All tables must have a primary key, and the key can be either a simple key like this or a composite key. For a simple key, the HASH of the key value determines the “partition” in the database storage where the item is stored. For a composite key, one must be a HASH key determining the “partition”, and the second must be a RANGE key which is used as a sort key to determine the position within the partition. For details see https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.CoreComponents.html#HowItWorks.CoreComponents.PrimaryKey and https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/WorkingWithTables.Basics.html

In this case we want to make queries not supported by the primary index. DynamoDB supports two additional sorts of index, the LocalSecondaryIndex, and GlobalSecondaryIndex. For details on what this means see the documentation linked previously.

In this case we are defining two GlobalSecondaryIndexes, each with a simple key. See https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_GlobalSecondaryIndex.html

The Projection field determines which item fields are copied into the index entries. In this case we specify that ALL fields are copied.

The ProvisionedThroughput fields describe how much processing resource is assigned to handling queries. One difference between Local and Global indexes is that Local indexes use the processing resources assigned to the table, while Global indexes require their own processing resources. Apparently this fact causes higher cost (fees) on the monthly bill to AWS.

Now that we have the table defined, we have one more task in the serverless.yml file. We must grant permissions to access the table. You’ll find this all through AWS, that every service uses “roles” to grant access to resources, and govern the sort of access.

To that end, change the provider section as so:

The top part is as before. The environment section passes environment variables to AWS services including the Lambda functions. In this case we’re passing the table name and the deployment region to the Lambda function so that we do not have hard-coded values in the function code.

The iamRoleStatements section is an array of roles to grant permissions. The role descriptions listed in the provider section affect the entire service. It is also possible to add role definitions to individual functions, to specify the permissions granted to that function.

In this case we are using one role for the entire service. In theory it is best to assign specific roles to each portion of the service. It’s the Principle of Least Privilege, in which each portion of an application has only the required permissions to do its work and no more. If a miscreant were to break into your application, the more barriers that can be erected against implementing mischief the safer is the entire application. In this case we’ll do only the one.

See: https://serverless.com/framework/docs/providers/aws/guide/iam/

The Resource section is an array of references to resources. In effect what’s being declared is the permissions being granted on access to the named resources.

Each AWS resource is named by an ARN, and you’ll see ARN strings all through the AWS console GUI’s. If we were constructing this service by clicking around the AWS GUI, we could use the ARN values directly. Instead we’re using Serverless, and the ARN’s are generated in the background, Serverless does not show them, and the ARN’s can easily change. What if we generate a Serverless service, publish its code on Github, and other people use the same service definition. The ARN’s generated for our IAM account are not applicable to those other folks, because other ARN’s will be generated in their IAM account.

In short, the serverless.yml file cannot have hard-coded ARN strings. Instead we must dynamically look-up the ARN’s, which is what is happening here.

The strings “Fn::GetAtt” and “Fn::Join” are CloudFormation operations. See: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference.html

For the Fn::Join operation, think of the “join” function in languages like JavaScript. That is, it takes an array of items and concatenates them together to produce a string. The structure is actually an array of two items, the first being a delimiter string, and the second being the array of items to concatenate. In the output string, the delimiter string will be concatenated in-between each item. In this case “/” is the delimiter, and so the items will be /-separated.

The Fn::GetAtt operation gets an attribute of something. In this case the thing whose attribute is being retrieved is the DynamoDB table, named by its resource name in the serverless.yml file. The attribute is the ARN for the table.

In short, this role is granted various DynamoDB permissions, and affects access to the DynamoDB table and to the two indexes. The ARN for an index is the ARN for the table concatenated with the string “/index/INDEXNAME”.

Node.js Lambda function code to access DynamoDB

In serverless.yml we have the DynamoDB table defined, as well as access permissions. We’ve defined four functions corresponding to four REST API methods. The next step is writing the code to glue between the REST method and the database.

We’ll again completely replace the entire code of handler.js with this new code. Start by importing the AWS Node.js SDK

The MESSAGES_TABLE and AWS_DEPLOY_REGION variables receive the environment variables mentioned earlier.

The Node.js module named aws-sdk is automatically available in Node.js Lambda containers. Normally to use a non-Node.js-core module we’d have to setup a package.json listing dependencies on packages in the npm repository. But, AWS pre-configures that module to be available. With this module we can access the entire suite of AWS API’s.

For DynamoDB there are two client objects.

The DynamoDB object: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB.html

And the DocumentClient: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB/DocumentClient.html

The DocumentClient is preferable because it simplifies expressing objects to DynamoDB, and simplifies the objects supplied by DynamoDB. We’ll get to what that means in a second.

The functions provided by the DynamoDB/DocumentClient objects correspond to the abstract DynamoDB API: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/CurrentAPI.html

Let’s start with the handler.js function to create a chat-room message:

Because the Lambda function is implemented using Node.js 8.10, we can use an async function, and the framework will correctly work with the async function. Unfortunately the AWS Node.js SDK does not provide functions which play well in async functions, namely they do not support returning Promise objects. Therefore we must wrap the AWS SDK function calls in a Promise as shown here.

EDIT: Turns out that the AWS Node.js SDK can return a Promise, and the resulting code is much cleaner. Read about this further along.

The HTTP response will be formed from the object returned from this function. The statusCode field becomes the HTTP status. The body field becomes the body of the response. It’s possible to specify an array of headers, which we do not show here.

For details, see: https://serverless.com/framework/docs/providers/aws/events/apigateway/

When we experimented earlier with the service, we saw that the incoming event.body field had a JSON string, and that we must parse that string. Since JSON.parse can throw exceptions, we must catch any thrown exception and turn it into an error on the REST API.

Each AWS SDK function takes a params object declaring the operation to perform. In this case we are using the “put” function to store an item in the database. See https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB/DocumentClient.html#put-property

We use the DocumentClient “put” function instead of the DynamoDB “putItem” function because of how it simplifies specifying object properties. In both cases the params.Items object is what will be stored in the database. In the code above, we might spell it out more clearly since it used an ES6 shortcut:

If we had used the DynamoDB “putItem” function, this object would have to be specified with type specifiers, as so:

These are the same type specifiers as we used earlier when defining the table. When DynamoDB receives the values it must know the data type so the value can be correctly stored. With the DocumentClient, the data type is deduced, while with the DynamoDB client the data type must be explicitly documented. It’s a lot easier on the programmer for the DocumentClient to deduce the data types, so let’s do that.

Now let’s add a handler method to retrieve a single message object:

We’re being very careful here in checking for potential errors so we can return good quality HTTP status codes.

In this case the params object takes a Key object that describes which item to retrieve from the table.

This is another instance where DocumentClient and DynamoDB return different result objects. If we had used the DynamoDB “getItem” method we would have received an object with type specifiers. To be friendly to our caller we would have had to munge the object to remove the type specifiers. But, since we are using DocumentClient, it does that work for us and we can simply respond with the Item object.

The response sent to our caller must have the object encoded as JSON, hence the use of JSON.stringify.

For documentation, see: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB/DocumentClient.html#get-property

Now we need a handler function to handle searching for messages in a given chatroom. Add this to handler.js:

The query function takes a rather comprehensive params object with a lot of options. See: https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB/DocumentClient.html#query-property

In this case we are querying a GlobalSecondaryIndex, named in the IndexName field. The KeyConditionExpression field describes how to match parameters to the keys to select items from the table. Of course “room” is a field name, and “:room” is a parameter that comes from elsewhere. The ExpressionAttributeValues object lists the parameter values that can be plugged into the KeyConditionExpression string.

Follow the code backwards and you’ll see that event.pathParameters.room comes from the URL request path. We are using a parameterized URL in this REST API method. Parameters in the URL arrive in the pathParameters object.

Finally, we have a very similar handler function for querying messages written by a user:

This is the same as the previous function, with a few obvious substitutions made to match the userIndex.

UPDATE: Getting the AWS SDK to return Promises

A reader of this article asked why we used “await new Promise” rather than just getting the DynamoDB methods to return a Promise. Doing so is of course preferable since Promises play very well in async/await functions. The AWS SDK documentation, however, did not discuss getting a Promise out of a method call.

Turns out this is possible, see https://aws.amazon.com/blogs/developer/support-for-promises-in-the-sdk/

Using the AWS SDK methods, leaving off the callback function causes the method to instead return an AWS.Request object. That object has a method, promise, which starts the request and then returns a Promise.

The change is pretty simple:

Deployment and testing

Now that our code is written, deploy it as so:

And now we can go back to Postman or another REST testing client, and exercise the API.

For example a POST on https://OBSCURED.execute-api.OBSCURED.amazonaws.com/dev/messages with a suitable JSON body, gets added to the database. You can then turn around and issue a GET request on the same URL with a query string “?messageId=message5” and receive this response:

Doing so proves that the createChatMessage and getMessage functions work.

What happens if no messageId is specified? The response has a 502 status code, which may be the incorrect status code. What about a messageId that doesn’t exist? Another error is returned.

To test getRoomMessages, make a GET request on https://OBSCURED.execute-api.OBSCURED.amazonaws.com/dev/messages/room/{room}, substituting a room name for “{room}”. For example, querying for the room “everywhere” we might get this result:

Where querying for the room “nowhere” gives an empty array.

An issue to put in your backlog is validating the datePosted field to ensure it contains a valid date string.

Testing getUserMessages is similar but using the URL https://OBSCURED.execute-api.OBSCURED.amazonaws.com/dev/messages/user/{userId}, substituting a user ID for the userId parameter. The behavior is similar to the room query.

Looking behind the curtain

Serverless used AWS CloudFormation to build the service, so let’s take a look at what was built for us. Like in the Wizard of Oz, the public face of our service is the serverless.yml file and associated code, but there is a different story available by poking around the AWS GUI.

Log in to your AWS console, and navigate to the CloudFormation service. You’ll see a list of “Stacks”, so click on the name in your serverless.yml file.

The Stacks screen has a lot of subsections, and it will show the Events section by default. Click on the Resources section, to open up the list of things built for this service. It might look like so:

If this looks like an eye test, it’s because we had to zoom the browser to 70% to get this to fit in one browser window. For what looks like a simple service, the Serverless framework sure created a lot of moving parts.

In many cases the second column is a link leading to the GUI to view the particular service. Of especial use is the AWS::Logs::LogGroup resources which capture logging messages from each sub-service. Is “sub-service” the right word? When the Lambda functions execute “console.log”, the text ends up in the corresponding log, for example.

Another useful stop is the DynamoDB UI:

Finally, for each of the API Gateway methods there is a section dealing with that method.

While it isn’t feasible to edit the API definition, you can inspect what was created, and more importantly you can use the Test screen to test the API.

Suppose the response you’re getting from the HTTP request is confusing. The Test screen can give you a view from closer to the actual implementation.

Conclusion

We’ve learned that the Serverless framework is an excellent way to create a cloud service with the AWS platform. What we did not see is that most of what we did with Serverless can be implemented similarly by directly using CloudFormation. Begging the question of why should we use Serverless in the first place?

What Serverless brings to the table is the ability to rehost the service on other platforms. CloudFormation is obviously specific to the AWS platform. Serverless supports many other similar platforms from other providers.

With a modest sized serverless.yml file, and a modest sized Node.js module, the Serverless framework created an impressively long list of services. This is a powerful result, reminding us of a scene in the movie Fame.

Fame is an early 1980’s movie about the New York City high school that’s focused on the performing arts (music, dance, acting, comedy, etc). One of the music students had a passion for synthesizers and electronic music, but was forced to play in a regular orchestra in the school. One day he complained to the teacher about having to play such a simple instrument (a Violin) that plays only one note at a time. With his synthesizers at home, he said, one can play a whole orchestra at once from one keyboard.

That’s what the Serverless framework offers — the ability to field a whole cloud service stack from a few small source files.