What problem are we trying to solve?

As an API producer I would like to restrict my REST APIs to specific clients. These restrictions should be dynamically configurable with a simple interface.

I feel if I skip the background then I won’t do justice to the problem, I promise it will be a quick read.

The Platforms Engineering team that I am part of has built an entitlement system (developed using NodeJS & Express) which has become the quintessential backbone providing user authorization data. This entitlement system exposes a bunch of REST APIs that other internal applications can integrate with. We expose a lot of “public” APIs to all of our users, meaning as long as you have a valid OAuth token we allow the APIs to be called. However there are certain APIs that we need to keep private i.e. only certain (OAuth) clients could call them.

Initially we had just couple of clients that needed access to these private endpoints. So the simplest and typical agile approach was to embed the check for client restriction at the endpoint level itself. i.e. something like

router = require ( 'express' ). Router (); router . put '/endpoint-1' , ( req , res , next ) -> if [ 'CLIENT-A' , 'CLIENT-B' ]. indexOf ( req . headers . client_id ) < 0 res . status ( 403 ). send ( 'Invalid client' ); else proceedWithLogic ; router . get '/endpoint-1' , ( req , res , next ) -> if [ 'CLIENT-A' ]). indexOf ( req . headers . client_id ) < 0 res . status ( 403 ). send ( 'Invalid client' ); else proceedWithLogic ;

This enabled the MVP (minimum viable product) to be shipped out and satisfy the immediate need. However, soon, with the increase in popularity of this entitlement system, more clients needed access to these private endpoints. We still couldn’t just expose them to all of our users, so we needed a more flexible way to manage this. We also didn’t want to have to hardcode a new client id every time and redeploy.

That’s for the history….

Design philosophy

Given that our system is built on NodeJS & Express we decided to build an express middleware that would intercept every request to our APIs and perform request validation against a cached route configuration made available to the middleware at server startup and can be refreshed on demand.

What does the configuration look like?

It’s a plain json document like below

[ { url : '/' methods : [ 'GET' , 'PUT' , 'POST' , 'DELETE' ] clientIds : [ 'CLIENT-A' ] } { url : '/endpoint-1' methods : [ 'POST' , 'PUT' , 'DELETE' ] clientIds : [ 'CLIENT-A' , 'CLIENT-B' ] } { url : '/endpoint-1' methods : [ 'GET' ] clientIds : [] } ]

As you might have guessed, with above configuration we are trying to limit POST, PUT & DELETE requests to '/endpoint-1' to the clients CLIENT-A & CLIENT-B

Aiming for minimal configuration

Our main goal was to keep the configuration minimal, because we all know how quickly the configuration can get messy and a nightmare to handle.

So our design aimed for evaluating the restriction matches starting from most distinct to partial matches. What do I mean by this?

e.g. If I want to secure all POST, PUT DELETE request to CLIENT-A & CLIENT-B to below endpoints:

1. /endpoint-1 2. /endpoint-1/{id} 3. /endpoint-1/{id}/sub-endpoint

Then I simply define the config at the most partial match level, so in the above case restricting at /endpoint-1/ will cause this level plus any child routes to be restricted these clients e.g:

url : '/endpoint-1' methods : [ 'POST' , 'PUT' , 'DELETE' ] clientIds : [ 'CLIENT-A' , 'CLIENT-B' ]

You might have noted, I have limited the access to the base route '/' to just one client CLIENT-A, that way we do not accidentally expose a new endpoint that we mean to keep private.

url : '/' methods : [ 'GET' , 'PUT' , 'POST' , 'DELETE' ] clientIds : [ 'CLIENT-A' ]

So how we make another endpoint, say /endpoint-2 , public? Good question! just configure as below:

url : '/endpoint-2' methods : [ 'GET' , 'PUT' , 'POST' , 'DELETE' ] clientIds : []

I hope by now you get the idea on how we provide the routes configuration.

If you have been with me so far, I am sure you would be interested in knowing how the validations are actually performed. And this is the code that actually does the trick.

findMatchingRoute = ( url , method ) -> matchedRoute = appRoutes . find ( route ) -> pattern = route . url . replace ( /\?/g , matchPattern ) + queryParamPattern new RegExp ( pattern ). test ( url ) and method in route . methods if not matchedRoute matchUrl = if url . lastIndexOf ( '/' ) > 0 url . slice ( 0 , url . lastIndexOf ( '/' )) else "/" if url isnt matchUrl then findMatchingRoute ( matchUrl , method ) else matchedRoute else matchedRoute clientIdHasAccess = ( clientId , originalUrl , method = 'POST' ) -> route = findMatchingRoute ( originalUrl , method ) not route or route . clientIds . length is 0 or clientId in route . clientIds validator = ( req , res , next ) -> if clientIdHasAccess ( req . headers [ clientKey ], req . originalUrl , req . method ) next () else res . status ( 403 ). send ( "Invalid Client" )

Basically for each request we perform the below algorithm:

Find a matching route for the current request using pattern matching. If a matched route config is found, then check for any client restrictions on it. If client restrictions exist, then check if the current request is made by one of the allowable clients. If yes, then just let the call through by invoking the next middleware function. Otherwise throw a 403 FORBIDDEN error.

And use the above module in the core project like so:

app = require ( 'express' )() app . use '/' , validator

So far we have solved two problems:

1. Minimal configuration of restricted routes. 2. No more hard coding of client ids.

One more problem still remains and that is how do we get new clients added to the configuration at runtime?

Well that wasn’t such a hard problem to solve, we chose to store our config in an external database, so we can modify the configuration as new client or routes get added. We load this config at startup of the entitlement server, cache this configuration in the validating module and refresh that cache at a set interval (say 10 mins).

So there you go. now we have all our requirements satisfied.

We have a way to secure our APIs to specific clients by means of an easy configuration and one that can be refreshed at runtime.

And guess what, this module is now available as open source. Please check out the project express-client-validator on Github.