Icons made by Freepix from https://www.flaticon.com.

Serverless computing continues to grow in the industry with increased speed to market, and agility for developers. In response, cloud architects, devops engineers, and infrastructure developers move up the technical stack, finding new ways to empower product teams. These in-house solutions solve pain-points, enable agility, and maintain security. I am pleased to announce our latest solution, and Neiman Marcus’ newest open source project: PSM.

PSM, short for Parameter Store Manager, is a security enabled simple REST service to manage application configuration in AWS SSM Parameter Store. In this blog I will take you through the PSM application, and what’s under the hood.

You can get right to the code here…

Update 01.26.2020

Hemanth Jayaraman, Omar Shabir, and I spoke about PSM, and serverless security at DevSecOps days austin!

Background

Serverless Computing

Neiman Marcus continues its growth toward the cutting edge of ecommerce and application technology, by embracing serverless technology. More and more applications served to our customers take advantage of progressive web apps, NoSQL databases, static content, and distributed services behind API’s.

These technologies abstract underlying resources from the application, meaning less to manage. This is fantastic for speed to market. Keeping up with these trends, tooling has also evolved. Neiman Marcus leverages Serverless Framework on AWS to provision application services. Serverless Framework does a fantastic job of putting code onto the web with simple configuration.

Sample serverless.yml, used in Serverless Framework to provision services.

Secrets & Configuration Management

While working with serverless technologies and Serverless Framework over the years, some consistent patterns have emerged in how we write our infrastructure. One of these patterns, includes using AWS SSM Parameter Store for configuration and secrets management.

Applications require deployment and run-time configuration, which might differ in various environments or stages. By moving configuration and secrets to an external data store, the application and deployment code can remain DRY, and secure while following infrastructure-as-code best practices: Keeping the configuration with the code.

A Pain-Point

Interestingly, an unforeseen pain-point developed. When moving configuration to parameter store, developers would now require access to push their own configuration. How do we achieve this, while still limiting access to our AWS accounts? How does this change our deployment workflow?

Enter PSM

It became clear, that an internal application should be developed for pushing configuration to parameter store and allow for it to be run in CI/CD. PSM was developed to solve these pain-points, while maintaining security and agility.

Under the Hood

Functions

PSM uses the following three lambda functions:

Encrypt — Encrypt secrets via KMS CMK to securely store in source control

Update — Update configuration in AWS SSM Parameter Store

View — Retrieve and view configuration from AWS SSM Parameter Store

Encryption

Secrets are an important part of application configuration and was a necessary requirement for the success of our fast-paced deployments and changes.

Every function within PSM handles secrets well. The encrypt function will encrypt a string against a CMK, which cannot be decrypted without elevated access. The update function has access to decrypt that secret and put it in parameter store as a secure string. Finally, the view function retrieves configuration, encrypting secure strings against the CMK.

Calling the Application

The application sits behind an API, while lambda does all the magic behind the scenes. The encrypt function is a simple POST or GET API call. The POST API will encrypt given data, and the GET API will generate a random value that is encrypted. The update and view functions take the appId and stage query string parameters. All these choices make for a simple, templated approach that can easily be replicated in automation.

Example curl command to update configuration with PSM

Workflow & Usage

It’s all about the workflow. Typically, developers encrypt their secrets with the encrypt function and store it in json formatted configuration files, pushing them to source control. Those configuration files reside in a config directory, with the name of the file matching the stage name. This enables stage specific configuration, and automation templating.

Config file example

Example repository structure

Next, within automation, there should a be stage that puts configuration into parameter store, calling the update function. Neiman Marcus leverages parameterized CI builds, which makes moving between stages and environments easier. With parameterized builds, stage specific configuration is deployed depending on the where an application is in the deployment lifecycle. For example, QA configuration can be deployed during the normal QA application deployment process.

Now at run-time, or during the application build, configuration is fetched from parameter store.

Calling parameters in your serverless.yml

Conclusion

Through some creative thinking, Neiman Marcus has enabled self-service configuration and secrets management that can work as fast as developers require. We’ve seen a dramatic increase in speed, while maintaining security.

Hopefully this blog has been a helpful look at simple configuration management, but also inspired you to see where your own in-house solutions can solve these types of problems.

Please check out our project on github!

Thank you for your time!

CRD