Photo by Émile Perron on Unsplash

Throughout the last couple of years I’ve been in a number of Node.js projects and one of the recurring issues is the way people manage the configuration and secrets.

First of all that these are the major aspects of what I’m trying to achieve with configuration / secrets management.

Flexibility

Allow the software developers to change the configuration. Most of the times it’s Ops duty to manage the configuration but this adds an unnecessary overhead and slows down the debugging / fixing process. A lot of the times it’s developers who define the configuration options and thus they’re the ones knowing best the proper configuration values. Consider the following case:

Application built in Scala (Akka) uses typesafe-config for server configuration and specifies the threadpool / dispatcher options with it. At one point application has started facing severe performance issues. Because Ops don’t have enough insights into application’s internal architecture (they have no idea about threadpools and they know even less about which Actors / Futures run at which pools) the fixing flow would look like the following:

Software engineer would replicate the problem locally Software engineer would find the cause of the problem and after some playing around would come up with proper settings for threadpools Software engineer would reach out to Ops to change the production configuration accordingly

In this process step 3 is unnecessary. Moreover, given that proper CI/CD pipeline is setup and proper development practices are enforced — step 3 wouldn’t go through peer review or be tracked in any way

Even more widespread case is when the developer needs to raise the debuglevel for a particular service. Why not give a developer an option to do it himself?

Given that there is a peer review this is potentially just as breaking as any other code change

Security

Secret values (DB passwords, API access keys, etc) should never appear on the repo. Period.

This aspect is a bit contradictory with “Flexibility” but it’s quite easily solvable with most configuration libraries. How? You have all non-secret values on a repo and then inject secret values during the deployment (I prefer using ENV variables for it)

Verbosity

It’s very useful to avoid having to redefine the whole configuration for every environment. Most configuration systems allow to do that by defining the config files processing order. They normally start with the most generic config file and then override the config values with more specific config files. If the library allows to do that — I would take that any day

Example: your DB connection pool setting are the same across all the environments (let’s say min: 2, max: 10). But for the production environment you want more aggressive pooling (min: 10, max: 80). They way you’d do that with config

default.js:

const config = {

database: {

client: 'mysql',

connection: {

host: 'localhost',

user: '',

password: '',

database: '',

},

pool: {

min: 2,

max: 10,

},

},

}



module.exports = config;

production.js:

const config = {

database.pool: {

min: 10,

max: 80,

},

}



module.exports = config;

Simplicity

A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away. Antoine de Saint-Exupery

Of course there are a number of solutions that solve at least some of the above problems. The questions is do you really want to install an additional component into your infrastructure just to solve that problem. Additional components come with increased complexity which leads to increased error rate, increased debugging time and reaction time

Some of the solutions I’ve used:

HashiCorp Vault — good for secret management but a bit of an overkill unless you have over 9000 microservices in tens of landscapes Chef/Puppet/Ansible for configuration management — again in my experience, those are the great tools for setting up the servers but when it comes down to custom software configuration something always break apart Consensus systems — ZooKeeper/etcd/consul. Some people use them to store all the configuration there and make applications interface with those. Most of the times people do that when they already have one of those systems in place (e.g. people running Kubernetes have the etcd out of box).

I’ve done most of those with mixed success. There’s a famous Russian idiom “to hammer the nails with a microscope” and I always felt like that was exactly what I was doing

My way

In Node.JS there’s a nice configuration library that does all the trickery and deceit for me. Project structure:

/

— package.json

— /src

— /config

— — /local.js

— — /default.js

— — /development.js

— — /production.js

— — /<otherenv>.js

— — /custom-environment-variables.js

The file content is as follows:

local.js — contains full configuration, should be put under .gitignore and never appear on a repo

and never appear on a repo default.js — contains the top-level generic configuration keys that are shared by all enviroments with the secret values blanked out

<env.js> — contains the specific overrides for a particular environment. Environment is defined using NODE_ENV environment variable

environment variable custom-environment-variables.js — maps the secret configuration keys to env variables to be later injected at runtime

I normally use it with Kubernetes that allows to store the secret values and inject them using environment variables in a convenient way

Example repo is here

Happy hacking and don’t reinvent the wheel!