Monday, May 23, 2016 By Chris Mikkelson

Introduction

Go's standard library support for JSON is an expedient method of implementing configuration files. An application can define a configuration struct type containing all the tunable elements, and load JSON from a file into an instance of this struct, for example:

type Config struct { ServerUrl string APIKey string MaxSessions int } func readConfig ( filename string ) ( * Config , error ) { // initialize conf with default values. conf := & Config { Url : "http://localhost:8080/" , MaxSessions : 10 } b , err := ioutil . ReadFile ( "./conf.json" ) if err != nil { return nil , err } if err = json . Unmarshal ( b , conf ); err != nil { return nil , err } return conf , nil }

This is already an improvement over loading JSON into an untyped dictionary structure, as would happen in Javascript or Python, in that json.Unmarshal provides some minimal validation. If configured with:

{ "MaxSessions": "quite a few" }

json.Unmarshal will return a error indicating that "quite a few" is not a valid integer, and the program can handle this error at startup rather than later in the runtime.

Customizing

The Go JSON library supports Unmarshaling into custom types through an interface type json.Unmarshaler :

type Unmarshaler interface { UnmarshalJSON ([] byte ) error }

If our config struct contains a field of a type satisfying json.Unmarshaler , that field's UnmarshalJSON method will be called to fill in the field. This has a number of uses which can make config handling more pleasant.

Loading Values From Elsewhere

One issue with the above example is the API key in the config. Configs get copied and pasted and checked in to version control, and over time this runs the risk of leaking API keys. A simple solution is to have the API key stored in a separate file, and read it in, and store this filename in the config instead:

type Config struct { ... APIKeyFile string }

but this requires a separate step to load the API key from the file and store it elsewhere. If you have multiple config values needing the same handling, this gets repetitive. A custom Unmarshaler streamlines this process dramatically.

type FileString string func ( f * FileString ) UnmarshalJSON ( b [] byte ) error { var s string err := json . Unmarshal ( b , & s ) if err != nil { return err } f , err := ioutil . ReadFile ( s ) if err != nil { return err } val := strings . TrimSpace ( string ( f )) * f = FileString ( val ) return nil } type Config struct { // ... APIKey FileString // ... }

Now, the Config APIKey field is a string read from the file named in the config.

Validation and Parsing

The above example adds some extra error conditions to loading the config, namely returning appropriate errors if the API key file does not exist or can't be read. We can take this effect a step further and validate that the ServerUrl parameter contains a valid URL using the net/url library.

type Url string func ( u * Url ) UnmarshalJSON ( b [] byte ) error { var s string err := json . Unmarshal ( b , & s ) if err != nil { return err } * u = Url ( s ) _ , err = url . Parse ( s ) return err } type Config struct { ServerUrl Url // ... }

With this, if the user supplies an invalid URL string for the URL parameter, it will be flagged as an error. The ServerURL field will always be populated with a valid URL string

Note that we are calling the parser above, but throwing away its results. Those results are useful, we should keep them around!

type Url struct { * net . URL } func ( u * Url ) UnmarshalJSON ( b [] byte ) error { var s string err := json . Unmarshal ( b , & s ) if err != nil { return err } u . URL , err = url . Parse ( s ) return err }

Now, the ServerURL field is a struct embedding a *net.URL , with all the fields and methods of that type. The string value is available with .ServerUrl.String() .

Bringing it all together

The above two techniques can be combined to load more complicated configurations. For example, the crypto/tls library provides a Config structure that many TLS-supporting libraries use. The Config structure contains parsed forms of certificates and CA certificates. With a custom Unmarshaler , we can have the user supply file names for certificates and keys, and load them into a tls.Config :

type certFiles struct { KeyFile , CertFile string } type tlsConfigFiles struct { CertFile string KeyFile string CACertFile FileString } type TLSConfig struct { * tls . Config } func ( t * TLSConfig ) UnmarshalJSON ( b [] byte ) error { var cf tlsConfigFiles err := json . Unmarshal ( b , & cf ) if err != nil { return err } pool := x509 . NewCertPool () if ! pool . AppendCertFromPEM ([] byte ( cf . CACertFile )) { return errors . New ( "invalid CA Cert" ) } cert , err := tls . LoadX509KeyPair ( cf . CertFile , cf . KeyFile ) if err != nil { return err } t . Config = & tls . Config { RootCAs : pool , Certificates : [] Certificate { cert }, } return nil } struct Config { //... TLS TLSConfig }

With this, any library call needing a tls config can use conf.TLS.Config .

Conclusion

The Unmarshaler interface in the encoding/json Go library is a powerful abstraction. It allows you to intercept the parsing of a loosely-structured JSON object and validate or transform the underlying values into the form the application needs them.

Although this post was JSON-centric, analogous Unmarshaler interfaces appear in the standard XML library and the leading third party YAML libraries so the above techniques would work for XML and YAML configs.

Chris Mikkelson is a Senior Distributed Systems Engineer at Farsight Security, Inc.