First of all, its name comes from scala.meta + Morpheus from The Matrix, for some reason. As Andrea put it when he wrote the first version of metarpheus:

Morpheus is the god of dreams. And it takes our Scala API definitions and transforms it into a type-checked JS API client, or documentation. It morphs Scala into other forms.

Back on track: at buildo, we mainly work on web applications including custom Scala APIs and JavaScript (React) clients. The first step in our journey to produce a client-consumable API contract is to look for it in our Scala source code, and extract it into an intermediate representation.

Parsing and Extracting with scala.meta

Thanks to the awesome scala.meta, metarpheus is able to extract domain models (represented as Scala type definitions) and api routes (spray or akka-http DSL).

Metarpheus uses the scala.meta syntactic API to parse all the relevant scala files and obtain a complete syntax tree out of it. The AST also includes comments, with precise locations: it means we can include them in our intermediate output and, for instance, reproduce them in the final generated JS code.

A thorough review of scala.meta features is out scope for this post but, if you are curious, be sure to take a look at all the things you can do with its powerful api, and all that’s been done already. You can dig into metarpheus code as well (BEWARE: do it at your own risk).

Having obtained the full AST for all the source files of interest, the next step is to extract our API routes, and the “models” (case classes and enums) that are referenced by each route. In other words, we obtain all the API operations and parts of the domain model — the ones exposed to the clients of the API, and thus part of the specification.

Generating JS Code

Once we parsed and extracted all the relevant models and routes, we produce a language-agnostic intermediate representation in JSON.

Given this intermediate representation, other tools can then generate JS code representing type definitions, a JS HTTP client, and reference API documentation.

Speaking of Scala-to-JS, here’s the thing, “full-stack”: this scala code…

case class User(email: String, password: String) class UserRouterImpl extends Router {

override val route = {

// GET /user?id

(path("user") & parameter('id.as[Int]).as(UserId) /**

get a single user

@param id user id

*/)(returns[User].ctrl(userController.user))

}

}

…becomes this JS code

// generated model in JS: export const User = t.struct({

email: t.String,

password: t.String

}); // example usage of the generated HTTP client: api.usersController_getById({

params: { id: 42 }

}) // Promise<User>

A few things to note before proceeding:

we are looking at the full-stack beast (from Scala to JS). As already said, the process also involves an intermediate JSON representation, and we can get more than just JS out of it (more about this later).

In the Scala snippet above, the returns[User].ctrl part is custom DSL. The Spray DSL is very powerful; in our code we put some restrictions on it, and we require e.g. userController.user to adhere to a specific interface. This fact also has the nice side-effect of facilitating metarpheus' parsing job.

The JS code uses tcomb for runtime type-checking in JS. We have a fairly comprehensive list of “standard” mapping from Scala types to tcomb types:

a bunch of “primitive” types, e.g. String , Boolean , etc.

, , etc. a case class becomes a t.struct() (just as in the example above)

(just as in the example above) case enums are expressed as t.enums.of()

some generic types are handled, such as List[T] and Set[T]

and in addition to these, project-specific overrides can be provided via a configuration file

tcomb offers runtime type checking for JavaScript. We are currently experimenting with static type checking (flow and Typescript) on different projects, and already have a method in place to produce Typescript type definitions instead of JS/tcomb code. I must say, it was fairly easy to implement.

At a Higher Level

Let’s take a look at what are the differentiating points (and benefits?) of our custom solution, compared to other more popular ones.

First of all, metarpheus can generate an intermediate representation of the API contract, and then JS, straight out of Scala code. The workflow could be described as “generate client code directly from server code”. It simplifies the somewhat more traditional approach of “generate server and client code starting from a versioned definition file”. Note that this can work because all of our API code bases are written in a single language / router framework: we don’t share definitions among services implemented in different languages, server-side.

In our experience, it has worked out pretty well: the API-client development workflow is not impacted negatively, in particular considering that both Scala and JS people can write down (and iterate on) the server “stubs” (models and api routes) that are fairly simple to define in plain Scala.

Sure, there are other means to obtain the same (generation of the API specification from server code) with other solutions. For instance, using Swagger core annotations. With metarpheus though, we can write Scala code without additional annotations or major restrictions on syntax, basically the same code you’d write if it wasn’t there at all. Our specification is plain Scala code.

One last brief point, about versioning. At buildo, Scala APIs are versioned alongside JS clients, in the same git repository. The developer is in charge of regenerating the type definitions and client code before submitting a PR. Our CIs typically run through a step, called “metarpheus-diff”, where server-client parity is checked by re-generating the definitions and diffing against the versioned ones. In other words, when part of the API changes even slightly, the CI will tell you.

Why not Swagger?

You might be asking, and rightfully so, why not just using Swagger and be done with it? As already said above: we like the idea that our Scala code can be the source of truth itself, the specification.

And well, actually, if needed, we do use Swagger. A fairly common case is the one involving generation of API reference docs. In this case we resort to some of the awesome tools available in the Swagger world: it’s just a matter of “translating” our intermediate language into a valid Swagger specification.

Here below, the JSON intermediate representation produced by the example above ( User and GET /users/:id )

{

"models": [{

"name": "User",

"members": [{

"name": "username",

"tpe": { "name": "String" }

}, {

"name": "password",

"tpe": { "name": "String" }

}]

}],

"routes": [{

"method": "get",

"route": [{

"str": "users"

}, {

"routeParam": {

"tpe": { "name": "Integer" }

}

}],

"returns": { "name": "User" },

"desc": "get a single user",

"name": ["userController", "user"]

}]

}

Using metarpheus-swagger we can easily produce a swagger.yml out of this, and then the docs. The produced Swagger for this example would look something like:

definitions:

User:

type: "object"

properties:

username:

description: "username"

type: "string"

password:

description: "password"

type: "string"

paths:

/users/me:

get:

summary: "gets the currently logged user"

parameters:

-

in: "path"

required: true

type: "integer"

responses:

200:

description: "a User"

schema:

$ref: "#/definitions/User"

Should I Build My Own Custom Solution?

It depends.

As any custom solution, it must be evaluated carefully. For us, it has worked well. One reason is that we use a single server-side language, and the same stack is repeated across many different projects, without profound variations.

This article is meant to show that, if your team is ok with trading generality and standardization in exchange for a solution that fits exactly your stack and workflow needs, it’s indeed possible to build and maintain a custom solution like ours. And lots of fun, too!

Summing Up

Hope you enjoyed reading through the article :) To sum it up, these are the main points I tried to convey:

If you don’t need to support many different server-side languages/frameworks, an intermediate specification language is not strictly needed: you can generate outputs for different client languages, documentation and other artifacts straight out of the API source code.

By leveraging scala.meta it’s possible to extract and generate an API contract and, in turn, client code, leaving the Scala code intact: no need for ad-hoc annotations.

Overall, being very custom for our stack, metarpheus has been serving us pretty well for a few years now, and required a relatively low effort in maintaining and evolving it.

Happy API specification!

—

If you want to work in a place where we care about the quality of our development workflow, take a look at https://buildo.io/careers