Simple workflow for building web service APIs

APIs are at the core of server-client communications, and well-defined API contracts are essential to the overall experience of client developer communities. At Yahoo, we have explored the best methods to develop APIs - both external (like apiary, apigee, API Gateway) and internal. In our examination, our main focus was to devise a methodology that provides a simple way to build new server endpoints, while guaranteeing a stable, streamlined integration for client developers. The workflow itself can be used with one of many domain-specific languages (DSL) for API modeling (e.g. Swagger, RAML). The main driver for this project was a need to build a new generation of Flickr APIs. Flickr has had a long tradition of exposing rich capabilities via our API and innovating. One of Flickr’s contributions to this domain was inventing an early version of OAuth protocol. In this post, we will share a simple workflow that demonstrates the new approach to building APIs. For the purpose of this article we will focus on the Swagger, although the workflow can easily be adapted to use on another DSL.

Our goals for developing the workflow:

Standardize parts of the API but allow for them to be easily replaced or extended

Maximize automation opportunities Auto-generation of documentation SDKs API validation tests

Drive efficiency with code re-usability

Focus on developer productivity Well-documented, easy to use development tools Easy to follow workflow Room for customization

Encourage collaboration with the open source community Use open standards Use a DSL for interface modeling



What we tried before and why it didn’t work

Let’s take a look at two popular approaches to building APIs.

The first is an implementation-centric approach. Backend developers implement the service thus defining the API in the code. If you’re lucky there will be some code-level documentation - like javadoc - attached. Other teams (e.g., frontend, mobile, 3rd party engineers) have to read the javadoc and/or the code to understand the API contract nuances. They need to understand the programming language used, and the implementation has to be publicly available. Versioning may be tricky and the situation can get worse when multiple development teams work on different services that represent a single umbrella-project. There’s a chance that their APIs will fall out-of-sync, be it by implementing rate-limiting headers or by using different versioning models.

The other approach is to use in-house developed DSL and tools. There are already a slew of mature open source DSLs and tools available on the market, and opting for this route may be more efficient. . Swagger is a perfect example. Many engineers know it and you can share your early API design with the open source community and get feedback. Also, there’s no extra learning curve involved so the chances that somebody will contribute are higher.

The new workflow

API components

Let’s start by discussing the API elements we work with and what roles they play in the workflow:

Figure 1 - API components

API specification : the centerpiece of each service, including an API contract described in a well-defined and structured, human-readable format.

: the centerpiece of each service, including an API contract described in a well-defined and structured, human-readable format. Documentation : developer-friendly API reference, a user’s guide and examples of API usage. Partially generated from API specification and partially hand-crafted.

: developer-friendly API reference, a user’s guide and examples of API usage. Partially generated from API specification and partially hand-crafted. SDKs (Software Development Kits) : a set of programming language-specific libraries that simplify interactions with APIs. They typically abstract out lower layers (e.g. HTTP) and expose API input and output as concepts specific to the language (e.g. objects in Java). Partially generated from the API specification.

: a set of programming language-specific libraries that simplify interactions with APIs. They typically abstract out lower layers (e.g. HTTP) and expose API input and output as concepts specific to the language (e.g. objects in Java). Partially generated from the API specification. Implementation : the business logic that directs how APIs provide functionality. Validated against API specification via API tests.

: the business logic that directs how APIs provide functionality. Validated against API specification via API tests. API tests: tests validating the implementation against the specification.

API specification - separating contract from implementation

An API specification is one of the most important parts of a web service. You may want to consider keeping the contract definition separate from the implementation because it will allow you to:

Deliver the interface to customers faster (and get feedback sooner)

Keep your implementation private while still clearly communicating to customers what the service contract is

Describe APIs in a programming language-agnostic way to allow the use of different technologies for implementation

Work on SDKs, API documentation and clients in parallel to implementation

There are a few popular domain-specific languages that can be used for describing the contract of your service. At Flickr, we use Swagger, and keep all Swagger files in a GitHub repository called “api-spec”. It contains multiple yaml files that describe different parts of the API - both reusable elements and service-specific endpoint and resource definitions.

Figure 2 - Repository holding API specification

To give you a taste of Swagger here’s how a combined yaml file could look like:

https://gist.github.com/norbertpotocki/ace6642e007ef252facc1b228da46478 Figure 3 - sample Swagger YAML file

One nice aspect of Swagger is the Swagger editor. It’s a browser-based IDE that shows you a live preview of your API documentation (generated from the spec) and also provides a console for querying mock backend implementation. Here’s how it looks like:

Figure 4 - API editor

Once the changes are approved and merged to master, a number of CD (continuous delivery) pipelines kick in: one per each SDK that we host and another pipeline for generating documentation. There is also an option of triggering a CD pipeline generating stubs for backend implementation but the decision is left up to the service owner.

The power of documentation

Documentation is the most important yet most unappreciated part of software engineering. At Yahoo, we devote lots of attention to documenting APIs, and in this workflow we keep the documentation in a separate GitHub repository. GitHub offers a great feature called GitHub Pages that allows us to host documentation on their servers and avoid building a custom CD pipeline for documentation. It also gives you the ability to edit files directly in the browser. GitHub pages are powered by Jekyll, which serves HTML pages directly from the repository. You use Markdown files to provide content, select a web template to use and push it to the “gh-pages” branch:

Figure 5 - documentation repository

The repo contains both a hand-crafted user’s guide and an automatically generated API reference. The API reference is generated from the API specification and put in a directory called “api-reference.

Figure 6 - API reference directory

The process of generating the API reference is executed by a simplistic CD pipeline. Every time you merge changes to the master branch of the API specification repository, it will assemble the yaml files into a single Swagger json file and submit it as a pull-request towards the documentation repository. Here’s the simple node.js script that does the transformation:

https://gist.github.com/norbertpotocki/0627f642064e91af09e2 Figure 7 - merging multiple Swagger files

And a snippet from CD pipeline steps that creates the pull-request:

https://gist.github.com/norbertpotocki/9bb6ca3b3fd6a84b2153 Figure 8 - generate documentation pull-request

The “api-reference” directory also contains the Swagger UI code, which is responsible for rendering the Swagger json file in the browser. It also provides a console that allows you to send requests against a test backend instance, and comes in very handy when a customer wants to quickly explore our APIs. Here’s how the final result looks:

Figure 9 - Swagger UI

Why having SDKs is important

Calling an API is fun. Dealing with failures, re-tries and HTTP connection issues - not so much. That’s where services which have a dedicated SDK really shine. An SDK can either be a thin wrapper around an HTTP client that deals with marshalling of requests and responses, or a fat client that has extra business logic in it. Since this extra business logic is handcrafted most of the time, we will exclude it from the discussion and focus on a thin client instead.

Thin API clients can usually be auto-generated from API specifications. We have a CD pipeline (similar to the documentation CD pipeline) that is responsible for this process. Each SDK is kept in a separate GitHub repository. For each API specification change, all SDKs are regenerated and pushed (as pull-requests) to appropriate repositories. Take a look at the swagger-codegen project to learn more about SDK generation.

It’s worth mentioning that the thin layer could also be generated in runtime based on the Swagger json file itself.

API implementation

The major question that pops out when implementing an API is: should we automatically generate the stub code? From our experience - it may be worth it, but most often it’s not. API stub scaffolding saves you some initial work when you add a new API. However, different service owners prefer to structure their code in various manners (packages, class names, how code is divided between REST controllers, etc.) and thus it’s expensive to develop a one-size-fits-all generator.

The last topic we want to cover is validating implementation against API specification. Validation happens via tests (written in Cucumber) that are executed with every change to the implementation. We validate API responses schema, different failure scenarios (for valid HTTP status code usage), returned headers, rate-limiting mechanism, pagination and others. To maximize code-reuse and simplify test code, we use one of the thin SDKs for API calls within tests.

Figure 10 - Implementation validation

Summary

In this article, we provided a simple, yet comprehensive, overview for working with APIs that we use at Flickr, and examined the key features, including clear separation of different system components (specification, implementation, documentation, sdks, tests), developer-friendliness and automation options. We also presented the workflow that binds all the components together in an easy to use, streamlined way.