This makes easy to debug problems, for example, big latencies spikes, and understand the full cycle of an API request and all the services implicated.

APIcast and Observability

APIcast (https://github.com/3scale/apicast) is an Open Source API Gateway whose main focus areas are high performance and extensibility. It is part of the Redhat 3scale API Management solution, and is used by hundreds of companies around the world to expose their APIs in a secure and controlled way.

It’s based on Openresty, which is a full-fledged web platform that integrates the standard Nginx core, LuaJIT and Lua Libraries with 3rd-party Nginx modules. So we combine the performance and robustness of NGINX with an easy way to extend/change it’s functionality by using LUA.

APIcast is being deployed in multiple scenarios, for example a basic and common deployment, where APIcast is the entrypoint of your API and all requests go through it:

But more and more, our users are deploying APIcast as the API Gateway of big and complex microservices deployments either at the front or between specific services.

As you can imagine, we are talking about hundreds of microservices interacting with each other, and each external requests ends up creating several requests between services. This raises new challenges, and one of them is: “Observability”

Why this request was so slow?

Which service is creating those latency peaks?

Which services are interacting when a user hits this endpoint?

Why the request failed?

Who initiated this request?

…

Right now, answering those questions is slow and complex, as the users start debugging via logs and trying to correlate everything.

We need to start looking at how microservices are monitored, and distributed tracing shows up as the pattern that helps us to get insights into what is going on inside the system.

What is required to add distributed tracing to our microservice deployment?

Each external requests should have a unique request ID attached, usually via a HTTP header

attached, usually via a HTTP header Each service should forward the request ID to other services

to other services Each service should output the request ID in the logs

Each service should record additional information , like start and end time of the request.

, like start and end time of the request. Logs will need to be aggregated somewhere, and provide a way to parse via HTTP request ID.

For APIcast, that is simple, attach a unique ID as a HTTP Header and we are done…

But the user will need to define a log format across all the services, what parameters to be stored in the log, the http header to use, how to capture the start/end time of each requests…correlate them, check how to do that in third party applications, try to modify those…

So we ask ourselves, Shouldn’t there be a “standard” way of doing this?

And that’s where OpenTracing comes into the scene.

OpenTracing

OpenTracing (http://opentracing.io) offers consistent, vendor-neutral APIs, Open source libraries and documentation for developers to instrument and add different tracing implementations into their applications.

Currently the OpenTracing project provides libraries for the following languages:

Go

Python

Javascript

Java

C#

Objective-C

C++

Ruby

PHP

And there’s a huge community pushing new contributions, check: https://github.com/opentracing-contrib.

Once you have instrumented your application by adding the OpenTracing libraries, you will need to choose the tracing implementation, these are the supported ones:

Jaeger

Appdash

LightStep

Hawkular

(…)

We decided to go with Jaeger, as it is part of the CNF, has Openshift-ready deployments https://github.com/jaegertracing/jaeger-openshift, and Red Hat is contributing to it.

Tracer: Jaeger

Jaeger is a distributed tracing system developed by Uber Technologies, inspired by Dapper and Openzipkin. It supports multiple storage backends, is OpenTracing native and includes a modern web UI and it’s designed for performance.