Jaeger Tracing on Kubernetes with ASP.NET Core and Traefik

possibly the hard way

This is a follow up to an article I wrote earlier which used the Jaeger all-in-one deployment. This tutorial will demonstrate a production ready Jaeger deployment, as well as some improved .net core code. Unlike the earlier article, I’ve split up the Jaeger Collector, Query, and Agent into separate components with Elasticsearch for storage. We’ve also added support for Jaeger Tracing to Traefik to follow traces through our edge router.

Its worth mentioning that the Jaeger space is changing quickly, and there might be easier ways to setup Jaeger on your cluster, like this.

Jaeger Agent

The Jaeger agent listens for spans from the client and sends them to the Jaeger collector.

The agent yaml is basically the same as the template with a few changes to add resource limits and to embed the configuration in a secret, which is better if you have sensitive configuration data, like an elasticsearch password. If you are considering whether to use a sidecar or daemonset, this is a good blog post that compares the differences. We chose to go with a daemonset, and as a daemonset we will have access to the agent ip via the Kubernetes downwards api, which looks like this:

access to agent/node ip through env variable.

This gives pods the ability to access the node ip on which they are running, i.e the jaeger agent host ip.

Jaeger Collector

The Jaeger collector pulls traces from the Jaeger agents and validates, indexes, transforms, and stores them. For our storage we used an existing Elasticsearch cluster.

Again, this is similar to the collector template provided here except we added resource limits and requests (we found that the collector was not very resource intensive) and pulled config values from a secret. If your Elasticsearch instance is accessed with TLS then you need to mount your certs on the Jaeger collector and include the --es.tls flag. We also mounted a sampling strategy json file to enable remote sampling. Remote sampling allows you to dynamically control the sampling strategy for different services and endpoints directly from the collector. This means you can change your trace throughput without having to restart your services.

Jaeger Query

We made a couple small changes here, namely hiding the service behind an ingress, so that we could easily expose our UI at an organization hostname. We also secured the Jaeger UI with basic authentication. Similar to the Jaeger collector we need the --es.tls flag and mounted certs.

What did it take to configure jaeger to use elasticsearch? Basically you need to tell the Jaeger Query and Collector where to look for the certs like so:

Configure Traefik with Jaeger

If you want to follow traces through your edge router, in this case Traefik, through to your service, here is how you can do it. The following lines can be added to your traefik.toml file. We set the tracing type to be rate limiting and sample 1/1000 requests.

Traefik also needs to know the Jaeger Agent host and port. For us, these values lived in environment variables (see the Jaeger Agent section above). Which means that we couldn’t set the localagenthostport or the samplingserverurl configuration values in the toml file, and instead had to pass them in as args to the docker image.

Once Treafik is setup the next step is to link its traces to your services. As an edge router is should appear as the parent to all service child spans.

Asp.Net Core Code

To make it easier for other developers to use Jaeger I established some default values and configuration that can be used during dependency injection in startup.cs . To do this I first defined some default Jaeger options like so:

Furthermore, I implemented an IServiceCollection extension which simplifies the initial Jaeger setup.

Our defaults include a GuaranteedThroughputSampler and a reporter configured with the jaeger agent host and port. Furthermore, adding services.AddOpenTracing() will enable automatic tracing of our endpoints. However, this means your /health and /metrics endpoints will also generate traces. So to avoid the bloat from these traces you can ignore them all together by specifying ignore patterns. Now that we’ve got the ease-of-use code in place we can setup Jaeger with the following simple lines in startup.cs :

Override defaults by setting options.SamplingRate and options.LowerBound .

Thanks for reading, and hopefully this simplifies the process of using Jaeger tracing for you.

Additional Resources