Last month I published a tutorial on how to implement Zipkin-based distributed tracing using the Kubernete-native Ambassador API gateway and the Java OpenTracing ‘Microdonuts’ application, and after a few requests I wanted to loop back around now and add LightStep support to this.

Introducing LightStep [x]PM

For those new to LightStep, it is a SaaS-based distributed tracing platform (and more) that is aiming to provide the next generation of Application Performance Monitoring (APM) — [x]PM. If you’re interested, I wrote about some of the history in the distributed tracing and APM spaces on InfoQ recently. The satellite-based architecture of the LightStep platform is interesting, as is the UI and diagnostic tooling provided, and I encourage the interested reader to check out their comprehensive docs.

In essence, the LightStep [x]PM data processing pipeline has two major components: the Satellites, which are standalone software appliances running within a customer’s network or VPC, and the LightStep Engine, the SaaS component. All spans generated by instrumented clients and servers are sent to a pool of satellites where they are processed and temporarily stored during trace assembly. The LightStep Engine queries the satellites, records aggregate information about spans, directs the assembly process, and stores traces durably.

You can experiment with the SaaS platform via the LightStep website, and you will need to create an account and get an access token in order to programmatically send span data to LightStep.

Configuring the OpenTracing Java ‘Microdonuts’ Application

If you clone my fork of the java-opentracing-walktrough GitHub repo and list the directory contents, you should see something similar to this:



...

$ cd java-opentracing-walkthrough

$ ls

Dockerfile README.md kubernetes-ambassador microdonuts

LICENSE client kubernetes-ambassador-lightstep $ git clone git@github.com :danielbryantuk/java-opentracing-walkthrough.git...$ cd java-opentracing-walkthrough$ lsDockerfile README.md kubernetes-ambassador microdonutsLICENSE client kubernetes-ambassador-lightstep

Most of the content is the same from my original tutorial, and you can read this if you want more details. The primary additions for the LightStep integration are located within the kubernetes-ambassador-lightstep folder. If you navigate to here and list the contents, you should see something similar to this:

cd kubernetes-ambassador-lightstep/

$ ls

ambassador-rbac.yaml ambassador-service.yaml lightstep-config.yaml lightstep-tracing.yaml microdonut.yaml tracing-config.yaml

The LightStep tracing config is located within the lightstep-tracing.yaml file. If you open this YAML file within your favourite editor, this is what you should see:

---

apiVersion: v1

kind: Service

metadata:

name: lightstep

annotations:

getambassador.io/config: |

---

apiVersion: ambassador/v0

kind: TracingService

name: tracing

service: "my-collector-grpc.mydomain.com:443"

driver: lightstep

config: {

access_token_file: /config/lightstep_api_key.txt

}

spec:

type: ExternalName

externalName: my-collector-grpc.mydomain.com

You’ll recognise the Ambassador annotation and most of the TracingService details from the previous tutorial, but you’ll also notice that in addition to changing the driver from zipkin to lightstep , I have also defined the TracingService to report the traces to an external URI (my-collector-grpc.mydomain.com, which is where my LightStep trace collector is running outside of the Kubernetes cluster) and specified an access_token_file property within the TracingService config.

The observant among you will have most likely been pondering, “how do we get the /config/lightstep_api_key.txt file into Ambassador? Great question. Ambassador simply runs as a container within a pod, which is in turn defined within a Deployment, just like any other Kubernetes service. Therefore you can mount a ConfigMap as a volume into the Ambassador container with the relevant data.

Let’s first define a ConfigMap, an example of which you can find in the lightstep-config.yaml file:

--- kind: ConfigMap

apiVersion: v1

metadata:

name: lightstep-config

data:

lightstep_api_key.txt: <your key here>

If you are following along, and have a LightStep account, then replace <your key here> with your actual key text. With the ConfigMap defined, you can now map this into the Ambassador container. If you look back into the ambassador-rbac.yaml file, you can see at the bottom of the config the volumeMounts and volumes defined (note the … shows where I have truncated the config to improve readability here):

---

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

name: ambassador

spec:

replicas: 3

template:

...

spec:

serviceAccountName: ambassador

...

volumeMounts:

- name: lightstep-config-volume

mountPath: /config

restartPolicy: Always

volumes:

- name: lightstep-config-volume

configMap:

name: lightstep-config

With all of these pieces of the puzzle complete, you can now deploy Ambassador with LightStep Tracing into Kubernetes. I’m going to use Google’s hosted GKE service (with pre-emptible VMs, just to keep the cost low), but you can use whatever flavour of Kubernetes that works for you.

$ gcloud container clusters create ambassador-tracing-demo --preemptible

...

Creating cluster ambassador-tracing-demo...done.

... $ kubectl create clusterrolebinding cluster-admin-binding-new \

--clusterrole cluster-admin --user < my GCP user name> clusterrolebinding "cluster-admin-binding-new" created $ kubectl apply -f lightstep-config.yaml,ambassador-rbac.yaml

configmap "lightstep-config" created

service "ambassador-admin" created

clusterrole "ambassador" created

serviceaccount "ambassador" created

clusterrolebinding "ambassador" created

deployment "ambassador" created

Let’s now go ahead and deploy the LightStep tracing config:

$ kubectl apply -f lightstep-tracing.yaml

service "lightstep" created

You can now deploy a simple httpbin Service to the cluster, which instructs Ambassador to proxy requests from the /httpbin/ route to the external httpbin.org website.

$ kubectl apply -f ambassador-service.yaml

service "ambassador" created

Now you can query Kubernetes for the Ambassador Service LoadBalancer external IP address. Here is what I saw:

I can see that Ambassador is exposed on the external IP of 35.239.206.165, and so I can make a request to my httpbin service, querying for the IP by specifying the ‘ip’ endpoint in the httpbin API, like so:

$ curl 35.239.206.165/httpbin/ip

{

"origin": "146.148.98.46"

}

You can curl your Ambassador httpbin endpoint a few times in order to make sure you have created a few traces. If you now log in to your LightStep account and navigate to the LiveView, you should see something similar to this:

This view provides an good overview of current traces, both the latency via the histogram in the top half of the page, and the additional individual trace information in the lower half of the page. You can click on the individual traces to get additional information on the trace:

Let’s now deploy the MicroDonut application and associated ConfigMap with our configuration tracing. You’ll note now that I have updated the Java application’s tracing config to send its trace data to LightStep, separate from the work we have already done to configure Ambassador. The tracing-config.yaml file now contains this:

--- kind: ConfigMap

apiVersion: v1

metadata:

name: tracing-config

data:

tracer_config.properties: |

public_directory=../client // Selector for the below config blocks

tracer=lightstep // Jaeger config

jaeger.reporter_host=localhost

jaeger.reporter_port=5775 // Zipkin config

zipkin.reporter_host=zipkin

zipkin.reporter_port=9411 // LightStep config

lightstep.collector_host=< your lightstep collector host >

// The collector_protocol value is either "http" or "https"

lightstep.collector_protocol=http

lightstep.collector_port=8081

lightstep.access_token=< your access token >

If you are following along, you will need to replace <your access token > with you actual text token.

Let’s deploy this into Kubernetes:

You should now be able to access the MicroDonut app via your web browser via the Ambassador external IP and the route /microdonut/ e.g.:

Before you get too hungry, you can order some virtual donuts by clicking on the images, and pressing the “order” button that is displayed below the images. Do this a few times to generate a few traces, and navigate to the LightStep LiveView:

That’s it! Now you can explore and experiment with Ambassador, LightStep, and MicroDonuts.

Conclusion

You can learn more about Ambassador at https://www.getambassador.io, and about the Ambassador Distributed Tracing feature in the Ambassador docs. Visit https://go.lightstep.com/Try-xPM.html to try LightStep today! If you have any questions, please join our Slack, drop us a line in the comments below, or @getambassadorio on Twitter.

Acknowledgements (a Big Thanks!)

I would like to express a big thanks to Alex Gervais from AppDirect for all of his help during this work. Not only did Alex provide guidance, but he has previously done most of the work on the open source Ambassador Tracing contributions, and also fixed a recent bug with the integrations. Many thanks also to the LightStep team, who gave us access to a trial account and also provided useful feedback on the article as it was being written.