When you move to the cloud and start deploying applications onto Kubernetes one of the first challenges you have to overcome is getting user traffic into your cluster, and hence this is why we created the Ambassador API gateway. The next set of challenges, however, often relate to being able to understand what is happening within your system during steady-state operation, and also, critically, what is occurring when things are going wrong. This is all about making your system observable, and implementing observability often starts at the edge of your system, where the user begins their interaction with your applications. We’ve already written about how Ambassador supports distributed tracing, and how you can view ingress logs, and in this article you will learn about how to collect and visualize metrics with Prometheus and Grafana.

Deployment

This guide will focus on deploying Prometheus and Grafana alongside Ambassador in Kubernetes using the Prometheus Operator.

Note: Both Prometheus and Grafana can be deployed as standalone applications outside of Kubernetes. This process is well documented in the documentation for their individual projects.

Ambassador

Ambassador makes it easy to output Envoy-generated statistics to Prometheus.

Starting with Ambassador 0.71.0 , Prometheus can scrape statistics directly from Envoy’s /metrics endpoint, which removes the need to configure Ambassador to output statistics to StatsD, which was previously the case.

If you are upgrading from an older version of Ambassador, it’s worth noting that the statistics scraped from the /metrics endpoint are not the same as the ones scraped from StatsD, and you will need to configure Envoy to output statistics to a separate collector before being scraped by Prometheus. You can use the Prometheus StatsD Exporter to do this.

1. Deploy the StatsD Exporter in the `default` namespace

2. Configure Ambassador to output statistics to statsd

In the Ambassador deployment, add the `STATSD_ENABLED` and `STATSD_HOST` environment variables to tell Ambassador where to output statistics.

...

env:

- name: AMBASSADOR_NAMESPACE

valueFrom:

fieldRef:

fieldPath: metadata.namespace

- name: STATSD_ENABLED

value: "true"

- name: STATSD_HOST

value: "statsd-sink.default.svc.cluster.local"

...

This will configure Ambassador to output statistics to the Prometheus StatsD exporter.

Prometheus Operator

The Prometheus Operator for Kubernetes provides an easy way to manage your Prometheus deployment using Kubernetes-style resources with custom resource definitions (CRDs).

Deploy the Prometheus Operator

To deploy the Prometheus Operator, you can clone the repository and follow the instructions in the README, or simply apply the published YAML with kubectl .

kubectl apply -f https://raw.githubusercontent.com/coreos/prometheus-operator/master/bundle.yaml

2. Deploy Prometheus by creating a Prometheus CRD.

First, create RBAC resources for your Prometheus instance

kubectl apply -f https://www.getambassador.io/yaml/monitoring/prometheus-rbac.yaml

Then, copy the YAML below and save it in a file called prometheus.yaml

---

apiVersion: v1

kind: Service

metadata:

name: prometheus

spec:

type: ClusterIP

ports:

- name: web

port: 9090

protocol: TCP

targetPort: 9090

selector:

prometheus: prometheus

---

apiVersion: monitoring.coreos.com/v1

kind: Prometheus

metadata:

name: prometheus

spec:

ruleSelector:

matchLabels:

app: prometheus-operator

serviceAccountName: prometheus

serviceMonitorSelector:

matchLabels:

app: ambassador

resources:

requests:

memory: 400Mi

Create the Prometheus instance with kubectl :

kubectl apply -f

3. Create a ServiceMonitor

Finally, we need to tell Prometheus where to scrape metrics from. The Prometheus Operator easily manages this using a ServiceMonitor CRD. To tell Prometheus to scrape metrics from Ambassador’s /metrics endpoint, copy the following YAML to a file called ambassador-monitor.yaml and apply it with kubectl .

If you are running a version of Ambassador higher than 0.71.0 and want to scrape metrics directly from the /metrics endpoint running in the default namespace:

---

apiVersion: monitoring.coreos.com/v1

kind: ServiceMonitor

metadata:

name: ambassador-monitor

namespace: monitoring

labels:

app: ambassador

spec:

namespaceSelector:

matchNames:

- default

selector:

matchLabels:

service: ambassador-admin

endpoints:

- port: ambassador-admin

If you are scraping metrics from a statsd-sink deployment:

---

apiVersion: monitoring.coreos.com/v1

kind: ServiceMonitor

metadata:

name: statsd-monitor

namespace: monitoring

labels:

app: ambassador

spec:

namespaceSelector:

matchNames:

- default

selector:

matchLabels:

service: statsd-sink

endpoints:

- port: prometheus-metrics

At this point, Prometheus is now correctly configured to gather metrics from Ambassador.

Helm

You can also use helm to install Prometheus via the Prometheus Operator. The default helm chart will install Prometheus and configure it to monitor your Kubernetes cluster.

This section will focus on setting up Prometheus to scrape statistics from Ambassador, and it’s worth noting that the configuration of the helm chart and analysis of statistics from other cluster components is outside of the scope of this documentation.

Install the Prometheus Operator from the helm chart

helm install -n prometheus stable/prometheus-operator

2. Create a ServiceMonitor

The Prometheus Operator Helm chart creates a Prometheus instance that is looking for ServiceMonitor s with label: release=prometheus .

If you are running a version higher than 0.71.0 and want to scrape metrics directly from the /metrics endpoint of Ambassador running in the default namespace:

---

apiVersion: monitoring.coreos.com/v1

kind: ServiceMonitor

metadata:

name: ambassador-monitor

namespace: monitoring

labels:

release: prometheus

spec:

namespaceSelector:

matchNames:

- default

selector:

matchLabels:

service: ambassador-admin

endpoints:

- port: ambassador-admin

If you are scraping metrics from a statsd-sink deployment:

---

apiVersion: monitoring.coreos.com/v1

kind: ServiceMonitor

metadata:

name: statsd-monitor

namespace: monitoring

labels:

release: prometheus

spec:

namespaceSelector:

matchNames:

- default

selector:

matchLabels:

service: statsd-sink

endpoints:

- port: prometheus-metrics

Prometheus is now configured to gather metrics from Ambassador.

Notes on the Prometheus Operator

The Prometheus Operator creates a series of Kubernetes Custom Resource Definitions (CRDs) for managing Prometheus in Kubernetes.

+============================+=====================================+

| Custom Resource Definition | Description |

+============================+=====================================+

| AlertManager | An AlertManager handles alerts sent |

| | by the Prometheus server. |

+----------------------------+-------------------------------------+

| PrometheusRule | Registers altering and reporting |

| | rules with Prometheus. |

+----------------------------+-------------------------------------+

| Prometheus | Creates a Prometheus instance. |

+----------------------------+-------------------------------------+

| ServiceMonitor | Tells Prometheus where to scrape |

| | metrics from. |

+----------------------------+-------------------------------------+

CoreOS has published a full API reference to these different CRDs.

Grafana

Grafana is an open source graphing tool for plotting data points. Grafana allows you to create dynamic dashboards for monitoring your ingress traffic statistics collected from Prometheus.

We have published a sample dashboard you can use for monitoring your ingress traffic. Since the statistics from the /metrics and /stats endpoints are different, you will see a section in the dashboard for each use case.

Note: If you deployed the Prometheus Operator via the helm chart, a Grafana dashboard is created by default. You can use this dashboard or set grafana.enabled: false and follow the instructions below.

To deploy Grafana behind Ambassador: replace {{AMBASSADOR_IP}} with the IP address of your Ambassador service, copy the YAML below, and apply it with kubectl :

---

apiVersion: extensions/v1beta1

kind: Deployment

metadata:

name: grafana

namespace: default

spec:

replicas: 1

template:

metadata:

labels:

app: grafana

component: core

spec:

containers:

- image: grafana/grafana:6.2.0

name: grafana-core

imagePullPolicy: IfNotPresent

resources:

limits:

cpu: 100m

memory: 100Mi

requests:

cpu: 100m

memory: 100Mi

env:

- name: GR_SERVER_ROOT_URL

value: {{AMBASSADOR_IP}}/grafana

- name: GF_AUTH_BASIC_ENABLED

value: "true"

- name: GF_AUTH_ANONYMOUS_ENABLED

value: "false"

readinessProbe:

httpGet:

path: /login

port: 3000

volumeMounts:

- name: grafana-persistent-storage

mountPath: /var

volumes:

- name: grafana-persistent-storage

emptyDir: {}

Now, create a service and Mapping to expose Grafana behind Ambassador:

---

apiVersion: v1

kind: Service

metadata:

name: grafana

namespace: monitoring

annotations:

getambassador.io/config: |

---

apiVersion: ambassador/v1

kind: Mapping

name: grafana-mapping

prefix: /grafana/

service: grafana.monitoring

spec:

ports:

- port: 80

targetPort: 3000

selector:

app: grafana

component: core

Now, access Grafana by going to {AMBASSADOR_IP}/grafana/ and logging in with username: admin : password: admin .

Import the provided dashboard by clicking the plus sign in the left side-bar, clicking New Dashboard in the top left, selecting Import Dashboard , and entering the dashboard ID(10434).

Viewing Statistics

So far you have created an environment where you have Ambassador deployed as an API gateway, Prometheus scraping and collecting metrics from Ambassador, and a Grafana dashboard to view these metrics.

You can easily view a sample of these statistics via the Grafana dashboard at {AMBASSADOR_IP}/grafana/ , and logging in with the credentials above

The example dashboard you installed above displays statistics about the API response codes, number of connections, connection length, and number of registered services.

To view the full set of statistics available to Prometheus, you can access the Prometheus UI by running:

kubectl port-forward -n monitoring service/prometheus 9090

and going to http://localhost:9090/ from a web browser.

In the UI, you can click the dropdown to see all of the stats Prometheus is able to scrape from Ambassador.

The Prometheus data model is, at its core, time-series based. Therefore, it makes it easy to represent rates, averages, peaks, minimums, and histograms. Review the Prometheus documentation for a full reference on how to work with this data model.

Next Steps

With this tutorial complete, you can now begin creating a richer monitoring and observability experience at your system’s edge, based on your specific requirements. I always recommend implementing baseline metrics, such as the Golden Signals, as defined by the Google SRE team, but after you have these covered, you will inevitably want to implement other signals that are important to your business.

You can learn more about Ambassador at https://www.getambassador.io. If you have any questions, please join our Slack, drop us a line in the comments below, or @getambassadorio on Twitter.