As part of their organization’s digital transformation, more and more customers are electing to use a managed Kubernetes service, like Amazon EKS, as their container-orchestration system of choice to deploy, scale, and manage microservices.

As the number of microservices grow within an application, it becomes difficult to pinpoint the exact location of errors, re-route traffic after failures, and safely deploy code changes. A service mesh, like AWS App Mesh, makes it easy to run services by providing consistent visibility and network traffic controls for services built across multiple types of compute infrastructure such as Amazon EC2, Amazon EKS, and AWS Fargate.

Before service mesh, developers had to use a language-specific library to implement features like backoffs, or retries, now they can decouple these features from the application and spend more time on the business logic and less time on non-functional requirements.

But operation or SRE teams still need to create and manage access to the applications from outside the Kubernetes cluster. Ingress controller has become a popular solution to solve this type of challenge.

Today, we will take a look at Gloo (from solo.io) which is a feature-rich, Kubernetes-native ingress controller, and API Gateway based on Envoy. Gloo is exceptional in its function-level routing, its support for legacy apps, microservices, and serverless applications. It can easily run on AWS using AWS App Mesh.

In this blog post:

We will create an EKS cluster with add-ons to create and manage AWS App Mesh automatically.

We will deploy a sample application using AWS App Mesh.

We will install and configure Gloo as the Ingress Controller.

And finally we will use Gloo for a canary deployment.

Prerequisites

Before starting, we need to install the following tools on our local computer:

kubectl 1.13

awscli (installed and configured with us-west-2 as the default Region)

aws-iam-authentificator

eksctl

glooctl (0.20.10 at the time of writing this article)

Getting started

We will start by creating an EKS cluster in the us-west-2 Region using a YAML config file:

#create eks config file cat <<EoF > eks-config.yaml apiVersion: eksctl.io/v1alpha5 kind: ClusterConfig metadata: name: GlooDemo region: us-west-2 version: "1.13" nodeGroups: - name: ng1-GlooDemo instanceType: m5.large desiredCapacity: 2 iam: withAddonPolicies: autoScaler: true #allow access AWS app mesh appMesh: true #allow access to AWS X-Ray xRay: true #allow access to AWS CloudWatch cloudWatch: true EoF #create eks cluster eksctl create cluster --auto-kubeconfig -f eks-config.yaml #load the EKS cluster config export KUBECONFIG=${HOME}/.kube/eksctl/clusters/GlooDemo

The cluster creation will take up to 15 minutes.

To simplify the creation and the management of AWS App Mesh, we will install two add-ons:

aws-app-mesh-controller-for-k8s that will manage AWS App Mesh resources for a Kubernetes cluster.

that will manage AWS App Mesh resources for a Kubernetes cluster. aws-app-mesh-inject that will be responsible for automatically inject the App Mesh container as a sidecar. To enable sidecar injection for a namespace, it is necessary to label the namespace with appmesh.k8s.aws/sidecarInjectorWebhook=enabled

#install aws-app-mesh-controller-for-k8s kubectl apply -f https://raw.githubusercontent.com/aws/aws-app-mesh-controller-for-k8s/v0.1.1/deploy/all.yaml

Now we will confirm that the Kubernetes custom resources for AWS App Mesh were created with the following command:

kubectl get crd NAME CREATED AT eniconfigs.crd.k8s.amazonaws.com 2019-10-26T00:51:33Z meshes.appmesh.k8s.aws 2019-10-26T00:59:52Z virtualnodes.appmesh.k8s.aws 2019-10-26T00:59:52Z virtualservices.appmesh.k8s.aws 2019-10-26T00:59:53Z

The aws-app-mesh-inject add-on needs to know the name of the mesh before being installed.

We will use color-mesh as our mesh name, and to help with visibility and tracing, we will also add the X-Ray container as a sidecar:

#export mesh name export MESH_NAME="color-mesh" #install X-Ray sidecar export INJECT_XRAY_SIDECAR="true" export ENABLE_STATS_TAGS="true" export ENABLE_STATSD="true" #install the sidecar injector curl https://raw.githubusercontent.com/aws/aws-app-mesh-inject/master/scripts/install.sh | bash

Deploying a mesh connected sample application

The sample application consists of two components:

ColorGateway – A simple HTTP service written in Go that is exposed to external clients and that responds to http://service-name:port/color. The gateway responds with a color retrieved from color-teller and a histogram of colors observed at the server that responded up to the point when you made the request.

– A simple HTTP service written in Go that is exposed to external clients and that responds to http://service-name:port/color. The gateway responds with a color retrieved from color-teller and a histogram of colors observed at the server that responded up to the point when you made the request. ColorTeller – A simple HTTP service written in Go that is configured to return a color. Multiple variants of the service are deployed. Each service is configured to return a specific color.

ColorGateway isn’t aware of the multiple variants of the CollorTeller component. AWS App Mesh will expose one CollorTeller virtual service with three virtual nodes and three routes.

To deploy the sample application, apply the following file to your Kubernetes cluster with the following command.

#install sample application kubectl apply -f https://raw.githubusercontent.com/fmedery/blogs/GlooPost1/eks-appmesh-gloo/color.yaml

You can verify all the deployed objects using the following command (notice that two extra containers are running in each pod: one for AWS X-Ray and one for AWS App Mesh.

kubectl -n appmesh-demo get all NAME READY STATUS RESTARTS AGE pod/colorgateway-957f89b6b-wrcsb 3/3 Running 0 45s pod/colorteller-759fc757cc-72kbw 3/3 Running 0 44s pod/colorteller-black-6c5dd7689c-lxqzl 3/3 Running 0 43s pod/colorteller-blue-58dbf546d5-gz9t7 3/3 Running 0 43s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/colorgateway ClusterIP 10.100.209.207 <none> 9080/TCP 45s service/colorteller ClusterIP 10.100.8.175 <none> 9080/TCP 44s service/colorteller-black ClusterIP 10.100.100.216 <none> 9080/TCP 44s service/colorteller-blue ClusterIP 10.100.41.247 <none> 9080/TCP 43s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/colorgateway 1/1 1 1 46s deployment.apps/colorteller 1/1 1 1 45s deployment.apps/colorteller-black 1/1 1 1 44s deployment.apps/colorteller-blue 1/1 1 1 44s NAME DESIRED CURRENT READY AGE replicaset.apps/colorgateway-957f89b6b 1 1 1 46s replicaset.apps/colorteller-759fc757cc 1 1 1 45s replicaset.apps/colorteller-black-6c5dd7689c 1 1 1 44s replicaset.apps/colorteller-blue-58dbf546d5 1 1 1 44s NAME AGE mesh.appmesh.k8s.aws/color-mesh 48s NAME AGE virtualnode.appmesh.k8s.aws/colorgateway 48s virtualnode.appmesh.k8s.aws/colorteller 48s virtualnode.appmesh.k8s.aws/colorteller-black 47s virtualnode.appmesh.k8s.aws/colorteller-blue 47s NAME AGE virtualservice.appmesh.k8s.aws/colorgateway.appmesh-demo 46s virtualservice.appmesh.k8s.aws/colorteller.appmesh-demo 47s

Installing Gloo

Now that our sample application has been installed, we will install Gloo ingress gateway.

#install gloo gateway using the gloo command line interface glooctl install gateway

We can get an overview of all of the resources running using the command below (notice that Gloo automatically provisioned an AWS Elastic Load Balancer ).

kubectl get all -n gloo-system NAME READY STATUS RESTARTS AGE pod/discovery-799484bcc4-nr2v2 1/1 Running 0 49s pod/gateway-certgen-kr5gv 0/1 Completed 0 56s pod/gateway-proxy-v2-6476c4759f-v82vm 1/1 Running 0 48s pod/gateway-v2-8679b5cb57-t4smr 1/1 Running 0 49s pod/gloo-5578cbc65b-dqqk9 1/1 Running 0 50s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/gateway ClusterIP 10.100.7.155 <none> 443/TCP 51s service/gateway-proxy-v2 LoadBalancer 10.100.37.255 aada551fcf78c11e9b9fe02975132263-1637777701.us-west-2.elb.amazonaws.com 80:30454/TCP,443:32704/TCP 50s service/gloo ClusterIP 10.100.222.125 <none> 9977/TCP,9988/TCP,9966/TCP 51s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/discovery 1/1 1 1 49s deployment.apps/gateway-proxy-v2 1/1 1 1 48s deployment.apps/gateway-v2 1/1 1 1 49s deployment.apps/gloo 1/1 1 1 50s NAME DESIRED CURRENT READY AGE replicaset.apps/discovery-799484bcc4 1 1 1 49s replicaset.apps/gateway-proxy-v2-6476c4759f 1 1 1 48s replicaset.apps/gateway-v2-8679b5cb57 1 1 1 49s replicaset.apps/gloo-5578cbc65b 1 1 1 50s NAME COMPLETIONS DURATION AGE job.batch/gateway-certgen 1/1 7s 57s NAME AGE mesh.appmesh.k8s.aws/color-mesh 2m

Now, let’s have a look at the pods:

pod/discovery-799484bcc4-nr2v2 : this component is responsible for dynamically discovering services to which Gloo can route (upstreams).

: this component is responsible for dynamically discovering services to which Gloo can route (upstreams). pod/gateway-proxy-v2-6476c4759f-v82vm : this is the Envoy proxy

: this is the Envoy proxy pod/gateway-v2-8679b5cb57-t4smr : this component allows users to configure an Envoy Proxy and also generates configuration that the Gloo control plane can use to generate Envoy configuration through xDS

: this component allows users to configure an Envoy Proxy and also generates configuration that the Gloo control plane can use to generate Envoy configuration through xDS pod/gloo-5578cbc65b-dqqk9 : an event-driven component responsible for generating configuration for and serving the core xDS services and configuration of custom Envoy filters.

Because of this decoupling, developers contributing to Gloo can easily add support for other architectures like Knative for example.

Allowing Ingress connectivity using Gloo

Let’s dig a little deeper into two of Gloo core concepts Virtual Service and Upstream:

Virtual Service defines a set of route rules that live under a domain or set of domains. Route rules consist of a matcher, which specifies the kind of function calls to match (requests and events, are currently supported), and the name of the destination (or destinations) to route them to.

defines a set of route rules that live under a domain or set of domains. Route rules consist of a matcher, which specifies the kind of function calls to match (requests and events, are currently supported), and the name of the destination (or destinations) to route them to. Upstream defines destinations for routes. Upstreams tell Gloo what to route to. Upstreams are automatically discovered by Gloo.

We will first verify that the ColorGateway Virtual Service was discovered by Gloo.

glooctl get upstreams|grep colorgateway | appmesh-demo-colorgateway-9080 | Kubernetes | Accepted | svc name: colorgateway | | appmesh-demo-colorgateway-v1-9080 | Kubernetes | Accepted | svc name: colorgateway |

We will create a Gloo Virtual Service called colorgateway and a route that will redirect the path /appmesh/color to the virtual service appmesh-demo-colorgateway-9080 using /color as his path.

#create a virtual service called colorgateway glooctl create virtualservice --name colorgateway #add a route to the virtual service glooctl add route \ --name colorgateway \ --path-prefix /appmesh/color \ --prefix-rewrite /color \ --dest-name appmesh-demo-colorgateway-9080 +-----------------+--------------+---------+------+----------+-----------------+---------------------------------------------+ | VIRTUAL SERVICE | DISPLAY NAME | DOMAINS | SSL | STATUS | LISTENERPLUGINS | ROUTES | +-----------------+--------------+---------+------+----------+-----------------+---------------------------------------------+ | colorgateway | colorgateway | * | none | Accepted | | /appmesh/color -> | | | | | | | | gloo-system.appmesh-demo-colorgateway-9080 | | | | | | | | (upstream) | +-----------------+--------------+---------+------+----------+-----------------+---------------------------------------------+

Now we will try to connect to the application.

#verify that the load balancer URL exists #creating a load balancer can take a few minutes glooctl proxy url http://aa1706959c82111e9bdfc02f316f0629-1493319117.us-west-2.elb.amazonaws.com:80 #test the connectivity for i in {1..10}; do curl $(glooctl proxy url)/appmesh/color; echo; done {"color":"blue", "stats": {"black":0.3,"blue":0.13,"white":0.57}} {"color":"white", "stats": {"black":0.29,"blue":0.13,"white":0.58}} {"color":"white", "stats": {"black":0.28,"blue":0.12,"white":0.6}} {"color":"black", "stats": {"black":0.31,"blue":0.12,"white":0.58}} {"color":"black", "stats": {"black":0.33,"blue":0.11,"white":0.56}} {"color":"white", "stats": {"black":0.32,"blue":0.11,"white":0.57}} {"color":"blue", "stats": {"black":0.31,"blue":0.14,"white":0.55}} {"color":"blue", "stats": {"black":0.3,"blue":0.17,"white":0.53}} {"color":"black", "stats": {"black":0.32,"blue":0.16,"white":0.52}} {"color":"black", "stats": {"black":0.34,"blue":0.16,"white":0.5}}

We can now connect to the ColorGateway from outside of the EKS cluster using the route /appmesh/color

Canary Deployment using Gloo weighted destinations

With this section, we will introduce another important concept of Gloo, Upstream Group:

Upstream Group is a top-level object, that let you logically groups upstreams , giving you the ability to address them as a group in distinct VirtualServices

This is a common requirement for Canary deployments where you want all calling routes to forward traffic equally across the two service versions.

For this example, we will start by deploying a new version of the gateway called colorteller-gatewayV2.

#deploy colorteller-gatewayV2 kubectl apply -f https://raw.githubusercontent.com/fmedery/blogs/GlooPost1/eks-appmesh-gloo/color-gateway-v2.yaml #verify if Gloo was able to discover it glooctl get upstream | grep gatewayv2 |appmesh-demo-colorgatewayv2-9080 | Kubernetes| Accepted| svc name: colorgatewayv2| |appmesh-demo-colorgatewayv2-v2-9080| Kubernetes| Accepted| svc name: colorgatewayv2|

Now we will create an upstream group called upstreamgroup-gateway and add routes that will split the traffic between colorteller-gateway (80%) and colorteller-gatewayv2 (20%).

#delete the virtual service previously created glooctl delete vs --name colorgateway #recreate the colorgateway virtual service with an upstream group has its backend kubectl apply -f https://raw.githubusercontent.com/fmedery/blogs/GlooPost1/eks-appmesh-gloo/01_canary.yaml upstreamgroup.gloo.solo.io/upstreamgroup-gateway created virtualservice.gateway.solo.io/colorgateway configured #verify the upstreamgroup has been created glooctl get upstreamgroup --name upstreamgroup-gateway +-----------------------+----------+--------------+----------------------------------+ | UPSTREAM GROUP | STATUS | TOTAL WEIGHT | DETAILS | +-----------------------+----------+--------------+----------------------------------+ | upstreamgroup-gateway | Accepted | 100 | destination type: Upstream | | | | | namespace: gloo-system | | | | | name: | | | | | appmesh-demo-colorgateway-9080 | | | | | weight: 80 % total: 0.80 | | | | | | | | | | | | | | | destination type: Upstream | | | | | namespace: gloo-system | | | | | name: | | | | | appmesh-demo-colorgatewayv2-9080 | | | | | weight: 20 % total: 0.20 | +-----------------------+----------+--------------+----------------------------------+ #verify that we have a route pointing to the upstreamgroup glooctl get vs --name colorgateway +-----------------+--------------+---------+------+----------+-----------------+-----------------------------------+ | VIRTUAL SERVICE | DISPLAY NAME | DOMAINS | SSL | STATUS | LISTENERPLUGINS | ROUTES | +-----------------+--------------+---------+------+----------+-----------------+-----------------------------------+ | colorgateway | | * | none | Accepted | | /appmesh/color -> upstream group: | | | | | | | | upstreamgroup-gateway.gloo-system | +-----------------+--------------+---------+------+----------+-----------------+-----------------------------------+ #test the new route for i in {1..10}; do curl $(glooctl proxy url)/appmesh/color; echo; donex` {"color":"white", "stats": {"black":0.32,"blue":0.35,"white":0.33}} {"color":"white", "stats": {"black":0.31,"blue":0.34,"white":0.34}} {"color":"blue", "stats": {"black":0.31,"blue":0.36,"white":0.34}} {"colorV2":"white", "statsV2": {"black":0.24,"blue":0.57,"white":0.19}} {"color":"blue", "stats": {"black":0.3,"blue":0.37,"white":0.33}} {"color":"black", "stats": {"black":0.31,"blue":0.36,"white":0.33}} {"color":"blue", "stats": {"black":0.31,"blue":0.37,"white":0.32}} {"color":"black", "stats": {"black":0.32,"blue":0.37,"white":0.32}} {"colorV2":"blue", "statsV2": {"black":0.23,"blue":0.59,"white":0.18}} {"color":"blue", "stats": {"black":0.31,"blue":0.38,"white":0.31}}

When we are confident that the new version is behaving as expected, we can increase the traffic sent to colorteller-gatewayV2.

#update the weight of each route to 50% kubectl apply -f https://raw.githubusercontent.com/fmedery/blogs/GlooPost1/eks-appmesh-gloo/02_canary.yaml upstreamgroup.gloo.solo.io/upstreamgroup-gateway configured #verify the weight for each route has been updated to 50% glooctl get upstreamgroup --name upstreamgroup-gateway +-----------------------+----------+--------------+----------------------------------+ | UPSTREAM GROUP | STATUS | TOTAL WEIGHT | DETAILS | +-----------------------+----------+--------------+----------------------------------+ | upstreamgroup-gateway | Accepted | 100 | destination type: Upstream | | | | | namespace: gloo-system | | | | | name: | | | | | appmesh-demo-colorgateway-9080 | | | | | weight: 50 % total: 0.50 | | | | | | | | | | | | | | | destination type: Upstream | | | | | namespace: gloo-system | | | | | name: | | | | | appmesh-demo-colorgatewayv2-9080 | | | | | weight: 50 % total: 0.50 | +-----------------------+----------+--------------+----------------------------------+ #test the route again for i in {1..10}; do curl $(glooctl proxy url)/appmesh/color; echo; done {"colorV2":"white", "statsV2": {"black":0.22,"blue":0.57,"white":0.22}} {"color":"blue", "stats": {"black":0.31,"blue":0.38,"white":0.31}} {"color":"white", "stats": {"black":0.3,"blue":0.38,"white":0.32}} {"colorV2":"white", "statsV2": {"black":0.21,"blue":0.54,"white":0.25}} {"colorV2":"blue", "statsV2": {"black":0.2,"blue":0.56,"white":0.24}} {"color":"black", "stats": {"black":0.31,"blue":0.37,"white":0.31}} {"colorV2":"white", "statsV2": {"black":0.19,"blue":0.54,"white":0.27}} {"colorV2":"white", "statsV2": {"black":0.19,"blue":0.52,"white":0.3}} {"colorV2":"black", "statsV2": {"black":0.21,"blue":0.5,"white":0.29}} {"color":"black", "stats": {"black":0.32,"blue":0.37,"white":0.31}}

Now we will route all the traffic to colorteller-gatewayv2 by removing colorteller-gateway from the upstream group:

# update the upstream group kubectl apply -f https://raw.githubusercontent.com/fmedery/blogs/GlooPost1/eks-appmesh-gloo/03_canary.yaml #verify the upstreamgroup glooctl get upstreamgroup --name upstreamgroup-gateway +-----------------------+----------+--------------+-------------------------------------+ | UPSTREAM GROUP | STATUS | TOTAL WEIGHT | DETAILS | +-----------------------+----------+--------------+-------------------------------------+ | upstreamgroup-gateway | Accepted | 100 | destination type: Upstream | | | | | namespace: gloo-system | | | | | name: | | | | | appmesh-demo-colorgatewayv2-9080 | | | | | weight: 100 % total: 1.00 | +-----------------------+----------+--------------+-------------------------------------+ #test one last time for i in {1..10}; do curl $(glooctl proxy url)/appmesh/color; echo; done {"colorV2":"black", "statsV2": {"black":0.24,"blue":0.48,"white":0.28}} {"colorV2":"blue", "statsV2": {"black":0.23,"blue":0.5,"white":0.27}} {"colorV2":"white", "statsV2": {"black":0.23,"blue":0.48,"white":0.29}} {"colorV2":"blue", "statsV2": {"black":0.22,"blue":0.5,"white":0.28}} {"colorV2":"black", "statsV2": {"black":0.24,"blue":0.48,"white":0.27}} {"colorV2":"white", "statsV2": {"black":0.24,"blue":0.47,"white":0.29}} {"colorV2":"white", "statsV2": {"black":0.23,"blue":0.46,"white":0.31}} {"colorV2":"black", "statsV2": {"black":0.25,"blue":0.44,"white":0.31}} {"colorV2":"blue", "statsV2": {"black":0.24,"blue":0.46,"white":0.3}} {"colorV2":"blue", "statsV2": {"black":0.24,"blue":0.47,"white":0.29}

Gloo is now routing all the traffic hitting /appmesh/color to colorteller-gatewayv2 .

Conclusion

In this post, we demonstrated how Gloo can transparently interact with AWS App Mesh and how easily developers can create and manipulate access to their applications. We also showcase how to use Gloo to create a canary deployment.

To learn more about Gloo advanced features, follow this link.

In an upcoming post, we will take a look at another solo.io product: supergloo, the Service Mesh Orchestration Platform.