In today’s blogpost we’re going to be discussing ingress and egress gateways. First, we’ll cover the basics, then we’ll go into detail and explore how they work through a series of practical examples. Ingress and egress gateways are load balancers that operate at the edges of any network receiving incoming or outgoing HTTP/TCP connections. Ingress gateways make it possible to define an entry points into an Istio mesh for all incoming traffic to flow through. Egress gateways are similar: they define exit points from the mesh, but also allow for the application of Istio features to the traffic exiting the mesh. Some examples of these features are monitoring, routing rules and retries.

In Istio, both gateways are based on Envoy . Envoy handles reverse proxying and load balancing for services running inside a service mesh’s network, and also for external services outside the mesh. An Istio gateway in a Kubernetes cluster consists of, at minimum, a Deployment and a Service . For an ingress gateway the latter is typically a LoadBalancer -type service, or, when an ingress gateway is used solely within a cluster, a ClusterIP -type service. For an egress gateway the service type is almost always ClusterIP .

Although Istio can be configured to support Kubernetes Ingress Resources, a better approach would be to use Istio’s custom resources ( Gateway , VirtualService ). That way you can use Istio features for more than internal services, including ingresses, giving you access to way more features than you’d have with just Kubernetes’ Ingress Resources.

The Gateway resource describes the port configuration of the gateway deployment that operates at the edge of the mesh and receives incoming or outgoing HTTP/TCP connections. The specification describes a set of ports that should be exposed, the type of protocol to use, TLS configuration – if any – of the exposed ports, and so on. For more information about Gateways, see the Istio documentation. VirtualService defines a set of traffic routing rules to apply when a host is addressed. Each routing rule defines matching criteria for the traffic of a specific protocol. If the traffic matches a routing rule, then it is sent to a named destination service defined in the registry. For example, it can route requests to different versions of a service or to a completely different service than was requested. Requests can be routed based on the request source and destination, HTTP paths and header fields, and weights associated with individual service versions. For more information about VirtualServices, see the Istio documentation.

Istio Ingress and Egress Gateways, showtime 🔗︎

Our only prerequisite before exploring these concepts through examples is the creation of a Kubernetes cluster.

You can create a Kubernetes cluster on five different cloud providers, or on-premise via the free developer version of the Pipeline platform. But you can also bring your own cluster.

Install Istio using Backyards 🔗︎

The easiest way to install a production ready Istio and a demo application on a brand new cluster is to use the Backyards CLI.

Register for an evaluation version and run the following command to install the CLI tool ( KUBECONFIG must be set for your cluster):

> curl https://getbackyards.sh | sh && backyards install -a --run-demo

This command installs Istio with the Banzai Cloud open-source Istio operator, then installs Backyards itself, as well as an application for demonstration purposes. After the installation has finished, the Backyards UI will automatically open and send some traffic to the demo application. Issuing this one simple command causes Backyards to start a new Istio mesh in just a few minutes!

You can read more about the latest Backyards release here

Using the main ingress gateway 🔗︎

Remember, as we talked about earlier in this post, ingress gateways enable us to expose services to the external world. Accordingly, an ingress gateway serves as the entry point for all services running within the mesh.

Determining the ingress IP 🔗︎

In order to expose a service, you must first know the external IP of the ingress gateway. Fortunately, the Banzai Cloud Istio operator helps us with this.

❯ kubectl -n istio-system get istio mesh NAME STATUS ERROR GATEWAYS AGE mesh Available [18.184.240.108 18.196.72.62] 15m

In general, you should manually set an external hostname that points to these addresses, but for demo purposes you can use xip.io, which is a domain name that provides wildcard DNS for any IP address.

Expose a service through the ingress gateway 🔗︎

The demo application that comes with Backyards contains several microservices. The frontpage service serves as the entry point of that application. Now, let’s create a Gateway and a VirtualService resource to expose the frontpage service.

If you’re using xip.io, the external hostname for the service is going to be either frontpage.18.184.240.108.xip.io or frontpage.18.196.72.62.xip.io .

Create Gateway resource 🔗︎

The following Gateway resource configures listening ports on the matching gateway deployment.

apiVersion : networking.istio.io/v1alpha3 kind : Gateway metadata : name : frontpage namespace : backyards-demo spec : selector : istio : ingressgateway # use Istio default gateway implementation servers : - port : number : 80 name : http protocol : HTTP hosts : - "frontpage.18.184.240.108.xip.io" - "frontpage.18.196.72.62.xip.io"

Create VirtualService resource 🔗︎

The following VirtualService resource configures routing for the external hosts within the mesh.

apiVersion : networking.istio.io/v1alpha3 kind : VirtualService metadata : name : frontpage namespace : backyards-demo spec : hosts : - "frontpage.18.184.240.108.xip.io" - "frontpage.18.196.72.62.xip.io" gateways : - frontpage http : - route : - destination : port : number : 8080 host : frontpage.backyards-demo.svc.cluster.local

Access the service on the external address 🔗︎

Try to access the service on the external address you just configured, on host frontpage.18.184.240.108.xip.io . If everything is set correctly, the following command will return an HTTP 200 status code.

❯ curl -i frontpage.18.184.240.108.xip.io HTTP/1.1 200 OK content-type: text/plain date: Sun, 02 Feb 2020 19:15:14 GMT content-length: 9 x-envoy-upstream-service-time: 16 server: istio-envoy frontpage

Create and use multiple ingress gateways 🔗︎

Having one ingress and egress gateway to handle incoming and outgoing traffic from the mesh is part of a basic Istio installation and has been supported by the Banzai Cloud Istio operator from day one, but in large enterprise deployments our customers typically use Backyards with multiple ingress or egress gateways.

The Banzai Cloud Istio operator has an Istio custom resource that defines mesh configurations. The main ingress/egress gateways are part of the specifications of that resource.

spec : ... gateways : egress : enabled : true maxReplicas : 1 minReplicas : 1 ports : - name : http2 port : 80 protocol : TCP targetPort : 80 - name : https port : 443 protocol : TCP targetPort : 443 replicaCount : 1 sds : enabled : false image : docker.io/istio/node-agent-k8s: 1.4.3 serviceType : ClusterIP enabled : true ingress : enabled : true maxReplicas : 1 minReplicas : 1 ports : - name : status-port port : 15020 protocol : TCP targetPort : 15020 - name : http2 port : 80 protocol : TCP targetPort : 80 - name : https port : 443 protocol : TCP targetPort : 443 - name : tls port : 15443 protocol : TCP targetPort : 15443 replicaCount : 1 sds : enabled : false image : docker.io/istio/node-agent-k8s: 1.4.3 serviceType : LoadBalancer ...

One way to support multiple gateways would have been to add support for specifying them in the existing custom resource. But we chose a radically different approach for the following reasons:

a separate controller should reconcile gateways, as there could be multiple gateways in multiple namespaces

RBAC: having a separate CR allows us to properly control who can manage gateways, without having permissions to modify other parts of the Istio mesh configuration

Thus, we have added a new CRD to the Banzai Cloud Istio operator, called the MeshGateway , that can be used to add and configure a new Istio ingress or egress gateway into the mesh. When you create a new MeshGateway CR, the Banzai Cloud Istio operator will take care of configuring and reconciling the necessary resources, including the Envoy deployment and its related Kubernetes service.

Create a new service in the default namespace and expose it 🔗︎

To demonstrate how to create and use multiple ingress gateways, let’s add a simple service to the default namespace.

❯ kubectl apply -f https://raw.githubusercontent.com/banzaicloud/istio-operator/release-1.4/docs/federation/multimesh/echo-service.yaml deployment.extensions/echo created service/echo created

It would be possible to expose this echo service through the existing ingress gateway, similar to the way we would for the frontpage service, but let’s assume we need to expose this service on port 8000, without modifying the existing ingress gateway.

Create a new ingress gateway using the MeshGateway resource 🔗︎

Apply the following resource and the operator will create a new ingress gateway deployment, and a corresponding service.

apiVersion : istio.banzaicloud.io/v1beta1 kind : MeshGateway metadata : name : echo-ingress namespace : default spec : maxReplicas : 1 minReplicas : 1 ports : - name : http port : 8000 protocol : TCP targetPort : 8000 serviceType : LoadBalancer type : ingress

The MeshGateway resource automatically labels the created Service and Deployment resources with the gateway-name and gateway-type labels and their corresponding values.

Get the echo ingress gateway IP 🔗︎

❯ kubectl -n default get meshgateways echo-ingress NAME TYPE SERVICE TYPE STATUS INGRESS IPS ERROR AGE echo-ingress ingress LoadBalancer Available [18.185.173.38 18.197.110.20] 29m

Create Gateway and VirtualService resources 🔗︎

Just like in the first example, the following Gateway and VirtualService resources are necessary to configure listening ports on the matching gateway deployment.

apiVersion : networking.istio.io/v1alpha3 kind : Gateway metadata : name : echo namespace : default spec : selector : gateway-name : echo-ingress gateway-type : ingress servers : - port : number : 8000 name : http protocol : HTTP hosts : - "echo.18.197.110.20.xip.io" - "echo.18.185.173.38.xip.io" --- apiVersion : networking.istio.io/v1alpha3 kind : VirtualService metadata : name : echo namespace : default spec : hosts : - "echo.18.197.110.20.xip.io" - "echo.18.185.173.38.xip.io" gateways : - echo http : - route : - destination : port : number : 80 host : echo.default.svc.cluster.local

Access the echo service on the external address 🔗︎

The service should be accessible on host echo.18.197.110.20.xip.io and port 8000 .

❯ curl -i echo.18.197.110.20.xip.io:8000 HTTP/1.1 200 OK date: Sun, 02 Feb 2020 19:22:15 GMT content-type: text/plain server: istio-envoy x-envoy-upstream-service-time: 1 Hostname: echo-68578cf9d9-874rz ...

Our ability to easily create ingress gateways gives you fine-grained control over how services are exposed to the outside world. That way, teams can manage the exposure of their own services without running the risk of misconfiguring the services of other teams. Another way of tackling this potential issue is to have separate load balancer configurations with, for example, different port level settings.

Accessing External Services 🔗︎

Direct outbound traffic 🔗︎

Any traffic that’s outbound from a pod with an Istio sidecar will also pass through that sidecar’s container, or, more precisely, through Envoy. Therefore, the accessibility of external services depends on the configuration of that Envoy proxy. By default, Istio configures the Envoy proxy to passthrough requests for unknown services. Although this provides a convenient way of getting started with Istio, its generally a good idea to put stricter controls in place.

To read more about the Sidecar object configuration, check out this informative blog post:.

Check the default outbound traffic policy 🔗︎

❯ kubectl -n istio-system get istio mesh -o jsonpath = '{@.spec.outboundTrafficPolicy.mode}{"

"}' ALLOW_ANY

This traffic policy should be set to ALLOW_ANY by default.

Make an HTTP request to httpbin.org from the echo service container in the default namespace 🔗︎

❯ kubectl -n default exec -ti deploy/echo -- curl -sD /dev/stderr http://httpbin.org -o/dev/null |head -5 Defaulting container name to echo-service. Use 'kubectl describe pod/echo-68578cf9d9-874rz -n default' to see all of the containers in this pod. HTTP/1.1 200 OK date: Sun, 02 Feb 2020 20:43:40 GMT content-type: text/html; charset=utf-8 content-length: 9593 server: envoy

This should work fine, since, by default, every sidecar sends traffic towards unknown services through its passhtrough proxy.

Change the default outbound traffic policy to block unknown services 🔗︎

Now we’re going to demonstrate a more controlled way of enabling access to external services. Change the spec.outboundTrafficPolicy.mode option from the ALLOW_ANY mode to the REGISTRY_ONLY mode in the mesh Istio resource in the istio-system namespace.

❯ kubectl -n istio-system patch istio mesh -p '{"spec":{"outboundTrafficPolicy":{"mode":"REGISTRY_ONLY"}}}' --type = 'merge' istio.istio.banzaicloud.io/mesh patched

Make another HTTP request to httpbin.org from the echo service container in the default namespace 🔗︎

❯ kubectl -n default exec -ti deploy/echo -- curl -sD /dev/stderr http://httpbin.org -o/dev/null |head -5 HTTP/1.1 502 Bad Gateway location: http://httpbin.org/ date: Sun, 02 Feb 2020 20:53:44 GMT server: envoy content-length: 0

Now we’re getting a 502 response code, since now the traffic towards external services is blocked and it is going through Envoy’s blackhole cluster.

Create a ServiceEntry to allow HTTP access to httpbin.org 🔗︎

ServiceEntry resources enable adding additional entries into Istio’s internal service registry, so that auto-discovered services in the mesh can access/route to these manually specified services. A service entry describes the properties of a service (DNS name, VIPs, ports, protocols, endpoints). These services could be external to the mesh (for example, web APIs) or mesh-internal services that are not part of the platform’s service registry. For more information about the ServiceEntry resource, see the Istio documentation.

Apply the following ServiceEntry to allow for HTTP access to httpbin.org

apiVersion : networking.istio.io/v1alpha3 kind : ServiceEntry metadata : name : httpbin-ext namespace : default spec : hosts : - httpbin.org ports : - number : 80 name : http protocol : HTTP resolution : DNS location : MESH_EXTERNAL

Make another HTTP request to httpbin.org from the echo service container in the default namespace 🔗︎

❯ kubectl -n default exec -ti deploy/echo -- curl -sD /dev/stderr http://httpbin.org -o/dev/null |head -5 Defaulting container name to echo-service. Use 'kubectl describe pod/echo-68578cf9d9-874rz -n default' to see all of the containers in this pod. HTTP/1.1 200 OK date: Sun, 02 Feb 2020 21:13:40 GMT content-type: text/html; charset=utf-8 content-length: 9593 server: envoy

Outbound traffic through an egress gateway 🔗︎

As you probably recall from earlier in this blogpost, egress gateways are exit points from the mesh that allow us to apply Istio features. This includes applying features like monitoring and route rules to traffic that’s exiting the mesh.

Use cases 🔗︎

Let’s take a quick look at some use cases.

Consider an organization which requires some, or all, outbound traffic to go through dedicated nodes. These nodes could be separated from the rest of the nodes for the purposes of monitoring and policy enforcement.

Now imagine a cluster where the application nodes don’t have public IPs, so the in-mesh services that run on them cannot access the internet directly. Defining an egress gateway and routing egress traffic through it, then allocating public IPs to the gateway nodes would allow for controlled access to external services.

Create an egress gateway with the MeshGateway resource 🔗︎

Apply the following resource and the Istio operator will create a new egress gateway deployment and a corresponding service.

apiVersion : istio.banzaicloud.io/v1beta1 kind : MeshGateway metadata : name : egress namespace : default spec : maxReplicas : 1 minReplicas : 1 ports : - name : http port : 80 protocol : TCP targetPort : 80 serviceType : ClusterIP type : egress

Create a Gateway resource for the egress gateway 🔗︎

Similar to the ingress gateway configuration, a Gateway resource must be created that will be a bridge between Istio configuration resources and the deployment of a matching gateway.

Apply the following Gateway resource to configure the outbound port, 80, on the egress gateway that was just defined in the previous step.

apiVersion : networking.istio.io/v1alpha3 kind : Gateway metadata : name : egress namespace : default spec : selector : gateway-name : egress gateway-type : egress servers : - port : number : 80 name : http protocol : HTTP hosts : - "*"

Define a VirtualService resource to direct traffic from the sidecars to the egress gateway 🔗︎

Apply the following VirtualService to direct traffic from the sidecars to the egress gateway and also from the egress gateway to the external service.

apiVersion : networking.istio.io/v1alpha3 kind : VirtualService metadata : name : httpbin-egress spec : hosts : - httpbin.org gateways : - egress - mesh http : - match : - gateways : - mesh port : 80 route : - destination : host : egress.default.svc.cluster.local port : number : 80 - match : - gateways : - egress port : 80 route : - destination : host : httpbin.org port : number : 80

Resend an HTTP request to httpbin.org from the echo service container in the default namespace 🔗︎

❯ kubectl -n default exec -ti deploy/echo -- curl -sD /dev/stderr http://httpbin.org/headers -o/dev/null |head -5 Defaulting container name to echo-service. Use 'kubectl describe pod/echo-68578cf9d9-874rz -n default' to see all of the containers in this pod. HTTP/1.1 200 OK date: Sun, 02 Feb 2020 22:20:28 GMT content-type: application/json content-length: 1850 server: envoy

Output should be the same as earlier, but if we check the logs of the egress gateway, it shows that the request actually went through the egress gateway.

❯ kubectl -n default logs -l gateway-name=egress,gateway-type=egress -c istio-proxy | tail -1 [2020-02-02T22:20:29.191Z] "GET /headers HTTP/1.1" 200 - "-" "-" 0 1850 173 173 "10.20.5.1" "curl/7.47.0" "bf26bf45-4a0b-4d78-a968-98515a890a91" "httpbin.org" "52.202.2.199:80" outbound|80||httpbin.org - 10.20.5.16:80 10.20.5.1:58584 - -

Ingress and egress gateways are core concepts of a service mesh. Although Istio itself provides the basic building blocks, having an easy and simple way to create and manage multiple mesh gateways is a must. The Banzai Cloud Istio operator provides support with a new CRD called MeshGateway . Give it a try, and quickstart your Istio experience with Backyards!

Banzai Cloud Istio operator is a simple way to deploy, manage and maintain Istio service meshes, even in multi-cluster topologies.

About Backyards 🔗︎

Banzai Cloud’s Backyards is a multi and hybrid-cloud enabled service mesh platform for constructing modern applications. Built on Kubernetes, our Istio operator and the Banzai Cloud Pipeline platform gives you flexibility, portability, and consistency across on-premise datacenters and on five cloud environments. Use our simple, yet extremely powerful UI and CLI, and experience automated canary releases, traffic shifting, routing, secure service communication, in-depth observability and more, for yourself.

Banzai Cloud is changing how private clouds are built: simplifying the development, deployment, and scaling of complex applications, and putting the power of Kubernetes and Cloud Native technologies in the hands of developers and enterprises, everywhere.

#multicloud #hybridcloud #BanzaiCloud