The Datawire and ORY teams have recently been discussing the challenges of API access control in a cloud native environment, the highlights of which I capture below in a Q&A. There are many possible solutions (with associated benefits and tradeoffs), and our friends at ORY have put together a tutorial on how to use their Oathkeeper with Ambassador running on Kubernetes, which I have also included below.

Introduction

The web application and web service landscape is changing radically as large software companies are making their internal infrastructure and software development and operation practices open to the public. Initiatives such as the Cloud Native Computing Foundation, and open source standards and software like Istio and Kubernetes, are making a big impact on how software is developed and operated. Go, the programming language written and maintained by Google, shines with it’s toolchain. However, some of the tools behave differently than expected and it may cost you several hours of debugging and experimenting to find the arguments and execution orders.

This affects also access control — which many developer’s have a love-hate relationship with — too. In the past, we have relied on language-level APIs provided by libraries such as OmniAuth, Spring Security, and PassportJS. These libraries will always have their place in the developer’s toolbox. As applications grow and companies move away from monoliths to the Service Mesh, using these libraries isn’t quite so easy any more.

As you move to a distributed service architecture, you move away from integrating with local libraries and SDKs, and towards calling services that operate on the network. This happens naturally as you adopt more languages (e.g. you are using the best language for each use case) and start more services. This, combined with differing zones of trust across your network, obviously impact how you perform access control as well.

Q&A: Aeneas Rekkas, CEO of ORY, on the Challenges of “Zero Trust” Access Control

In order to set the scene, I recently sat down with Aeneas Rekkas, founder and CEO of ORY, and explored the concepts mentioned in more details:

Datawire: Can you explain what you mean by “Zero Trust API Access Control” please?

Aeneas Rekkas: Companies usually differentiate between an internal network (intranet) and an external network (internet). The intranet is typically within the physical premises of the company (e.g. office) and most, if not all, traffic from within that network is trusted without question. This is, for apparent reasons, a bad security practice. Advanced persistent threats (APTs) reek havoc in such environments as, once they are in the network, they comprimes everything else. Another issue is the rise of remote work and bring your own device. Both do not play well with the idea of giving internal and external traffic different privileges.

Zero Trust API Access Control defines that each API is protected, regardless of where the traffic comes from. Instead of having an file server API that can be accessed by anoyne from within the intranet, the file server should instead require valid credentials. These credentials should typically come from one source (Authorization Server) and could be API Keys, Bearer Tokens, TLS Client Certificates, or HTTP Basic Authorization. Because this system will be deployed with every service, it is a good idea to use an open source service specifically designed for this use case, such as ORY Oathkeeper.

DW: What are the advantages of using ORY over other solutions?

AR: We started ORY because developer tools are clunky and so is application security. ORY’s vision is to improve the way developers approach application security with an open source ecosystem. All services are written cloud-first and adhere to principles like 12-factor design. Besides extremely low resource consumption, every service deploys in seconds due to a ~12mb docker image size. The ORY ecosystem is being used in production by SME and F200 alike and has been acclaimed for its straight-forward design.

DW: Why did you choose to integrate with the Ambassador API Gateway? What has your experience working with Ambassador been like?

AR: Ambassador is the perfect vehicle to implement Zero Trust API Access Control on top of. With its Kubernetes-first approach it perfectly fits in the Istio Service Mesh. Because we don’t want to reinvent the wheel and write another API Reverse Proxy, integrating with Ambassador was a no-brainer. The community and maintainers from DataWire are exceptionally responsive and helpful. You should give it a try!

DW: What are the future plans for ORY?

AR: We’re just getting started. Besides working on a new open source Identity Management product called ORY Hive, we are planning a cloud environment service for running the ORY Ecosystem with zero friction and managed security. Our vision is to make cutting-edge application security as available as AWS EC2 or GCP Compute Engine!

Tutorial: API Access Control with Ambassador and Oathkeeper

Let’s try and make some of the concepts discussed a little more concrete, by setting up an example with Ambassador and ORY Oathkeeper on Kubernetes. Before you go ahead, you’ll need to:

Make sure you have access to Kubernetes — either via minikube, Docker Desktop, amanaged Kubernetes, or any other type of Kubernetes deployment.

Make sure kubectl is configured and pointed to your Kubernetes deployment.

is configured and pointed to your Kubernetes deployment. Download the ORY Oathkeeper CLI and put it in your PATH.

On Mac or Linux you will need to make the binary executable (and you may also want to rename it to something more convenient): $ mv oathkeeper-darwin-amd64 oathkeeper && chmod u+x oathkeeper

Ambassador

Ambassador is a Kubernetes-native API Gateway built on the Envoy Proxy. Ambassador supports a wide variety of features needed in an edge proxy, e.g., rate limiting, distributed tracing, dynamic routing, metrics, and more. Ambassador also includes an authentication API where you can plug in an external authentication service. This is the API that we will be using in this post.

Deploying & Configuring Ambassador

The first step is confirming that kubectl is set up properly:

$ kubectl get service kubernetes

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 21m

I recommend first completing the Ambassador Getting Started tutorial on thegetambassador.io website, but I have included the core steps to set Ambassador up for this tutorial here (these steps were correct as of publication in August 2018).

To deploy Ambassador in your default namespace, first you need to check if Kubernetes has RBAC enabled:

$ kubectl cluster-info dump --namespace kube-system | grep authorization-mode

If you see something like --authorization-mode=Node,RBAC in the output, then RBAC is enabled.

Note: If you’re using Google Kubernetes Engine with RBAC (which is the default for all new clusters), you will need to grant permissions to the account that will be setting up Ambassador. To do this, get your official GKE username, and then grant cluster-admin role privileges to that username:

$ kubectl create clusterrolebinding my-cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud info --format="value(config.account)")

If RBAC is enabled:

$ kubectl apply -f https://getambassador.io/yaml/ambassador/ambassador-rbac.yaml

Without RBAC, you can use:

$ kubectl apply -f https://getambassador.io/yaml/ambassador/ambassador-no-rbac.yaml

Defining the Ambassador Service

Ambassador is deployed as a Kubernetes service. Create the following YAML and put it in a file called ambassador-service.yaml.

---

apiVersion: v1

kind: Service

metadata:

name: ambassador

spec:

ports:

-

port: 80

selector:

service: ambassador

type: LoadBalancer

Deploy this service with kubectl :

$ kubectl apply -f ambassador-service.yaml

The YAML above creates a Kubernetes service for Ambassador of type LoadBalancer. All HTTP traffic will be evaluated against the routing rules you create. Note that if you're not deploying in an environment where LoadBalancer is a supported type, you'll need to change this to a different type of service, e.g., NodePort.

Creating your first route

Create the following YAML and put it in a file called httpbin.yaml:

---

apiVersion: v1

kind: Service

metadata:

annotations:

getambassador.io/config: |

---

apiVersion: ambassador/v0

kind: Mapping

name: httpbin_mapping

prefix: /httpbin/

service: httpbin.org:80

host_rewrite: httpbin.org

name: httpbin

spec:

ports:

-

name: httpbin

port: 80

Then, apply it to the Kubernetes with kubectl :

$ kubectl apply -f httpbin.yaml

When the service is deployed, Ambassador will notice the getambassador.io/config annotation on the service, and use the Mapping contained in it to configure the route. (There's no restriction on what kinds of Ambassador configuration can go into the annotation, but it's important to note that Ambassador only looks at annotations on Kubernetes services.)

In this case, the mapping creates a route that will route traffic from the /httpbin/ endpoint to the public httpbin.org service. Note that you are using the host_rewrite attribute for the httpbin_mapping — this forces the HTTP Host header, and is often a good idea when mapping to external services.

Testing the Mapping

To test things out, you'll need the external IP for Ambassador (it might take some time for this to be available):

$ kubectl get svc -o wide ambassador

Eventually, this should give you something like:

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE

ambassador 10.11.12.13 35.36.37.38 80:31656/TCP 1m

You should now be able to use curl to httpbin:

$ curl 35.36.37.38/httpbin/ip

{

"origin": "< your IP address >"

}

or on minikube:

$ minikube service list

|-------------|----------------------|------------------------------|

| NAMESPACE | NAME | URL |

|-------------|----------------------|------------------------------|

| default | ambassador | http://192.168.178.108:32548 |

| default | ambassador-admin | http://192.168.178.108:30428 |

|-------------|----------------------|------------------------------|



$ curl http://192.168.178.108:32548/httpbin/ip

{

"origin": "< your IP address >"

}

When you have found your Ambassador IP, I would recommend placing this into an appropriate variable e.g.

$ export AMBASSADOR_IP=192.168.178.108:30428

ORY Oathkeeper

ORY Oathkeeper is a cloud native Identity & Access Service. As such, it evaluates incoming HTTP request based on a set of rules, decides whether the request should be allowed or not, and converts the session data to a consumable format. Decisions are made by consulting two deciders: Authenticators and Authorizers.

Authenticators look for access credentials in the HTTP header — for example abearer token, and implement business logic which validate those credentials. ORY Oathkeeper currently ships with different authenticators:

The JWT authenticator looks for the bearer token in the HTTP header and treats the value as a JSON Web Token. You can define which signature verification algorithm (HS256, RS256, …) should be used and provide the required key(s).

looks for the bearer token in the HTTP header and treats the value as a JSON Web Token. You can define which signature verification algorithm (HS256, RS256, …) should be used and provide the required key(s). The OAuth 2.0 Token Introspection authenticator extracts the bearer token from the HTTP header and performs the OAuth 2.0 Token Introspection flow. This authenticator works great with ORY Hydra!

For a complete list of implemented authenticators, head over to the ORY Oathkeeper developer guide..

Authorizers use the session state returned by the authenticator to authorize the request. This could be by consulting an Access Control List (ACL), Role Based Access Control (RBAC), or more advanced Access Control Policy Definitions like the one provided by ORY Keto.

Credential Issuers convert the session state returned by authenticators to an easily consumable format. The session state can be converted to a JSON Web Token signed with a private/public keypair, to HTTP Headers, and to HTTP Cookies.

ORY Oathkeeper has two operational modes. One is a reverse proxy which can be deployed as a sidecar or in close proximity to the API Gateway. The second is as anAPI which is connected to the API Gateway of your choice. For this tutorial, you will exclusively look at the API operation mode.

Deploying and Configuring ORY Oathkeeper

First you need to create a secret which will be used to sign the ID Token. The secret must be 32 characters long:

$ kubectl create secret generic ory-oathkeeper --from-literal=CREDENTIALS_ISSUER_ID_TOKEN_HS256_SECRET=<your-secret>

# For example:

# $ kubectl create secret generic ory-oathkeeper --from-literal=CREDENTIALS_ISSUER_ID_TOKEN_HS256_SECRET=dYmTueb6zg8TphfZbOUpOewd0gt7u0SH

Next, deploy the ORY Oathkeeper Service and Deployment in “API mode”.

This configuration sets up the ORY Oathkeeper API with an in-memory database (please note, that restarting the service will remove all existing data!). ORY Oathkeeper can connect to other database backends such as MySQL or PostgreSQL for persistence.

This configuration additionally creates a ClusterIP service which makes it available from the Kubernetes-internal network.

But you want the service to be accessible from the outside world as well! To do that we’ll fetch the yaml definition

and open it in a text editor. The first section reads the service definition of ORY Oathkeeper:

---

apiVersion: v1

kind: Service

metadata:

name: ory-oathkeeper

spec:

ports:

-

name: http-ory-oathkeeper

port: 80

targetPort: http-api

selector:

app: ory-oathkeeper

type: ClusterIP

This configuration does not include metadata for Ambassador. Let’s change that and make ORY Oathkeeper’s API available to the outside world. In a production deployment, you wouldn’t do this under normal circumstances, and instead you would expose this API only internally, or with some type of access control in place — for example Ambassador + ORY Oathkeeper!

Ok, let’s define a mapping that makes ORY Oathkeeper available through ambassador. To do so, the metadata of the service needs to be updated:

---

metadata:

name: ory-oathkeeper

annotations:

getambassador.io/config: |-

---

apiVersion: ambassador/v0

kind: Mapping

name: ory-oathkeeper_mapping

prefix: /ory-oathkeeper/

service: ory-oathkeeper

The complete file should now look like this:

---

apiVersion: v1

kind: Service

metadata:

name: ory-oathkeeper

annotations:

getambassador.io/config: |

---

apiVersion: ambassador/v0

kind: Mapping

name: ory-oathkeeper_mapping

prefix: /ory-oathkeeper/

service: ory-oathkeeper

spec:

ports:

-

name: http-ory-oathkeeper

port: 80

targetPort: http-api

selector:

app: ory-oathkeeper

type: ClusterIP

[... rest of the file ...]

Let’s re-apply the configuration:

$ kubectl apply -f oathkeeper-api.yaml

Now you can check if the ORY Oathkeeper is alive via the Ambassador route you have created, and you can also list all access rules via the Oathkeeper CLI you downloaded earlier (for now just an empty array):

$ curl http://${AMBASSADOR_IP}/ory-oathkeeper/health/alive

{"status":"ok"} $ oathkeeper rules --endpoint http://${AMBASSADOR_IP}/ory-oathkeeper list

[]

Next, you will define an access rule for accessing ORY Oathkeeper’s API. To keep things simple, you will require no authentication or authorization to access the API. Let’s echo to a new file access-rule-oathkeeper.json:

cat <<EOT > access-rule-oathkeeper.json

[{

"id": "oathkeeper-access-rule",

"match": {

"url": "http://${AMBASSADOR_IP}/ory-oathkeeper/<.*>",

"methods": ["GET", "POST", "PUT", "DELETE", "OPTIONS", "HEAD"]

},

"authenticators": [{ "handler": "anonymous" }],

"authorizer": { "handler": "allow" },

"credentials_issuer": { "handler": "noop" }

}]

EOT

You need to make sure that the value of match.url (here http://${AMBASSADOR_IP}/ory-oathkeeper/<.*> ) has the host and port where ambassador is available to you. If you set the environment variable previously, this is the case. ${AMBASSADOR_IP} would be, for example, the IP:Port you can find with minikube service list. The rule itself is very simple, it matches all requests with prefix http://${AMBASSADOR_IP}/oathkeeper-api/ and does not enforce any authentication (“anonymous” allows access by unauthorized clients), allows all requests, and does not transform the authorization header. You will set up a more sophisticated rule in the next sections.

Let’s import this rule into ORY Oathkeeper:

$ oathkeeper rules --endpoint http://${AMBASSADOR_IP}/ory-oathkeeper import access-rule-oathkeeper.json

Now you are ready to activate the external auth service in Ambassador. To do so, you add another section to the annotations you downloaded earlier as file oathkeeper-api.yaml:

---

apiVersion: ambassador/v0

kind: AuthService

name: authentication

auth_service: ory-oathkeeper

path_prefix: /judge

allowed_headers:

- Authorization

The complete file should now look like this:

---

apiVersion: v1

kind: Service

metadata:

annotations:

getambassador.io/config: |

---

apiVersion: ambassador/v0

kind: Mapping

name: ory-oathkeeper_mapping

prefix: /ory-oathkeeper/

service: ory-oathkeeper

---

apiVersion: ambassador/v0

kind: AuthService

name: authentication

auth_service: ory-oathkeeper

path_prefix: /judge

allowed_headers:

- Authorization

name: ory-oathkeeper

spec:

ports:

-

name: http-ory-oathkeeper

port: 80

targetPort: http-api

selector:

app: ory-oathkeeper

type: ClusterIP

[... rest of file …]

And re-apply the configuration:

$ kubectl apply -f oathkeeper-api.yaml

If you retry the command from earlier

$ oathkeeper rules --endpoint http://${AMBASSADOR_IP}/ory-oathkeeper list

[{

"authenticators": [{ "handler": "noop" } ],

[...]

You will notice that the request passes and you will also see the access rule you just created! Now, if you try to call the httpbin service, the request will fail with a 404 because no access rule has been configured for this service:

$ curl http://${AMBASSADOR_IP}/httpbin/

{"error":{"code":404,"status":"Not Found","request":"84a2b164-7229-4f69-a0cd-227611c07128","message":"Requested url does not match any rules"}}

Let’s change that by creating a simple access rule in file access-rule-httpbin.json for the httpbin service (Don’t forget to replace the URL with your Ambassador IP and port number):

cat <<EOT > access-rule-httpbin.json

[{

"id": "httpbin-access-rule",

"match": {

"url": "http://${AMBASSADOR_IP}/httpbin/<.*>",

"methods": ["GET"]

},

"authenticators": [{ "handler": "anonymous" }],

"authorizer": { "handler": "deny" },

"credentials_issuer": { "handler": "noop" }

}]

EOT

The access rule is very similar to the one you created for ORY Oathkeeper. This time however, you are using a simple authorizer that denies all requests. Let’s import the rule and see what happens when you request the httpbin service.

$ oathkeeper rules --endpoint http://${AMBASSADOR_IP}/ory-oathkeeper import access-rule-httpbin.json $ curl http://${AMBASSADOR_IP}/httpbin/

{"error":{"code":403,"status":"Forbidden","request":"fa893865-35dc-47fd-9907-52da8664c242","message":"Access credentials are not sufficient to access this resource"}}

Ok, so authorization was not granted. Let’s update the rule and allow all requests:

cat <<EOT > access-rule-httpbin.json

[{

"id": "httpbin-access-rule",

"match": {

"url": "http://${AMBASSADOR_IP}/httpbin/<.*>",

"methods": ["GET"]

},

"authenticators": [{ "handler": "anonymous" }],

"authorizer": { "handler": "allow" },

"credentials_issuer": { "handler": "noop" }

}]

EOT

Import the file again and execute curl:

$ oathkeeper rules --endpoint http://${AMBASSADOR_IP}/ory-oathkeeper import access-rule-httpbin.json $ curl http://${AMBASSADOR_IP}/httpbin/

<!DOCTYPE html>

<html lang="en">

[...]

It worked! There are obviously many more authentication and authorization strategies. However, you have barely touched the surface. For example, you can authenticate OAuth 2.0 Access Tokens using the OAuth 2.0 Token Introspection Authenticator. A list of all the possible handlers can be found in the ORY Oathkeeper documentation.

If you’re looking for an OAuth 2.0 Server that just works, you should check out ORY Hydra immediately. All ORY products integrate very well with one another but can also work completely standalone. The ORY team are also working on an ORY Oathkeeper Authorizer that works with the Open Policy Agent (OPA). If you find this interesting, check out the GitHub issuefor this.

Conclusion

In this tutorial you succesfully deployed Ambassador and ORY Oathkeeper to Kubernetes and set up different access rules that grant or deny access to the upstream httpbin service!

Keep an eye out for a follow up blogpost that will introduce ORY Hydra and ORY Keto. This will explain how to set up all four services in Kubernetes for a full-stack, cloud native access control system. Sign up to the ORY newsletter to be notified when the blogpost is released.

You can learn more about Ambassador at https://www.getambassador.io, and about the Ambassador Authentication options in the docs. If you have any questions, please join our Slack, drop us a line in the comments below, or @getambassadorio on Twitter.