Google Cloud PubSub is a fully managed service for real-time messaging and streaming data. Till now it was one of the most reliable components in our big data pipeline. But it’s less obvious that you can use it as a way to propagate changes throughout the system in a micro-service architecture.

While big data pipelines are in general high throughput, change propagation could be less chatty. For these scenarios PubSub has a nice feature called Push Subscriptions. Instead of writing specialised code in the pull model to pull messages from a subscription, a push subscription just let’s PubSub do a post on an HTTP endpoint.

PubSub Push

The PubSub Push Subscription feature takes a lot of the complexity out of the equation. No need to think about acknowledgments, just make sure you can handle the messages fast enough and return a 204. PubSub will make sure to keep the messages coming or back-off when your service can’t keep up. One caveat is that only one message is pushed at a time, but that’s ok for low throughput scenario’s. Remember: you don’t have to write any special code.

Create a new API key in the Cloud console

But, there is always a but… what about security? The PubSub FAQ hints on only accept messages that are accompanied by a secret key. But this implies writing extra code. Secret keys also means managing and rotating your keys, again extra code. Unless…

What if you could again move the key handling out of your code? Well you can, by using Cloud Endpoints.

Endpoints

Cloud Endpoints is a proxy you can put in front of your micro-service that can handle your authentication, monitoring, logging and tracing. It also has supports for API keys and is easy to use. The only thing you need to write for your endpoint is a OpenAPI file ( previously known as swagger ). The file needs to get pushed to your project using

gcloud service-management deploy openapi.yaml

In the example below we have a /api/hellopy/subscriptions/foobar push endpoint we want to secure. Simply by adding a security definition with a definition of api_key the endpoint proxy will handle all the key verification.

swagger: "2.0"

...

paths:

...

"/api/hellopy/subscriptions/foobar":

post:

description: "PubSub push."

operationId: "pubsub_foobar"

produces:

- "application/json"

responses:

204:

description: "Ack"

security:

- api_key: []

...

securityDefinitions:

api_key:

type: "apiKey"

name: "key"

in: "query"

In the Cloud project you need to create a credentials API key in the API manager. The key can then be used as a query parameter in the subscription url. That’s it.

Use the API key in the URL

Now you can start testing your endpoints by pushing some messages on the topic. Make sure to keep and eye on the logs to see if your messages come through.

Aside the zero code solution, you also have the ability to rotate the key by pressing Regenerate key in the API manager. The API manager gives you a grace period of 24 hours in which you have time to update the keys in your subscriptions, it will keep accepting the original key during that time.