Creating the pipeline

If you remember from the video (or not if you’ve not watched it), this is the pipeline that we will be building:

You can start by creating the 3 buckets we will use for this demo: xray-data-in, xray-data-in-processed and xray-data-in-anonymized.

Setting up Bucket notifications to Kafka

Now to bucket notifications! To help you with these steps, you will find a Postman collection of all the requests, “Ceph notifications pipeline.postman_collection.json” in the utils folder. Whether you use it or not, don’t forget to set your credentials (user Access Key and Secret Key) to be able to interact with the Gateway. In Postman you have to edit the Collection parameters (authorization tab), or you can set them request-by-request.

There are two different steps involved:

Create a Kafka topic: that’s a simple POST request to the gateway, passing the required parameters:

It will return the topic information in the form of arn:aws:sns:my-store::storage

Create the notification: that’s a PUT request to the gateway on the bucket where you want to have notifications enabled. In our example it was xray-data-in:

with the parameters passed as XML in the body of the request (including the topic you just created):



<TopicConfiguration>

<Id>storage</Id>

<Topic>arn:aws:sns:my-store::storage</Topic>

</TopicConfiguration>

</NotificationConfiguration> http://s3.amazonaws.com/doc/2006-03-01/ "> storage arn:aws:sns:my-store::storage

As simple as that!

Deploying the KNative Serving container

It’s the container that will do the image analysis, and eventually its anonymization. The source code is available in the examples/xray/xray-event-container folder of the repo. Here is the Python code itself:

It’s pretty much self-explanatory and commented but if you have any question please leave a comment. You can build it yourself, or use the image at quay.io/guimou/xray:master

After creating a project/namespace “xray”, we will create a secret with the credentials needed to access the different buckets that the container will interact with (don’t forget to create them! You should have xray-data-in, xray-data-processed and xray-data-anonymized for this demo). Here is the “secret.yaml” file (you will have to enter actual key/secret values):

apiVersion: v1

kind: Secret

metadata:

name: s3-secret

namespace: xray

stringData:

AWS_ACCESS_KEY_ID: replace_me

AWS_SECRET_ACCESS_KEY: replace_me

That we deploy with:

oc create -f secret.yaml

Then we define our service with “service-xray.yaml” (you will have to enter the right service_point URL):

apiVersion: serving.knative.dev/v1alpha1

kind: Service

metadata:

name: xray

namespace: xray

spec:

template:

spec:

containers:

— image: quay.io/guimou/xray:master

env:

— name: AWS_ACCESS_KEY_ID

valueFrom:

secretKeyRef:

name: s3-secret

key: AWS_ACCESS_KEY_ID

— name: AWS_SECRET_ACCESS_KEY

valueFrom:

secretKeyRef:

name: s3-secret

key: AWS_SECRET_ACCESS_KEY

— name: service_point

value: http://your.gateway.url

That we deploy with:

oc create -f service-xray.yaml

Deploying the KNative Eventing source

That’s the component that will consume our Kafka topic and pass the message to the Service we just created.

It is defined by “kafkasource.yaml”:

apiVersion: sources.eventing.knative.dev/v1alpha1

kind: KafkaSource

metadata:

name: kafka-source

namespace: xray

spec:

consumerGroup: xray

bootstrapServers: my-cluster-kafka-bootstrap.kafka:9092

topics: storage

sink:

apiVersion: serving.knative.dev/v1alpha1

kind: Service

name: xray

And again deployed with:

oc create -f kafkasource.yaml

Everything is now ready and should work!