Serverless upload to the object storage

In this article I will demonstrate how to write a simple python function which will download an image from Unsplash and upload to Minio, S3 compatible object storage.

To follow and execute code in this article you will need MicroK8s, Kubeless and Python3. You can read introduction to serverless on MicroK8s in my article here:

Unsplash

Unsplash is a very popular marketing service ;-) — providing thousands of royalty free photos for free.

If you do not have Unsplash account, please create one.

Next, register a new application: https://unsplash.com/oauth/applications/new.

Register new app in Unsplash

After this you will receive your access key (which we will use later):

API Credentials.

We can verify is the key if working by executing curl command followed by jq:



$ curl -s

{

"desc": "5 bottles of DOSE Juice on a shelf"

} $ export UNSPLASH_ACCESS_KEY="..."$ curl -s https://api.unsplash.com/photos/\?client_id\=$UNSPLASH_ACCESS_KEY |jq '.[0] |{desc: .description}'"desc": "5 bottles of DOSE Juice on a shelf"

Minio

Minio is an open source object storage server, compatible with S3. An interesting feature is that it can also act as a gateway between other objects storage systems.

We can install Minio using Kubernetes, but before we need to prepare a Persistent Volume:

$ cat pv.yaml

kind: PersistentVolume

apiVersion: v1

metadata:

name: task-pv-volume

labels:

type: local

spec:

storageClassName: standard

capacity:

storage: 10Gi

accessModes:

- ReadWriteOnce

hostPath:

path: "/tmp/data"

Next, I generated yaml file for here: Minio https://www.minio.io/kubernetes.html. What you need to do is to uncomment storageClassName and fill it with correct value, in my case it was standard (same as in above pv.yaml file):

storageClassName: standard

and enter values into the Minio access key and secret key.

$ kubectl create -f minio-deployment.yaml

To verify:

$ kubectl get services

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE

kubernetes ClusterIP 10.152.183.1 <none> 443/TCP 4d

minio-service LoadBalancer 10.152.183.228 <pending> 9000:32302/TCP 72m

Python function for Kubeless

Now, most interesting part is coming — we are going to write some code :) To do it there are two helpful apis:

As we can see in Minio docs, it is using special python package. Let’s see if this package is available in Kubeless by trying to execute a function with simply importing minio:

$ cat /tmp/up.py

import minio def hello(event, context):

print (event)

return event['data'] $ kubeless function deploy up --runtime python3.6 --from-file up.py --handler up.hello INFO[0000] Deploying function...

INFO[0000] Function up submitted for deployment

INFO[0000] Check the deployment status executing 'kubeless function ls up'

$ kubeless function ls

NAME NAMESPACE HANDLER RUNTIME DEPENDENCIES STATUS

hello default test.hello python3.6 1/1 READY

up default up.hello python3.6 0/1 NOT READY

$ kubeless function ls

NAME NAMESPACE HANDLER RUNTIME DEPENDENCIES STATUS

hello default test.hello python3.6 1/1 READY

up default up.hello python3.6 1/1 READY

$ kubeless function call up --data "Let's pray..."

ERRO[0000] {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"no endpoints available for service \"up:http-function-port\"","reason":"ServiceUnavailable","code":503}



FATA[0000] the server is currently unable to handle the request

It does not have, which was expected :) Everyone who used to work with AWS Lambda for sure does remember building python packages, fitting them into specific size and uploading along with Lambda. Here, we can skip it and provide the dependencies as extra parameter:

$ cat requirements.txt

minio

requests

$ kubeless function deploy up --runtime python3.6 --from-file /tmp/up.py --dependencies requirements.txt --handler up.hello

Now, when we list our function we will see our packages on the list of the dependencies:

$ kubeless function ls up

NAME NAMESPACE HANDLER RUNTIME DEPENDENCIES STATUS

up default up.hello python3.6 minio 0/1 NOT READY

requests

$ kubectl get pods

NAME READY STATUS RESTARTS AGE



up-5f6cbd8bb7-957ls 0/1 Init:1/2 0 2m5s

Describe the pod to see that it is building new docker image:

$ kubectl describe pod up-5f6cbd8bb7-957ls ...Events:

Type Reason Age From Message

---- ------ ---- ---- -------

Normal Scheduled 6m26s default-scheduler Successfully assigned default/up-5f6cbd8bb7-957ls to t480

Normal Pulled 6m25s kubelet, t480 Container image "kubeless/unzip@sha256:f162c062973cca05459834de6ed14c039d45df8cdb76097f50b028a1621b3697" already present on machine

Normal Created 6m25s kubelet, t480 Created container

Normal Started 6m25s kubelet, t480 Started container

Normal Pulling 6m24s kubelet, t480 pulling image "python:3.6"

Normal Pulled 4m36s kubelet, t480 Successfully pulled image "python:3.6"

Normal Created 2m3s (x2 over 4m34s) kubelet, t480 Created container

Normal Started 2m3s (x2 over 4m33s) kubelet, t480 Started container

Normal Pulled 2m3s kubelet, t480 Container image "python:3.6" already present on machine

Quick check if the dependencies are working (in the code we are importing minio module):

$ kubeless function call up --data "Let's pray..."

Let's pray...

Since dependencies are working, I prepared simple python script to get an image from Unsplash — which I uploaded here, I do not copy paste it here due to the a poor monospace font support in Medium — where the most important parts namely are, downloading the image:

and uploading it to deployed in Kubernetes Minio:

This code I deployed as a function:

$ kubeless function delete up

$ kubeless function deploy up \

--runtime python3.6 --from-file up.py \

--dependencies requirements.txt --handler up.hello \

--env MINIO_ACCESS_KEY=$MINIO_ACCESS_KEY \

--env MINIO_SECRET_KEY=$MINIO_SECRET_KEY \

--env MINIO_API_URL=$MINIO_API_URL \

--env UNSPLASH_ACCESS_KEY=$UNSPLASH_ACCESS_KEY

Final call of the function (which will output name of the file after successful upload):

$ kubeless function call up Deer_1546895482.jpg

And check in the Minio web panel:

Everything works :) I hope you enjoyed reading this.