Kubeflow has matured dramatically with its last two releases, not only becoming much more stable but also in the creation of Kubeflow Pipelines.

Kubeflow Pipelines take Kubeflow beyond TensorFlow based Machine Learning and creating a robust method for building just about any kind of pipeline in a Kubernetes Native way. This is a great improvement on other Workflow engines (like Airflow) as it enables fine grained control over access control secrets, volume mounts in a code-first way (suitable for deployment via CI/CD).

Kubeflow Pipelines provides robust infrastructure suitable for building enterprise grade ETL/ELT pipelines and is definitely worth spending an hour or so of experimentation.

While the official Kubeflow getting started guide (CLI) is great, it skips over a number of steps which are important for those less familiar with the tooling / Google Cloud.

Development environment setup

Install Python 3 (via Anaconda)

This assumes a linux / debian based environment

sudo apt-get update; apt-get install -y wget bzip2 mkdir tmp cd tmp curl -O https://repo.anaconda.com/archive/Anaconda3-2019.07-Linux-x86_64.sh bash Anaconda3-2019.07-Linux-x86_64.sh cd .. rm -r tmp conda init

Note

You will need to add your installation directory ( `./bin` ) to your PATH

You may need to specify which shell to initialize (e.g. bash / zsh)

You will need to restart your terminal

Create a conda environment pointing to Python 3.7

conda create --name kubeflow-env python=3.7 conda activate kubeflow-env

Install Kubeflow Python SDK

pip install https://storage.googleapis.com/ml-pipeline/release/latest/kfp.tar.gz --upgrade

Setup Authentication

Create OAuth Client Credentials:

Follow steps here:

How To: https://cloud.google.com/iap/docs/authentication-howto

Add Credentials: https://console.cloud.google.com/apis/credentials

Make sure that you select “Web application” as the type

You will need to enter authorised redirect domain: https://iap.googleapis.com/v1/oauth/clientIds/$(CLIENT_ID):handleRedirect



Authenticate with Google Cloud

# Set up the application-default credentials for gcloud gcloud auth application-default login # Make sure you that you also have a valid interactive login (for kubectl) gcloud auth login

Create Environment Variables for Auth

Credentials can be found here

# If using Cloud IAP, create environment variables from the # OAuth client ID and secret that you obtained earlier: export CLIENT_ID=XXXXXXXXXXXXXXXXX.apps.googleusercontent.com export CLIENT_SECRET=XXXXXXXXXXXXXXXXX

Setup kfctl

The kfctl tool is the Kubeflow version of kubectl and is responsible for creating and deploying the manifests required to deploy Kubeflow to a Kubernetes cluster.

Install kfctl (Mac)

wget https://github.com/kubeflow/kubeflow/releases/download/v0.6.2/kfctl_v0.6.2_darwin.tar.gz tar -xvf kfctl_v0.6.2_darwin.tar.gz cp kfctl $CONDA_PREFIX/bin chmod +x $CONDA_PREFIX/bin/kfctl

Install kfctl (Linux)

wget https://github.com/kubeflow/kubeflow/releases/download/v0.6.2/kfctl_v0.6.2_linux.tar.gz tar -xvf kfctl_v0.6.2_linux.tar.gz cp kfctl $CONDA_PREFIX/bin chmod +x $CONDA_PREFIX/bin/kfctl

Initialise kfctl

This step generates all the manifests required to create a new Kubernetes cluster and install all the services that Kubeflow requires (e.g. istio).

# The following command is optional, to make kfctl binary easier to use. export PATH=$PATH:/mnt/c/gcp # The zone you wish to create a new Kubernetes Cluster in export ZONE=australia-southeast1-a # The project you want to export PROJECT=<PROJECT_NAME> # Set KFAPP to the name of your Kubeflow application. See detailed # description in the text below this code snippet. export KFAPP=<CLUSTER_NAME> # Run this command for the default installation which uses Cloud IAP: kfctl init ${KFAPP} --platform gcp --project ${PROJECT}

Create the cluster & install kubeflow on it

cd ${KFAPP} kfctl generate all -V --zone ${ZONE} kfctl apply all -V

Notes:

You may need to increase the CPU quota here: https://console.cloud.google.com/iam-admin/quotas

If there are issues with authentication with the new cluster (e.g. your application-default and interactive gcloud logins have different email addresses) you might need to manually run: gcloud container clusters get-credentials $KFAPP --zone $ZONE --project $PROJECT In order to generate the k8s cluster authentication files



Login to your new Kubeflow page

This can take up to 20 minutes to deploy, so be patient. Once it has deployed successfully, you should be able to access the Kubeflow Admin here: https://<CLUSTER_NAME>.endpoints.<PROJECT_NAME>.cloud.goog/

Troubleshooting

Delete everything / start again

Delete the deployments from Deployment Manager: https://console.cloud.google.com/dm/deployments?

Next steps