If you are looking for a tool to deploy machine learning models as production web services, then ‘Cortex’ could be a good option to try. This open-source platform is an alternative to serving models with AWS SageMaker or creating your own model deployment platform over AWS services like Elastic Container Service (ECS), Elastic Kubernetes Service (EKS), and Elastic Compute Cloud (EC2) and even open-source projects like Docker, Kubernetes, and TensorFlow.

Key Features

Multi framework: Cortex supports multiple frameworks, including TensorFlow, PyTorch, scikit-learn, XGBoost, and more.

Cortex supports multiple frameworks, including TensorFlow, PyTorch, scikit-learn, XGBoost, and more. Autoscaling: Cortex automatically does load balancing for API’s to handle production workloads.

Cortex automatically does load balancing for API’s to handle production workloads. Infrastructure: Cortex can run inference both on CPU or GPU infrastructure.

Cortex can run inference both on CPU or GPU infrastructure. Rolling updates: Cortex updates APIs post-deployment without any downtime.

Examples of Cortex deployments

Sentiment analysis: Deploy a BERT model for sentiment analysis using Cortex

Image classification: Deploy an Inception model to classify images using Cortex

Install

Prerequisites (Copied from https://www.cortex.dev/install)

# install the CLI on your machine bash -c "$(curl -sS https://raw.githubusercontent.com/cortexlabs/cortex/0.11/get-cli.sh)" # provision infrastructure on AWS and spin up a cluster cortex cluster up

Deploy a model

# clone the Cortex repository git clone -b 0.11 https://github.com/cortexlabs/cortex.git # navigate to the iris classifier example cd cortex/examples/sklearn/iris-classifier # deploy the model to the cluster cortex deploy # view the status of the deployment cortex get --watch # get the API's endpoint cortex get classifier # classify a sample curl -X POST -H "Content-Type: application/json" \ -d '{ "sepal_length": 5.2, "sepal_width": 3.6, "petal_length": 1.4, "petal_width": 0.3 }' \ <API endpoint>

Github: https://github.com/cortexlabs/cortex

Website: https://www.cortex.dev

Tutorial: https://www.cortex.dev/iris-classifier

Examples: https://github.com/cortexlabs/cortex/tree/0.11/examples