Running applications in production can be tricky. This post proposes an opinionated checklist for going to production with a web service (i.e. application exposing HTTP API) on Kubernetes.

This checklist was shaped by my own limited experience in Zalando, so please see it only as template for your own checklist adopted to your environment. Some items can be considered optional or aspirational, depending on context.

General Application's name, description, purpose, and owning team is clearly documented (e.g. in a central application registry or wiki)

Application's criticality level was defined (e.g. "tier 1" if the app is highly critical for the business)

Development team has sufficient knowledge/experience with the technology stack

Responsible 24/7 on-call team is identified and informed

Go-Live plan exists incl. steps for potential rollback

Application Application's code repository (git) has clear instructions on how to develop, how to configure, and how to contribute changes (important for emergency fixes)

Code dependencies are pinned (i.e. hotfix changes do not accidentally pull in new libraries)

All relevant code is instrumented with OpenTracing or OpenTelemetry

OpenTracing/OpenTelemetry semantic conventions are followed (incl. additional company conventions)

All outgoing HTTP calls have a defined timeout

HTTP connection pools are configured with sane values according to expected traffic

Thread pools and/or non-blocking async code is correctly implemented/configured

Database connection pools are sized correctly

Retries and retry policies (e.g. backoff with jitter) are implemented for dependent services

Circuit breakers are implemented

Fallbacks for circuit breakers are defined according to business requirements

Load shedding / rate limiting mechanisms are implemented (could be part of provided infrastructure)

Application metrics are exposed for collection (e.g. to be scraped by Prometheus)

Application logs go to stdout/stderr

Application logs follow good practices (e.g. structured logging, meaningful messages), log levels are clearly defined, and debug logging is disabled for production by default (with option to turn on)

Application container crashes on fatal errors (i.e. it does not enter some unrecoverable state or deadlock)

Application design/code was reviewed by a senior/principal engineer

Security & Compliance Application can run as unprivileged user (non-root)

Application does not require a writable container filesystem (i.e. can be mounted read-only)

HTTP requests are authenticated and authorized (e.g. using OAuth)

Mechanisms to mitigate Denial Of Service (DOS) attacks are in place (e.g. ingress rate limiting, WAF)

A security audit was conducted

Automated vulnerability checks for code / dependencies are in place

Processed data is understood, classified (e.g. PII), and documented

Threat model was created and risks are documented

Other applicable organizational rules and compliance standards are followed

CI/CD Automated code linting is run on every change

Automated tests are part of the delivery pipeline

No manual operations are needed for production deployments

All relevant team members can deploy and rollback

Production deployments have smoke tests and optionally automatic rollbacks

Lead time from code commit to production is fast (e.g. 15 minutes or less including test runs)

Kubernetes Development team is trained in Kubernetes topics and knows relevant concepts

Kubernetes manifests use the latest API version (e.g. apps/v1 for Deployment)

for Deployment) Container runs as non-root and uses a read-only filesystem

A proper Readiness Probe was defined, see blog post about Readiness/Liveness Probes

No Liveness Probe is used, or there is a clear rationale to use a Liveness Probe, see blog post about Readiness/Liveness Probes

Kubernetes deployment has at least two replicas

A Pod Disruption Budget was defined (or is automatically created, e.g. by pdb-controller)

Horizontal autoscaling (HPA) is configured if adequate

Memory and CPU requests are set according to performance/load tests

Memory limit equals memory requests (to avoid memory overcommit)

CPU limits are not set or impact of CPU throttling is well understood

Application is correctly configured for the container environment (e.g. JVM heap, single-threaded runtimes, runtimes not container-aware)

Single application process runs per container

Application can handle graceful shutdown and rolling updates without disruptions, see this blog post

Pod Lifecycle Hook (e.g. "sleep 20" in preStop ) is used if the application does not handle graceful termination

) is used if the application does not handle graceful termination All required Pod labels are set (e.g. Zalando uses "application", "component", "environment")

Application is set up for high availability: pods are spread across failure domains (AZs, default behavior for cross-AZ clusters) and/or application is deployed to multiple clusters

Kubernetes Service uses the right label selector for pods (e.g. not only matches the "application" label, but also "component" and "environment" for future extensibility)

There are no anti-affinity rules defined, unless really required (pods are spread across failure domains by default)

Optional: Tolerations are used as needed (e.g. to bind pods to a specific node pool) See also this curated checklist of Kubernetes production best practices.

Monitoring Metrics for The Four Golden Signals are collected

Application metrics are collected (e.g. via Prometheus scraping)

Backing data store (e.g. PostgreSQL database) is monitored

SLOs are defined

Monitoring dashboards (e.g. Grafana) exist (could be automatically set up)

Alerting rules are defined based on impact, not potential causes

Testing Breaking points were tested (system/chaos test)

Load test was performed which reflects the expected traffic pattern

Backup and restore of the data store (e.g. PostgreSQL database) was tested