Kubectl Unix Pipes: Manage Multiple Clusters With Ease

• By Duffie Cooley

kubectl enables system administrators to interact with Kubernetes using a well-designed command-line interface. The common interactions include running services, grabbing pod logs, or identifying nodes inside of a cluster. For most Kubernetes users, these interactions are second nature.

The kubectl tool has a number of advanced features that can empower users to filter data, aggregate data, or even forward cluster services to a local machine. These tools can be powerful for debugging production users or debugging your application. In our last post, we showed some shell tips for kubectl, and how kubeconfig contexts can provide flexibility for your cluster configurations. Let's explore another set of advanced examples that you can apply to deepen your knowledge of Kubernetes.

All of these tricks assume you have a working Kubernetes cluster and kubectl tool. If you don't have a Kubernetes cluster, we suggest Tectonic for AWS or bare metal and minikube for local laptop development.

Pod operations with kubectl

All of these examples make use of the Kubernetes API to pull specific information about pods. One way to build a query is to grab all the relevant data and then figure out what the jsonpath expression would be. For example, run kubectl get pods --all-namespaces -o json to see all the data we have to filter on for the 'Filter Kubernetes pods by time' example.

If you don't have an application running, try creating pods labelled run=shop and start a service listening on port 80:

kubectl run shop --replicas=2 --image quay.io/coreos/example-app:v1.0 --port 80 --expose

Then you can see what we are doing with jsonpath .

For more on jsonpath , check out the Kubernetes.io jsonpath support documentation.

Filter Kubernetes pods by time

kubectl get pods --all-namespaces --sort-by='.metadata.creationTimestamp' -o jsonpath='{range .items[*]}{.metadata.name}, {.metadata.creationTimestamp}{"

"}{end}'

Find Kubernetes Pod by Label Selector and Fetch the Pod Logs

Given a namespace "your-namespace" and a label query that identifies the pods you are interested in, you can get the logs for all of those pods. If the pod isn't unique, it will fetch the logs for each pod in parallel.

ns='<your-namespace>' label='<yourkey>=<yourvalue>' kubectl get pods -n $ns -l $label -o jsonpath='{range .items[*]}{.metadata.name}{"

"}{end}' | xargs -I {} kubectl -n $ns logs {}

Find a Kubernetes pod by label selector and port-forward locally

Given a namespace "your-namespace" and a label query that identifies the pods you are interested in connecting to, this command will connect to the first pod by name. Ensure that you replace 8080 with your pod's port.

ns='<your-namespace>' label='<yourkey>=<yourvalue>' kubectl -n $ns get pod -l $label -o jsonpath='{.items[1].metadata.name}' | xargs -I{} kubectl -n $ns port-forward {} 8080:80

Node operations with kubectl

By combining jq with kubectl JSON output, you can make complex queries, like filtering all resources by their create date.

Count the number of pods on a Kubernetes node

Often, high level statistics can help in debugging. This command will count all of the pods on each node:

kubectl get pods --all-namespaces -o json | jq '.items[] | .spec.nodeName' -r | sort | uniq -c

Filter nodes by label

You can use label queries on nodes. This is often used in configuring deployments that need certain constraints. For more info on selectors, see: kubectl explain deployment.spec.selector

kubectl get nodes -l 'master' or kubectl get nodes -l '!master'

You can list all labels with the --show-labels argument to any Kubernetes object. kubectl get nodes --all-namespaces --show-labels

Get a list of pods for each node

This will generate a JSON document that has a Kubernetes node name, and then a list of all of the pod names running on the node. This is very useful for debugging placement or load issues.

kubectl get pods --all-namespaces -o json | jq '.items | map({podName: .metadata.name, nodeName: .spec.nodeName}) | group_by(.nodeName) | map({nodeName: .[0].nodeName, pods: map(.podName)})'

Get the external IP for Kubernetes nodes

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name} {.status.addresses[?(@.type=="ExternalIP")].address}{"

"}{end}'

SSH into nodes with Fabric

Kubernetes has a database of nodes in the cluster which you can query with kubectl get nodes . This is a powerful database for automation and integration with existing tools. One such tool is the Fabric SSH utility, which is known as a fabfile.py . Introduced by CoreOS, this integrates Kubernetes nodes and Fabric together, enabling really powerful capabilities like SSH'ing into all machines running in a particular AWS failure domain.

fab -u core -R failure-domain.beta.kubernetes.io/zone=us-west-2a -- date

Learn more at the Fabric Kubernetes Nodes project.

Many other resources exist for working with the kubectl command line interface. Be sure to take a look at the cheat sheet in the kubernetes.io docs section as well!

Take a look at our recent webinar where we covered all of these features.