Looking at the configuration of a Kubernetes node sounds like a simple thing, yet it not so obvious.

The arguments kubelet takes come either as command line parameters or from a configuration file you pass with --config . Seems, straight forward to do a ps -ex | grep kubelet and look in the file you see after the --config parameter. Simple, right? But… are you sure you got all the arguments right? What if Kubernetes defaulted to a value you did not want? What if you do not have shell access to a node?

There is a way to query the Kubernetes API for the configuration a node is running with: api/vi/nodes/<node_name>/proxy/cofigz . Lets see this in a real deployment.

Deploy a Kubernetes Cluster

I am using the Canonical Distribution of Kubernetes (CDK) on AWS here but you can use whichever cloud and Kubernetes installation method you like.

juju bootstrap aws

juju deploy canonical-kubernetes

..and wait for the deployment to finish

watch juju status

Change a Configuration

CDK allows for configuring both the command line arguments and the extra arguments of the config file. Here we add arguments to the config file:

juju config kubernetes-worker kubelet-extra-config='{imageGCHighThresholdPercent: 60, imageGCLowThresholdPercent: 39}'

A great question is how we got the imageGCHighThreshholdPercent literal. At the time of this writing the official upstream docs point you to the type definitions; a rather ugly approach. There is an EvictionHard property in the type definitions, however, if you look at the example of the upstream docs you see the same property is with lowercase.

Check the Configuration

We will need two shells. On the first one we will start the API proxy and on the second we will query the API. On the first shell:

juju ssh kubernetes-master/0

kubectl proxy

Now that we have the proxy at 127.0.0.1:8001 on the kubernetes-master, use a second shell to get a node name and we query the API:



kubectl get no

curl -sSL " juju ssh kubernetes-master/0kubectl get nocurl -sSL " http://localhost:8001/api/v1/nodes/ /proxy/configz" | python3 -m json.tool

Here is a full run:

juju ssh kubernetes-master/0

Welcome to Ubuntu 18.04.1 LTS (GNU/Linux 4.15.0-1023-aws x86_64)

* Management:

* Support: * Documentation: https://help.ubuntu.com * Management: https://landscape.canonical.com * Support: https://ubuntu.com/advantage System information as of Mon Oct 22 10:40:40 UTC 2018 System load: 0.11 Processes: 115

Usage of /: 13.7% of 15.45GB Users logged in: 1

Memory usage: 20% IP address for ens5: 172.31.0.48

Swap usage: 0% IP address for fan-252: 252.0.48.1

http://www.ubuntu.com/business/services/cloud Get cloud support with Ubuntu Advantage Cloud Guest: 0 packages can be updated.

0 updates are security updates. Last login: Mon Oct 22 10:38:14 2018 from 2.86.54.15

To run a command as administrator (user "root"), use "sudo <command>".

See "man sudo_root" for details.

NAME STATUS ROLES AGE VERSION

ip-172-31-14-174 Ready <none> 41m v1.12.1

ip-172-31-24-80 Ready <none> 41m v1.12.1

ip-172-31-63-34 Ready <none> 41m v1.12.1

ubuntu@ip-172-31-0-48:~$ curl -sSL "

{

"kubeletconfig": {

"syncFrequency": "1m0s",

"fileCheckFrequency": "20s",

"httpCheckFrequency": "20s",

"address": "0.0.0.0",

"port": 10250,

"tlsCertFile": "/root/cdk/server.crt",

"tlsPrivateKeyFile": "/root/cdk/server.key",

"authentication": {

"x509": {

"clientCAFile": "/root/cdk/ca.crt"

},

"webhook": {

"enabled": true,

"cacheTTL": "2m0s"

},

"anonymous": {

"enabled": false

}

},

"authorization": {

"mode": "Webhook",

"webhook": {

"cacheAuthorizedTTL": "5m0s",

"cacheUnauthorizedTTL": "30s"

}

},

"registryPullQPS": 5,

"registryBurst": 10,

"eventRecordQPS": 5,

"eventBurst": 10,

"enableDebuggingHandlers": true,

"healthzPort": 10248,

"healthzBindAddress": "127.0.0.1",

"oomScoreAdj": -999,

"clusterDomain": "cluster.local",

"clusterDNS": [

"10.152.183.93"

],

"streamingConnectionIdleTimeout": "4h0m0s",

"nodeStatusUpdateFrequency": "10s",

"nodeLeaseDurationSeconds": 40,

"imageMinimumGCAge": "2m0s",

"imageGCHighThresholdPercent": 60,

"imageGCLowThresholdPercent": 39,

"volumeStatsAggPeriod": "1m0s",

"cgroupsPerQOS": true,

"cgroupDriver": "cgroupfs",

"cpuManagerPolicy": "none",

"cpuManagerReconcilePeriod": "10s",

"runtimeRequestTimeout": "2m0s",

"hairpinMode": "promiscuous-bridge",

"maxPods": 110,

"podPidsLimit": -1,

"resolvConf": "/run/systemd/resolve/resolv.conf",

"cpuCFSQuota": true,

"cpuCFSQuotaPeriod": "100ms",

"maxOpenFiles": 1000000,

"contentType": "application/vnd.kubernetes.protobuf",

"kubeAPIQPS": 5,

"kubeAPIBurst": 10,

"serializeImagePulls": true,

"evictionHard": {

"imagefs.available": "15%",

"memory.available": "100Mi",

"nodefs.available": "10%",

"nodefs.inodesFree": "5%"

},

"evictionPressureTransitionPeriod": "5m0s",

"enableControllerAttachDetach": true,

"makeIPTablesUtilChains": true,

"iptablesMasqueradeBit": 14,

"iptablesDropBit": 15,

"failSwapOn": false,

"containerLogMaxSize": "10Mi",

"containerLogMaxFiles": 5,

"configMapAndSecretChangeDetectionStrategy": "Watch",

"enforceNodeAllocatable": [

"pods"

]

}

} ubuntu@ip-172-31-0-48:~$ kubectl get noNAME STATUS ROLES AGE VERSIONip-172-31-14-174 Ready 41m v1.12.1ip-172-31-24-80 Ready 41m v1.12.1ip-172-31-63-34 Ready 41m v1.12.1ubuntu@ip-172-31-0-48:~$ curl -sSL " http://localhost:8001/api/v1/nodes/ip-172-31-14-174/proxy/configz " | python3 -m json.tool"kubeletconfig": {"syncFrequency": "1m0s","fileCheckFrequency": "20s","httpCheckFrequency": "20s","address": "0.0.0.0","port": 10250,"tlsCertFile": "/root/cdk/server.crt","tlsPrivateKeyFile": "/root/cdk/server.key","authentication": {"x509": {"clientCAFile": "/root/cdk/ca.crt"},"webhook": {"enabled": true,"cacheTTL": "2m0s"},"anonymous": {"enabled": false},"authorization": {"mode": "Webhook","webhook": {"cacheAuthorizedTTL": "5m0s","cacheUnauthorizedTTL": "30s"},"registryPullQPS": 5,"registryBurst": 10,"eventRecordQPS": 5,"eventBurst": 10,"enableDebuggingHandlers": true,"healthzPort": 10248,"healthzBindAddress": "127.0.0.1","oomScoreAdj": -999,"clusterDomain": "cluster.local","clusterDNS": ["10.152.183.93"],"streamingConnectionIdleTimeout": "4h0m0s","nodeStatusUpdateFrequency": "10s","nodeLeaseDurationSeconds": 40,"imageMinimumGCAge": "2m0s","imageGCHighThresholdPercent": 60,"imageGCLowThresholdPercent": 39,"volumeStatsAggPeriod": "1m0s","cgroupsPerQOS": true,"cgroupDriver": "cgroupfs","cpuManagerPolicy": "none","cpuManagerReconcilePeriod": "10s","runtimeRequestTimeout": "2m0s","hairpinMode": "promiscuous-bridge","maxPods": 110,"podPidsLimit": -1,"resolvConf": "/run/systemd/resolve/resolv.conf","cpuCFSQuota": true,"cpuCFSQuotaPeriod": "100ms","maxOpenFiles": 1000000,"contentType": "application/vnd.kubernetes.protobuf","kubeAPIQPS": 5,"kubeAPIBurst": 10,"serializeImagePulls": true,"evictionHard": {"imagefs.available": "15%","memory.available": "100Mi","nodefs.available": "10%","nodefs.inodesFree": "5%"},"evictionPressureTransitionPeriod": "5m0s","enableControllerAttachDetach": true,"makeIPTablesUtilChains": true,"iptablesMasqueradeBit": 14,"iptablesDropBit": 15,"failSwapOn": false,"containerLogMaxSize": "10Mi","containerLogMaxFiles": 5,"configMapAndSecretChangeDetectionStrategy": "Watch","enforceNodeAllocatable": ["pods"

Summing Up

There is a way to get the configuration of an online Kubernetes node through the Kubernetes API (api/v1/nodes/<node_name>/proxy/configz). This might be handy if you want to code against Kubernetes or you do not want to get into the intricacies of your particular cluster setup.

References