I already described how you can get started with Kubernetes natively on vSphere using the kube-up/kube-down mechanism. This was pretty straight-forward, but not ideal as it was not very reliable or easy to follow. Since writing that piece, Kubernetes have moved on to a new deployment mechanism called kubernetes-anywhere. In this post, I will show you how to deploy Kubernetes onto a vSphere environment with a vSAN datastore, using the kubernetes-anywhere utility. All of this is done from a Photon OS VM. Now in my previous example, I used the Photon OS OVA, which is a trimmed down version of the OS. In this example, I have deployed my Photon OS VM using the full ISO image, meaning that all of the tooling I might need (git, awk, etc) are already included. This just saves me a little time. If you want to use the cut-down distribution, check out how to add the additional tooling to the OS with tdnf in my previous post. Now, I have broken up this deployment into a number of distinct parts, but in a nutshell, the steps can be summarized as follows:

Setup guest from where deployment will be done, e.g. Photon OS distro Download OVA for K8S Download “kubernetes-anywhere” Create build environment Make configuration file for deploying K8S on vSphere Deploy K8S on vSphere Get access to the K8S dashboard

Let’s look at these steps in more detail. If you do not need all of this detail, there is an excellent write-up on how to get started with K8S on vSphere found here.

Part 1: Deploy a Photon OS VM with full ISO image

There were only a few things I needed to do with the full ISO image. One was to give it more disk space, for cloning and building. I allocated a 100GB VMDK during VM creation. The next thing I did was to login on the console and enable root logins over SSH, and the final item was to enable and start docker. That’s it. Of course, you can also use many other distros if you wish, but I’m getting to like Photon OS for this sort of testing.

Part 2: Download Photon OS image for VMs to be used by Kubernetes

You can download the OVA from here, or you can deploy it directly from the vSphere Web Client:

Make sure it is on the same vSphere environment where you plan to run K8S. This OVA is another Photon OS image, and will be used to deploy master and node VMs to run Kubernetes. Do not change the name, and do not power it on.

Part 3: git clone kubernetes-anywhere

Login in to your Photon OS VM, install git with “tdnf install git” and then run a git clone of kubernetes-anywhere.

root@photon [ ~ ]# git clone https://github.com/kubernetes/kubernetes-anywhere Cloning into 'kubernetes-anywhere'... remote: Counting objects: 4092, done. remote: Total 4092 (delta 0), reused 0 (delta 0), pack-reused 4092 Receiving objects: 100% (4092/4092), 3.97 MiB | 642.00 KiB/s, done. Resolving deltas: 100% (2573/2573), done. Checking connectivity... done.

Part 4: make docker-dev

Now we build a new container environment for the deployment. This is why docker must be enabled and started on the Photon OS VM. This step takes a long time, and generates a lot of output. I’ve put a few snippets here, so you know what to expect. Although it takes a long time to complete the very first time it is run, it is only a few seconds subsequently to make new environments since the container images have all been downloaded.

root@photon [ ~ ]# cd kubernetes-anywhere root@photon [ ~/kubernetes-anywhere ]# make docker-dev docker build -t kubernetes-anywhere:v0.0.1 . Sending build context to Docker daemon 205.3 kB Step 1 : FROM mhart/alpine-node:6.4.0 6.4.0: Pulling from mhart/alpine-node e110a4a17941: Pull complete d6d25f5b0348: Pull complete Digest: sha256:0a7f08961c8dfaf42910a5aa58549c840bb93fdafab5af3746c11f1cd4062bda Status: Downloaded newer image for mhart/alpine-node:6.4.0 ---> ecd37ad77c2b Step 2 : RUN apk add --update bash ---> Running in f50126adb9b2 fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz (1/5) Installing ncurses-terminfo-base (6.0-r7) (2/5) Installing ncurses-terminfo (6.0-r7) (3/5) Installing ncurses-libs (6.0-r7) (4/5) Installing readline (6.3.008-r4) (5/5) Installing bash (4.3.42-r5) Executing bash-4.3.42-r5.post-install Executing busybox-1.24.2-r9.trigger OK: 14 MiB in 18 packages ---> f06561950212 Removing intermediate container f50126adb9b2 Step 3 : ADD ./util/docker-build.sh /opt/ ---> f37f66f9fcfe Removing intermediate container 1da43e7f98a1 Step 4 : RUN /opt/docker-build.sh ---> Running in c8b41a59eb47 fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz + apk add --update git build-base wget curl jq autoconf automake pkgconfig ncurses-dev libtool gperf flex bison ca-certificates fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz (1/38) Upgrading musl (1.1.14-r11 -> 1.1.14-r14). 81300K .......... .......... .......... .......... .......... 99% 203K 0s 81350K ......... 100% 164M= 3m21s 2017-02-15 13:35:14 (405 KB/s) - '/usr/local/bin/kubectl' saved [83312488/83312488] + chmod +x /usr/local/bin/kubectl + cd /tmp + git clone https://github.com/google/jsonnet.git Cloning into 'jsonnet'.. . Saving to: 'terraform_0.7.2_linux_amd64.zip' 0K .......... .......... .......... .......... .......... 0% 987K 16s . 2017-02-15 13:36:11 (8.28 MB/s) - 'terraform_0.7.2_linux_amd64.zip' saved [16332841/16332841] + wget https://releases.hashicorp.com/terraform/0.7.2/terraform_0.7.2_SHA256SUMS --2017-02-15 13:36:11-- https://releases.hashicorp.com/terraform/0.7.2/terraform_0.7.2_SHA256SUMS . Removing intermediate container c3e8ad7bf718 Step 5 : WORKDIR /opt/kubernetes-anywhere ---> Running in 4d419a2d773f ---> 368033c43cdd Removing intermediate container 4d419a2d773f Step 6 : ADD . /opt/kubernetes-anywhere/ ---> 6d16fcb2ed3d Removing intermediate container eb77d6546eeb Step 7 : CMD make ---> Running in c00e6d5d1c94 ---> 32497ffce687 Removing intermediate container c00e6d5d1c94 Successfully built 32497ffce687 Starting Kuberetes Anywhere deployment shell in a container docker run -it --rm --env="PS1=[container]:\w> " --net=host --volume="`pwd`:/opt/kubernetes-anywhere" kubernetes-anywhere:v0.0.1 /bin/bash [container]:/opt/kubernetes-anywhere>

Part 5: make config – Configure the vSphere cloud provider for Kubernetes This is where you tell “kubernetes-anywhere” that you are deploying against a vSphere environment, and this is where you will fill in the details for the vSphere environment. You will need the IP/FQDN of your vCenter server, plus credentials. You will also need datacenter name, cluster name and datastore name. As you can see from this setup, I an deploying onto a vSAN datastore. The other major difference from the defaults is the phase 2 installer container. Do not pick the default, but instead use the one that we have highlighted here. I have highlighted in red where you will need to change from the default, but obviously these are taken from my setup, so you will have to modify to match your own. In phase 3, the addons, Y (for Yes) is the default, so you can just hit return. [container]:/opt/kubernetes-anywhere> make config CONFIG_="." kconfig-conf Kconfig * * Kubernetes Minimal Turnup Configuration * * * Phase 1: Cluster Resource Provisioning * number of nodes (phase1.num_nodes) [4] cluster name (phase1.cluster_name) [kubernetes] * * cloud provider: gce, azure or vsphere * cloud provider: gce, azure or vsphere (phase1.cloud_provider) [ vsphere ] * * vSphere configuration * vCenter URL Ex: 10.192.10.30 or myvcenter.io (without https://) (phase1.vSphere.url) [ vcsa-06.rainpole.com ] vCenter port (phase1.vSphere.port) [443] vCenter username (phase1.vSphere.username) [ administrator@vsphere.local ] vCenter password (phase1.vSphere.password) [ xxxxxxx ] Does host use self-signed cert (phase1.vSphere.insecure) [true] Datacenter (phase1.vSphere.datacenter) [ Datacenter ] Datastore (phase1.vSphere.datastore) [ vsanDatastore ] Specify a valid Cluster, Host or Resource Pool in which to deploy Kubernetes VMs. (phase1.vSphere.resourcepool) [ Cluster ] Number of vCPUs for each VM (phase1.vSphere.vcpu) [1] Memory for VM (phase1.vSphere.memory) [2048] Name of the template VM imported from OVA (phase1.vSphere.template) [KubernetesAnywhereTemplatePhotonOS.ova] Flannel Network (phase1.vSphere.flannel_net) [172.1.0.0/16] * * Phase 2: Node Bootstrapping * installer container (phase2.installer_container) [ docker.io/ashivani/k8s-ignition:v4 ] docker registry (phase2.docker_registry) [gcr.io/google-containers] kubernetes version (phase2.kubernetes_version) [v1.4.8] bootstrap provider (phase2.provider) [ignition] * * Phase 3: Deploying Addons * Run the addon manager? (phase3.run_addons) [Y/n/?] Run kube-proxy? (phase3.kube_proxy) [Y/n/?] Run the dashboard? (phase3.dashboard) [Y/n/?] Run heapster? (phase3.heapster) [Y/n/?] Run kube-dns? (phase3.kube_dns) [Y/n/?] # # configuration written to .config # make: '.config' is up to date. [container]:/opt/kubernetes-anywhere>

Part 6: make deploy – Roll out Kubernetes

This is the last step to get K8S deployed. This rolls out the master and node VMs for Kubernetes, and builds the framework. Again, I have included some snippets, but this does take a little time as well, especially the master (which I’ve seen take between 12min and 15min).

[container]:/opt/kubernetes-anywhere> make deploy util/config_to_json .config > .config.json make do WHAT=deploy-cluster make[1]: Entering directory '/opt/kubernetes-anywhere' ( cd "phase1/$(jq -r '.phase1.cloud_provider' .config.json)"; ./do deploy-cluster ) .tmp/vSphere-kubernetes.tf data.template_file.cloudprovider: Refreshing state... tls_private_key.kubernetes-root: Creating... . tls_private_key.kubernetes-admin: Creating... . tls_private_key.kubernetes-node: Creating... . vsphere_folder.cluster_folder: Creating... datacenter: "" => "Datacenter" existing_path: "" => "<computed>" path: "" => "kubernetes" tls_private_key.kubernetes-master: Creating... . vsphere_folder.cluster_folder: Creation complete vsphere_virtual_machine.kubevm2: Creating... datacenter: "" => "Datacenter" disk.#: "" => "1" disk.705700330.bootable: "" => "true" disk.705700330.controller_type: "" => "scsi" disk.705700330.datastore: "" => "vsanDatastore" disk.705700330.iops: "" => "" disk.705700330.keep_on_remove: "" => "" disk.705700330.key: "" => "<computed>" disk.705700330.name: "" => "" disk.705700330.size: "" => "" disk.705700330.template: "" => "KubernetesAnywhereTemplatePhotonOS.ova" disk.705700330.type: "" => "thin" disk.705700330.uuid: "" => "<computed>" disk.705700330.vmdk: "" => "" domain: "" => "vsphere.local" enable_disk_uuid: "" => "true" folder: "" => "kubernetes" linked_clone: "" => "false" memory: "" => "2048" memory_reservation: "" => "0" name: "" => "node1" network_interface.#: "" => "1" network_interface.0.ip_address: "" => "<computed>" network_interface.0.ipv4_address: "" => "<computed>" network_interface.0.ipv4_gateway: "" => "<computed>" network_interface.0.ipv4_prefix_length: "" => "<computed>" network_interface.0.ipv6_address: "" => "<computed>" network_interface.0.ipv6_gateway: "" => "<computed>" network_interface.0.ipv6_prefix_length: "" => "<computed>" network_interface.0.label: "" => "VM Network" network_interface.0.mac_address: "" => "<computed>" network_interface.0.subnet_mask: "" => "<computed>" resource_pool: "" => "Cluster" skip_customization: "" => "true" time_zone: "" => "Etc/UTC" uuid: "" => "<computed>" vcpu: "" => "1" vsphere_virtual_machine.kubevm4: Creating... . vsphere_virtual_machine.kubevm1: Creating... . vsphere_virtual_machine.kubevm5: Creating... . vsphere_virtual_machine.kubevm3: Creating... . tls_private_key.kubernetes-root: Creation complete tls_self_signed_cert.kubernetes-root: Creating... . tls_self_signed_cert.kubernetes-root: Creation complete tls_private_key.kubernetes-node: Creation complete data.tls_cert_request.kubernetes-node: Refreshing state... tls_private_key.kubernetes-master: Creation complete tls_locally_signed_cert.kubernetes-node: Creating... . tls_locally_signed_cert.kubernetes-node: Creation complete tls_private_key.kubernetes-admin: Creation complete data.tls_cert_request.kubernetes-admin: Refreshing state... tls_locally_signed_cert.kubernetes-admin: Creating... . tls_locally_signed_cert.kubernetes-admin: Creation complete vsphere_virtual_machine.kubevm2: Still creating... (10s elapsed) vsphere_virtual_machine.kubevm4: Still creating... (10s elapsed) vsphere_virtual_machine.kubevm1: Still creating... (10s elapsed) vsphere_virtual_machine.kubevm5: Still creating... (10s elapsed) vsphere_virtual_machine.kubevm3: Still creating... (10s elapsed) . null_resource.master: Still creating... (14m0s elapsed) null_resource.master (remote-exec): 96 75.8M 96 73.5M 0 0 94543 0 0:14:01 0:13:35 0:00:26 277k null_resource.master (remote-exec): 97 75.8M 97 73.7M 0 0 94772 0 0:13:59 0:13:36 0:00:23 278k null_resource.master (remote-exec): 97 75.8M 97 73.9M 0 0 94856 0 0:13:58 0:13:37 0:00:21 254k null_resource.master (remote-exec): 97 75.8M 97 74.1M 0 0 94981 0 0:13:57 0:13:38 0:00:19 233k null_resource.master (remote-exec): 97 75.8M 97 74.3M 0 0 95093 0 0:13:56 0:13:39 0:00:17 215k null_resource.master (remote-exec): 98 75.8M 98 74.5M 0 0 95234 0 0:13:55 0:13:40 0:00:15 203k null_resource.master (remote-exec): 98 75.8M 98 74.6M 0 0 95355 0 0:13:54 0:13:41 0:00:13 185k null_resource.master (remote-exec): 98 75.8M 98 74.8M 0 0 95501 0 0:13:52 0:13:42 0:00:10 196k null_resource.master (remote-exec): 98 75.8M 98 75.1M 0 0 95639 0 0:13:51 0:13:43 0:00:08 196k null_resource.master (remote-exec): 99 75.8M 99 75.2M 0 0 95727 0 0:13:51 0:13:44 0:00:07 194k null_resource.master: Still creating... ( 14m10s elapsed) null_resource.master (remote-exec): 99 75.8M 99 75.3M 0 0 95782 0 0:13:50 0:13:45 0:00:05 181k null_resource.master (remote-exec): 99 75.8M 99 75.5M 0 0 95844 0 0:13:50 0:13:46 0:00:04 170k null_resource.master (remote-exec): 99 75.8M 99 75.6M 0 0 95903 0 0:13:49 0:13:47 0:00:02 157k null_resource.master (remote-exec): 99 75.8M 99 75.8M 0 0 95969 0 0:13:48 0:13:48 --:--:-- 148k null_resource.master (remote-exec): 100 75.8M 100 75.8M 0 0 95991 0 0:13:48 0:13:48 --:--:-- 141k null_resource.master: Provisioning with 'local-exec'... . Apply complete! Resources: 19 added, 0 changed, 0 destroyed. The state of your infrastructure has been saved to the path below. This state is required to modify and destroy your infrastructure, so keep it safe. To inspect the complete state use the `terraform show` command. State path: ./.tmp/terraform.tfstate make[1]: Leaving directory '/opt/kubernetes-anywhere' KUBECONFIG="$(pwd)/phase1/$(jq -r '.phase1.cloud_provider' .config.json)/.tmp/kubeconfig.json" ./util/validate Validation: Expected 5 healthy nodes; found 0. (10s elapsed) Validation: Expected 5 healthy nodes; found 0. (20s elapsed) Validation: Expected 5 healthy nodes; found 5. (30s elapsed) Validation: Success! KUBECONFIG="$(pwd)/phase1/$(jq -r '.phase1.cloud_provider' .config.json)/.tmp/kubeconfig.json" ./phase3/do deploy + case "${1:-}" in + deploy + gen + cd ./phase3 + mkdir -p .tmp/ + jsonnet --multi .tmp/ --tla-code-file cfg=../.config.json all.jsonnet .tmp/dashboard-deployment.json .tmp/dashboard-svc.json .tmp/heapster-deployment.json .tmp/heapster-svc.json .tmp/kube-dns-deployment.json .tmp/kube-dns-svc.json .tmp/kube-proxy.json + mkdir -p /tmp/kubectl/ + HOME=/tmp/kubectl + kubectl apply -f ./.tmp/ deployment "kubernetes-dashboard" created service "kubernetes-dashboard" created deployment "heapster-v1.2.0" created service "heapster" created replicationcontroller "kube-dns-v19" created service "kube-dns" created daemonset "kube-proxy" created [container]:/opt/kubernetes-anywhere>

When this step completes, the master and nodes VMs should now be visible in the vSphere web client, in a folder called kubernetes:

Part 7 – Get access to the K8S dashboard

The kubectl commands will allow you to figure out which node and port are being used to run the dashboard. You can then open a browser and point to it. The first step is to download “kubectl”, and then point it at your K8S configuration.

[container]:/opt/kubernetes-anywhere> curl -LO https://storage.googleapis.com/kubernetes-release/release/v1.4.8/bin/linux/amd64/kubectl [container]:/opt/kubernetes-anywhere> chmod u+x kubectl [container]:/opt/kubernetes-anywhere> export KUBECONFIG=phase1/vsphere/.tmp/kubeconfig.json [container]:/opt/kubernetes-anywhere> kubectl describe service kubernetes-dashboard --namespace=kube-system| grep -i NodePort Type: NodePort NodePort: <unset> 31402 /TCP [container]:/opt/kubernetes-anywhere> kubectl get pods --namespace=kube-system| grep -i dashboard kubernetes-dashboard-1763797262-abexo 1/1 Running 0 13m [container]:/opt/kubernetes-anywhere> kubectl describe pod kubernetes-dashboard-1763797262-abexo --namespace=kube-system| grep Node Node: node4/ 10.27.51.132

Now if I point my browser at http://10.27.51.132:31402, I can launch my K8S dashboard.