We often need to simulate devices — sensors and vehicles — to test our IoT systems, see how well they manage load, how they deal with errors, etc.

Each simulated device pretends to be an independent entity with it’s own settings i.e. battery level, upload frequency, service outages, persistent memory, etc. Managing this at scale can be very challenging.

But we found that with StatefulSets in Kubernetes it’s actually super easy. With it we can manage credentials for hundreds of simulated devices and scale up/down as needed. What we are doing now would be a torture with webjobs.

To Deploy this tutorial:

Dowload IotHubCredentials.json from the git repository

PS C:\Users\vladi\Downloads> kubectl create secret generic sim-sensor-credentials --from-file=IoTHubCredentials.json

secret "sim-sensor-credentials" created

PS C:\Users\vladi\Downloads> kubectl create -f https://raw.githubusercontent.com/VladimirAkopyan/IoTSimulator/master/Kubernetes.yaml

configmap "sim-sensor-settings" created

service "simulated-sensors-service" created

statefulset "simulated-sensors" created

What are StatefulSets?

In a normal ReplicaSet a pod is temporary, holds no persistent data, has a random name, and if it dies, it will be replaced by a different pod with a different name.

StatefulSets were designed to manage statefull applications that save data on disk. In a StatefulSet pods are numbered 0, 1, 2, 3 … and if one dies, it will be recreated with the same name. They can also be matched with Persistent Volumes to save data to disk, but in this example we don’t actually need to use disk.

Step By Step

Each application container is simulating one device. Each pod is numbered and we use that determine which device this pod should simulate. We provide all pods with a list of all credentials — say a hundred of them. The pod will find it’s appropriate credentials in that list and use them to connect to the IoT system and do it’s job.

Below is a definition for a StatefulSet that provides name of the pod to the application as an environmental variable. If you want more pods, you just change the number of replicas in the yaml file.

As long as your list of credentials is large enough, and your cluster has enough capacity, there is no limit to how many completely independent devices you can simulate by just changing one number at any time.

apiVersion: v1

kind: Service

metadata:

name: simulated-sensors-service

labels:

app: simulated-sensors-service

spec:

clusterIP: None

selector:

app: simulated-sensors-service

---

apiVersion: apps/v1

kind: StatefulSet

metadata:

name: simulated-sensors

spec:

replicas: 5

serviceName: "simulated-sensors-service"

selector:

matchLabels:

app: sim-sensor

template:

metadata:

labels:

app: sim-sensor

spec:

containers:

- name: test-container

image: clumsypilot/iotsimulator

env:

- name: POD_NAME

valueFrom:

fieldRef:

fieldPath: metadata.name

In fact we literally use the podNames as DeviceID for IoThub — setup separately

Managing Settings and Credentials

Sometimes we want to change settings, for example simulate a high-load situation, by cranking up the data rate on each device. We want to be able to change settings without touching the source. Also we don’t want to store auth credentials in the source, so let’s explain how we do that properly in Kubernetes.

We are using a .Net Core application to simulate devices but i tried to keep it language agnostic as much as possible.

Put settings in Config Maps

The application will read settings from Environmental Variables, but the best way to manage them is through a ConfigMap.

apiVersion: v1

kind: ConfigMap

metadata:

name: sim-sensor-settings

namespace: default

data:

APPINSIGHTS_INSTRUMENTATIONKEY: ####

ReadingsDelay: "30"

ImageDelay: "3600"

Under the data section, add whatever settings you wish to manage. To create this config map for the cluster:

kubectl create -f ConfigMap.yaml

Also make changes to the StatefulSet definition as follows:

apiVersion: v1

kind: Service

metadata:

name: simulated-sensors-service

labels:

app: simulated-sensors-service

spec:

clusterIP: None

selector:

app: simulated-sensors-service

---

apiVersion: apps/v1

kind: StatefulSet

metadata:

name: simulated-sensors

spec:

replicas: 5

serviceName: "simulated-sensors-service"

selector:

matchLabels:

app: sim-sensor

template:

metadata:

labels:

app: sim-sensor

spec:

containers:

- name: test-container

image: clumsypilot/iotsimulator

envFrom:

- configMapRef:

name: sim-sensor-settings

env:

- name: POD_NAME

valueFrom:

fieldRef:

fieldPath: metadata.name

Verify that the settings are being attached to the pod correctly:

Provide Credentials through Secrets

Secrets are better at storing sensitive data, and as we have a large amount of it, it’s best to provide it as a file, IoTHubCredentials.json

{

"Credentials": {

"simulated-sensors-0": "************",

"simulated-sensors-1": "************",

"simulated-sensors-2": "************",

"simulated-sensors-3": "************",

"simulated-sensors-4": "************"

}

}

Just like CongfigMaps, secrets are key-value pairs. In the config map we specified each key-value pair individually.In this case the filename is the key, and it’s contents are the “value”. You can have several files in one secret.

kubectl create secret generic sim-sensor-credentials --from-file=./IoTHubCredentials.json

In this case we actually want to provide this information as a file, so we will do that by mounting the secret as a volume. When you do that, each key stored in the secret is represented as a file.

Paths: linux path starting with s slash, like /secrets/file.json is absolute, whereas secrets/file.json is relative to current application directory.

apiVersion: v1

kind: Service

metadata:

name: simulated-sensors-service

labels:

app: simulated-sensors-service

spec:

clusterIP: None

selector:

app: simulated-sensors-service

---

apiVersion: apps/v1

kind: StatefulSet

metadata:

name: simulated-sensors

spec:

replicas: 5

serviceName: "simulated-sensors-service"

selector:

matchLabels:

app: sim-sensor

template:

metadata:

labels:

app: sim-sensor

spec:

volumes:

- name: secrets

secret:

secretName: sim-sensor-credentials

containers:

- name: test-container

image: clumsypilot/iotsimulator

envFrom:

- configMapRef:

name: sim-sensor-settings

volumeMounts:

- name: secrets

mountPath: /secrets

readOnly: true

env:

- name: POD_NAME

valueFrom:

fieldRef:

fieldPath: metadata.name

Now you can read the file in /secrets/IoTHubCredentials.json

Read this data with .Net Core

The easiest way to read this information in .Net is using nuget packages:

Microsoft.Extensions.Configuration

Microsoft.Extensions.Configuration.Abstractions

Microsoft.Extensions.Configuration.EnvironmentVariables

Microsoft.Extensions.Configuration.Json

They help manage configuration of the application — ConfigutationBuilder creates a dictionary out of data found in config files and environmental variables. Each subsequent .Add will add data to the dictionary, overwriting any values that came in the file before it. We are supplying a file appsettings.json to give the application some default and enable testing and building the application on the developer machine without mucking about with environmental variables.

IConfiguration config = new ConfigurationBuilder()

.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true) /

.AddJsonFile("/secrets/ IoTHubCredentials .json", optional: true)

.AddEnvironmentVariables()

.Build();

The secret and environmental variables only come in effect when applicaiton is run in a kubernetes cluster.

Console.WriteLine($"Pod Name is: {config["POD_NAME"]}");

Console.WriteLine($"Application Insights Key: {config["APPINSIGHTS_INSTRUMENTATIONKEY"]}");

Console.WriteLine($"Image delay is: {config["ImageDelay"]}");

Console.WriteLine($"Readings delay is: {config["ReadingsDelay"]}"); string devicekeyPath = $"IoTHubCreds:{config["POD_NAME"]}";

Console.WriteLine($"IoT Credentials are: {config[devicekeyPath]}");

Debugging

I’ve built this using alpine container, they are small but don;t contain normal bash. You can exec into them using:

kubectl exec -it simulated-sensors-4 /bin/sh

ConfigMaps are easy to update using kubectl edit configmaps/sim-sensor-setting but secrets are a bit harder — we usually create them from file.

kubectl create secret generic sim-sensor -credentials --from-file=./ IoTHubCredentials .json --dry-run -o yaml | kubectl apply -f -

Be ware that updating Secrets and ConfigMaps will update the values visible to the application residing in container, but won’t retart the container.

A typical application will read these values once at startup, and then never check if they’ve changed.

Generally a good technique is to treat these as immutable objects. Create a ConfigMapv1.1 and update your deployment or statefullset to the new value. In that case the normal kubectl safety mechanisms have your back.