Is it hot in here or is it just me?

The IoL citizens need to know the weather, so let’s build a weather station!

In this article, I am going to cover several technologies that are common to the Internet of Things such as messaging, sensors, embedded code and data analysis. There are tons of similar examples on the web, but I’m using this to walk through the full IoT concepts with a familiar sensor.

Goal

Build a standalone system to sense the light, temperature, humidity and compute the heat index

Display on an attached LCD screen

Send the data to a central system to store and analyze the data

Overview

Microcontroller

Cactus Micro Rev2 – Arduino

Arduino compatible w/ ESP8266 WiFi module

Resources: http://wiki.aprbrother.com/wiki/Cactus_Micro_Rev2

This little device is what got this project started. In the quest for a true “Internet of Things” solution, I found an Arduino with an ESP8266 WiFi module built-in. This is a huge time and space saver, costs only a few dollars (~$11) and is perfect for small sensor projects. The Cactus Micro is made by April Brothers. I easily picked one up on eBay and another one at Tindie.

It took a while for me to get started with this, because there were two product versions, Rev1 and Rev2, which require slightly different code (lesson learned). It also requires the very important step of programming the attached ESP8266 independently before programming the normal Arduino sketch. I also thought I fried it because I chose the wrong board type while uploading the code to the device, which confused the boot loader. The Cactus Micro is basically a clone of the “LilyPad Arduino USB” Moral of the story? Follow these instructions very carefully!

In preparation for the project code, follow these steps to configure the Cactus Micro.

Step 1 – Make the Arduino an ESP8266 Programmer

LilyPad Arduino USB as the board type. Using Arduino IDE 1.0.6 , upload the following sketch and program it to the connected Cactus Micro. Useas the board type.

This puts the Arduino portion of the Cactus Micro in a special mode so that we can write code to the attached ESP8266 module. I had to use an older version of the Arduino IDE for compatibility reasons.

Tip

If you selected the wrong board type, (i.e Arduino Uno), then the Cactus Micro will likely not boot up properly, and you will not see the serial interface available to your computer. To solve this, press the reset button twice as you hit the sketch upload button. It will place the Arduino into a recovery mode, the IDE will automatically detect it, and it will upload the sketch. I lost days to this problem, and actually purchased another device because I thought I fried it! I had accidentally selected LilyPad Arduino instead of LilyPad Arduino USB.

Step 2 – Program the ESP8266

Program the ESP8266 with the espduino firmware using esptool.

Clone the espduino source code

git clone https://github.com/tuanpmt/espduino cd espduino

Download the esptool. I just copied the contents into the espduino folder so I can run the following line on my Macbook Pro. Note that “tty.usbmodem1421” is the serial port of the Cactus Micro device.

./esptool.py -p /dev/tty.usbmodem1421 write_flash 0x00000 esp8266/release/0x00000.bin 0x40000 esp8266/release/0x40000.bin

Step 3 – Program the Arduino with a regular sketch

To use the WiFi functionality, add the following code to your Arduino Sketches. Be sure that the espduino library is also installed into your Arduino IDE.

This code will be included in the final solution, but I’ve isolated it so it can be used with other projects.

// ESP8266 WiFi #include <espduino.h> #define PIN_ENABLE_ESP 13 #define SSID "yourSSID" #define PASS "yourPassword" void wifiCb(void* response) { uint32_t status; RESPONSE res(response); if(res.getArgc() == 1) { res.popArgs((uint8_t*)&status, 4); if(status == STATION_GOT_IP) { Serial.println("WIFI CONNECTED"); wifiConnected = true; } else { Serial.println("WIFI OFFLINE"); wifiConnected = false; } } } void setup() { // Write to ESP8266 Serial1.begin(19200); // Write to Console Serial.begin(19200); // Enable ESP8266 esp.enable(); delay(500); esp.reset(); delay(500); while(!esp.ready()); // Setup WiFi esp.wifiCb.attach(&wifiCb); esp.wifiConnect(SSID, PASS); } void loop() { esp.process(); if(wifiConnected) { // Do Something } }

Display

LCD

To see the current status and sensor data, I’ve connected a 16×2 LCD screen. These are really common, so there is a ton of documentation and programming libraries to quickly include it into projects.

Sensors

Temperature & Humidity – DHT11

The DHT11 sensor is a super easy thermometer and humidity sensor that provides a digital reading. An easy to use library can get this working in no time.

Light Sensor – Photoresistor

The photoresistor is variable resistor controlled by light and will provide an analog reading from 0 – 1023.

Circuit

I basically used this Fritzing diagram as the basis to connect it all together, the pin numbers were adjusted in the final code to comply with the GPIO options of the Cactus Micro so don’t take the pin numbers literally. I also added an LED for a status light.









Messaging – IoT communication

In order to send Internet of Things data, I researched the several types of messaging protocols and solutions out there. There are a number of standards based protocols, such as MQTT, AMQP, øMQ, etc, which focus on distributed messaging in a Publish/Subscription model. There are also web-oriented methods such as REST and WebSockets. Finally, there are platforms like PubNub, Pusher, HiveMQ and OctoBlu that provide a full service offering that may use one or many of these protocols or simulate their concepts with proprietary SDKs.

I’ve used PubNub to build much of my IoL city, but I want to learn more about how the IoT truly works, so I’m purposely looking to test a standard messaging protocol for this project. After getting a feel for the options out there, its seems like MQTT is easy to work with, and is common for IoT projects.

MQTT

Message Queue Telemetry Transport (MQTT) is one of the most popular messaging protocols because it’s simple to use and highly scalable. It is based on a Publish/Subscription model, where a central broker binds the client connections, and clients will either publish and/or subscribe to data topics. There are numerous libraries, documents and open-source brokers to help you get going.

MQTT – Publish / Subscription model

The basic communication occurs by connecting to an MQTT broker, then either publishing or subscribing to a messaging topic. The topics are structured in a way that makes it easy to subscribe to an entire message stream using a hierarchical topic namespace. By using either the # or + wildcards, the data can be filtered as needed. More info from the Mosquito website here.

Examples

/sensors/london/weather/temp

/sensors/birmingham/weather/humidity

/sensors/+/weather/temp

/sensors/#

MQTT – Broker

The broker is the server that will accept messages, filter data and resend them.

Mosca and Mosquito are two popular open-source brokers that are also easy to use.

HiveMQ is a an enterprise platform that offers enhanced SLA and scalability options. They also have really good documentation about the general concepts of MQTT. I encourage you to read their blog which explains the protocol in detail.

I’ve selected Mosca, since it’s written with NodeJS and that’s my primary environment. It provides a standalone application or can be embedded into other NodeJS projects, which is fantastic. Here is a pretty good article to get Mosca up and running.

MQTT – Client

The client is any end device or system that sends and receives the actual data.

The client could be an Arduino, Raspberry Pi, web browser or server that might send sensor data, robotic commands or maybe it’s a system that stores the data into a database for future analysis.

For this project, I’m using the Arduino MQTT client library included with the espduino code found here. Another popular option is to use the pubsubclient library.

I will also use the built-in Node-RED MQTT nodes to easily subscribe and publish messages.



Data Storage

Now that I have data moving around, I would like to store it into a database. There are two major types of databases to chose from, SQL (relational) or NoSQL (non relational). SQL databases such as MS SQL, Oracle and MySQL typically require a defined schema. This is fine when you know what type of data to expect. But in the age of websites, and IoT sensors, the data tends to be much more dynamic. The alternative is a NoSQL database such as MongoDB or DynamoDB where the information can be stored as key:value pairs. This is great for web services which commonly use the JSON format.

MongoDB

This project will use MongoDB which can easily be installed into a Node-RED service or any server really.

Messaging Flow

This is an overview of how my weather station will communicate with my broader network, where the Mosca broker will relay the data to my Node-RED client. Node-RED will then store the data into a MongoDB database and also provide a simple graph using a Google Chart Node-RED node.





Data Analysis

With the messages stored into a database, its time to turn “data” into “information”. As a simple test, I installed a Google Chart node into Node-RED so I could generate some pretty pictures. Cool!





Programming

Arduino Code

Full code Gist

/** * This program sends weather data to an MQTT broker, and displays the messages onto the attached LCD screen * Hardware: Cactus Micro Rev2 - http://wiki.aprbrother.com/wiki/Cactus_Micro_Rev2 * Written by: Cory Guynn with some snippets from public examples 😉 * http://www.InternetOfLego.com */ // Global Variables unsigned long time; // used to limit publish frequency // LED Status #define LEDRED 10 // LCD Screen #include <LiquidCrystal.h> // initialize the library with the numbers of the interface pins LiquidCrystal lcd(8, 9, 4, 5, 6, 7); // ESP8266 WiFi #include <espduino.h> #define PIN_ENABLE_ESP 13 #define SSID "yourSSID" #define PASS "yourPW" // MQTT Messaging #include <mqtt.h> ESP esp(&Serial1, &Serial, PIN_ENABLE_ESP); MQTT mqtt(&esp); boolean wifiConnected = false; #define mqttBroker "yourbrokeraddress" // DHT11 Temperature & Humidity Sensor #include "DHT.h" #define DHTPIN 2 #define DHTTYPE DHT11 DHT dht(DHTPIN, DHTTYPE); // Photoresistor Ligh Detection int lightPin = A0; // LDR Sensor /******************* // Functions *******************/ void wifiCb(void* response) { uint32_t status; RESPONSE res(response); if(res.getArgc() == 1) { res.popArgs((uint8_t*)&status, 4); if(status == STATION_GOT_IP) { Serial.println("WIFI CONNECTED"); lcd.clear(); lcd.print("WiFi ONLINE"); lcd.setCursor(0, 1); lcd.print(SSID); mqtt.connect(mqttBroker, 1883, false); wifiConnected = true; //or mqtt.connect("host", 1883); /*without security ssl*/ } else { wifiConnected = false; mqtt.disconnect(); lcd.clear(); lcd.print("WiFi OFFLINE"); } } } void mqttConnected(void* response) { Serial.println("MQTT Connected"); //mqtt.subscribe("/topic/0"); //or mqtt.subscribe("topic"); /*with qos = 0*/ mqtt.publish("/sensors/iolcity/weather/news", "weatherstation is online"); } void mqttDisconnected(void* response) { Serial.println("MQTT Disconnected"); } void mqttData(void* response) { RESPONSE res(response); Serial.print("Received: topic="); String topic = res.popString(); Serial.println(topic); lcd.clear(); lcd.home(); lcd.print(topic); Serial.print("data="); String data = res.popString(); Serial.println(data); lcd.setCursor(1, 1); lcd.print(data); } void mqttPublished(void* response) { } void setup() { // initialize leds pinMode(LEDRED, OUTPUT); // set up the LCD's number of columns and rows: lcd.begin(16, 2); // Print a message to the LCD. lcd.print("Weather Station"); delay(2000); lcd.clear(); Serial1.begin(19200); Serial.begin(19200); esp.enable(); delay(500); esp.reset(); delay(500); while(!esp.ready()); Serial.println("Setup MQTT client"); if(!mqtt.begin("DVES_duino", "admin", "Isb_C4OGD4c3", 120, 1)) { Serial.println("Failed to setup mqtt"); while(1); } Serial.println("Setup MQTT lwt"); mqtt.lwt("/lwt", "offline", 0, 0); //or mqtt.lwt("/lwt", "offline"); /*setup mqtt events */ mqtt.connectedCb.attach(&mqttConnected); mqtt.disconnectedCb.attach(&mqttDisconnected); mqtt.publishedCb.attach(&mqttPublished); mqtt.dataCb.attach(&mqttData); /*setup wifi*/ Serial.println("Setup WiFi"); esp.wifiCb.attach(&wifiCb); esp.wifiConnect(SSID, PASS); Serial.println("System Online"); } void loop() { esp.process(); if(wifiConnected) { /******************* // Sensors *******************/ // publish sensor reading every 5 seconds if (millis() > (time + 5000)) { time = millis(); // blink LED to indicate sensor read digitalWrite(LEDRED, HIGH); // turn the LED on (HIGH is the voltage level) delay(500); // wait digitalWrite(LEDRED, LOW); // turn the LED off by making the voltage LOW // Read light sensor int l = analogRead (lightPin); l = map (l, 0, 1023, 0, 100); // scaled from 100 - 0 (lower is darker) // Reading temperature and humidity float h = dht.readHumidity(); // Read temperature as Celsius (the default) float t = dht.readTemperature(); // Read temperature as Fahrenheit (isFahrenheit = true) float f = dht.readTemperature(true); // Check if any reads failed and exit early (to try again). if (isnan(h) || isnan(t) || isnan(f)) { Serial.println("Failed to read from DHT sensor!"); return; } // Compute heat index in Fahrenheit (the default) float hif = dht.computeHeatIndex(f, h); // Compute heat index in Celsius (isFahreheit = false) float hic = dht.computeHeatIndex(t, h, false); /******************* // Print to LCD ********************/ lcd.clear(); lcd.home(); lcd.print("Humidity: "); lcd.print(int(h)); lcd.print(" % "); lcd.setCursor(0, 1); lcd.print("Temp: "); lcd.print(int(t)); lcd.print(" *C "); /******************* // Publish to MQTT ********************/ // Convert data to character array char tChar[10]; char hChar[10]; char hicChar[10]; char lChar[10]; dtostrf(t, 4, 2, tChar); dtostrf(h, 4, 2, hChar); dtostrf(hic, 4, 2, hicChar); dtostrf(l, 4, 2, lChar); // Publish data character array to MQTT topics mqtt.publish("/sensors/iolcity/weather/humidity", hChar); mqtt.publish("/sensors/iolcity/weather/temperature", tChar); mqtt.publish("/sensors/iolcity/weather/heatindex", hicChar); mqtt.publish("/sensors/iolcity/weather/light", lChar); // Convert data to JSON string String json = "{\"data\":{" "\"humidity\": \"" + String(h) + "\"," "\"temperature\": \"" + String(t) + "\"," "\"heatindex\": \"" + String(hic) + "\"," "\"light\": \"" + String(l) + "\"}" "}"; // Convert JSON string to character array char jsonChar[100]; json.toCharArray(jsonChar, json.length()+1); // Publish JSON character array to MQTT topic mqtt.publish("/sensors/iolcity/weather/json", jsonChar); /******************* // Print to Console ********************/ Serial.println(" "); Serial.println("Data"); Serial.println(json); } } }

Node-RED

Overview

This flow subscribes to all weather messages from the iolcity weather station and performs several tasks with the data.

It uses a “switch” node to direct the messages based on their specific topic.

The “json” topic messages are converted from a JSON string to a JSON object so that it can be properly manipulated. A timestamp and cleansing of the data structure is managed by a function object before it is committed to the mongodb database.

To quickly see the data, I’ve created an HTML node to respond to a GET request “/showweather”. It will query all the data from the mongodb collection. I use the Google Chrome app Postman to review this data. This also helps to ensure that the data has been stored/formatted properly.

Finally, Google Chart nodes are used to present the data as an Annotation Chart.

Flow

Copy and paste this text into your Node-RED import tool to use this flow.

[{"id":"9dcc379f.6233c8","type":"mqtt-broker","z":"22b6f4a1.dd490c","broker":"localhost","port":"1883","clientid":"","usetls":false,"verifyservercert":true,"compatmode":true,"keepalive":"15","cleansession":true,"willTopic":"","willQos":"0","willRetain":"false","willPayload":"","birthTopic":"","birthQos":"0","birthRetain":"false","birthPayload":""},{"id":"6ed1115c.912ef","type":"mongodb","hostname":"127.0.0.1","port":"27017","db":"test","name":""},{"id":"352330bd.cadcd","type":"mongodb out","z":"22b6f4a1.dd490c","mongodb":"6ed1115c.912ef","name":"mongodb save","collection":"weatherstation","payonly":true,"upsert":true,"multi":false,"operation":"insert","x":701,"y":377,"wires":[]},{"id":"767f711c.89809","type":"mqtt in","z":"22b6f4a1.dd490c","name":"/sensors/iolcity/weather/#","topic":"/sensors/iolcity/weather/#","broker":"9dcc379f.6233c8","x":148,"y":341,"wires":[["fef20750.010df8","f1caad2f.0e355"]]},{"id":"f1caad2f.0e355","type":"debug","z":"22b6f4a1.dd490c","name":"/sensors/iolcity/weather/#","active":false,"console":"false","complete":"payload","x":602,"y":339,"wires":[]},{"id":"323088d4.cdcf78","type":"mongodb in","z":"22b6f4a1.dd490c","mongodb":"6ed1115c.912ef","name":"weatherstation","collection":"weatherstation","operation":"find","x":306,"y":744,"wires":[["7574f0c.f8a8b1","62df7b8.f9d2084"]]},{"id":"d15eddda.2ea12","type":"mongodb in","z":"22b6f4a1.dd490c","mongodb":"6ed1115c.912ef","name":"weatherstation","collection":"weatherstation","operation":"find","x":339,"y":614,"wires":[["d13af397.2ec51"]]},{"id":"d13af397.2ec51","type":"http response","z":"22b6f4a1.dd490c","name":"weather response","x":673,"y":612,"wires":[]},{"id":"fc855614.037aa8","type":"http in","z":"22b6f4a1.dd490c","name":"[get] /showweather","url":"/showweather","method":"get","swaggerDoc":"","x":119,"y":615,"wires":[["d15eddda.2ea12"]]},{"id":"613f2ff4.9ec0d","type":"comment","z":"22b6f4a1.dd490c","name":"Weather Station Data in HTML/JSON View","info":"","x":185,"y":569,"wires":[]},{"id":"624cc3f9.9db33c","type":"chart request","z":"22b6f4a1.dd490c","charttype":"AnnotationChart","path":"/mqttchart","refresh":"","formatx":"","formaty":"","attribs":[{"name":"date","type":"date"},{"name":"temperature","type":"number"},{"name":"humidity","type":"number"},{"name":"heatindex","type":"number"}],"x":109,"y":740,"wires":[["323088d4.cdcf78"]]},{"id":"62df7b8.f9d2084","type":"debug","z":"22b6f4a1.dd490c","name":"chart debug","active":true,"console":"false","complete":"payload","x":692,"y":764,"wires":[]},{"id":"7574f0c.f8a8b1","type":"chart response","z":"22b6f4a1.dd490c","x":703,"y":722,"wires":[]},{"id":"fef20750.010df8","type":"switch","z":"22b6f4a1.dd490c","name":"","property":"topic","rules":[{"t":"eq","v":"/sensors/iolcity/weather/json"},{"t":"eq","v":"/sensors/iolcity/weather/temperature"},{"t":"eq","v":"/sensors/iolcity/weather/humidity"},{"t":"eq","v":"/sensors/iolcity/weather/heatindex"}],"checkall":"true","outputs":4,"x":141,"y":444,"wires":[["f93b30c1.06c4d","7d147084.82eb9"],["7f7d9349.80826c"],["3a479471.c5b86c"],["9795c56e.686a38"]]},{"id":"f93b30c1.06c4d","type":"debug","z":"22b6f4a1.dd490c","name":"/sensors/iolcity/weather/json","active":true,"console":"false","complete":"payload","x":624,"y":442,"wires":[]},{"id":"7f7d9349.80826c","type":"debug","z":"22b6f4a1.dd490c","name":"/sensors/iolcity/weather/temperature","active":false,"console":"false","complete":"payload","x":636,"y":478,"wires":[]},{"id":"3a479471.c5b86c","type":"debug","z":"22b6f4a1.dd490c","name":"/sensors/iolcity/weather/humidity","active":false,"console":"false","complete":"payload","x":625,"y":512,"wires":[]},{"id":"9795c56e.686a38","type":"debug","z":"22b6f4a1.dd490c","name":"/sensors/iolcity/weather/heatindex","active":false,"console":"false","complete":"payload","x":631,"y":547,"wires":[]},{"id":"1a29061a.e5d6fa","type":"function","z":"22b6f4a1.dd490c","name":"timestamp and format data","func":"msg.payload.data.date = new Date();

msg.payload = msg.payload.data;

return msg;","outputs":1,"noerr":0,"x":477,"y":395,"wires":[["352330bd.cadcd","546686fe.ab9978"]]},{"id":"7d147084.82eb9","type":"json","z":"22b6f4a1.dd490c","name":"","x":286,"y":395,"wires":[["1a29061a.e5d6fa"]]},{"id":"546686fe.ab9978","type":"debug","z":"22b6f4a1.dd490c","name":"json debug","active":true,"console":"false","complete":"payload","x":690,"y":409,"wires":[]},{"id":"d1b0dc7d.2e4f2","type":"comment","z":"22b6f4a1.dd490c","name":"Weather Station in Google Chart view - /mqttchart","info":"","x":214,"y":693,"wires":[]},{"id":"68c9dd4c.973624","type":"comment","z":"22b6f4a1.dd490c","name":"Weather Station MQTT Flow","info":"","x":141,"y":300,"wires":[]}]

Bonus: Real-time Graphs

ThingSpeak

Now I am going to send the streaming data to the ThingSpeak service via Node-RED to collect and visualize the data in real-time!

I signed up for a free ThingSpeak account because they have a simple to use platform for collecting sensor data. With the power of Matlab, this system can quickly create charts and visualizations to add context to the sensor data stream.

After creating an account, I then created a channel and named the fields.

Node-RED

This flow pulls in the JSON data from the MQTT stream. It is then parsed into a JSON object where I can extract the various parameters. The values are then set to the corresponding “field” name that ThingSpeak will use for the charts. Finally the the fields are added to the GET URL and sent to the HTTP Request.

And now for the weather…

View the real-time weather data from my public ThingSpeak channel:

Gallery