Over the course of the last few weeks, I've been putting some time to learning data aggregation and display, using various applications and what you see above is the end result for my 3D printer farm. I'm finally at a stage with it where I think it's displaying relevant information so here's my tutorial on how to set it up.

Writers Note:

The "tutorial" assumes you have all the following applications installed and ready to setup with a modicum of knowledge about them:

1. Node-Red

2. InfluxDB

3. Grafana

4. OctoPrint-MQTT Plugin - Available to install from the Plugin Store.

5. OctoPrint-Filament Manager Plugin - Available to install from the Plugin Store.

6. A running MQTT Server - Mine is local only with no username and password so if you have a different setup you'll have to use your brain to connect octoprint up to it as I won't detail it here.

6. A little knowledge about all those things mentioned above. Yes I will produce steps to follow but I won't explain what a MQTT broker is for example.

I will however detail the setup of the MQTT plugin, create the required flow's in node-red and how to clean that information to pass to influxDB, then finally displaying all that in Grafana. To get these programs installed you'll have to use the available tutorials on the internet. This is just to simplify the tutorial and cut out any requirements to mention systems/setups a user may have.

Please also note I'm still learning this myself, feel free to contact me if you have better ways of acheiving this as I'll admit the setup is rather convoluted.

OctoPrint-MQTT

The MQTT Plugin can be installed from the Octoprint plugin manager, just click the "Get More..." button on the plugin page and search for MQTT. Once installed and you've restarted your instance you will see a new menu option called "MQTT"

Click on the settings as there are a few we need to change from the default configuration.

The first screen you will see if for the Broker information, you can enter your MQTT host information here. So the host will be the MQTT server's IP address and the Port will be the standard port more than likely.

Everything else on this page I left as default, as I have no username/password or SSL for local stuff. Next we need to navigate to the Topics section and set those up.

Under the General header you will see the section depicted above. This base topic can pretty much be anything you want. Any topic within my home currently starts with "ocelot" and then I've just split that by printer with the "cr10" part. Feel free to go crazy with this, but remember to replace whatever you chose for the rest of the tutorial.

The only other setting I use within this was in the Progress message section below General. Scroll down and make sure you have "Activate progress message" and "Include extended information" checked. Click save and you have completed the initial setup of OctoPrint-MQTT, next we need to go over to setup Node-Red with a flow to capture all this information. Don't forget to restart Octoprint after changing the MQTT settings.

Pre-Requisite

Before continuing, and if you haven't already, it would be a good time to set up the influxDB database ready to capture all this data we're about to be sending it. I chose influxDB as it's a relatively simple time based database which will automatically tag every bit of data with a timestamp. There was also the massive fact that I'm lazy and already had a docker spun up on this due to collecting my UnRaid server's metrics. I set up a new database for each printer just to keep things segregated better, that is totally a personal preference and you could indeed just have 1 database for them all. It will require changing the javascript functions later in the script if you'd like to go about this, it is, however, out of the scope of this tutorial.

The following set of commands will create a database for you with a retention policy of 365 days (1 year).

influxdb - will log into your influx database session

CREATE DATABASE [PRINTER NAME] with DURATION 365d - Will create the required database. Replace [PRINTER NAME] inc the brackets with what you want your database to be called. I used cr10 for mine so my command was:

CREATE DATABASE cr10 with DURATION 365d

Node-Red and InfluxDB

Before continuing you will need the following pallets added to node-red:

1. MQTT-Node - Should be default install, but required to connect to our MQTT instance.

2. node-red-contrib-influxdb - So we can push data forward to the InfluxDB.

3. The last three nodes we will be using are the JSON node, the Function node and the Change node. These three should already be included in your Node-Red installation.

On the Node-Red side bar you will see a list of choices for nodes, we want the "input" section and we want to select the "mqtt" node as you can see to the left.

Grab the node and drag it onto a fresh Flow, and double click it to bring up the node's properties tab. Select the drop down tab next to Server and click "Add new mqtt-broker" if you don't already have this setup.

Give the MQTT server a name, anything you want. As you can see I was very original in my choice and called it "MQTT Server". I've only got 1 of these running in my house so no need to make up fancy names.

Fill in the "Server" details, so your MQTT servers IP address and Port. If you need to setup credentials you can do in the Security tab. I didn't change anything in the Messages tab.

Now we just need to select the server in the MQTT properties tab and setup the Topic for it. This is where it get's a little setup specific so bare with me.

So for this first node we will grab the hot end temperature. Now your's won't look like mine (mines bigger), just be sure to have remembered what you setup the start of the topic with in Octoprints MQTT settings. Mine was "ocelot/cr10". Then follow it with "/temperature/tool0". My printer only has 1 tool and a bed but the plugin will follow the normal octoprint designations so tool0,tool1 and so on.

Type all that in to the Topic section, and if you want then give the node a name. This is great for actually remembering "what node does what" as apposed to how I'd initially set it up, which was blank.

Note: "Don't do that".

If you wanna go a head and be awesome you can setup 2 more nodes. Each node can use the same MQTT server so you only have to set that up once, just select it with the new MQTT node you dropped into your flow. The following code can be added for "Topic" after what you entered in your MQTT plugins "Base Topic" setup.

temperature/bed - this will grab your bed temperature

progress/printing - this will grab a very good overview of your print status. Even more so because we turned on the extra parameters in the plugin settings. You can hook into all this seperately but with my lack of javascript knowledge, for working with variables/objects, I left it as simple as possible (for me anyway as this is probably spinning someones head right now).

So if you've copied me, you should now have 3 MQTT nodes each for Bed Temp, Hotend Temp and Print Progress. The next step is converting the MQTT output into something our Influx Database can understand, and for that we only need to use the "json" node. This resides in the "function" section and you can drop this straight into your Flow and connect it up to the hotend and bed temperature's we setup previously. You don't need to do anything to this node, it's good to go!

Next we need to drop in the "influxDB" node and connect up the json nodes to that.

The influx node works similar to the MQTT node, we must specify a server in the dropdown list. If you want, you can take the time now to create a specific server connection for each printer. I collect a lot of metrics so you'll see more to the left than just printers, but after you have created the server connection for your requirements we can select it and then fill in the "Measurement" section. Call this something like hotendTemp and bedTemp respectively, as you will need to differentiate them from the other metrics added to this database when we come to choose them in Grafana. When you've got two of these nodes setup, you can link them to the output side of the two JSON nodes we setup. This will complete the information feeding for our printers temperatures and look a little like the below image (hopefully).

For the "Print Progress" MQTT data we will need to drop in a Javascript function to convert it/error check it and also Drop some un-needed data before passing it to influx.

Grab a "function" and "change" node from the Function section and put them inbetween the Print Progress JSON node and the InfluxDB node you setup to receive this data. This should look like the below image.

My function script is below, feel free to modify that any which way you want, it is very basic as it just need's to sort out the data. I imagine this could be taken a lot further for other needs but for the dashboard they ain't needed maaaan. It's also got duplicate if statements that can easily be merged I'm just... as you know... Lazy!

Just make sure to change the printer name.

//Get Print Status if (msg.payload.printer_data.progress.printTimeLeft === null){ msg.payload.fileLength = 0; }else{ msg.payload.printProgress = msg.payload.printer_data.progress.completion; } if (msg.payload.printer_data.progress.printTimeLeft === null){ msg.payload.fileLength = 0; }else{ msg.payload.printTime = msg.payload.printer_data.progress.printTime; } if (msg.payload.printer_data.progress.printTimeLeft === null){ msg.payload.printTimeLeft = 0; }else{ msg.payload.printTimeLeft = msg.payload.printer_data.progress.printTimeLeft; } //Set print status msg.payload.printStatus = msg.payload.printer_data.state.text; //Get File Info if (msg.payload.printer_data.progress.printTimeLeft === null){ msg.payload.fileLength = 0; }else{ msg.payload.fileSize = msg.payload.printer_data.job.file.size; } if (msg.payload.printer_data.progress.printTimeLeft === null){ msg.payload.fileLength = 0; }else{ msg.payload.fileLength = msg.payload.printer_data.job.filament.tool0.length; } msg.payload.printer = "[ENTER PRINTER NAME HERE]"; return msg;

For the change node, all we need to do here is drop the msg.payload.printer_data which if you go to the node's properties can be done with the setup as displayed below.

That completes the sections for data aggregation, we should now have an influx database receiving MQTT data from our octoprint instance. Make sure to deploy the Node-Red set up, and all should be well, you can double check the debug screen to make sure no errors appear from the influxDB node, and your MQTT node will show "not connected" if it's had any issues with connection.

Repeat these steps for as many printers as you have, for now, we can move onto actually displaying this data into Grafana.

Grafana

Woohoo onto the stuff that actually produces some pretty... pretty results! On my Grafana dashboard I monitor 3 printers (was 5, but ya know... things die!) so a lot of it is catered around that. Basically, I wanted 1 screen for all and you may have different wishes. Grafana is quite robust and you should be able to present the data anyway you wish. I will explain how I did what you see in the main screenshot, then we can at least have a starting point for you to hopefully take my idea further than me!

Basic Grafana Set up

In the grafana configuration go to "Data Sources" and setup a new source for your printers database. Standard procedure as we have to Name the source, Give it a URL so http(s)://[SERVER IP]:[INFLUX PORT], and then finally tell it what database you are connecting to. Everything else I left at default but heck, you might use passwords and all that jazz.

Create a new Dashboard, I titled mine oh so descriptively "3D Printers" and left it at that. After that, we can add a Query and begin.

History

This uses the Graph visualisation.

History/Timeline Graph

The above graph I created with the intention of being able to see a history of my printers usage. A timeline which I can easily see the activity of my printers, if they were all on at once and how long a print took. Not anything specific but is very cool nonetheless. The image below displays the metrics you will need to select and the aggregator choices you'll need to make for the data to be displayed correctly.

History Graph Metrics

If you've got multiple printers your pulling data in from like me, use the "Queries to" to select "Mixed" which will allow you to select different databases to take the query from. Something to be aware of as well, if you look at the CR10s line just before the 5/11 in the Graph image above you will see breaks. These are a small caveat in the way I've done the Javascript, they are "null" values reported by the MQTT script, and I haven't investigated much further yet as to why it publishes them. You can remove these completely if you wish by changing the "time(30m)" value in the "GROUP BY" section of the query. "time(1h)" completely removed them for me but I didn't like how it displays quick prints so I changed left it at 30 minutes. You will also want to make sure to set up the Axis for percentage, as in the picture below.

AxisVisulisation Settings

Currently Printing

This uses the Table visualisation.

Currently Printing

This table uses a little more Grafana wizardry than the above but it will basically display anything that is currently active on the printers. In my Javascript way above, I added the printers name so I could use it in this table. The metrics are below but also remember to set the "Relative Time" option at the bottom of the Query screen, this will make sure we only grab the last set of data available. Mine is set to 1h as it fit my requirements.

Currently Printing Metrics

These settings will make sure to only pull the last entry available within the timeframe specified in the relative time setting. Make sure to set your FORMAT AS option as a table, and take a note of the field names, as we will make them "prettier" now.

To do this, go into the VisualiSation section and add a rule to the Column Styles section. If you click in the "Apply to columns named" box, you should be given suggestions of what you can use, if not then it refers to the metric name you used in the SELECT command above. If you have an operator ie. "last()" then that is appended to the name. I've included an example screenshot of my fileVolume() style. You will need to do one for each metric you have pulled, or be a heathen and leave them default!

Metric Style Settings

Printer Statistics

Again with the Table visualisation

Printer Statistics Table

This is an overview of the last year of use with my printers, and some extra random data I thought would be nice to throw in there considering I have that information available. Similar table to above, just make sure to FORMAT AS a Table, set up your metric styles as above and I also set my Relative time to 365d. As always, a screenie of my query.

Printer Statistics Metric

Looks a little more complicated as a query but it's really not, we're just leveraging some of the different selectors from influxDB. They are somewhat self-explanatory but I'll explain what the different ones are doing here.

last() - This will just select the last value for the field in the database, we used it on the previous tables as we wanted to see the last value at that specific time. Here we have no time field so we are just displaying the last value.

spread() - This is an aggregation field and will count all the available fields values in a given timeframe. So give me the accumulated time my printer has been running for.

distinct() - Will select values once, so if a filename has been sent 100 times over the course of 20 mins, we will only see that once.

count() - This will count the values within a field.

math() - I love this selector as it allows you to do any calculation with a field, and it's exactly how I got the total print/running costs of my printer. The printTime() field is in seconds, so first we need to divide that value by 86400 so we get how many day's our printer has been active for. I also know that my energy supplier's kW/h price is 0.1508 and I know after monitoring my printers use somewhere around 2.4 kW/h. I can then multiply that daily value by my price and to get an idea of what my printers cost in power. I gather everything else there is self-explanatory.



Update (06/06/2019):

I've just noticed the spread() selector I used above to calculate the total print time is only taking the difference between the lowest and highest value in the time range you specified. I've since changed this to say "Longest Print Time" for now until I have more time to delve deeper into Grafana.

Hotend and Bed Temperature Dials

This uses the Gauge visualisation

Temperature Guages

These are just used for a quick status on my printers temperatures. No real need for it bar just plain old nosiness. You have the option to display "target" and "actual" values from the metrics that are collected but I've only used the actual. I did have a graph on before that graphed all of thise and compared target vs acual, but I haven't seen the need for it after using it for a week. I'd generally be checking heating errors in Octoprint as it went but the data's there if you'd rather display these in graph format.

Metrics for Temperature Guage

Nothing too complex here, we're just selecting the last value from the actual metric on hotendTemp. I also fill in with the previous here as I originally had issues with it displaying a null value before I error checked with the if statements on my function. That is due to influxDB setting the format on a metric it's collected, the first in the sequence will set to a boolean, string or number and it then expects that for the rest of the data that is sent. It's probably not needed anymore but that's why it's there.

Visualisation settings for Guage

To get the little orange bar, and tell the gauge that it's actually a temperature that we wanted to display we need to head over the the visualisation settings again. I have set mine up with a max value of my known max hotend temperature, and then Thresholds to change the bar colour when getting around that. I often print at 200°C so that's the threshold for that, and I generally don't print over 230°C. I had plans to use Grafana's notification feature more here, but I will detail that in another tutorial when I've figured it out myself.

Camera's

The easiest of them all! Just use the Text visualisation and put it into HTML mode. You can use the script below to display your camera feed, just change the [CAMERA URL] and you will be golden.

<img id="webcam_image" data-bind="css: { flipH: settings.webcam_flipH(), flipV: settings.webcam_flipV() }, event: { load: onWebcamLoaded, error: onWebcamErrored }, visible: !webcamError()" src="[CAMERA URL]">

Wonderous shot of my spools and shelves

Spool Tracker Information

This one will require the use of an extra plugin from Octoprints directory. The links above for the github, but it's the Filament Manager plugin if you don't want to scroll. It is in the plugin directory and handily allows you to connect your spool database back to a PostgreSQL database. This database thankfully is also a connection point within Grafana. This wasn't planned when I was initially figuring out a way to do this, it's just one of those... oh! moments.

Spool Tracking Table

The database is a table again and it works pretty well as a quick overview of your spool metrics. Especially if your rather pedantic with your OCD and like to number your spools like me. The only thing I haven't figured out, because the filament manager doesn't store it, is the remaining quantity. I'd rather display that then the used quantity, you'd think it would be a simple case of subtracting one column from another but nooo... they have to have some complicated database way of doing it, OR! and a completely acceptable answer is that i'm being a dodo.

Below is a picture of the query I used for PostgreSQL.

Metrics for Spool Tracking

Conclusion

Well, that's it and hopefully, I haven't forgotten anything. I will do another write up once I figure out, or find needs for, extra metrics but for now that will do. I've quite enjoyed having this dashboard handy for a quick view into my printers world and I do have future plans for a tablet in the print room so it will be a nice place to display that data. I had written a javascript application to display this data and I was going to advance it further to allow for stopping and starting prints/other cool things with octoprint but for now I'm happy with this.

Update (31/05/2019): If anyone makes a Grafana dashboard with the above information I'd love to see your results so please use the comments section below to do so.

Update (06/06/2019) I've messed around with some new tracking devices and updated my dashboard a little. Here is a screenshot of the new one.