MQTT Part 4: Sniff meets Node-Red

So far we've used MQTT to publish sensor data from an Arduino, and subscribe to receive messages to display on a screen. Having written code to do each of those things, one of Sniff's strengths is that its pretty trivial to do both at the same time, just by running the script to do each at the same time. Here's the complete code to support both a screen and a sensor at the some time:

make i2c device 2

make display lcdi2c device 2

make displayFlush boolean

make spi device

make ethernet device D10

make networkMAC string "b6:ee:63:ed:95:cb"

make networkIP string "192.168.0.200"

make networkConnected boolean

make networkPort number

make networkPeer string

make mqtt device

make clientid string

make message string

make topic string

make retain boolean

when start

.set clientid to networkMAC

.set networkPeer to "192.168.0.108" #No DNS on Arduino!

.

.repeat until networkConnected

..tell mqtt to "connect"

.

.broadcast measureStuff

.

.set topic to "text"

.tell mqtt to "subscribe"

.set topic to "office/text"

.tell mqtt to "subscribe"

.forever

..tell mqtt to "loop"

..if not message = ""

...tell display to "show"

make thermometer dht11 device D2

make temperature number

make humidity number

when measureStuff

.forever

..tell thermometer to "read"

..

..set retain to yes

..set topic to "office/temperature"

..set message to [temperature]

..tell mqtt to "publish"

..

..set retain to yes

..set topic to "office/humidity"

..set message to [humidity]

..tell mqtt to "publish"

..

..wait 10 secs

You might notice that we're subscribed to both text and office/text. The intention here is that we might have several of these devices in different rooms - so we can publish a message to all of them at once, or to each individually.

With the physical/embedded side out of the way, we now need to think about what we want to do with that data we've collected. That means we probably have something running on a desktop machine. While we could easily write that in Sniff, we could also use Node Red.

Node Red is a javascript based system that is specifically designed for collecting messages, processing them and passing them on.  It works particularly well with MQTT, which is sort of to be expected given that they're developed by the same people.

Grab a copy of Node Red, and run through the first tutorial to get the basic idea...

time passes... OK - you're back! Great!

Now that you've got the general idea of Node Red make an MQTT input node, and tell it to subscribe to office/temperature on your server/broker. Connect the output to a debug node, and hit deploy.

When the Sniff code is running on the Arduino, you should see the temperature messages appearing in the debug window.

Next we can connect the MQTT input node to an MQTT output node which publishes the received temperature back to office/text, so that the measured temperature appears on the screen.

Now that we can fetch the data from the sensor, and push it back, we can add some processing the he middle. For example let's make the message a little friendlier. Just add a Function node between the two MQTT nodes.

We read the msg.payload, modify and and write it back before passing it on to the send node. In fact we can make any changes we like. For example we can indulge the British habit of reporting cold temperatures in C (it was -2 last night!), and warm temperatures in F (awesome holiday - 80 degrees all week!). There's little confusion as we simply never use the values between 20 and 50!

The next step is to handle temperature and humidity at the same time. Getting the data is easy, as we can just subscribe to office/+, but there's serious gotcha - this also matches office/text. This means when we publish the results, we'll trigger an input, which will trigger an output.... that would be bad. To get round that add a switch node, which only outputs its result if the msg.topic!="office/text".

We can just check the message topic to find out what kind of information we've received. Slightly less obvious is that when we receive an update to temperature, we'll need to combine it with the previously received humidity. Fortunalty this is pretty easy, as each node in the flow has a "context". Context is a variable that we can store values in, and they'll be there the next time the function is executed. When we receive a value we store it in the context, and then generate a new payload message from the stored values.

One final thing to deal with is the that our simple Sniff program only handles one line of text at a time, while out Javascript function bundles the Humidity and into a single packet. While there are lots of ways we could handle this, the easiest way is to have a split node, which separates the two lines into two separate messages. This produces our final flow:

And one the Arduino we see:

While its true, we could have coded all of this in Sniff on the Arduino, without any networking involved, and it would have been much simpler its worth remembering that these kinds of sensor/display devices might be deployed all round your house, school or factory. When a few hundred are boxed up, and bolted to the wall, and you need a ladder to even reach some of them, going out and reprogramming them because you want to make a minor change isn't really practical. It makes a lot of sense to keep those boxes as dumb as possible, while the "business logic" - the stuff you need to change, and develop is all in one place where you can get to it.

In addition by pushing the data out to Node Red we get an easy connection to all sorts of other data streams, like email, twitter, and websites.

MQTT Part 3: Fancy publishing

So far with MQTT we've published and subscribed to basic messages from both desktop and embedded Sniff.

make spi device

make ethernet device D10

make networkMAC string "b6:ee:63:ed:95:cb"

make networkIP string "192.168.0.200"

make networkConnected boolean

make networkPort number

make networkPeer string

make mqtt device

make clientid string

make message string

make topic string

make retain boolean

when start

.set clientid to networkMAC

.set networkPeer to "192.168.0.108" #No DNS on Arduino!

.

.repeat until networkConnected

..tell mqtt to "connect"

.

.broadcast measureStuff

.

.forever

..tell mqtt to "loop"

That's the basic framework we need to create a connection to the broker, and keep everything ticking over. However this time around we're going to publish the temperature and humidity from a dht11 temperature and humidity sensor out to the world.

make retain boolean

make thermometer dht11 device D2

make temperature number

make humidity number

when measureStuff

.forever

..tell thermometer to "read"

..

..set retain to yes

..set topic to "office/temperature"

..set message to [temperature]

..tell mqtt to "publish" 

..

..set retain to yes

..set topic to "office/humidity"

..set message to [humidity]

..tell mqtt to "publish" 

..

..wait 60 secs

We've already included a broadcast in the start script to kick this script off, so all we need to do is measure the temperature by telling the thermometer to "read", and publish that data. Then we wait a minute and do it again.

Topic Hierarchy

However there are a couple of new tricks here that are worth looking at. Firstly look at the names of my topics - they've got a / in them, which allows us to group topics together. In this case I'm reporting  information about my office, so I'm grouping then under that name.

When someone subscribes they can subscribe to specific topics, but they can also subscribe to groups of topics, so "office/#" means every topic that begins "office/". That would include office/temperature and office/humidity but it would also include "office/lights/1/brightness". You can only use # as the last character of a subscription as it basically means "everything in this category".

On the other hand, maybe I'm interested in the temperature of lots if different rooms. In which case I can subscribe to "+/temperature", or if I want the brightness of all the lights in the house "+/lights/+/brightness". Each + represents exactly one level in the hierarchy, but you can use it anywhere in the path.

Retained Messages

The other thing to consider is that we're only publishing a measurement once per minute. That's fairly sensible, as air temperature doesn't change that quickly, so no need to flood the network. However what if a client connects to the broker and subscribes to  "+/temperature" to work out the average temperature of the house. It would have to wait up to a minute before it got messages from all the sensors, which would be frustrating. We could go further and have sensors push out values only when they change, which would leave our averaging program waiting for as long as it took before it gets all the data it needs (and it shouldn't need to know how many sensors there are either, so it would never know its got them all).

To get round this problem we've created a variable called "retained", and set that to yes before each call to publish. When a message is published in retained mode the broker sends it out to subscribers as normal, but keeps a copy. If any new subscribers come along, then they immediately get the last retained message on that topic, so in this case subscribing to "+/temperature" would immediately generate a set of messages with the last known temperature of every sensor. 

Last Will and Testament

Now that messages can be retained, we've created a new problem: My office publishes a message in the middle of the night, and tells the broker that its 16 degrees. It then gets disconnected from the network because a ferret unplugs the network cable (yes that really did happen to me!). The next morning its warmer, but when the heating subscribes to check the temperature, the last good value sent says its cold, so the heating turns on full blast until I reset the sensor.

Fortunately MQTT has a mechanism to check that everything is still running smoothly. Even when its not publishing, the client and broker occasionally send messages to each other just to make sure that they're still alive. If the broker spots that a client isn't working any more it can clean up a little bit.

However as the client is already disconnected the only way it can control what happens during this cleanup is by leaving a will - instructions for a message that should be sent upon its death. To make this happen our connection code might looks like:

.set retain to yes

.set topic to "office/temperature"

.set message to ""

.

.repeat until networkConnected

..tell mqtt to "connect"

When we connect this message (an empty string) gets sent to the broker, but the broker doesn't forward it to clients. Rather it hangs onto it, and if at some point in the future it looses connection to the client unexpectedly then it publishes the will-message on the dead clients behalf. Here I've set the message to "" to erase any previous temperature recorded, so it won't be used hours later, when its invalid.

If we had a client which published messages to control some kind of actuator, then the will-message could be used to reset the actuator to some kind of safe/recoverable state.

Note that will messages work with retain so if the will is retained, then its value will persist forever, while a non-retained-will, just gets send out at the time of failure (and an older retained value may persist and get sent to new subscribers!).

If you want to deliberately shut down cleanly then you can tell mqtt to "disconnect" which doesn't trigger the will, but in that scenario you can send whatever messages you like to leave the system in valid state.

There are still a few tricks that MQTT can do, but that's most of what you should need to start generating useful MQTT data sources. Next time I'll look at hooking that up to something server side.

MQTT Part 2: Embedded!

MQTT is pretty handy for letting Sniff programs communicate just running on computers, but its real purpose is to pass messages between embedded devices like Arduino's which are sitting on the network, doing a single simple task - either measuring something (a temperature sensor?), or controlling something (a heating valve?).

The current Sniff implementation of MQTT runs great on Arduino, and it should run on other systems like MBED, though we've not had a chance to try that out yet. It requires a Wiznet5100 based ethernet interface, like the Arduino Ethernet Shield. Support to other network devices "real soon now".

So if you grab an Arduino, an Ethernet shield, and already have an MQTT broker set up from the previous tutorial, we'll begin.

Setup

First we need to set up the ethernet device:

make spi device

make ethernet device D10

make networkMAC string "b6:ee:63:ed:95:cb"

make networkIP string "192.168.0.200"

make networkConnected boolean

make networkPort number

make networkPeer string

Note that we also made an SPI device to handle the low level hardware, and we're using D10 as the select pin for the WizNet (as per Ethernet Shield). These chips come without a Mac address, so you'll need to generate one. It doesn't really matter what it is, but it has to be unique, so every board has its own. You also need an IP address, which you should get from your network administrator.

Subscribe

With that done, everything else is pretty much as it was previously.

make mqtt device

make clientid string

make message string

make topic string

when start

.set clientid to networkMAC

.set networkPeer to "192.168.0.108" #No DNS on Arduino!

.repeat until networkConnected

..tell mqtt to "connect"

.

.say "connected"

.

.set topic to "text"

.tell mqtt to "subscribe"

.forever

..tell mqtt to "loop"

..if not message = ""

...say message

I've done a couple of things slightly different, just because we're not running on a computer. First off I've used the MAC address as the clientID - you don't have to, but its a good way to ensure you've got a unique value. Next, I've used the raw IP address of the server rather than its hostname, as we don't have an easy way to look it up like we do on a "real" computer. In the desktop version I tried to connect, and gave up if I failed, but on an embedded device, we probably just want to keep trying until it works, which is what I've done here.

Control

Of course the real fun comes when we hook up some other hardware:

make i2c device

make display lcdi2c device 2 #2 line display

make displayFlush boolean

when start
etc

.

.forever

..tell mqtt to "loop"

..if not message = ""

...tell display to "show"

Now we're pushing the receive message onto an LCD screen connected via i2c. That's all it takes to do something really useful and exciting - now you can publish messages from any client, and they appear on the screen. You could have 1 screen, and dozens of sensors pushing alert messages to it, or dozens of screens with a central server sending messages to a dozen subscribers... or a thousand! This is essentially the architecture that bus and railway digital info boards would use.

You can publish from Arduino just as you would from the desktop too - once setup, MQTT works identically on both platforms, so setting up a sensor and publishing the results to the world is about as easy as it gets! However there are a few more tricks still to learn, so there's at least one more MQTT post to go...

MQTT Basics

MQTT isn't particularly well know, but if you're building things with Arduino's then it probably does something very useful to you, in a quick and some fashion - it moves data around between embedded controllers and servers with the minimum of fuss. It's an industrial standard which means its battle tested, but it also means it can be a bit intimidating reading some of the documentation. Its actually really simple.

There are two parts to any MQTT system: The server or "broker" that manages everything, and the clients that do the work. Clients will usually be attached to hardware to either measure or control something, but there might also be some larger clients, doing logging, analysis and providing overall control. We'll be mainly looking at the small side of the system (though you can write the "big" side in Sniff too). For pushing data out of the MQTT system you can use something like Node-Red, which has full MQTT support, making it easy to link IoT/Embedded tech to the bigger world.

Setup

The first thing you'll need to do is set up a broker. These can potentially be massive servers, as the whole thing is designed to scale to thousands or even millions of clients, but assuming you just want to drop a few smartThings around your house we can use a Raspberry Pi - in fact this is the best use I've ever found for a Pi! To set it up as a server just run

sudo apt-get install mosquitto 

That's it! It should install the server and start it running. You should probably tweak the config a little (at some point you should turn on the security features!), but basically it works perfectly out of the box.

Next we need a client. We're going to write our own clients in a few seconds, but its handy to have a debug tool. I grabbed a copy of MQTT.fx, but there are plenty to choose from (including for phones which is kind of handy of you're wandering round the building testing an install). Node-Red will work too, though its a bit fancy for just basic debugging. Connect your test client to the server, and nothing much should happen! No errors, and its all good.

Connecting

Now lets write some Sniff. We'll start with something running on the computer first to illustrate a the basic concepts of MQTT.

make mqtt device

make clientid string

make topic string

make message string

make networkConnected boolean

make networkPeer string

First we need to make an MQTT device, and some variables to control it. There are a few more parameters we can use but the essential ones are here - the rest can be left as default a lot of the time. Then we need to connect to our server:

when start

.set clientid to "sniffListenClient"

.set networkPeer to "raspberrypi.local."

.tell mqtt to "connect"
.

.if not networkConnected

..say "connect failed"

..stop script

.say "connected"

The only tricky bit here is the clientid. It needs to be something completely unique. If you run two versions of the same program with the same id then it won't work. For test purposes we can put anything in here, but for real world uses it should be a name which is specific to that client, on that particular piece of hardware.

Subscribing

Finally we can get to some real stuff - messages. Every message has a topic, and clients tell the server that they're interested in a particular topic. Then they get sent all the messages on that topic.

.

.set topic to "text"

.tell mqtt to "subscribe"

.

Here I've said I'm interested in all messages with the topic "text". Then all we have to do is wait for messages to arrive:

.
.forever

..tell mqtt to "loop"

..if not message = ""

...say message

Messages get received when we call "loop", but its important to call loop regularly even if you're just sending messages, as it handles lots of behind the scenes networking too. If "loop" finds a message it sets the values of topic and message, otherwise they'll both be empty strings. In principle you can send any kind of data over MQTT, but Sniff doesn't handle raw data very well, so it can only send and receive ASCII data. Having things in a readable format is probably better for small scale projects anyway.

Now we've reached the point where something like MQTT.fx comes in handy - select the publish tab, enter the topic "text" in the top box, and add some longer text into the main area. When you hit publish the Sniff program should receive the message and print it out!

It's that simple! You can have any number of clients (with unique ids) connected at the same time, all connected to the same server, and they'll all receive the message, so you can control them all by just publishing a message to the server.

Publishing

To Publish messages to the server, we first need to establish the connection, and start calling "loop":

#Call loop to receive messages forever

when listen

.forever

..tell mqtt to "loop"

when start

.set clientid to "sniffSendClient"

.set networkPeer to "raspberrypi.local."

.tell mqtt to "connect"

.

.if not networkConnected

..say "connect failed"

..stop script

.say "connected"

.

.broadcast listen

The easiest way to do this is to create a new script called listen, and once we're connected we start running that, using broadcast to kick it off. It will now do all the housekeeping in the background, and the mains script can continue on and do its thing.

.

.forever

..ask "Message?" and wait

..set topic to "text"

..set message to answer

..tell mqtt to "publish"

..say message

To publish a message first we ask what the message should be (which is returned in answer). Then we set the topic to "text" because that's what our other clients are subscribed to, and the message to whatever we just typed in. To push those out to all the other clients, just tell mqtt to "publish", and its done!

One gotcha to be careful of is that "listen" is running at the same time as the main script, and it can change the values of topic and message when we're not expecting it (this is called a race condition, and its a real thing that proper computer scientists worry about a lot). The way Scratch and Sniff are designed this is minimised as different scripts are only allowed to run at specific times. Here we need to watch out that listen can run during an ask, or a say (or a wait, but that's pretty obvious!). That's why we print the message out after its been sent. If we say the message before sending, listen will sneak in while we're printing it out. It will print OK, but the publish probably won't work. Just remember to set topic and message immediately before you publish and you're guaranteed to be OK.

And we're done...

for now! That is all you need to know to use MQTT in Sniff, but there are few more fancy features that will come in handy. You probably also want to know about using this on Arduino (same code, different setup), but I'll write them up next time!

Release 29 : Back to school!

Teaching has started again, which has pushed this release back a bit. Finally I've got a quiet few days to push out the latest fixes and features!

The first thing you'll notice if you use SniffPad is that syntax highlighting has improved a bit... variable names are now blue, which is handy as it means there's a visual indicator that you've spelt a variable name wrong when you type it, rather than getting a weird error message some time later. You might have noticed this already if you've tried live.sniff.org.uk.

The headline feature is of course MQTT support. There'll be more posts on that over the next few days, but if you want to place Arduino's to measure and control things using a network, then you should be very excited. Combining Sniff and MQTT makes building network sensor about 8 lines of code, and it will integrate with whatever control system you want to run on your central servers.

There's also the usual round of bug fixes and tweaks that are really important, but too small to mention.

Head over to the downloads page to grab a copy.