MQTT Basics

MQTT isn't particularly well know, but if you're building things with Arduino's then it probably does something very useful to you, in a quick and some fashion - it moves data around between embedded controllers and servers with the minimum of fuss. It's an industrial standard which means its battle tested, but it also means it can be a bit intimidating reading some of the documentation. Its actually really simple.

There are two parts to any MQTT system: The server or "broker" that manages everything, and the clients that do the work. Clients will usually be attached to hardware to either measure or control something, but there might also be some larger clients, doing logging, analysis and providing overall control. We'll be mainly looking at the small side of the system (though you can write the "big" side in Sniff too). For pushing data out of the MQTT system you can use something like Node-Red, which has full MQTT support, making it easy to link IoT/Embedded tech to the bigger world.

Setup

The first thing you'll need to do is set up a broker. These can potentially be massive servers, as the whole thing is designed to scale to thousands or even millions of clients, but assuming you just want to drop a few smartThings around your house we can use a Raspberry Pi - in fact this is the best use I've ever found for a Pi! To set it up as a server just run

sudo apt-get install mosquitto 

That's it! It should install the server and start it running. You should probably tweak the config a little (at some point you should turn on the security features!), but basically it works perfectly out of the box.

Next we need a client. We're going to write our own clients in a few seconds, but its handy to have a debug tool. I grabbed a copy of MQTT.fx, but there are plenty to choose from (including for phones which is kind of handy of you're wandering round the building testing an install). Node-Red will work too, though its a bit fancy for just basic debugging. Connect your test client to the server, and nothing much should happen! No errors, and its all good.

Connecting

Now lets write some Sniff. We'll start with something running on the computer first to illustrate a the basic concepts of MQTT.

make mqtt device

make clientid string

make topic string

make message string

make networkConnected boolean

make networkPeer string

First we need to make an MQTT device, and some variables to control it. There are a few more parameters we can use but the essential ones are here - the rest can be left as default a lot of the time. Then we need to connect to our server:

when start

.set clientid to "sniffListenClient"

.set networkPeer to "raspberrypi.local."

.tell mqtt to "connect"
.

.if not networkConnected

..say "connect failed"

..stop script

.say "connected"

The only tricky bit here is the clientid. It needs to be something completely unique. If you run two versions of the same program with the same id then it won't work. For test purposes we can put anything in here, but for real world uses it should be a name which is specific to that client, on that particular piece of hardware.

Subscribing

Finally we can get to some real stuff - messages. Every message has a topic, and clients tell the server that they're interested in a particular topic. Then they get sent all the messages on that topic.

.

.set topic to "text"

.tell mqtt to "subscribe"

.

Here I've said I'm interested in all messages with the topic "text". Then all we have to do is wait for messages to arrive:

.
.forever

..tell mqtt to "loop"

..if not message = ""

...say message

Messages get received when we call "loop", but its important to call loop regularly even if you're just sending messages, as it handles lots of behind the scenes networking too. If "loop" finds a message it sets the values of topic and message, otherwise they'll both be empty strings. In principle you can send any kind of data over MQTT, but Sniff doesn't handle raw data very well, so it can only send and receive ASCII data. Having things in a readable format is probably better for small scale projects anyway.

Now we've reached the point where something like MQTT.fx comes in handy - select the publish tab, enter the topic "text" in the top box, and add some longer text into the main area. When you hit publish the Sniff program should receive the message and print it out!

It's that simple! You can have any number of clients (with unique ids) connected at the same time, all connected to the same server, and they'll all receive the message, so you can control them all by just publishing a message to the server.

Publishing

To Publish messages to the server, we first need to establish the connection, and start calling "loop":

#Call loop to receive messages forever

when listen

.forever

..tell mqtt to "loop"

when start

.set clientid to "sniffSendClient"

.set networkPeer to "raspberrypi.local."

.tell mqtt to "connect"

.

.if not networkConnected

..say "connect failed"

..stop script

.say "connected"

.

.broadcast listen

The easiest way to do this is to create a new script called listen, and once we're connected we start running that, using broadcast to kick it off. It will now do all the housekeeping in the background, and the mains script can continue on and do its thing.

.

.forever

..ask "Message?" and wait

..set topic to "text"

..set message to answer

..tell mqtt to "publish"

..say message

To publish a message first we ask what the message should be (which is returned in answer). Then we set the topic to "text" because that's what our other clients are subscribed to, and the message to whatever we just typed in. To push those out to all the other clients, just tell mqtt to "publish", and its done!

One gotcha to be careful of is that "listen" is running at the same time as the main script, and it can change the values of topic and message when we're not expecting it (this is called a race condition, and its a real thing that proper computer scientists worry about a lot). The way Scratch and Sniff are designed this is minimised as different scripts are only allowed to run at specific times. Here we need to watch out that listen can run during an ask, or a say (or a wait, but that's pretty obvious!). That's why we print the message out after its been sent. If we say the message before sending, listen will sneak in while we're printing it out. It will print OK, but the publish probably won't work. Just remember to set topic and message immediately before you publish and you're guaranteed to be OK.

And we're done...

for now! That is all you need to know to use MQTT in Sniff, but there are few more fancy features that will come in handy. You probably also want to know about using this on Arduino (same code, different setup), but I'll write them up next time!

Release 29 : Back to school!

Teaching has started again, which has pushed this release back a bit. Finally I've got a quiet few days to push out the latest fixes and features!

The first thing you'll notice if you use SniffPad is that syntax highlighting has improved a bit... variable names are now blue, which is handy as it means there's a visual indicator that you've spelt a variable name wrong when you type it, rather than getting a weird error message some time later. You might have noticed this already if you've tried live.sniff.org.uk.

The headline feature is of course MQTT support. There'll be more posts on that over the next few days, but if you want to place Arduino's to measure and control things using a network, then you should be very excited. Combining Sniff and MQTT makes building network sensor about 8 lines of code, and it will integrate with whatever control system you want to run on your central servers.

There's also the usual round of bug fixes and tweaks that are really important, but too small to mention.

Head over to the downloads page to grab a copy.

Lego IR "train" controller

Just a quick update on the previous post where we used the Lego IR remote to control and Arduino. I've not got my hands on a Lego "train" controller which has two dials and two buttons. Pointing one of those at an Arduino with the IR receiver attached, produces a fairly simple set of codes.

The Left dial generates 100 and 101 as its turned in each direction, while right dial generates 116/117. When used with the regular IR receiver these mean speed up/slow down on the red and blue channels but you can use them in your code to mean anything you like. Each turn will generate a burst of codes, so you're likely to receive several at a time. This is intended to make sure that the code actually gets through. You might need to ignore repeats of a code if they arrive very close together, as its not multiple turns, but just multiple signals.

The left button generates 72,  while the right button generates 88. Somewhat surprisingly pressing both butts at the same time generates 31, which to the Lego decoder means brake both motors. The numbers don't appear to make sense as 31 is a code to control both motors, while 72/88 are codes to control individual motors so are sent in a very different way.

Unlike the regular Lego remote, the codes are never combined (except for 72+88=31), so if you press and hold a button, then turn a dual, you can no longer detect if the button is pressed. Turning both dials at once, results in both sets of messages being sent, rather than just a combined signal as per the regular remote.

Channels (1-4) work report in the irProtocal variable, so you can use up to four remotes at a time!

Receiving Lego IR

In the previous post, we used an IR LED to control Lego Power Functions motors, which was pretty useful, but what about going the other way - using a Lego PF remote to control an Arduino?

That standard Lego remote is really nice for controlling Robots as it has two levers, giving you forward/back tank style steering. There's a handy "reverse" switch for each lever, and best of all there's a channel selector, so you can operate up to four remotes at the same time. This is always a problem in workshops where everyone gets the same kit of parts, and everyones remote controls everyone else robots!

At about £7.50 from the Lego store, they're pretty reasonably priced if you're buying them for yourself, or even a classroom, though unfortunatly our workshop budget won't stretch to giving them away, as we do with the regular £1 remotes.

Lego uses a unique (but well documented) protocol, so which we've added to the regular receiveIR device, so if you have build a robot with an IR receiver you don't need to change anything, other than detect the new keypress codes.

Working with IR is cheap and easy - just get a  tsop-4828 receiver for about 50p, and wire it to a data pin on an Arduino (note that the transmit code should work on any micro controller, but the receive as some Arduino specific bits):

The code to read from an device, works exactly the same as it always did:

make remote receiveIR device D2

make irProtocol number

make keyPressed number

when start

.forever

..tell remote to "read"

..if not keyPressed = 0

...say join "Protocol:" [ irProtocol ]

...say join "KeyVal  :" [ keyPressed ]

...say ""

Now, if you point a lego transmitter at it you'll start getting key codes. The channel is returned in the variable irProtocal, so you can easily run multiple controllers at once. If you're really interested in making sense of keyPressed value then there's documentation available from lego. The value returned is the middle 2 nibbles of the data packet, with the escape bit tagged on the beginning.

While its possible to decode the packet based on the documentation, its probably easier to just look at the values that are being sent when each button/lever is pressed, and actual accordingly.

If we look at the codes sent by the regular remote, a value of 16 is sent when nothing is pressed. In fact you can subtract 16 from every received code and things start to make sense: Left stick generates 0 in the centre, 1 when forwards and 2 when backwards. Add these to the 16, that this remote always sends and you're in business. The right stick works the same except we multiple by four, so the value sent is 16+4*rightStick+leftStick.

Of course you can still use most other regular remotes to, but the Lego ones are a bit nicer. The code is now on live.sniff.org.uk along with an example and will by in the next desktop release.

Power Functions IR

Lego WEDO is great - you can control power functions motors, but its a bit expensive, and you need a computer. An alternative and cheaper way to control power functions is with the IR control, which you costs £11.50  (important hint: NEVER buy Lego on Ebay or Amazon without checking the lego site first. It's usually much cheaper. The logo consumer store is also much cheaper than the edu store, even though you're buying the same component from the same company). Of course you'll still need some motors and lights and the remote control, but you probably have some of those anyway. Either way you're in business for about £40 rather than £135 (though of course if you buy a few more bits you'll get free shipping, so a few extra parts is saving money in the long run).

It turns out that while the lego IR system is a bit quirky, its well documented, and pretty easy to control from an Arduino or similar. All you need is an IR LED, which cost a few pennies - wire it to one of the pins with a 500ohm(ish) resistor in series so you don't break anything, and you'r good to go.

To code for it in Sniff, just make a device:

make remote transmitLegoIR device D13

make blueValue number

make redValue number

make channel number

I've called it "remote" and attached it to pin 13. Then I powered up the receiver as normal, and connected a motor to the red output. You'll notice that there's an orange slider on the side of the receiver, which sets the channel. If you've only got one then you can just keep it set to 1, otherwise match the channel in the code to the receiver you want to control.

All I want to do is run a motor back and forth, so here's the code:

when start

.set channel to 1

.set blueValue to 0

.forever

..set redValue to sin of (timer*100)

..tell remote to "send"

..wait 0.1 secs

All the code does is set the blue output to 0 (Stop), then goes into a loop. The red value is set to range from -1 to +1: full reverse to full forwards, changing with time. Once you've set the red and blue values as you want them, just tell the remote to "send" and it fires of a few IR pulses and the Power functions should do whatever you command!

Normally at this point I'd tell you that the code would be in the next update of Sniff (and it will be!), but if you go to live.sniff.org.uk now you'll find the example code already there, and you can compile it for Arduino online!

Sniff Live For Arduino

Our initial implementations of Sniff Live were aimed at getting regular programs compiling and running in the browser, so it would be easier for new users to write a few simple Sniff programs without having to install the system. On Windows in particular installation is a bit of work (it's MUCH easier if you're running Linux or Mac).

However the most fun projects we do with Sniff involve external hardware - usually the Arduino Uno, so getting that working with Sniff Live was something we always had as part of the plan, and now its ready for you to try out.

If you head over to live.sniff.org.uk  you'll see that on the front page there's a suggestion to download the Loader app.  Download and unzip it somewhere. The Loader app is the easiest way to upload intel hex files to an Uno, though if you have any other method you prefer that will work too.

Then log in as usual, and take a copy of the blink example using the "copy examples" pop up in the top left. Once you've got that, press the Arduino button in the editor to compile your code for arduino. If you were running Sniff on your own computer, this would also do the flashing for  you, but unfortunately it can't because now the code is being compiled on our server, while the arduino is connected to your PC!

Once the code is compiled, press the "run" button, either in the editor, or in the sidebar (the run link might not appear straight away). This should download the hex file to your computer. The exact details of what this looks like will depend on your browser and its settings, but you should save the file, and open it.

Your PC probably doesn't know what to do with a "hex" file, so set up the file association so that it opens in the "UnoLoader.exe" that you downloaded earlier. From then on when you double click a hex file, or tell your browser to "Open" it, then it should upload the code straight to your arduino without any further intervention from you.

If UnoLoader doesn't work, then check that you have an Uno connected, and that it is appearing as a COM port. If UnoLoader does fail, then its window will stay open and it will tell you whats going wrong. Check out what it says the problem is, and if you can't figure it out, let us know.

Sniff Live!

We've been fairly quiet recently, as we've been working on a big project. It's not exactly a secret (as we've mentioned it in a previous post), but we're just about ready to make a bit more of an announcement about it...

You can now program Sniff entirely in a browser!

After a few year of trying to make the install as easy and as portable as possible, we now have a solution which requires nothing installed on your machine. Just go to http://live.sniff.org.uk, and enter your name to create an account. From there you'll be taken to the online IDE, which is basically the regular Sniffpad IDE but now everything is online.

This isn't intended as a replacement of the regular Sniff implementation - more as a way for you to get started, so it's slightly more limited, but hopefully being able to try stuff out and online will make it an easy way to get started.

To run a program, just press the compile button, and then the run button. Run requires you to enable pop-ups as it runs the program in a new window. Alternatively if a program has been compiled previously a run link will appear in the left sidebar, which doesn't require pop-up permissions.

On the left sidebar you can copy some demo's which are preloaded onto the system, or edit programs you've previously worked on. You can also upload/download code or images. Images can be edited using the paint package, though it is limited to quite small images. Any code you write can load and save files to your account on the server - either text files or image files.

The system can technically compile code for Arduino Uno too, but currently it there's no way to actually execute that code on a board. We're working on that but its a bit tricky!!! 

The system is still in its development phase, so expect things to change/break, but hopefully it should make it a lot easier to use Sniff on a wider range of computers.

Check it out at: http://live.sniff.org.uk