Live Video on the Pi

In Release 22 we added a "video" device, which allows you to capture live video using a Raspberry Pi and PiCam and access the data live in Sniff. The code may also work on other Linux platforms, and with other webcams, but for now we've focused on getting it working with the specific hardware that people are most likely to have. We also hope to follow up with support for other OS'es but video capture is a tricky thing...

However in Sniff (on a Pi!) its really easy:

make display window device

make displayX number

make displayY number

when start

.forever

..tell display to "get event"

..wait 0.1 secs

make webcam device

when start

.forever

..tell webcam to "update"

..set displayX to 320

..set displayY to 240

..tell webcam to "draw"

First we make a window, and poll if for events (to keep the OS happy). Then we can just grab a frame by telling the webcam to "update". Then we draw it on screen by selecting the middle of the screen and telling the webcam to "draw". Thats it! You should now have a live feed from your camera, on screen!

In addition to just drawing the image, you can access the pixel values in it, so we could scale it down and copy it to a regular bitmap using:

make bitmap device

make scale number

when copyVidToBitmap

.make x number

.make y number

.set scale to 4

.set displayX to 640/scale

.set displayY to 480/scale

.tell bitmap to "new"

.repeat 640/scale using x

..repeat 480/scale using y

...set displayX to x*scale

...set displayY to y*scale

...tell webcam to "get pixel"

...set displayX to x

...set displayY to y

...tell bitmap to "set pixel"

We could save this out as a BMP, but in the demo we draw the smaller version over the larger one, and make it spin around!

..broadcast copyVidToBitmap and wait 

.. 

..set bitmapValue to 10 

..tell bitmap to "turn by" 

..set displayX to 220 

..set displayY to 140 

There are probably a few wrinkles to work out (and it probably works best on a Pi2!), but at least for basic stuff, getting your live video feed into Sniff is really easy. How about writing the code to do your own green screen on the images, or edge detect the data to make a live line-drawing?

Release 22 (Sorry for the bugs in 21!)

Its Sniff release time again, and 22 squashes a few bugs in Release 21... Most of there are concerned with cross platform issues within the IDE, so you might not have encountered them depending on your platform (Mac users can be smug and annoying here... Linux had some low driver bugs while on Windows the delete key stopped working!).

However its not all internals stuff... we've got the source to an updated version of the IK demo we posted recently, and if you've got a Raspberry Pi (and who hasn't) now might be a fun time to turn it back on and play with the new camera API. This might work with other cameras on other Linux platforms, but it works best with a Pi Cam! We'll have a write up of that soon.

Head over to the Downloads page and get the update!

Inverse Kinematics (IK) for your Robot Arm

A basic robot arm has a number of segments of (usually) fixed length. Between each of these is a joint which can rotate. We move the arm by changing the angles at each of the joints. If we know the length of each segment, and the angles then we can start at the base of the robot, and calculate the position of the end of the arm:

when getEndOfSegementPosition

.make count number

.set displayX to 0

.set displayY to 0

.set angle to 0

.

.repeat segementNumber using count

..change angle by item count of segementAngles

..change displayX by item count of segementLengths * cos of angle

..change displayY by item count of segementLengths * sin of angle

This is called Forward Kinematics or FK, and its also used in computer animation to position and draw a characters limbs. It's provides complete, precise control. However its also a bit slow and tedious. If we want a character (robot arm) to press a button, then we need to mess around with all of the angles until we get the hand in the right place.

Wouldn't it be easier if we could just tell the computer where we want the end of the arm to be, and let it figure out all the necessary angles for us? This is Inverse Kinematics, as we're calculating the required input from the desired output (whereas in FK we apply the inputs to see the output - forwards!).

If your arm only has one or two joints then you can perhaps work this out analytically (using maths!), but if you're arm is lots of segments, then finding the best set of angles for all of them becomes tricky. Especially as there are likely to be many possible sets of angles that would all get the same result. 

One commonly used way of tackling the project is to use Cyclic Coordinate Descent (CCD):

[WELMAN C.: Inverse Kinematics and Geometric Constraints for Articulated Figure Manipulation. Master’s thesis, Simon Fraser University, 1989] . This works well, and is simple enough to implement that we can do it in Sniff.

We look at a joint somewhere along the arm and consider the position of the end effector, compared to the target. We then rotate that joint to minimise the difference between the current position and the desired position of the end effector. Because we can only move one joint, the optimal angle for the joint is always to orient the arm so that the vector from the joint to the end effector is pointing towards the target. When the desired and actual locations line up then we've done the best we can with that single joint.

Lets try that again. Consider joint B. We'd like to move C to T, but the best we can do is C'.

make jointNumber number

when updateJoint

.make sX number

.make sY number

.make dX number

.make dY number

.make desiredAngle number

.make actualAngle number

.make deltaAngle number

.

.set segementNumber to jointNumber - 1

.broadcast getEndOfSegementPosition and wait

.set sX to displayX

.set sY to displayY

.

.set dX to mouseX-sX

.set dY to mouseY-sY

.set desiredAngle to atan of (dY/dX)

.

.set segementNumber to length of segementLengths

.broadcast getEndOfSegementPosition and wait

.set dX to displayX-sX

.set dY to displayY-sY

.set actualAngle to atan of (dY/dX)

.

.set deltaAngle to desiredAngle-actualAngle

.

.set angle to item jointNumber of segementAngles

.change angle by deltaAngle

.if angle >90

..set angle to 90

.if angle <-90

..set angle to -90

.replace item jointNumber of segementAngles with angle

To do that in Sniff, we just get the end of the previous section (the position of the joint), find the vector to the target, and calculate the desiredAngle. Then we do the same, but using the end position of the arm to find the actualAngle. Calculate the difference and add it to the angle of this joint.

You'll notice that there's some code to stop angle being greater or less than 90 degrees - this is a constraint. One of the neat bits about this approach is that its really easy to tweak it, in this case to prevent the arm bending more than 90degrees at any single joint but we could have different constraints for each joint, or contain deltaAngle so that it can't move to quickly.

Of course its quite likely that we can't get to correct position by moving just one joint, so we apply this adjustment to each joint in turn:

...set jointNumber to length of segementLengths

...repeat length of segementLengths

....broadcast updateJoint and wait

....change jointNumber by -1

However note that we start at the end of the arm, and move back to the root. Intuitively you can think about reaching for something: you move your wrist first, and only move your whole arm from your shoulder when finer motions can't to the job.

Typically you'd apply this whole process a few and the arm will end up pretty much where its supposed to be!

Try it!!! The full code will be in the next release of Sniff.

The code works pretty well for a basic implementation, but it has a few glitches. Tweaking the constraints can dramatically help the quality of the final position. However the main issue is that we're using the atan function to calculate an angle from the ration of dy/dx. While its certainly true that tan(theta)=dy/dx its less clear cut that theta=atan(dy/dx). This can break, not only because dx might be zero, but also because it assumes that we're working positive numbers. We can't tell the difference between 4/5 and -4/-5 because they're the same, but the required angles are 180 degrees out!! In C we'd use atan2(y,x) which does the divide for itself, so it works reliably for all angles. The full code includes an implementation of atan2, which makes everything nice and stable.

For a real arm we might also want to do the whole thing in 3D, which is much harder, but if you've got one of the simple USB robot arms, then they only support pivoting at the base and at the grabber. The main body of the arm remains in a 2D plane. You can therefore calculate the rotation of the base required to place the target in the plane of the arm, then treat the rest as a 2D problem. I've not got one of these but it should simple to adapt this code to control one.

XKCD Velociraptor Problem

I'm always looking for fun problems to solve in Sniff, and I recently came across post on Wired,

which in turn stole its idea from XKCD. So I'm going to steal it and do it in Sniff! (seriously read the others first).

The question is how long can you survive a velociraptor attack, given a 40m head start? Velociraptors have a top speed of 25m/s and accelerate at 4m/s/s. You have a top speed of 6m/s and accelerate "quickly".

This is a fun question... not just because it has dinosaurs in it, because it ends in inevitable carnage, or even that it parodies typical maths/physics questions, but because it's actually something we can code easily - as they do in the wired article. The code actually demonstrates some of the basics of physics sims, as used for both science and games (so I get to tag this post as Algorithms, Science and Games!). 

make vX number

make hX number

make vV number

make hV number

make vVmax number

make hVmax number

make vA number

make hA number

make dt number

make t number

when start

.set vX to -40

.set hX to 0

.

.set vV to 0

.set hV to 0

.

.set vVmax to 25

.set hVmax to 6

.

.set vA to 4

.set hA to 1000

.

.set dt to 0.1

.set t to 0

We start by putting all of the information in the question into variables. Not only does this make things clearer in the code, but it means we can tweak them to try out different scenarios. Variables beginning h mean human, while variables beginning v mean velociraptor. The velociraptor starts 40 to the left, everyone starts at 0 speed. We set Vmax as per the question.

In the Wired article the interpret the human "quickly reaching your top speed" to mean an arbitrary 3m/s/s. I've chosen to interpret "quickly" in the spirit of physics textbooks where "light" is a code word for massless, "smooth" really means frictionless, and "quickly" really means "instantly". I've there for made hA=1000.

t for time, starts at zero. The standard "numeric" way to handle these kinds of problems is simply to use small time steps, so dt is 0.1.

Now the fun bit:

.repeat until not vX<hX

..change t by dt

..

..

..change vV by vA*dt

..if vV>vVmax

...set vV to vVmax

..

..change hV by hA*dt

..if hV>hVmax

...set hV to hVmax

..

..

..change vX by vV*dt

..change hX by hV*dt

..

.say [vX]

We're going to keep going while the velociraptors position is less then the humans. Each time round the loop represents a tiny fraction of a second.

Acceleration is rate of change of velocity so vV increases by vA each second. We're not looking at a whole second, so vV changes by vA*dt each time around. However vV can't be bigger than vVmax, so if it is we set it back.

Now we know the new velocity, we can calculate how far we move: vV*dt, and similarly or hV. 

With that in place we find we survive in impressive  38 seconds. Using the more conservative acceleration of 3m/s for the human the Wired use you only survive about 30 seconds. This code and the wired python code disagree very slightly as the wired code lets vV become greater than vVmax! However the answers get closer together as we make dt smaller.

This is an important point - this sort of code is only an approximation but even with simple code we get good results if we make dt small enough. Making dt bigger makes it less accurate, but your code runs faster.  The trick is to find ways to make dt bigger without sacrificing accuracy, but thats getting a bit advanced.

One of the main ways you might use this kind of code is in a game to simulate some kind of object being fired. In that case you have velocity and accretion in X and Y directions.

.set accX to 0

.set accY to -9.8

.repeat until posY<0

..change posX by velX*dt

..change posY by velY*dt

..

..change velX bu accX*dt

..change velY by accY*dt

Typically the acceleration in X is negligible, while the acceleration in Y is due to gravity. Drop this into a game and you've got your first physics engine!

Sniff Sprites Reference Material

At the game jam last week everything went really well, and we had lots of fun. We've previously produced some good worksheets (which you can download) which provide a good introduction to working with sprites. In sessions we've done, we've provided a basic empty example, and following a brief introduction to Sniff we just let the kids loose to work through the cards. That cards are numbered, but if you'd prefer to shuffle them around they work pretty well in more or less any order.

Kids can work through these examples in a few hours, and start to get the basics of a game. However when they come to make more complex games they'll find they want to do things which aren't explained on the cards. That's fine - we expected that, and explaining stuff on a "need to know" basis is actually a really great way of teaching. Kids take stuff in really quickly when they actually think they're going to use what they're being told (see how fast a kids who wants to make a ballistics game can learn about trigonometry and forces - months of teaching condensed into minutes).

However the problem with that is that is that when they "need to know" there needs to be someone who does know! In this case Michelle and I were there, and we've both worked on developing the sprite resources, so we could quickly address problems. If you want to learn more about using the sprite system we'd refer you to the examples posted here (we've added a "game" label so you should be easily able to find the relevant posts).

However we don't have a definitive reference. Sniff does come with a manual (in the docs folder!) which includes documentation for some of the arduino devices, but the sprite system is complex enough to warrant documentation of its own, so we've now written some.

Here's a reference manual for Sniff Sprites which lets all of the methods you can use with the Sprite and SpriteManager classes. Once you've got to grips with the basics, this is where you should look to find out the exact details of what features are available.

IPACA Game Jam

Last week we held a game jam event at Bournemouth University with 15 KS3 kids from IPACA (Isle of Portland Aldridge Commity Academy). We've been working up to this for a few months, rolling out various supporting code and worksheet resources, so that (we hoped!) we could teach them Sniff to the level of having some kind of working game within 2 days.

That's a big challenge, which wasn't helped by traffic on the A35 from Weymouth to Bournemouth on the first morning... so they arrived almost 2 hours late. That's a big chunk of time out of the first day! Second day was able to start a bit more promptly by we still ended up with about 9 hours to do a project we thought was ambitious in 12 hours!

Sniff's actually based in the NCCA at Bournemouth University (You may have heard of us from the NESTA NextGen Report) so we had use of one of the Linux labs there (not sure of the spec, but I believe there's around 1.6 Terabytes of RAM installed in the room!). We were able to divide everyone up into groups of 3, and give them their own work area with one machine each. While its hard (on this scale) to have more than one person working directly on code, this meant that in their teams they could have 1 person developing assets, 1 programming, and a 3rd assisting/coordinating). This worked really well.

We ran a smaller pilot session at the end of the summer term, which some of the students had attended, but most of them hadn't used Sniff before, so I kicked of the session with a 20minute crash course on Sniff. This generally goes something along the lines of getting them to run sniffpad, and then throwing a few examples at them:

when start
.say "hello"

when start
.repeat 10
..say "hello"

make count number
when start
.set count to 1
.repeat 10
..say [count]

make count number
when start
.set count to 1
.repeat 100
..if count mod 5 = 0
...say "fizz"
..else
...say [count]

You might recognise this final example as a simplified version of fizz buzz - in fact its just fizz (which is much easier to code). Look closely and you'll see what a great example it is - sequence, iteration, selection. It's all there. It also lets me talk about indention, typing, and with a bit of tweaking you can also explain the broadcast mechanism (which even kids who claim to be "expert" at Scratch seem to have rarely used).

While I'd normally take a little longer to cover this it really doesn't take very much time to throw down some basic Sniff examples. The general reaction is "OK - I get it", can we make a game now. If you've laid the foundations in Scratch then there's no need to spend much time "teaching  Sniff" - you can move on quickly to making things, and learn the rest of Sniff indirectly.

To do that we've got a bunch of worksheets/cards that Michelle developed over the summer. Throwing them straight into sprites is a little unfair, as its quite a complex library, but their enthusiasm for coding games helps make up for that.

You can download the cards

These contain fragments of code to do all sorts of useful things with Sprites. However they're not complete programs, and also require small modifications depending on how you want to use them. We give out a basic framework which contains all the generic boiler plate where all the devices are made, and then kids can work through the cards to implement all the basic features they'll need in a game. Though there's a general "order" to the cards (they're numbered!) you don't need to cover them all sequentially, so you can hold back certain bits of code until they're needed.

It took an hour or two for everyone to more or less get through these and produce some kind of crazy moving object that they could control with the keyboard.

Sandwiches and crisps were provided by the CDE (Centre for Digital Entertainment - thanks Dan!), and as we were running late, Phil Wilkinson (a DEng student attached to the CDE/Bournemouth and IPACA) asked everyone to discuss what sort of game they might like to make for the rest of the workshop over lunch. Phil is interested in how getting kids to make games can be used to explore social issues with chi. For this sessions we wanted something fairly lightweight, so he decided "to teach something relevant, something modern. The Internet!" [those of you over the age of 16 who don't see why that's funny, are welcome to google it. It may be NSFW depending on where you work!].

So we asked them to build a game around the theme of the internet. The idea was that it was open enough that they could make more or less anything, but there was at least a jumping off point for discussion. 

After lunch I introduced sniff paint, which proved very effective, and we immediately started getting some great artwork.

Over the course of the rest of the afternoon icons started to appear, and the some sort of vague game play ideas started to be discussed.

Day 2

On the morning of day 2, Phil worked with them to turn their rough game ideas into concepts that could be realised within a few hours, while I ran around fielding technical questions. Almost all of these were really solid questions about how they could implement features of their game. As we were doing this in an intense 2 day session, there was a lot about the sprite library that we didn't have time to teach in advance. However I you learn better if you already have a need for the knowledge, and I was able to explain features on-demand. Some better reference documentation would be useful here, or if you were working at a more relaxed pace, perhaps over a number of shorter sessions, you could introduce some of these in a more structured way. At around this time as they started to get seriously into development we encountered a bug in the Sprite "load costume" code, which I had to make a quick patch to, and distribute round the lab (yes it was stressful). However apart from that the system help up pretty well to some serious testing.

One concept we'd deliberatly kept back was how to dynamically create sprites at runtime. This is a more advanced feature, but only one team needed it.

Just after lunch one of the teams accidentally deleted their code - someone clicked save when they mean to click load and saved over their program. Michelle was able to work with them to quickly rewrite most of what they'd lost, but they never quite caught up. Things like this are always going to happen, as they were under quite a lot of pressure at this stage to get everything working. Some sort of automated backup script would probably be a good idea.

At 3pm we called time on the development to allow for everyone to go around and play everyone else games. While everyone would have benefited from "a little more time", pretty much all the teams had produced a game that was to some extent playable. If we'd had another day most of them could have been refined into something that was actually finished. My main concern was that at some point they might just get stuck and not know how to progress. In fact this never came close to happening - they all knew what they wanted to do and worked hard, and asked a lot of questions to get there.

Most importantly everyone seemed to have a fun 2 days, and the teams all worked well to get something working within the time limit. They went home enthusiastic about programming, and knowing that with a bit more practise and a less crazy schedule they could actually make their own game.

Here are 4 of the 5 games which you can play online:

Virus Defence (use the mouse to block the virus)

Ad Invaders (space invaders - cursor keys)

Internet Survival (another shooter variane - cursor keys)

Oops We Deleted Everything (they didn't get finished - cursor keys)

[At the end of the session I grabbed copies of all of the code, which I used to make these. They're exactly as they were written, and I've not tidied them up, so there's still a few bugs in there. The final game I seem to have copied the wrong files, as I don't have a working version of the game, which is a shame as it looked great.]

E-Compasses revisited

We've had support for the hmc5883 magnetometer for some time. This measures the magnetic field strength on 3 axis, which with a little bit of trigonometry can be turned into a compass. We used this on SniffBot so that it could be instructed to face North when in idle mode. This worked brilliantly sometimes, but a few times we tried it, it just behaved very badly. We put this down to the generally messy magnetic field caused by all the metal and electronic equipment we have around here.

However we were prompted to revisit it, by the announcement of the BBC Microbit. Despite this being promised to arrive in schools in the next few weeks, no one has yet seen one... slightly worrying. However we do know it includes a mag3110 magnetometer. This does exactly the same job as the hmc5883, and in fact they both run on i2c, so for all practical purposes its a direct drop in replacement. So in Sniff R21 we've added support for the mag3110. With the new driver you should be able to just change:

make magnetometer hmc5883 device

To:

make magnetometer mag3110 device

And you're good to go.

When we tried it, we found that the results were pretty unreliable, but a bit of research suggested we could improve things by calbrating the sensor. If you've got an iPhone and used the compass app, then you'll be familiar with this, where it asks you to wave the phone around. The problem is that the chip tends to be biased, outputting the results with a fixed offset.

From our experience (sample size 1) on the hmc5883 this is sometimes quite a small error, so we got OK results... sometimes! On the mag3110 this is typically quite a larger error so we need always need to compensate for it.

While your phone does some complex stuff, including using the accelerometer to track the movement of your phone during calibration, we can do a basic calibration by getting you to wave the chip around. If you move it enough we'll get max and minimum values on each axis which are Offset+field, and Offset-field. Taking the average gives is the offset, which we can then subtract from subsequent readings.

To make this happen we've added a new variable compassCalibrate. If you set it to "yes" then you can  still use the compass as normal but the readings are also used for calibration. Provided you're operating in a relatively constant magnetic field, this works really well, so if you just need a generic compass just set it to "yes".

when start

.set compassCalibrate to yes

.

.say "Calibrating"

.repeat 100

..tell magnetometer to "read"

..wait 0.1 secs

In the compass.sniff example code, we set compassCalibrate to yes, then take 100 readings over 10 seconds to get a baseline calibration. During this time you should turn the chip around as much as possible. However even after we leave this setup phase, and start collecting "real" readings, we leave calibration on, and in fact the accuracy does seem to improve with further use.

.forever

..tell magnetometer to "read"

..say join "Heading " [ heading ]

..say ""

..wait 1 secs

Where this approach will fail is if you actually want to measure a changing magnetic field. Large changes in the field will be partially calibrated out, as the sensor tries to correct for something it actually should be measuring. In which case you should probably calibrate initially in a constant field, then turn calibration mode off.

With this code in place the Mag3110 worked really well, so we went back and added the same calibration modes to the Hmc5883. Your old code should work just as it always did, but if you add a calibration phase, or simple let the device calibrate as you're using it you should see much better results.