Live Video on the Pi

In Release 22 we added a "video" device, which allows you to capture live video using a Raspberry Pi and PiCam and access the data live in Sniff. The code may also work on other Linux platforms, and with other webcams, but for now we've focused on getting it working with the specific hardware that people are most likely to have. We also hope to follow up with support for other OS'es but video capture is a tricky thing...

However in Sniff (on a Pi!) its really easy:

make display window device

make displayX number

make displayY number

when start

.forever

..tell display to "get event"

..wait 0.1 secs

make webcam device

when start

.forever

..tell webcam to "update"

..set displayX to 320

..set displayY to 240

..tell webcam to "draw"

First we make a window, and poll if for events (to keep the OS happy). Then we can just grab a frame by telling the webcam to "update". Then we draw it on screen by selecting the middle of the screen and telling the webcam to "draw". Thats it! You should now have a live feed from your camera, on screen!

In addition to just drawing the image, you can access the pixel values in it, so we could scale it down and copy it to a regular bitmap using:

make bitmap device

make scale number

when copyVidToBitmap

.make x number

.make y number

.set scale to 4

.set displayX to 640/scale

.set displayY to 480/scale

.tell bitmap to "new"

.repeat 640/scale using x

..repeat 480/scale using y

...set displayX to x*scale

...set displayY to y*scale

...tell webcam to "get pixel"

...set displayX to x

...set displayY to y

...tell bitmap to "set pixel"

We could save this out as a BMP, but in the demo we draw the smaller version over the larger one, and make it spin around!

..broadcast copyVidToBitmap and wait 

.. 

..set bitmapValue to 10 

..tell bitmap to "turn by" 

..set displayX to 220 

..set displayY to 140 

There are probably a few wrinkles to work out (and it probably works best on a Pi2!), but at least for basic stuff, getting your live video feed into Sniff is really easy. How about writing the code to do your own green screen on the images, or edge detect the data to make a live line-drawing?

Release 22 (Sorry for the bugs in 21!)

Its Sniff release time again, and 22 squashes a few bugs in Release 21... Most of there are concerned with cross platform issues within the IDE, so you might not have encountered them depending on your platform (Mac users can be smug and annoying here... Linux had some low driver bugs while on Windows the delete key stopped working!).

However its not all internals stuff... we've got the source to an updated version of the IK demo we posted recently, and if you've got a Raspberry Pi (and who hasn't) now might be a fun time to turn it back on and play with the new camera API. This might work with other cameras on other Linux platforms, but it works best with a Pi Cam! We'll have a write up of that soon.

Head over to the Downloads page and get the update!

Inverse Kinematics (IK) for your Robot Arm

A basic robot arm has a number of segments of (usually) fixed length. Between each of these is a joint which can rotate. We move the arm by changing the angles at each of the joints. If we know the length of each segment, and the angles then we can start at the base of the robot, and calculate the position of the end of the arm:

when getEndOfSegementPosition

.make count number

.set displayX to 0

.set displayY to 0

.set angle to 0

.

.repeat segementNumber using count

..change angle by item count of segementAngles

..change displayX by item count of segementLengths * cos of angle

..change displayY by item count of segementLengths * sin of angle

This is called Forward Kinematics or FK, and its also used in computer animation to position and draw a characters limbs. It's provides complete, precise control. However its also a bit slow and tedious. If we want a character (robot arm) to press a button, then we need to mess around with all of the angles until we get the hand in the right place.

Wouldn't it be easier if we could just tell the computer where we want the end of the arm to be, and let it figure out all the necessary angles for us? This is Inverse Kinematics, as we're calculating the required input from the desired output (whereas in FK we apply the inputs to see the output - forwards!).

If your arm only has one or two joints then you can perhaps work this out analytically (using maths!), but if you're arm is lots of segments, then finding the best set of angles for all of them becomes tricky. Especially as there are likely to be many possible sets of angles that would all get the same result. 

One commonly used way of tackling the project is to use Cyclic Coordinate Descent (CCD):

[WELMAN C.: Inverse Kinematics and Geometric Constraints for Articulated Figure Manipulation. Master’s thesis, Simon Fraser University, 1989] . This works well, and is simple enough to implement that we can do it in Sniff.

We look at a joint somewhere along the arm and consider the position of the end effector, compared to the target. We then rotate that joint to minimise the difference between the current position and the desired position of the end effector. Because we can only move one joint, the optimal angle for the joint is always to orient the arm so that the vector from the joint to the end effector is pointing towards the target. When the desired and actual locations line up then we've done the best we can with that single joint.

Lets try that again. Consider joint B. We'd like to move C to T, but the best we can do is C'.

make jointNumber number

when updateJoint

.make sX number

.make sY number

.make dX number

.make dY number

.make desiredAngle number

.make actualAngle number

.make deltaAngle number

.

.set segementNumber to jointNumber - 1

.broadcast getEndOfSegementPosition and wait

.set sX to displayX

.set sY to displayY

.

.set dX to mouseX-sX

.set dY to mouseY-sY

.set desiredAngle to atan of (dY/dX)

.

.set segementNumber to length of segementLengths

.broadcast getEndOfSegementPosition and wait

.set dX to displayX-sX

.set dY to displayY-sY

.set actualAngle to atan of (dY/dX)

.

.set deltaAngle to desiredAngle-actualAngle

.

.set angle to item jointNumber of segementAngles

.change angle by deltaAngle

.if angle >90

..set angle to 90

.if angle <-90

..set angle to -90

.replace item jointNumber of segementAngles with angle

To do that in Sniff, we just get the end of the previous section (the position of the joint), find the vector to the target, and calculate the desiredAngle. Then we do the same, but using the end position of the arm to find the actualAngle. Calculate the difference and add it to the angle of this joint.

You'll notice that there's some code to stop angle being greater or less than 90 degrees - this is a constraint. One of the neat bits about this approach is that its really easy to tweak it, in this case to prevent the arm bending more than 90degrees at any single joint but we could have different constraints for each joint, or contain deltaAngle so that it can't move to quickly.

Of course its quite likely that we can't get to correct position by moving just one joint, so we apply this adjustment to each joint in turn:

...set jointNumber to length of segementLengths

...repeat length of segementLengths

....broadcast updateJoint and wait

....change jointNumber by -1

However note that we start at the end of the arm, and move back to the root. Intuitively you can think about reaching for something: you move your wrist first, and only move your whole arm from your shoulder when finer motions can't to the job.

Typically you'd apply this whole process a few and the arm will end up pretty much where its supposed to be!

Try it!!! The full code will be in the next release of Sniff.

The code works pretty well for a basic implementation, but it has a few glitches. Tweaking the constraints can dramatically help the quality of the final position. However the main issue is that we're using the atan function to calculate an angle from the ration of dy/dx. While its certainly true that tan(theta)=dy/dx its less clear cut that theta=atan(dy/dx). This can break, not only because dx might be zero, but also because it assumes that we're working positive numbers. We can't tell the difference between 4/5 and -4/-5 because they're the same, but the required angles are 180 degrees out!! In C we'd use atan2(y,x) which does the divide for itself, so it works reliably for all angles. The full code includes an implementation of atan2, which makes everything nice and stable.

For a real arm we might also want to do the whole thing in 3D, which is much harder, but if you've got one of the simple USB robot arms, then they only support pivoting at the base and at the grabber. The main body of the arm remains in a 2D plane. You can therefore calculate the rotation of the base required to place the target in the plane of the arm, then treat the rest as a 2D problem. I've not got one of these but it should simple to adapt this code to control one.

XKCD Velociraptor Problem

I'm always looking for fun problems to solve in Sniff, and I recently came across post on Wired,

which in turn stole its idea from XKCD. So I'm going to steal it and do it in Sniff! (seriously read the others first).

The question is how long can you survive a velociraptor attack, given a 40m head start? Velociraptors have a top speed of 25m/s and accelerate at 4m/s/s. You have a top speed of 6m/s and accelerate "quickly".

This is a fun question... not just because it has dinosaurs in it, because it ends in inevitable carnage, or even that it parodies typical maths/physics questions, but because it's actually something we can code easily - as they do in the wired article. The code actually demonstrates some of the basics of physics sims, as used for both science and games (so I get to tag this post as Algorithms, Science and Games!). 

make vX number

make hX number

make vV number

make hV number

make vVmax number

make hVmax number

make vA number

make hA number

make dt number

make t number

when start

.set vX to -40

.set hX to 0

.

.set vV to 0

.set hV to 0

.

.set vVmax to 25

.set hVmax to 6

.

.set vA to 4

.set hA to 1000

.

.set dt to 0.1

.set t to 0

We start by putting all of the information in the question into variables. Not only does this make things clearer in the code, but it means we can tweak them to try out different scenarios. Variables beginning h mean human, while variables beginning v mean velociraptor. The velociraptor starts 40 to the left, everyone starts at 0 speed. We set Vmax as per the question.

In the Wired article the interpret the human "quickly reaching your top speed" to mean an arbitrary 3m/s/s. I've chosen to interpret "quickly" in the spirit of physics textbooks where "light" is a code word for massless, "smooth" really means frictionless, and "quickly" really means "instantly". I've there for made hA=1000.

t for time, starts at zero. The standard "numeric" way to handle these kinds of problems is simply to use small time steps, so dt is 0.1.

Now the fun bit:

.repeat until not vX<hX

..change t by dt

..

..

..change vV by vA*dt

..if vV>vVmax

...set vV to vVmax

..

..change hV by hA*dt

..if hV>hVmax

...set hV to hVmax

..

..

..change vX by vV*dt

..change hX by hV*dt

..

.say [vX]

We're going to keep going while the velociraptors position is less then the humans. Each time round the loop represents a tiny fraction of a second.

Acceleration is rate of change of velocity so vV increases by vA each second. We're not looking at a whole second, so vV changes by vA*dt each time around. However vV can't be bigger than vVmax, so if it is we set it back.

Now we know the new velocity, we can calculate how far we move: vV*dt, and similarly or hV. 

With that in place we find we survive in impressive  38 seconds. Using the more conservative acceleration of 3m/s for the human the Wired use you only survive about 30 seconds. This code and the wired python code disagree very slightly as the wired code lets vV become greater than vVmax! However the answers get closer together as we make dt smaller.

This is an important point - this sort of code is only an approximation but even with simple code we get good results if we make dt small enough. Making dt bigger makes it less accurate, but your code runs faster.  The trick is to find ways to make dt bigger without sacrificing accuracy, but thats getting a bit advanced.

One of the main ways you might use this kind of code is in a game to simulate some kind of object being fired. In that case you have velocity and accretion in X and Y directions.

.set accX to 0

.set accY to -9.8

.repeat until posY<0

..change posX by velX*dt

..change posY by velY*dt

..

..change velX bu accX*dt

..change velY by accY*dt

Typically the acceleration in X is negligible, while the acceleration in Y is due to gravity. Drop this into a game and you've got your first physics engine!

Sniff Sprites Reference Material

At the game jam last week everything went really well, and we had lots of fun. We've previously produced some good worksheets (which you can download) which provide a good introduction to working with sprites. In sessions we've done, we've provided a basic empty example, and following a brief introduction to Sniff we just let the kids loose to work through the cards. That cards are numbered, but if you'd prefer to shuffle them around they work pretty well in more or less any order.

Kids can work through these examples in a few hours, and start to get the basics of a game. However when they come to make more complex games they'll find they want to do things which aren't explained on the cards. That's fine - we expected that, and explaining stuff on a "need to know" basis is actually a really great way of teaching. Kids take stuff in really quickly when they actually think they're going to use what they're being told (see how fast a kids who wants to make a ballistics game can learn about trigonometry and forces - months of teaching condensed into minutes).

However the problem with that is that is that when they "need to know" there needs to be someone who does know! In this case Michelle and I were there, and we've both worked on developing the sprite resources, so we could quickly address problems. If you want to learn more about using the sprite system we'd refer you to the examples posted here (we've added a "game" label so you should be easily able to find the relevant posts).

However we don't have a definitive reference. Sniff does come with a manual (in the docs folder!) which includes documentation for some of the arduino devices, but the sprite system is complex enough to warrant documentation of its own, so we've now written some.

Here's a reference manual for Sniff Sprites which lets all of the methods you can use with the Sprite and SpriteManager classes. Once you've got to grips with the basics, this is where you should look to find out the exact details of what features are available.