Open source IR motion tracking - what would be the proprer image sensor

[Tomek]

Hi Guys.

[this is my first post by the way]

Introduction (probably just skip it)
I'm thinking about replicating a device called Track IR (http://www.naturalpoint.com/trackir/). Basically it's a game controller that consists of "special" camera and a clip for your headphones that has 3 IR leds. This allows to track your head movements which are replicated in a first person perspective games. Its particularly useful in simulation or racing games like MS Flight Simulator or Formula 1. Its really cool stuff.
There are some open source alternatives like FreeTrack http://www.free-track.net/english/,  but they are based on regular web cameras (or WiiMote) which introduce notable delay (piping full image through USB and processing it on PC) this in turn introduce motion sickness and vomiting .
Track IR has 1080p@120FPS image sensor and (most probably) FPGA that processes the image inside the device and sends only processed data down to PC (x & y cordinates of image of ir leds).

I have some experiance with web cam solution. I could buy original TrackIR, but I thought it would be much more fun to build proper one by my self .
Open source version might have more versatile application than just head tracking. Ha! It could track any thing.

Here comes technical stuff
I got stuck on very first step in my research - the image sensor.

All i know, is that this project requires camera with following characteristics:

High Resolution - high precision of movement reproduction
High FPS - allows smoothing of movement without introducing significant delay
Low Color depth - 1bit per color would be great if controlling of threshold was possible.
RAW output - ease of processing
up to $15 per piece - its meant to be Open Source
 
Of course when it comes to image sensors bandwidth is main limiting factor. thus I would like to trade color depth for HiRes and HiFPS.
The problem is  that most cameras that i can find on digikey (or any where else) allow for trade of  HiRes vs HiFPS. The "best" case scenario is that camera ic has some codec build-in to compress output.

so my question is: Is there any Image sensor that meets those requirements? Where I should look for it besides usual suspects?

thanks

[Legit-Design]

CastAR from http://technicalillusions.com/ is doing sub millimeter, "0.07mm tracking", with really cheap cellphone camera modules, I think they quoted in the range of 1-3$ price for a single camera module. They are doing it now with really small fpga, which is also doing other things, but they are going towards metalized gate array, which is a custom chip. I think they have patented (or will patent) the sub millimeter tracking with optical tricks, will that prevent it to be used in open source thing?

http://technicalillusions.com/?incsub_wiki=tracking
The fpga will process camera information. No need for high bandwidth tracking information, since it only sends the location in all degrees.

The way Jeri explained the leds in one of the videos is that they are just modulating them. One tracking pad on the table has 5 leds, four leds each corner, one led will tell the camera which is the corner to start from.

[Tomek]

Thanks Legit-Design
I totally forgot about CastAR. if they did that, it means its doable.  That's definitely a place to look for some answers.
BUT ( ) When I was talking about bandwidth limitation I meant throughput between camera and FPGA.(sorry, that probably wasn't clear). It's the limitation of how fast camera can push out data and what data it pushes.
I steel think that proper camera is the secret ingredient. I doesn't mean that it has to be something special. Its just a matter of feature set. I can choose from thousands  of image sensors, but without access to all data sheets to dig into minutia of particular devices capabilities I won't be able to achieve good quality tracking.
Its like Oculus Rift guys not sharing info about screens they are researching, or part numbers removed from components in oscilloscope in one of Daves videos.
I think proper part is the key here.

Any way, CastAR is definitely a great trail to follow. Thanks again

[hagster]

I worked on a project that did this about 15 years ago using some basic logic chips. We had an analogue bw camera and ran a threshold detector on the output. Using a fast timer we measured the time between the sync and the first bright spot. Worked pretty well but we could only track on it led. Also the led had to be the brightest thing in the scene. Should be easy with modern hardware.

[Legit-Design]

Like Jeri told on one of the CastAR interviews they are also using motion prediction with their technology. Since humans move in a more natural way, any movement will start to slow down before it will stop, so they actually know where subjects head is going to land before it actually lands there, same thing when movement starts.

[hagster]

You can't really predict the motion. If someone keeps their head still for 5 minutes how are you going to know when they will move it to the nearest millisecond.

What you can do is bound the likely acceleration. Jeri is probably using a Kaplan filter to model the process. This is how your satnav smoothes out it's measurement noise. It can predict when you will apply the brakes, but it knows that your decelleration will be bounded by the grip of your tires.

[Legit-Design]

You can't really predict the motion. If someone keeps their head still for 5 minutes how are you going to know when they will move it to the nearest millisecond.

That is not what I meant, you are being just silly.
Even if you know few milliseconds before where the head is going to land with reasonably probability it's still motion estimation/prediction. Point is not to know where it's going to land 5 minutes from now because that would be just silly. Head has a mass, mass doesn't usually accelerate and decelerate suddenly.

Since humans move in a more natural way, any movement will start to slow down before it will stop, so they actually know where subjects head is going to land before it actually lands there, same thing when movement starts.

[hagster]

Yes sorry, didn't mean to sound like I was having a go. You are exactly right that it's all limited by the mass of your head.

 The term prediction to me makes it sound like they are somehow second guessing your descisions. Estimation is a far better word to describe it. You can easily extrapolate the current trends into the future to make a prediction. The trouble is if you get it wrong you end up overshooting. To get the best results you need to model the dynamics of your process. The Kalman filter (correct spelling this time) is probably the optimum system for doing this. It is also really simple to impliment and is computationally very lightweight. Well worth anyone looking up if they need to do this sort of thing. As usual it's one of those things written by a mathematician in a special way to make it seem impenetrable. In fact it's really simple and can be done in 3 lines of code.

[Tomek]

Like Jeri told on one of the CastAR interviews they are also using motion prediction with their technology. Since humans move in a more natural way, any movement will start to slow down before it will stop, so they actually know where subjects head is going to land before it actually lands there, same thing when movement starts.

That's Interesting. I suppose they using it when dealing with quick and vast moves. Although considering that reasonable sensor should operate at 120FPS that shouldn't be necessary.

PS. Sorry for delay. "day" job ...  

[hagster]

It probably helps with accuracy. I guess the IR LED image on the camera spreads across a few pixels. They probably have to do some clever tricks to find the center of mass to get the accuracy. The filtering might help with this.

I wonder how they demodulate the code from the LED when it's moving accross the screen.

[xuio]

I wonder how they demodulate the code from the LED when it's moving accross the screen.

They could identify the LEDs while the image is steady and then just track position changes