The Daredevil Camera - seeing the world in sound

[Artlav]

Once upon a time a heard about a lens that could focus sound.
Ever since then i wanted to have an ability to see sound.
Imagine using such lenses to focus sound onto a plane of microphones.
Just like light in a camera.
One microphone is one pixel.

A decade and several concepts later, i finally made one (albeit with FFT instead of lenses).
http://www.ribbonfarm.com/2016/06/29/the-daredevil-camera/

"Show off your project" part aside, can anyone think of a practical application for a something like that?
Many people tell me to do a Kickstarter for it, but i just can't imagine any sort of a product out of it.
I'll probably just release in as an open-source project, once all the bugs are ironed out.

Any ideas on how to improve on this are also welcome.

[mikeselectricstuff]

Someone was talking about precisely this as an potentially interesting FPGA application recently on embedded.fm , I think it was episode 155 : http://embedded.fm/episodes/155

Not sure about useful applications, but I'm sure people would find it interesting, especially with a realtime display.

If you've already got it to the point of working, I'd say it would be worth costing up production of a batch of maybe 100 and trying it on kickstarter.
Submit it to hackaday - I bet  there will be some interest

[rs20]

So, this is Passive SONAR? Phased array?

Also, did you consider having the microphone arranged not in a rectangular grid? I have considered doing something similar in the past, and I concluded that a high frequency at an angle sufficient to induce a 360 degree phase shift on adjacent microphones will induce 720, 1440 etc on other microphones -- thus appearing to be dead ahead. I always imagined that I'd place my microphones pseudo-randomly, even if it slightly increased the complexity of analysis.

[mikeselectricstuff]

How about putting a directional ultrasonic transmitter in the middle and looking at the reflections?
Or an array of emitters.

As regards possible improvement, one thought is depending on the pitch,  it may be cheaper to do the mic arrays as strips, to improve PCB panel utilisation, and maybe make it more modular for experimenting with different array sizes.

For location of a single source, I wonder if a cross array may be effective as a lower cost option.

[mikeselectricstuff]

Another reason for kickstarting a batch of boards is that before sending them out you could do some experiments with  a really large array!

As regards getting data out quickly to a PC - usb3 is probably the answer  - FTDI do a USB3 interface chip.
Alternatively Gbit ethernet - quite commonly used for industrial vision cameras
 

[Artlav]

If you've already got it to the point of working, I'd say it would be worth costing up production of a batch of maybe 100 and trying it on kickstarter.

Not exactly - right now it stores on a bunch of microsd cards, which makes this barely practical.
I'll need to make a real-time display board first.

BTW, do you have any advice on how can you get 16 of 4 Mbit/s data streams into a PC?
I left a 3-pin header on the cell for that purpose.

Submit it to hackaday - I bet  there will be some interest

Should i personally?
These projects tend to find their own way to such places.
I.e. there is a thread on Hacker News - https://news.ycombinator.com/item?id=12004866

So, this is Passive SONAR?

Not exactly, but i guess it's similar to some extent.
It's not optimized for locating sound sources.

I always imagined that I'd place my microphones pseudo-randomly, even if it slightly increased the complexity of analysis.

Not sure how that would help - you'd still get the same aliasing issues, only with a more complex processing pipeline.

How about putting a directional ultrasonic transmitter in the middle and looking at the reflections?

Yeah, i was thinking about using a white noise "flashlight" to illuminate the scene, but it would need some work before the result is intelligible - i.e. a regular camera at the middle of the array synced a-la the visual camera in the FLIR thermal cameras.

it may be cheaper to do the mic arrays as strips,

What do you mean by strips?
8x1?
Back when i did the math, the cheapest option was the 8x8, since it would take the least amount of parts at most utilization.

For location of a single source, I wonder if a cross array may be effective as a lower cost option.

Perhaps, but i'm after images.
To see the world in sound, not to locate a gunshot in a city.

[mikeselectricstuff]

I'll need to make a real-time display board first.

Make it drive the cheap Chinese RGB LED panels designed for video walls - insanely cheap and available in a variety of shapes & sizes with a fairly standard interface.
And/or an LCD. Or VGA out.

BTW, do you have any advice on how can you get 16 of 4 Mbit/s data streams into a PC?

USB3 - look at the FTDI FT60x devices. Or raw packets on Gige - The Chinese video wall controllers for the LED tiles use gigabit Ethernet for links - I think they use an off-the-shelf GigE controller to send raw ethernet packets.

Submit it to hackaday - I bet  there will be some interest

Should i personally?

These projects tend to find their own way to such places.

Yes but a little push wouldn't hurt

it may be cheaper to do the mic arrays as strips,

What do you mean by strips?
8x1?
Back when i did the math, the cheapest option was the 8x8, since it would take the least amount of parts at most utilization.

Depends on the spacing as to how worthwhile it is.
For example if the array is 20mm pitch, then you could make, say,  10mm wide PCBs and get twice as many on the panel, saving bare PCB and assembly costs. The strips can probably also be two layer.
As probably only the FPGA needs 4 layer, this could be a major cost saving, especially for larger spacings
If you can figure out an interconnect/mounting scheme that doesn't cost more than the cost saving then it may be worth a serious look.

[mikeselectricstuff]

One tip with the SD cards - you may find that if you pre-erase every sector with FFs first before starting a recording, you may get more consistent write performance, as one reason for lumpy throughput is erase delays.
 

[NiHaoMike]

I remember reading in a science book that a balloon or beach ball filled with CO2 can be used as a lens for sound waves.

[tombi]

In terms of applications -

Could you overlay the image onto a light image? Like a thermal camera does.

Then (assuming the resolution is high enough) you could use it to pinpoint the source of noise. For example you could find a noisy bearing or fan belt on a motor. Or could you visualize the sound reflections in a room when setting up speakers. How about locating a buzzing component on a SMPS PCB. What about locating and measuring the major sources of noise from a computer (i.e. which fan is making the noise).

Tom

[ivaylo]

There is this - http://smins.co.kr/html/en/product/product_0201.html (discussed here before). If you manage to cut the price in say three, I say go kickstarter...

[mtdoc]

Really interesting project.

I wonder if deaf people would find it useful?

[bookaboo]

Neat project I've seen some people use commercial ultrasonic listeners (or at least some sort of sonic pickup) for air leak detection in pneumatics and vibration analysis. Maybe there's some niche industrial application in there.

Failing that would make an interesting interactive exhibition if pointed at an array of instruments.

[mikeselectricstuff]

Regarding the data rate to the host, to what extent could this be reduced by doing at least some initial process in the FPGA ?
Presumably depends on the size of FPGA fitted. I'm not familiar with the Altera range but it would be good to have the option to fit different size devices.
I think it's probably still worthwhile to have a means to get high bandwidth data so you can experiment with algorithms to later bake into the FPGA, e.g. for a standalone display.

Something that may be worth exploring if you want to look at making these - Altera seem to be trying to get on The Maker bandwagon (they were at Maker Faire SF), so there might be some scope to get a good deal on the FPGAs if you can get in touch with the right people there. It's a good example of something that pretty much has to be done on an FPGA, so could be a good showcase product, and would also be a good exhibition demo piece. 

Another thought - is there an existing reasonable-cost Altera devboard with USB3 and enough IO that you could just do a microphone array board to plug into it?
 

[mikeselectricstuff]

Neat project I've seen some people use commercial ultrasonic listeners (or at least some sort of sonic pickup) for air leak detection in pneumatics and vibration analysis. Maybe there's some niche industrial application in there.

Failing that would make an interesting interactive exhibition if pointed at an array of instruments.

Imagine a big mic array in front of an orchestra, doing realtime visualisations projected on a huge screen....

[Brutte]

That is an interesting project.

BTW, do you have any advice on how can you get 16 of 4 Mbit/s data streams into a PC?

Make it modular. I'd use an USB Hi-Speed bridge.
Either use a brain-dead FT232H ($3 a pop) or a custom uC (STM32F20x + ULPI or similar). The throughput is around 40MB/s (~320Mb/s).
Then you can buy a PCIe USB 2.0 host card (usually 4 hosts/card), I think a motherboard with 4 slots is not particularly uncommon. So that gear would allow pulling in 640MB/s

[rs20]

I always imagined that I'd place my microphones pseudo-randomly, even if it slightly increased the complexity of analysis.

Not sure how that would help - you'd still get the same aliasing issues, only with a more complex processing pipeline.

Did a quick sim, here's the PSF for a dead-ahead source with a 16x16 grid:



And here's the PSF with 256 randomly placed mics within the same square area:



I don't think the aliasing issues are the same.

EDIT: After all, if a 2x2 grid has aliasing issue, then adding further mics on the same pitch, they're not going to help! If 2 mics have a 360 degree phase difference, then adding a third on the same pitch gives you 720 degrees, not disambiguating the situation at all. The additional mics sharpen the image, but sharpen the aliased images as well. Randomly placed extra mics sharpen the image, but also kill the aliases (albeit at the cost of a higher "noise floor", although with 256 mics that's very low in any case). I'm not necessarily adovcating random placement as the best approach, there's probably a more principled/structured approach that is better. But random appears to certainly be better than a regular grid.

Regardless, I think you should continue with the grid approach for now since you already have it built, and I'm very excited to see the results!

EDIT: Fixed broken image link.

[ruffy91]

Aliasing is only a issue if the signal is purely periodic and harmonic.
As soon as it is non-harmonic or non-periodic (noise for example) you won't have aliasing.
So distributing the array random would allow to see more kinds of sound sources.

[mikeselectricstuff]

Some application ideas :
bat/bird/animal watching - imagine a view of some trees, where every bird sound marks the position of the bird, or zooms in on it.
Insect seek & destroy - guide a laser to zap mosquitoes etc.
Diagnosing mechanical/vibration issues (as shown in earlier link to commercial unit)

Tracking popele on stage etc. using ultrasonic beacon - might have some advantages over IR
Tracking ball path e.g. tennis, using wind noise from ball going through air

[andreymath]

Dear Artlav,

here is a practical usage of your invention:
track of the direction of the sound source.

It can potentially become a sound radar :-) :-) :-)
It will require some scientific/empirical research of the measurements......

Good luck,
 

[LaserSteve]

Oh, I hate to set you off onto this quest, but to make that an effective imager, you need two things to make your life easier.  A precise  means to move it along the Z axis, perhaps up to two meters or more,  and a local "coherent" reference sound.   Zone plates and Gas Balls also come to mind, but more as test sources rather then as lenses.   

Virtual TDOA by scanning the data off SD cards needs too much processing power. Adding the local coherent reference, allows the microphone diaphragm to become a mixer and phase detector.

I used to work for a  company that had world records in directivity and intensity of sound.  That lasted a year.. I had started on a similar scanner, but done with far less elements. You really only need a sparse array if you have some control over what your imaging.

When I worked for the company that did CT imaging, I had to learn  that for certain modalities of imaging, it was best to realize that each image involved exactly the same series of software processing steps, and rather then use a general purpose computer, small arrays of FPGAs were used.
In the past just boards full of TTL were used, mimicking certain processor instructions on a parallel, repetitive,  basis. On a modern CT machine, 6 Gigabytes per second of data were coming off the sensor down four fibers in parallel, no single processor made could handle that pipe.  FYI, the A/D converters for all 512 by 64 sensors  were all current to frequency converters,  which is the easy way to break the A to D barrier without buying massive arrays of flash AD converters. Its very easy to measure period over six orders of magnitude using an inverse counting scheme with some interpolation...



I might upload something tomorrow, if I can find the book in the piling system.    Your on to something very useful, and you don't know it yet. I cant find a good copy of Winston E. Kock's images on line, but I have a feeling you'll improve upon them. Dr. Kock's method used photo film as an integrator and storage medium. He imaged both near field  sound and microwaves using a swinging arm, a neon lamp, and a detector.  It looks like garbage until you add a local phase reference. These days that reference could be a noise  sequence as well, but um, getting busted for ITAR scares me some times.

Steve 

[mikeselectricstuff]

Here's another application - monitor the audience at quiet concerts and point a STFU gobo-light at the inconsiderate gits who insist on talking through the music.

[mark03]

Great project.  Now you are starting to get an idea of the complexities faced in 3D ultrasound imaging... just imagine a 64x64 (or larger) array, with each channel sampled at tens of MHz!

This is a perfectly ordinary sampling problem, only in three dimensions (x, y, and time) instead of one.  As in textbook 1-D sampling it's the highest frequency that matters.  The highest spatial frequency is 1/lambda (reciprocal of wavelength), attained when the source is 90 degrees off to the side.  A uniform grid with pitch < lambda/2 = c/(2*f), where c is the speed of sound in air and f is the highest [temporal] frequency component, will allow a perfect reconstruction of the continuous wavefield on the aperture (thank you Nyquist).

If the grid spacing does not meet this constraint, there will be aliases at certain angles, which are called "grating lobes".  If the individual microphones have some directivity of their own, this can be a fair trade-off, because the grating lobes can be partially suppressed by the inherent fall-off in microphone response, while allowing a larger aperture for the same number of microphones.  Larger aperture == better resolution.  Resolution in the far field should be on the order of lambda/D radians where D is the aperture size.

Random placement can be advantageous when the aperture is subsampled and you have undesirable grating lobes.  It will act to spread out the grating-lobe energy across the image, instead of being concentrated in the aliases.  Data processing is a lot less elegant and fun, though... no more FFTs for example.  I would recommend against it.

BTW if you want to save on the # of microphones you can theoretically get away with just one and reconstruct using synthetic-aperture techniques.  This is normally done in a pulse-echo system where you send out a signal, listen to the reflection, move, then repeat.  Of course it assumes the target is not moving/changing over the course of the experiment, and accurate knowledge of the tx/rx unit's path.  In grad school I wanted to try this rolling an equipment cart along the sidewalk in front of a tall building, setting off a firecracker every ~ 10 cm 

[m98]

Uhm, probably I got something completely wrong, but why couldn't you do the same thing with a binaural microphone and some "magic mathematics stuff" to exactly resolve the sound sources based on phase shift, amplitude and blauerts frequency bands?

[LaserSteve]

I found some of Dr. Kock's work on line.  Keep in mind this man worked on Microscopy, Beamforming,   Sonar, Radar, Microwaves, Loudspeaker Drivers, Crossover Networks,  and Early Music Synthesizers, as well as other things. He spent quite a bit of time at Bell Labs, as well. That man designed electronic keyboards in 1936!

http://blog.modernmechanix.com/mags/PopularScience/9-1950/sound.jpg

http://www.spirit-science.fr/doc_musique/FormesIMAG/07Kock.png

https://www.google.com/search?q=koch+acoustic+lens&biw=1440&bih=752&source=lnms&tbm=isch&sa=X&ved=0ahUKEwjqiI3pxNDNAhUIQSYKHeYlBqs4ChD8BQgHKAI#tbm=isch&q=winston+koch+acoustic+lens

His publications, patents,  and work with Dr. Harvey are amazing.

Off Topic:
http://120years.net/category/name/winston-e-kock/

Steve