Facilitated Echolocation

[longh]

Hi everybody, this is my first post!

Ever since college I've had a project in mind, and I have never figured out how to actually implement it. Here is a video of me talking about the project back then:



And here is my old project page:

https://sites.google.com/a/vt.edu/amp_lab/projects/facilitated-echolocation

Long story short, I want to see if humans' natural ability to navigate with echolocation could be improved with technology. People can in fact navigate by clicking their tongue and listening for the echos off objects. However, compared to animals that are good at echolocation (bats come to mind), we really stink at it. 

What my project would attempt is to do is replace the click of a person's tongue with an audio pulse from a tweeter or ultrasonic transducer. Also, I would use microphones with a high frequency response to record the echos at a high sampling rate (~100kHz), store the audio in a buffer, and then immediately begin playing the echos back to the user through headphones at a lower sampling rate (~10-20kHz).

Below are the actions that would take place when a pulse is sent out.

At time t=0:
    A loud ping is transmitted from an ultrasonic transducer (this part seems pretty easy).
    The ADC begins recording audio from two microphones at a high sampling rate (~100kHz for each channel) into a buffer.
    A DAC begins outputting the audio from the buffer at a much slower sampling rate (~20kHz) into a pair of headphones.

At time t=0.1s:
    The ADC stops recording new data

At time t=0.5s:
    The DAC has output all of the samples and stops

At this point, the user can press a button to initiate the transmission of another ping.

The main difficulty I have is figuring out how to sample audio at such a high frequency and store it in a buffer. I just purchased a Mojo FPGA board and am hoping to use it to shuffle data from the ADC to some kind of memory, and then from the memory back to the DAC. But I am totally new to FPGAs.

Previously I did get this working by buying a fast sound card with a 192kHz sampling rate and just recording the pulse with a laptop. However, this required me to press record, stop the recording, and then play back the audio after converting it to a slower speed.

I want the system to be real time and portable, hence me buying the FPGA. I envision that a user would only have to press a button, then they would immediately start hearing the echos at a reduced sampling rate through their headphones.

My questions are:
1. What kind of ADC should I look into
2. What kind of memory would be best
3. Can the Mojo FPGA handle this task, or is there something better suited
4. what kind of DAC should I look into?

Thanks so much for your help!!
-Hunter

[PA0PBZ]

Did you ever think about down converting the ultrasound (like a radio down converts the received signal to an IF frequency)? It would be a lot easier to build, see for example bat detectors.

[longh]

Yes, I did consider doing this. However, I want to slow down the audio not only because I want the ultrasound to become audible, but also because I want to increase the time between the arrival of echos.

For me, if I make a clicking sound (hitting two small rocks together is a great way to do this) I can hear echos from objects at least 5m or so away from me. However, for closer objects the echo reaches my ear so quickly that I can't distinguish it from the initial click. My hope is that by slowing everything down by a factor of 5 or 10, my brain's range resolution would also be increased by that same factor (from a minimum of 5m to a minimum of 1/2m).

[janekm]

Also consider checking out the mmWave chips TI came out with recently: http://www.ti.com/sensing-products/mmwave/overview.html (the industrial ones are cheaper than the automotive ones). Converting the output from these chips to audio could give a really interesting “picture” of the environment.

[longh]

Cool, I'll have to look into those. For now however, I want to get this audio system working.

So for the ADC I'm thinking maybe this AD7357: http://www.analog.com/en/products/analog-to-digital-converters/precision-adc-20msps/simultaneous-sampling-ad-converters/ad7357.html#product-overview
For the memory, I wonder maybe flash would be best?

[phil from seattle]

There are lots of ways to indicate distance other than time delay.  An audible frequency, for example. Also, you don't need to be linear if you use delayed sound so you can increase sensitivity in the more important distances.  Using some sort of actuator that presses the skin could be better as it wouldn't interfere with hearing.

[longh]

Yes, that is true. However, I want to try it this way first. It seems like the main difficulty will be writing to memory fast enough from the ADC.

[dmills]

I have seen blind people doing echolocation by snapping their fingers.

No real need for an FPGA to do this, a modest micro is easily capable (But it will need a modest amount of ram and two I2S ports that can be setup to be clocked independently, an alternative is one port but then you will need to record for 0.1 second then change the rate and playback).

100kHz * 0.1 seconds * 2 bytes per sample = 20kB * 2 channels = 40kB, not a lot by today's standards.

A small ARM dev board, so analogue doings and then it is a small matter of typing.

Regards, Dan.

[StillTrying]

There's a few 10 second sound recording chips out there, perhaps you could get one to playback at half speed.

[longh]

Thanks for the input! So dmills, can you help me find such a ARM dev board that has the independently clocked I2S ports? I'm not really familiar with ARM devices at all, I've only used AVR microcontrollers. Also I've never used I2S before.

I tried looking on Digikey for STM32 development boards, but there are a million to choose from! https://www.digikey.com/en/product-highlight/s/stmicroelectronics/stm32-kits

[RandallMcRee]

There is a recent crowd supply offering for an open oscilloscope. It incorporates an FPGA. Your requirements seem well within its capabilities.

https://www.crowdsupply.com/andy-haas/haasoscope

I think going with a platform such as this will be a lot more useful to you than futzing with your own A/D. Great idea, don't get derailed with overly complicated hardware.

[mark03]

Most of the new data-converter products coming out of Analog Devices and Linear Technology (actually the same company now) these days are fast successive-approximation (SAR) ADCs running in the low MHz range.  I use an LT2378-16 in my DSP-based bat receiver and it works fine, but there are lots of choices.

Any reasonably fast ARM Cortex-M MCU will have enough horsepower to process the samples, with an I2S peripheral built in for driving an audio DAC/codec.  I am using the STM32L4 series, mostly because I am familiar with that family from work projects.  You could pick up one of STMicro's "Discovery" boards which has the audio DAC already on board, then interface the ADC using spare pins.  (You might have to remove some parts to free up the pins you need; those boards are really packed with gizmos.)  Coming up to speed with a modern 32-bit MCU and its toolchain will involve a significant learning curve, but it is still easier than dealing with an FPGA and its toolchain, unless you have prior experience steering you in that direction.

You should have no trouble finding an MCU with enough on-board RAM to store and play back your recorded echoes.  As an example, the STM32L496 family have 320 kbytes, which is enough for more than one second with 16-bit samples.  Your echolocation range is 170 meters per second of recording, so you don't need more than a couple hundred milliseconds, do you?  Flash memory would be too slow anyway.

BTW, use caution when sourcing transducers for your project.  "Ultrasound transducers" for use in air are often narrowband, 40-kHz units.  You want broadband for echolocation, obviously.  I have found these to work quite well for speakers, although they are a bit expensive in the US:
https://www.kemo-electronic.de/en/Car/Speaker/L010-Piezo-Loudspeaker.php

For microphones, if you aren't too afraid of surface-mount packages, it appears that most bottom-port MEMS microphones have good response up to ~ 100 kHz.  Almost none of them are characterized above 20 kHz, but they work.  I am using an ICS-40720 in my DSP bat receiver.

Cool project.  The only real problem I can see is that you lose a lot of the directional cues normally provided by your ears and head, when you listen with headphones or earbuds.  This is why people with hearing aids often have trouble localizing sound and separating sources in a crowded room (the "cocktail party effect").  That could give an edge to the all-natural, tongue-clicking-plus-ears system.

[longh]

Thanks for the info!

Not having experience with 32-bit MCUs, the sheer number of options seem overwhelming.

Would you recommend this board from Digikey?: https://www.digikey.com/product-detail/en/stmicroelectronics/STM32L496G-DISCO/497-17136-ND/6697038

Here is the description of the board from ST's website: http://www.st.com/en/evaluation-tools/32l496gdiscovery.html

For this board will I need to purchase any additional tools or software such as a debugger or IDE?

It is pretty pricey at $73, but it is nice that it comes with an audio DAC on board.

In the past I used some Knowles MEMS microphones that worked very well. I also used piezo tweeters for the audio generation.

I agree about the loss of directional cues. I tested out how it would sound using Matlab, and when the playback speed is slowed down, sounds that are slightly to the right now sound directly to the right of you because of the increased delay. One kind of hokie idea I thought of is to place the microphones in a fake "bat's head". Basically a small foam ball 1/10 the diameter of my head so that when the audio is slowed down by a factor of 10 the delays make sense to my brain. Even if this works though and can allow left/right cues to be interpreted, you still won't have the ability to discern up/down cues which come from the way the sound reflects from your outer ear.

[Marco]

Why not just use a piezo ultrasonic transducer pair? You can simply do envelope detection on the received signal, so you don't need to work at high frequencies.

[longh]

This is a valid point. I would say the reason I don't want to do it this way is that I want to hear it with my own ears. I know that there are a ton of ways to measure distance with echolocation, but I want to use the computer between my ears! 

[Marco]

That's not what I mean, you can still do the time stretching and remodulation ... but you do it using the envelope signal from the ultrasonic receiver.

[longh]

Oh I see. So you could extract the envelope time series, stretch it out, then multiply it by a 1kHz carrier that you could easily hear. That is a good idea!

[mark03]

Would you recommend this board from Digikey?: https://www.digikey.com/product-detail/en/stmicroelectronics/STM32L496G-DISCO/497-17136-ND/6697038

That is pretty expensive for a Discovery board.  It really depends how you value your time vs your money.  You'll be spending many many hours getting to grips with the IDE and the processor, beside which $75 pales in comparison.  But you could also go cheaper.  ST basically has the Discovery boards, with lots of on-board peripherals like audio codecs, sensors, and sometimes a display, and the Nucleo boards, which are a little bit closer to "breakout boards" where there is less stuff included.  Both have the ST-LINK debugger so you can hook them straight up to a PC without needing to buy your own JTAG adapter.  The discovery boards can be helpful when you are new at this stuff, because you can try out their examples and play around with the on-board peripherals.  But if you want to interface your own hardware, you may not have a lot of pins left over.  For my bat receiver I used the STM32L476 discovery which I think was only $20.  But I had to unsolder the LCD and hack up some pin headers to get access to enough of the MCU pins for my project.  Now I would pick a Nucleo board instead.  Actually I just picked up the '496 Nucleo for my next project...

For this board will I need to purchase any additional tools or software such as a debugger or IDE?

You can set up a free toolchain and IDE based on gcc and Eclipse.  This will work just fine, but getting it set up properly can be quite challenging.  Various people have written up how-tos or offered custom plug-ins for Eclipse to make things easier.  Then there are hybrid free/commercial offerings like Keil (? not sure, never used it) or vendor-provided packages like NXP's LPC Xpresso.  Then the straight-up commercial offerings like IAR for $$$ but often with no-cost versions for limited code size.  Search the Microcontrollers and FPGAs forum, where this question comes up regularly.  Opinions are varied and held with degrees of religious fervor

[mark03]

That's not what I mean, you can still do the time stretching and remodulation ... but you do it using the envelope signal from the ultrasonic receiver.

Doesn't this pre-suppose a narrowband signal to begin with, like you'd get from those 40-kHz transducers?  I guess they are used for rangefinding...  but it seems to me this would be more or less equivalent to working down in the audible range to begin with.  You could just make audible (loud) clicks and then slow those down, no ultrasound required.

[Marco]

The small bandwidth of the receiver will smear the response of an individual reflection, but it won't time stretch the response as a whole.

[longh]

I think I'll go with the Nucleo board, looks a little less intimidating 

https://www.digikey.com/product-detail/en/NUCLEO-L496ZG-P/497-17227-ND/7065918

mark03,
You said you used a LT2378-16 ADC in your design. Did you use the $200 evaluation board they sell, or did you have to layout a PCB for it? I guess I could get the one that comes in a MSOP package and use a surfboard to make a prototype.

Here is the MSPO LT2378-16: https://www.digikey.com/product-detail/en/linear-technology/LTC2378CMS-16-PBF/LTC2378CMS-16-PBF-ND/2749246

It says that daisy chaining is possible, so I guess I could connect 2 in this way. Now I just need an audio DAC since the Nucleo doesn't have one.

[xani]

If you already plan to use a ton of processing power on it, maybe do a time-freqency mapping instead ?


Send a ultrasonic pulse, detect it and then map time of arrival (time since sonar pulse) to the frequency outputted ,and amplitude to amplitude + length of output signal, then just stretch the length of that

So object at 10cm would generate 4k, one at 50cm would generate 2k, one at 1m would generate 1k etc.

But having object at 1m and another one at 50cm would generate two-tone signal, with each tone's volume depending on object size (or rather how much it managed to reflect)

[mark03]

You said you used a LT2378-16 ADC in your design. Did you use the $200 evaluation board they sell, or did you have to layout a PCB for it? I guess I could get the one that comes in a MSOP package and use a surfboard to make a prototype.

Here is the MSPO LT2378-16: https://www.digikey.com/product-detail/en/linear-technology/LTC2378CMS-16-PBF/LTC2378CMS-16-PBF-ND/2749246

It says that daisy chaining is possible, so I guess I could connect 2 in this way. Now I just need an audio DAC since the Nucleo doesn't have one.

I did not get the ADC eval board.  For something relatively straightforward like this (and being a hobby project) I usually just design a PCB, sleep on it, go back and review carefully, and take my chances.  I'm attaching a schematic here, but keep in mind I had some unique requirements.

Driving the LTC2378 serial bus was tricky with the STM32L4 because the MCU's SPI peripheral is pretty limited.  I ended up using one of the timers to generate the right number of SPI clock pulses every time it sees the conversion-complete signal edge from the ADC.  The MCU SPI is then configured in slave mode.  This was the only solution I could find to have the ADC send data into a DMA buffer in the background.  That, in turn, was pretty important to me as I wanted to run up to 1 MSPS and still have plenty of CPU cycles for downconversion and filtering.  You could probably get away with an interrupt-driven solution, which would make things a lot simpler.

IIRC Linear/Analog have some fast SAR ADCs that come in dual or quad configurations.  You might try their selector tool and see what pops up.

BTW xani has an interesting idea---that is a lot like automotive radars.  They transmit a sawtooth chirped FM signal, and multiply that same signal by whatever they receive back.  Result is the multiplication of time-offset chirps, which is a constant tone whose frequency depends on the round-trip delay.  An acoustic version of this would be fun to play with.

[mark03]

I just had another idea.  You might consider dropping the external ADC completely for a first prototype.  The STM32L496 has two built-in ADCs with ~ 11 effective bits SNR.  They will run at ~ 5 MHz sampling rate, so you can oversample and get back some SNR.  The ADC peripheral has a built-in averager (up to 32x), but to do it right you can turn off the averager and pipe the ADC samples into the DFSDM (cascaded integrator comb filter).  There is a built-in data path for this on the '496, and you can decimate up to 80x with the sinc^3 filter with input in the 12 LSBs.  That should buy you an extra ~ 2.5 bits, which brings your SNR pretty much in line with the best MEMs mics.  Then all you need is one op-amp gain stage to match the microphone clipping level with the ADC max input voltage.  Shouldn't need much in the way of anti-alias filtering, just a simple RC.

[longh]

Wow, thanks everyone for the help!

So I ordered this board from DigiKey: https://www.digikey.com/product-detail/en/stmicroelectronics/NUCLEO-L496ZG-P/497-17227-ND/7065918

It's the  STM32L496ZG-P Nucleo dev board.

I found this tutorial for getting the STM32F4 toolchain working on Ubuntu: http://vedder.se/2012/07/get-started-with-stm32f4-on-ubuntu-linux/
I have a feeling that it might be a little bit easier to get the toolchain working with Linux than with Windows. I keep reading about how difficult it is to get an open source tool chain working with these boards, why is this? I know with simpler AVR chips the toolchain was super easy to setup. Just needed a notepad and AVRDude.

I like the idea of using the onboard ADC. Now I just need to find a suitable DAC for playing out the audio.