Real Time Pitch Shifting

[moffy]

I have been able to run a speed and memory test on a real time pitch shifting algorithm on an iMXRT1020EVK development board. It comes with an audio codec installed as well as the requisite USB. I used a USB Speaker/DAC example, and inserted the Pitch shift code to execute every 1024 samples to see if it could keep up in real time, while playing back a WAV file at 48kHz from the PC. I first had to fix some memory issues, because of the way the chip partitions it, but once that was done I was the able to run the code. The processor seems to have plenty of capacity left even while running the code.
The samples when processed are just dumped at present, (exact same code was run on the PC to process a WAV file to verify it), I still need to implement a sample rate converter (I have suitable code) then output it and synchronise it to the codec clock, probably by counting samples left in the buffer. I am amazed at the capabilities of the processors these days, and they only cost a few dollars.
I'll update as things progress.

[moffy]

Well I have added a cubic spline resampler to the timing loop and the processor still appears to have excess capacity, so it looks like this might actually work. Once, if, I get it working, I'll test it out on the Baptist church auditorium I attend, and see if it provides any remedy to the fairly bad feedback issues. I'll try various pitch shifts and listen for artifacts, while no one is there. Not sure what the results will be but a 6db improvement would be good.

[LaserSteve]

Anglican Church sound guy here.

If the sweeping pitch shifter helps but does not get you to 100% improvement, try the following:¹

 "Break" the corners of the auditorium for midrange with panels
of Owens  Corning 703 sound absorbing  glass fiber.  Back the panel frame with 1/8th or 1/4 inch plywood or MDF, and run a few stringers down the side and along the top as a frame. Have an air gap between the glass fiber and the back panel of say 1/8 inch.  Then when hanging the panel, have an air gap of 1/8 to 1/4" between between the top of the panel and the wall, so you have a slight amount of down tilt and no coupling to the wall.

We frame and cover ours with fireproof acoustic cloth from Guilford of Maine. Pick a burlap looking one. They will send you 10 samples Of your choice for picking the color.  The open weave vents the panel and serves a second purpose. Second purpose is when the artistically inclined non-technical ladies of the church complain about your un-needed ugly architecture ruining panels, you can let them embroider on them.  The right fabric has the same hole pattern as embroidery grids. Ask me how I know this.  :-)

The corners act as retroreflectors and feedback modes can  really build up between them. So put a panel on both sides of each corner. This bought us ~ 6 db of damping.

Next up. If your band loves hot  stage monitors more then they love the Lord, and many do, you need  20 linear  feet of 48x48 panels behind the band about 36" above the floor with a 12" gap between panels.

Then get an IPhone or PC and a calibrated reference MIC from Dayton Audio.

I used the reference Mic, A known good flat  reference speaker , and the Holm Impulse freeware
to sweep the room. If you correlate the echos in the sweep with the dimensions of the room, you can see where you might need a few more panels. Twenty feet is a carefully calculated number based on average wavelengths.

Flat reference speaker is easy. Put a 4 or 6" speaker in the middle of a 14 in  square panel of half inch plywood. Good enough for government work when mapping a hall.

If you get in any deeper add in a 1/3rd octave equalizer.

Close the lid on the big Piano and see if the resonance stops.  You might be surprised if the piano is near a speaker.

Wood slatting, also helps if designed correctly.

Switch to modest vertical line arrays if you can

When the band tells you they can't hear because you cut down the monitor volume, well show them how much over the OSHA daily exposure  limit they are. Every time I check , even C weighed, there are always a few band members who want it LOUD.

YOUR mileage may vary.

Two inches thickness of Owens  Corning  can be replaced by 4 inches of Roxul wool from Home Depot, if you have budget issues.

When you do your pitch shifter, sweep your shift up and down by 4-6 hz every few seconds.

I've done this at two churches and one school.

It helps if you can sweep the room with the congregation seated, that makes quite a difference over the empty space.

Do not over damp the room, leave some natural reverb.

After doing this I'm 15 dB down on average level during worship, It took s while, but the now  band loves that they can now hear each other.



Steve
 

[moffy]

Thanks LaserSteve for your advice. There are a number of simple remedies I would like to try, including some you mentioned, but unfortunately there is inertia to trying things that involve cost. The echos are pretty bad and some acoustic paneling is badly needed, but it just isn't going to happen at least in the short term. So I'm trying a less than ideal solution that might help softer talkers/singers be heard. If it works it will be a no cost no fuss fix.
Thanks again for the advice.

[LaserSteve]

We had no budget. My panel cost is 1/6th that of a commercial panel.  Found a carpenter in the congregation. Built the first test panels with him, donated the foam. The improvement was enough to get the music lovers rolling. Funding was found.

Second batch, Corning was too expensive at the time, we switched to the Rock Wool. Which is available locally in much of the world. Half the cost per panel, but takes more time to assemble.

The simplest Dayton reference mic with calibration file costs perhaps one  half of your DSP board.  IMM-6 costs 27$. If your willing to make some assumptions,  any electret mic element will work.
IMM-6, plugged into an Apple product, gives you really accurate levels, as Apple calibrates their tablets/IPods  on the production line.


The one third octave equalizer was 99$. Notching out the feedback causing 406 Hz  tone from the female vocalists, priceless.

Break one corner, even if you use only one panel straddling the corner, note the difference.  Usually you only need to do one or two corners.

Rock wool, made from slag and minerals similar to glass, is dirt cheap and does not burn. Standard pink attic insulation works well, but only has 1/3rd the acoustic  density, so you need a lot of it.

Guilford was used due to its fire rating, Burlap works just fine. However in a brick building with  wood arched framing, and  no sprinklers, fire resistance was on my mind.

I too, live on a budget. There are plenty of web sites on home made acoustic treatment. That is where I started.

Good luck... let us know how your system works, because I assure you, there is a demand for such things.

Steve

[SiliconWizard]

Well I have added a cubic spline resampler to the timing loop and the processor still appears to have excess capacity, so it looks like this might actually work.

So yes, those NXP MCUs are pretty powerful.

Now, it would have been more interesting here for everyone if you had given a bit more details about your project. I'm not specifically talking about sharing code (you don't have to if you don't want to or can't), but you could tell us more about the pitch shifting algorithm you implemented. For instance, unless you are doing pitch shifting by just changing the sample rate (which would explain the need for resampling in the end), it's not clear why you need a resampler. And there are several ways of implementing pitch shifting anyway, so always interesting to know which one you used.

[moffy]

Well I have added a cubic spline resampler to the timing loop and the processor still appears to have excess capacity, so it looks like this might actually work.

So yes, those NXP MCUs are pretty powerful.

Now, it would have been more interesting here for everyone if you had given a bit more details about your project. I'm not specifically talking about sharing code (you don't have to if you don't want to or can't), but you could tell us more about the pitch shifting algorithm you implemented. For instance, unless you are doing pitch shifting by just changing the sample rate (which would explain the need for resampling in the end), it's not clear why you need a resampler. And there are several ways of implementing pitch shifting anyway, so always interesting to know which one you used.

LaserSteve, I have seen some Youtube videos of people building sound panels with the rockwool and the results look impressive. We have a graphic equalizer, and I have done a sound check on the room acoustics with my Clarett 2 Pre usb soundcard, mic etc. I have a pretty good grasp of the issues but unfortunately that doesn't overcome the inertia issue. Thanks again for your suggestions, they are much appreciated.

That sounds reasonable. The code is just an embedded implementation of: https://github.com/dbry/audio-stretch
Which works in the time domain rather than the frequency, making it faster and less laggy than a frequency domain solution. The problem is not the pitch shifting, as you pointed out, but maintaining the same time for the pitch shifted sound is the issue. I must say I am not real sure how it works apart from a general overview, I just tested a number of implementations to see if they worked, then chose the one most likely to work in real time. Sorry, no great insight as yet, maybe later when and if I have it working.
The cubic spline resampler comes from: https://github.com/rspeyer/soundtouch
This is a frequency domain based pitch shifting library, which includes resampling filters in the "InterpolateCubic.cpp". Lovely code but too much and too laggy for realtime work.

LaserSteve, thanks for the suggestions I have all ready tested the room acoustics (REW software, laptop, mic etc), and we have a graphic equalizer but at present, with the constraints, this is probably the best realisable solution. Plus it is fun, and something I haven't done before and I would really like to know how practical it would be.

[SiliconWizard]

OK, thanks for the reference, I'll have a look.

As to cubic spline interpolation, this is something I've already talked about in other occasions. It's pretty decent and not too costly for real-time processing. I suppose this is using Catmull-Rom splines?

[moffy]

Finally I got the Cubic spline resampling filter working. Should have been easy, but often the things you think will be hard are actually easy and vice a versa. I was getting persistent crackling and was looking at everything, but it happened to be a simple matter of volume and intermittent saturation. For a 20 cent upward shift on a vocal/musical piece there are no noticeable artifacts. If I play back the shifted with the original at the same volume I get some strange metallic sounding phasing effects, so it will be interesting to hear what it is like in real life.
I need to transfer the revised resampling filter to the RT1020EVK, then the fun will start with running the input of the codec through the pitch shifter and then through the codec output in real time. Syncing to the output clock will be interesting. Fun times, fun times.

[jonpaul]

Bonjour a tous:

To see research  of this interesting topic, eg IEEE ASSP group for the theory and practical implémentations.

A historical note on DSP origins:

 The first digital signal processor was the SIGSALY top secret voice scrambler in 1942, Bell Labs, using 384 valves to preform 72 ADC.

Our friend Tony at Eventide invented the first digital pitch shifters, the Harmonizer (TM) in 1975  https://www.eventideaudio.com/50th-flashback-4-1-the-h910-harmonizer/

These hardware boxes still exist and some even still work! The firm sells plugins with the classic effects.

https://www.eventideaudio.com/plug-ins/anthology-xi/


Enjoy!

Jon

[voltsandjolts]

SIGSALY ...interesting.

Here is a paper about a SIGSALY re-construction...by some bloke called...ahem...Jon Paul

[moffy]

Well, I have got the real time pitch shifting working on the RT1020EVK. The source is a wav file on the PC which plays back over the RT1020EVK(it appears as a speaker/soundcard to Windows). The music is then processed in real time by the pitch shifter and resampler and output in real time to the audio codec at 48k. There are semi regular clicks in the output, not surprising as I haven't looked at synchronisation yet, but I should get there. The clicks could be tricky to resolve, guess I'll just have to work through it.
It has been a real benefit using the same code on the PC(without the hardware specific) for debugging purposes. Helped with the logic as well as choosing buffer sizes etc. Looking good.

[moffy]

Most of the glitches are gone now, one every say 10s(definitely synchronisation) or longer, had a memory issue which was easily fixed. The quality is surprisingly good.

[moffy]

Having proven that the concept is feasible and practical, I now have to write the code for a real Pitch Shifting device. I am thinking of two ways to control it:
1. Laptop/PC command line via a usb/serial connection.
2. Some switches and an LCD display(I2C)
There will really only be one variable to control i.e. the amount of pitch shift, and of course an ON/OFF.
If you have any thoughts about useful features let me know, I will consider them. Thanks.
This will probably take a little while.

[Brak01]

Hello,
I know its been almost two years since your last post - but I just today came upon the thread and am very interested.

I am a a sitar player (a 20 string instrument from India).
We normally tune our instrument to C#, which can cause some headaches when playing with Western instruments - requiring someone to re-tuning their instruments up or down a half-step, or doing some mental/finger gymnastics.

So I have been trying to find a solution to both shift my outgoing signal, but all the on-the-market options are either an over-kill in terms of functions (which makes it more expensive than needed), an under-kill by excluding intervals I would need. And many just sound awful.

So I am interested i your project as its approaching the issue from a different angle (time vs pitch), and you are only concerned with addressing this issue of pitch-shifting.

Hope to hear from you soon.

[jonpaul]

 harmonizer Eventide H910 vs H949
https://gearspace.com/board/geekzone/1325574-eventide-h910-h949-why-do-they-sound-way.html

Invented by my old friend and colleague Anthony Angelo.

Eventide H90 latest guitar pedal with harmonizer function

https://www.eventideaudio.com/pedals/h90/

ENJOY!


Jon

[moffy]

Thank you JonPaul, I managed to program a pitch shifting algorithm into a VST plugin, which shifts in realtime. I did some simulations using an Impulse Response filter using some real room impulse wave files and a feedback VST I wrote just for the simulation, all loaded into Cubase. It is surprisingly effective but can produce some artifacts if mixed with the non shifted direct performance. It depends upon how dominant the sound reinforcing is, I have yet to test it at our church hall, interesting experiment.

[jonpaul]

moffy,

pitch shifting artifacts are a classic DSP issue since decades.
See the Eventide flashback for history of all our effects.
https://www.eventideaudio.com/flashback-7-1-the-h949-harmonizer/
https://www.eventideaudio.com/category/legacy-products/flashbacks/

The latest Eventide plug-ins for Harmonizer (TM) have Tony's best optimizations.
https://store.eventideaudio.com/en-fr/products/h910-harmonizer-1
https://www.eventideaudio.com/plug-ins/h949-harmonizer/

Our Anthology Legacy XII  has all the classic effects, including my Instant Flanger, Omnipressor, etc. from 50 yrs ago,
https://www.eventideaudio.com/plug-ins/anthology-xii/



Enjoy,

Jon

[moffy]

Just a note about the quality of the pitch shifting algorithm. I was using a recording of Beethoven's 5th to test its polyphonic ability. If you listened to the pitch shifted audio, just a quarter of a semitone, by itself, there were not any noticeable artifacts beyond a slight sharpness, shifting up, but that was only really noticeable with A/B comparison between the shifted and non shifted audio. It was only when you mixed the original and shifted together that you got what sounded like phasing artifacts. My eldest daughter who is very musical, singing, piano, flute, oboe and now the harp also listened to the shifted audio and could only hear the slight sharpness for A/B comparison. I am impressed how effective the algorithm is, even though it is shifting by such a tiny amount.

[jonpaul]

The pitch shifting testing program material needs a variety of selection.

Quiet paino music , high and low pitch voice and song, flutes, low organ notes....

You need to try them all


Jon

[moffy]

The pitch shifting testing program material needs a variety of selection.

Quiet paino music , high and low pitch voice and song, flutes, low organ notes....

You need to try them all


Jon

Very good advice.

[jonpaul]

In fact the smaller the shift in pitch the more problematic the artifacts.

Lots of info from Tony's H949 flashback

https://www.eventideaudio.com/flashback-7-1-the-h949-harmonizer/

See the many AES, IEEE ASSP and other papers on pitch shifting.

Jon