changing sounds without gaps to multiple speakers

[Kip Keller]

Hi Goncalo,

   I am working on a part of our overall project to record neurons with OE, take movies with spinnaker and play sounds through our sound card to multiple speakers. The current problem is with playing the sounds - so I have focused there. We want to play a background sound (broad band noise from a soundfile) and, with a keystroke, interject a different soundfile and when that has finished resume the background noise. All of this needs to happen without gaps or clicks. Once I figure out how to do this with one audio output channel, I would like to extend it to multiple audio outputs, but that seems like a less immediate issue. And, of course, I will also want to write timestamps for these sounds to a CSV file to be able to lineup the OE spikes files, the movie and the sound stimuli later. But I haven't included that here either.

  I have attached my current workflow for this part of the project. It is almost totally based on your help to others in the group (THANKS!). Minor tweeks give me different sorts of errors, so these details are probably not (yet) important. I am just missing something more basic/obvious, I guess. Thanks for your help.

Kip Keller

[Gonçalo Lopes]

Hi Kip,

There are several angles to this question depending on what exactly you are trying to achieve. In any case overall the best way to trigger independent sounds currently is to use audio sources from OpenAL. I'm attaching a simple example for us to iterate on.

Basically the idea is we create a global audio context and load a bunch of sound buffers (drag-drop WAV files into the Buffers property of AudioResources).

Then every time we want to play a sound clip we create a new audio source in the mix into the context (e.g. inside each of the CreateObservable nodes):

Here we can configure the buffer name to be created and various properties of the audio source. For example, you probably want to set the initial state of the source to Playing.

Now here is where a number of things get tricky:

  1. Unfortunately the OpenAL API does not send out any event of when buffers have finished playing, so we either have to configure in advance the time we want to allocate for each clip, or keep asking what the state of the source is in a polling fashion. None of them is currently very satisfying, but I've converged currently on setting up a cleanup Timer which will always be higher than any audio clip to account for cleanup. The cleanup itself is done by the ResourceSubject operator which will dispose of the created source when the Timer fires.

  2. The bigger issue though is what to do about "clicks" and "gaps". I'm not sure what your intended goal was, but I suspect you might be hitting an issue which is very common when dealing with sound transitions. Because sound perception is so low-latency usually we can't simply "stop" a sound in its tracks as this will create an audible clicking effect as the sound card forces its levels down to Zero. Instead, we need to think about fading the sound smoothly (yet quickly) down to zero, possibly while we fade in the next sound (so-called cross-fade effect).

There is currently no nice way to do this in Bonsai, but OpenAL definitely supports controlling the levels and gains of sources, so it could be added.

What exactly would you like your transitions to be? Can sounds just be added in the middle of the current mixture, or do you really need interruption of the previous sound and continuation with the new sound?

Hope this helps.

--
You received this message because you are subscribed to the Google Groups "Bonsai Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bonsai-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bonsai-users/c564cc00-3bfb-4825-9039-123100091cb4n%40googlegroups.com.

[keller]

Hi Goncalo,

  Thanks very much for your time and help. I have waited a bit on answering to try to make my reply more intelligent. Maybe not.

  Your workflow works well after a couple of 'tweaks' that might be helpful to share. 1) I had to enter the 'Device Name' also in each CreateSource, not just the CreateAudioContext. 2) In order to run continuously, I had to set 'looping' to 'true' in each CreateSource. This, however, only does part of what I want (allows key-presses forever), but, because of the 'kill' from the Timer, this will not allow looping 'within' a keypress. Somehow, I would have to override the Timer's 'DueTime' for looping within a keypress

  This points to a problem of my (not) understanding. I see what the Timer is doing, but I don't understand how.

  On to 'cracks' and 'pops' and so on. I understand very well the problems of transitioning between sounds. In my first iteration the problem is minimized by my sounds all being 'white' noise or something buried in 'white' noise. And all the levels are roughly consistent. So the transitions are relatively buried. (Traced some other 'pops' to MATLAB & OEphys not letting go of the sound card until MATLAB & OE were actually closed - presumably this problem will go away as I ditch MATLAB for using the OE board directly in Bonsai). However,as you suggested, life will not always be so easy and so I do wonder how to go about fading in/out over maybe a millisecond or two. I thought to do this with a delay after any sound, a ramping, etc. - but, not only is this very klugie, I don't know how to play any two buffers at once (etc.). And I don't know OpenAL at all - or really how to access it. Stumped a bit, I guess.

  But again, thank you for your help.

Cheers,

Kip

[Gonçalo Lopes]

Hi Kip,

I see, if you want to support looping within a keypress, then maybe the following tweak is better (also attached):

Basically this lets all sounds be triggered independently, but also interrupts them as soon as any other sound is triggered (without using a fixed timer).

 Indeed Looping is set to True.

Re. the audio context device name, indeed if you need to target a specific audio device you need to specify this for all created sources. However, if you can get away with setting your output as the default audio device, you should also be able to leave the device name empty and it will pick up the default device.

For setting fade in/out the logic would be similar to this example, but with dynamic gain control of the sources. I realized not all properties might be exposed for this, will have a look.

Hope this helps.

[keller]

Goncalo,

   Thanks so much for this. Sorry for my delay in responding. This seems to work well - not I have to figure out the fade-in and fade-out. Hints appreciated!

Cheers,

Kip

[Gonçalo Lopes]

OpenAL actually allows you to regulate gain directly for each individual source. Unfortunately, setting these parameters is not exposed directly in Bonsai just yet. I've opened an issue to change this for next version: Add support for setting audio source gain and pitch · Issue #775 · bonsai-rx/bonsai (github.com)

In the meantime, you have two options:

1. Set global gain using UpdateListener

2. Use the available OpenTK API in a small C# script to modulate individual source gain

I'm attaching an example of such a script and project that should give you a fair idea of how it would work.

Hope this helps.