Streaming background music with audiobridge plugin

[Firas Abd Alrahman]

We have audiobridge implemented successfully, but we currently has a case that we need to stream background music (audio file) with microphone stream.

I am asking about suggestions or any useful idea.

Capturing mediastream and encoding from mp3 file is done successfully via Audio API, now what is the most possible/easier solution ?

1 - Mix background music stream with microphone stream (using audio api) and update janus localstream with new stream.

I started here to removeStream then add the new stream, but this will hangup directly so we need to re-create offer ... I can proceed if this is the right approach.

 

2 - Submit multiple streams / tracks with addstram (or addtrack).

I do not know how janus deal in this situation, I noticed that janus take only the first track, but maybe we can hack something here ? 

 

3 - Anything else.

browser support is not an issue, we are working with --enable-experimental-web-platform-features enabled. and chromium/crosswalk is enough.

I am not looking for ready solution, any suggestion/idea will be very appreciated.

Thank you!

[Mirko Brankovic]

I had discussion about same topic, if I'm not understanding you wrong, how to play moh to audiobridge user. In case he is only one in room or you someone mutes him ...

My best guess of what would be a nice approach is to redirect that user to streaming plugin where moh file will be started and running in background. Since that user will be receiving only audio and shouldn't be in audio mix.

First problem would be then if you would have thousands of different files I guess. 

Next challenge will be rtp timeout if you want that user to recvonly audio if you stick with audio bridge.

I think the best approach would be a custom plugin ...

--
You received this message because you are subscribed to the Google Groups "meetecho-janus" group.
To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janus+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mirko Brankovic]

or you ment to force moh file from client a to client b..

[Lorenzo Miniero]

We don't support multiple streams yet. If you need multiple streams, you'll need to create multiple PeerConnections, in this case a new participant for the music stream. Anyway, an answer to your problem typically depends on the nature of this background music. If it's something only the user has, you'll have to either play with the Audio API to see if you can mix in the browser before publishing, or publish twice as explained above. As an alternative, if synchronization isn't really important, why not play the audio music as a plain HTML5 audio (stored on a web server somewhere) in parallel to the WebRTC streams for all participants?

L.

[Firas Abd Alrahman]

Thank you Mirko, Thank you Lorenzo.

I understood that I can only publish one stream one track, but I was able to solve this, I will post the solution if someone interested.

1- I made some modifications to janus js library, in StreamsDone function before addstream, i added a callback function called hookstream, this allowed me to modify the stream before being published by janus and return another stream, as I failed to removestream/addstream this approach was complicated as need to re-create offer etc... so now I do in StreamsDone before config.pc.addStream (with some minor changes in callback class)

if(callbacks.hookstream){

var newstream = callbacks.hookstream(stream)

stream = newstream;

}

2- In attach audiobridge plugin I added callback function hookstream, now I used audioapi to get the stream form audio element, then mixed with mic stream coming from janus, and returned the new stream, this worked!! 

Users now able to select a background music file and played together with his mic voice!

My only problem is we need to activate enable-experimental-web-platform-features in chrome flags, but can not blame janus for that, it worked with firefox directly.

here is the code to mix the streams: 

https://pastebin.com/62a9kZeq

I think the hook is useful because now I can do some webapi works to janus stream like controlling the gain or anything ... I do not know if I can modify the stream before being published without touching janus js library, but it is good for me now.

Thank you again !

[Lorenzo Miniero]

There should be no need for that. In janus.js, you can provide your own MediaStream, you're not forced to use the getUserMedia it provides. Check the documentation for the exact syntax on how to do that. The main difference is that when closing the session you'll be responsible for closing the MediaStream, janus.js won't do it for you.

L.

[Firas Abd Alrahman]

hmmm, yes Lorenzo thank you, I just noticed that!

But I think the new approach is better for me, as for ex. when videoplugin attached, janus update the mediastream and take care about many stuff, I don't want to repeat that, I take the latest stream and mix it again with new audio and return it back.

so I don't need to handle usermedia and before/after operations.

I already had changes to janus js library to make it angular service, so the library code already touched.

Thank you!

[Firas Abd Alrahman]

I had some successful step with one problem:

I will share my experience here first

CatpureStream crashes chrome (latest version) and still not ready and never worked in android chrome browsers or crosswalk, Mixing Microphone stream with audio stream failed on android chrome/crosswalk,no error returned, but no sound streamed, just nothing happens, while it worked in web browsers.

for streams mixing find here topic no. 5  so all solutions to mix streams with one succeeded in desktop browsers, but failed in android browsers.

So I went with solution #2, attaching two audiobridges plugins, one for mic and another for streaming audio files (local files selected by file upload html element).

I used aurora audio decoder/player, and published the final stream to janus audiobridge in createOffer stage (stream parameter).

I attach audiobridge plugin for background music when user play file, and detach when stop

All went perfect!

The Problem: The publisher listen to the background music twice

The reason is clear, publisher has two users joined the room, one for mic (send and receive), one for background music (send only (remote stream muted))

audio go from background music stream and comeback to mic stream after being mixed with another room streams in server.

audio come back with delay, so it appears as echo.

Now I am just thinking of optimization! I noticed that Janus does not return my voice back to me when I talk, so Janus subtract user voice from mixed streams as user do not listen his voice back in his headphone.

Can I do something like that in my case? or anything else?

Thank you!

On Sunday, July 2, 2017 at 9:37:59 PM UTC+3, Firas Abd Alrahman wrote:

[Lorenzo Miniero]

Il giorno lunedì 10 luglio 2017 23:02:28 UTC+2, Firas Abd Alrahman ha scritto:

I had some successful step with one problem:

I will share my experience here first

CatpureStream crashes chrome (latest version) and still not ready and never worked in android chrome browsers or crosswalk, Mixing Microphone stream with audio stream failed on android chrome/crosswalk,no error returned, but no sound streamed, just nothing happens, while it worked in web browsers.

for streams mixing find here topic no. 5  so all solutions to mix streams with one succeeded in desktop browsers, but failed in android browsers.

So I went with solution #2, attaching two audiobridges plugins, one for mic and another for streaming audio files (local files selected by file upload html element).

I used aurora audio decoder/player, and published the final stream to janus audiobridge in createOffer stage (stream parameter).

I attach audiobridge plugin for background music when user play file, and detach when stop

All went perfect!

The Problem: The publisher listen to the background music twice

The reason is clear, publisher has two users joined the room, one for mic (send and receive), one for background music (send only (remote stream muted))

audio go from background music stream and comeback to mic stream after being mixed with another room streams in server.

audio come back with delay, so it appears as echo.

Now I am just thinking of optimization! I noticed that Janus does not return my voice back to me when I talk, so Janus subtract user voice from mixed streams as user do not listen his voice back in his headphone.

Can I do something like that in my case? or anything else?

I don't understand the problem. If the other room is only used to allow you to publish music, and you mute that remote stream, you won't get anything back. If noone else contributes to the 2nd room (where it's only you publishing music) then there's no mixing happening either for the music room. If you hear echo, it's because one of the participants listens to both rooms, inclucind the music, and talks: the music somehow gets in their mic and sent back to you as part of the participant's contribution. If so, that's environmental echo, and there's nothing Janus can do about that. Users need to use headsets, or you have to play with the echo cancellation stuff in Chrome.

L.

[Firas Abd Alrahman]

Thank you Lorenzo.

everything happens in one room, I noticed that you suggested to have a room for music, I will try that.

But the following may describe my current situation.

Can I tell janus in mic stream offer, to ignore music stream for the talker only so it does not come back ?

[Lorenzo Miniero]

Not at the moment, no, we would need to modify the AudioBridge plugin to do that.

L.