Subscribing to some channels only

[Ancor Gonzalez Sosa]

As you may know, we use janus.js in the implementation of jangouts[1].

We are researching about a new "mode" to lower the CPU and bandwidth usage on the clients. We were wondering if this can be achieved:
 - All the publishers publish video + audio + data
 - Powerful clients behave like they do now, subscribing to all the channels for each publisher
 - Less powerful clients subscribe only to audio + data, ignoring the video stream

Is that possible? Right now, if the publisher offers video the subscribers always get it. It seems there is no way to refuse.

We tried adding a "videoRecv: false" to the "media" parameter of createAnswer during subscription, but it didn't do the trick.

Any tip?

Thanks

[1] https://github.com/jangouts/jangouts

--
Ancor González Sosa
https://github.com/ancorgs/

[Lorenzo Miniero]

Il giorno martedì 30 giugno 2015 14:56:22 UTC+2, Ancor Gonzalez Sosa ha scritto:

As you may know, we use janus.js in the implementation of jangouts[1].

We are researching about a new "mode" to lower the CPU and bandwidth usage on the clients. We were wondering if this can be achieved:
 - All the publishers publish video + audio + data
 - Powerful clients behave like they do now, subscribing to all the channels for each publisher
 - Less powerful clients subscribe only to audio + data, ignoring the video stream

Is that possible? Right now, if the publisher offers video the subscribers always get it. It seems there is no way to refuse.

We tried adding a "videoRecv: false" to the "media" parameter of createAnswer during subscription, but it didn't do the trick.

Passing videoRecv:false should actually work, as, at least in principle, it should have the browser reject the video track Janus is offering (m=video 0) while only keeping audio. If it's not working, it's either a bug in the janus.js library, or something is not working as it's supposed to in the core. I'm more inclined towards the former, because if the SDP contains "m=video 0", the browser itself will dump any incoming video packet (or at least it should).

Can you share an offer/answer in such a case?

As a side note, I think that there would still be the need for another fix this in either the core or the plugins. In fact, even if the medium has been denied, I think the videoroom plugin would still push the RTP packet to the core anyway, as the user is a subscriber in general and there's no check on what's been negotiated. The core may in turn forward the packet anyway, since the channel is available (video and audio would share the same channel audio alone uses). Not sure if this would happen, I'll have to look: if it's not there, the fix should be trivial enough though.

[Ancor Gonzalez Sosa]

El martes, 30 de junio de 2015, 15:13:06 (UTC+2), Lorenzo Miniero escribió:

Il giorno martedì 30 giugno 2015 14:56:22 UTC+2, Ancor Gonzalez Sosa ha scritto:

As you may know, we use janus.js in the implementation of jangouts[1].

We are researching about a new "mode" to lower the CPU and bandwidth usage on the clients. We were wondering if this can be achieved:
 - All the publishers publish video + audio + data
 - Powerful clients behave like they do now, subscribing to all the channels for each publisher
 - Less powerful clients subscribe only to audio + data, ignoring the video stream

Is that possible? Right now, if the publisher offers video the subscribers always get it. It seems there is no way to refuse.

We tried adding a "videoRecv: false" to the "media" parameter of createAnswer during subscription, but it didn't do the trick.

Passing videoRecv:false should actually work, as, at least in principle, it should have the browser reject the video track Janus is offering (m=video 0) while only keeping audio. If it's not working, it's either a bug in the janus.js library, or something is not working as it's supposed to in the core. I'm more inclined towards the former, because if the SDP contains "m=video 0", the browser itself will dump any incoming video packet (or at least it should).

Can you share an offer/answer in such a case?

Indeed, it looks like a bug in janus.js.
This is the affected line in jangouts (in its current form, without video ignoring) https://github.com/jangouts/jangouts/blob/master/src/app/components/room/room.service.js#L293
These are the sdps when using "video: false" in the "media" part of the call above. http://paste.opensuse.org/32557767
These are the sdps when using more explicit "videoRecv: false, videoSend: false". http://paste.opensuse.org/27143960

In both cases it's actually requesting the video, so you were right about the culprit.
 

[Lorenzo Miniero]

Il giorno martedì 30 giugno 2015 16:00:34 UTC+2, Ancor Gonzalez Sosa ha scritto:

El martes, 30 de junio de 2015, 15:13:06 (UTC+2), Lorenzo Miniero escribió:

Il giorno martedì 30 giugno 2015 14:56:22 UTC+2, Ancor Gonzalez Sosa ha scritto:

As you may know, we use janus.js in the implementation of jangouts[1].

We are researching about a new "mode" to lower the CPU and bandwidth usage on the clients. We were wondering if this can be achieved:
 - All the publishers publish video + audio + data
 - Powerful clients behave like they do now, subscribing to all the channels for each publisher
 - Less powerful clients subscribe only to audio + data, ignoring the video stream

Is that possible? Right now, if the publisher offers video the subscribers always get it. It seems there is no way to refuse.

We tried adding a "videoRecv: false" to the "media" parameter of createAnswer during subscription, but it didn't do the trick.

Passing videoRecv:false should actually work, as, at least in principle, it should have the browser reject the video track Janus is offering (m=video 0) while only keeping audio. If it's not working, it's either a bug in the janus.js library, or something is not working as it's supposed to in the core. I'm more inclined towards the former, because if the SDP contains "m=video 0", the browser itself will dump any incoming video packet (or at least it should).

Can you share an offer/answer in such a case?

Indeed, it looks like a bug in janus.js.
This is the affected line in jangouts (in its current form, without video ignoring) https://github.com/jangouts/jangouts/blob/master/src/app/components/room/room.service.js#L293
These are the sdps when using "video: false" in the "media" part of the call above. http://paste.opensuse.org/32557767
These are the sdps when using more explicit "videoRecv: false, videoSend: false". http://paste.opensuse.org/27143960

In both cases it's actually requesting the video, so you were right about the culprit.
 

Have you checked in the log where it says "Creating answer (iceDone=" which constraint is being passed to createAnswer? Looking at the code they should be right, and the constraints are printed at:

https://github.com/meetecho/janus-gateway/blob/master/html/janus.js#L1366

that is immediately after the line I mentioned. You may want to add some additional debug line to identify the viewer you're interested in, of course, as the same line will be printed for every PeerConnection you'll create.

L.

[Ancor Gonzalez Sosa]

El martes, 30 de junio de 2015, 16:08:22 (UTC+2), Lorenzo Miniero escribió:

Il giorno martedì 30 giugno 2015 16:00:34 UTC+2, Ancor Gonzalez Sosa ha scritto:

El martes, 30 de junio de 2015, 15:13:06 (UTC+2), Lorenzo Miniero escribió:

Il giorno martedì 30 giugno 2015 14:56:22 UTC+2, Ancor Gonzalez Sosa ha scritto:

As you may know, we use janus.js in the implementation of jangouts[1].

We are researching about a new "mode" to lower the CPU and bandwidth usage on the clients. We were wondering if this can be achieved:
 - All the publishers publish video + audio + data
 - Powerful clients behave like they do now, subscribing to all the channels for each publisher
 - Less powerful clients subscribe only to audio + data, ignoring the video stream

Is that possible? Right now, if the publisher offers video the subscribers always get it. It seems there is no way to refuse.

We tried adding a "videoRecv: false" to the "media" parameter of createAnswer during subscription, but it didn't do the trick.

Passing videoRecv:false should actually work, as, at least in principle, it should have the browser reject the video track Janus is offering (m=video 0) while only keeping audio. If it's not working, it's either a bug in the janus.js library, or something is not working as it's supposed to in the core. I'm more inclined towards the former, because if the SDP contains "m=video 0", the browser itself will dump any incoming video packet (or at least it should).

Can you share an offer/answer in such a case?

Indeed, it looks like a bug in janus.js.
This is the affected line in jangouts (in its current form, without video ignoring) https://github.com/jangouts/jangouts/blob/master/src/app/components/room/room.service.js#L293
These are the sdps when using "video: false" in the "media" part of the call above. http://paste.opensuse.org/32557767
These are the sdps when using more explicit "videoRecv: false, videoSend: false". http://paste.opensuse.org/27143960

In both cases it's actually requesting the video, so you were right about the culprit.
 

Have you checked in the log where it says "Creating answer (iceDone=" which constraint is being passed to createAnswer? Looking at the code they should be right, and the constraints are printed at:

https://github.com/meetecho/janus-gateway/blob/master/html/janus.js#L1366

that is immediately after the line I mentioned. You may want to add some additional debug line to identify the viewer you're interested in, of course, as the same line will be printed for every PeerConnection you'll create.

I checked in both Chromium and Firefox.

In both the mediaConstraints object looks sane, i.e.  { mandatory: { OfferToReceiveAudio: true, OfferToReceiveVideo: false } }

In both the answer SDP looks quite similar to the ones I already pasted, with a "a=recvonly" in the video part.

In Chromium the visible result is the expected one: audio but not video. In Firefox I receive audio and video.

Cheers.

[Ancor Gonzalez Sosa]

There is actually a quite relevant difference. In Chromium I see
m=video 0 RTP/SAVPF 100
which I have just learned that is the way to reject the channel (setting it to port 0).

In Firefox that line is
m=video 9 RTP/SAVPF 100

[Lorenzo Miniero]

Do you still get video anyway in Chromium, despite the video m-line being rejected? If so, apart from the already mentioned fix I must make in the Janus core, it's a bug in Chromium too, as it's parsing the video packets Janus sends instead of dropping them.

As for Firefox, I guess it's not digesting the constraints properly. IIRC they're either deprecated or spelled differently now, and so Firefox may be expecting something different, not sure about that. I'll have a look tomorrow.

L.

[Ancor Gonzalez Sosa]

I get no video. In Chromium everything works perfectly.

As for Firefox, I guess it's not digesting the constraints properly. IIRC they're either deprecated or spelled differently now, and so Firefox may be expecting something different, not sure about that. I'll have a look tomorrow.

Much appreciated.
 

[Lorenzo Miniero]

This patch:

https://github.com/meetecho/janus-gateway/commit/ca2ccd4a87efb207692a770a3b432d7fefd5f462

fixes the behaviour in the core when media are not available, and adds some additional options when creating listeners too. More specifically, when you create a listener, you can now also pass any of these boolean variables (audio, video, data), to instruct the plugin into whether or not the related medium should be sent to that specific listener no matter the source. For instance:

var listen = { "request": "join", "room": 1234, "ptype": "listener", "feed": id, "video": false };

remoteFeed.send({"message": listen});

will make sure the new listener will not receive video frames even if video has been negotiated or the publisher is sending it.

As to the janus.js bugs, it's only partially a bug in our JavaScript library. As I recalled, the syntax in Firefox is indeed different as the spec changed:

   Old syntax: {'mandatory': {'OfferToReceiveAudio': true, ...
   New syntax: {'offerToReceiveAudio': true, 'offerToReceiveVideo': true}

I changed my local janus.js to use the correct version when Firefox is being used but, while this works for publishers (which are sendonly as expected), this still doesn't work for rejecting a medium. If I pass {'offerToReceiveAudio': true, 'offerToReceiveVideo': false} to a recvonly PeerConnection, video should be rejected as we're not going to send it and we also don't want to receive it: instead, the video is still negotiated. This is probably a bug in Firefox (at least in Firefox stable, haven't checked Nightly) so I'll notify this on the mozill.dev.media group.

That said, for now for what you need the new boolean parameters to the listener creation should do the trick.

L.

[Ancor Gonzalez Sosa]

El miércoles, 1 de julio de 2015, 11:43:19 (UTC+2), Lorenzo Miniero escribió:

This patch:

https://github.com/meetecho/janus-gateway/commit/ca2ccd4a87efb207692a770a3b432d7fefd5f462

fixes the behaviour in the core when media are not available, and adds some additional options when creating listeners too. More specifically, when you create a listener, you can now also pass any of these boolean variables (audio, video, data), to instruct the plugin into whether or not the related medium should be sent to that specific listener no matter the source. For instance:

var listen = { "request": "join", "room": 1234, "ptype": "listener", "feed": id, "video": false };

remoteFeed.send({"message": listen});

will make sure the new listener will not receive video frames even if video has been negotiated or the publisher is sending it.

You absolutely rock! I have never seen such a responsive free software project. Thanks a lot.

And now one step further. :-)

The final goal is to remain subscribed to audio and data (de)activating the video stream whenever the subscriber wants to do it (several times per call). A "configure" request cannot be used because it's only accepted by the videoroom plugin when issued by a publisher, not by a subscriber. A "pause" request cannot be used because it will pause all the streams, no option to pause just some of them. Sending a new offer from the subscriber (as suggested here[1]) cannot be done because Janus will say "Offer already sent, not sending it again" (not to mention the already mentioned Firefox bug).

Any other alternative?

If I would have to implement that myself (hopefully not, I suck at C development), what do you suggest? Adding support for "configure" subscriber-side or extending "pause"?

As to the janus.js bugs, it's only partially a bug in our JavaScript library. As I recalled, the syntax in Firefox is indeed different as the spec changed:

   Old syntax: {'mandatory': {'OfferToReceiveAudio': true, ...
   New syntax: {'offerToReceiveAudio': true, 'offerToReceiveVideo': true}

I changed my local janus.js to use the correct version when Firefox is being used but, while this works for publishers (which are sendonly as expected), this still doesn't work for rejecting a medium. If I pass {'offerToReceiveAudio': true, 'offerToReceiveVideo': false} to a recvonly PeerConnection, video should be rejected as we're not going to send it and we also don't want to receive it: instead, the video is still negotiated. This is probably a bug in Firefox (at least in Firefox stable, haven't checked Nightly) so I'll notify this on the mozill.dev.media group.

Did I tell you that you rock? :-)
 

That said, for now for what you need the new boolean parameters to the listener creation should do the trick.

Thanks again.

Cheers.

[1] http://www.w3.org/2011/04/webrtc/wiki/images/archive/f/f7/20131112025841!W3C-sdp-2.pdf

[Lorenzo Miniero]

You're right, pause completely pauses all streams. And no, we don't support renegotiation as of yet (which is why you're getting that error when sending a new SDP).

To do what you need, it's just the matter of implementing a simple request that only manipulates the new audio/video/data flags, according to what's being requested, as they're the ones responsible for choosing what to provide a listener with. I guess something as a per-listener configure would do the trick.

I'll try and so something like this by the end of the week.

L.

[Ancor Gonzalez Sosa]

El jueves, 2 de julio de 2015, 10:17:00 (UTC+2), Lorenzo Miniero escribió:

You're right, pause completely pauses all streams. And no, we don't support renegotiation as of yet (which is why you're getting that error when sending a new SDP).

To do what you need, it's just the matter of implementing a simple request that only manipulates the new audio/video/data flags, according to what's being requested, as they're the ones responsible for choosing what to provide a listener with. I guess something as a per-listener configure would do the trick.

I'll try and so something like this by the end of the week.

Any news on this?

Cheers.

[Lorenzo Miniero]

No sorry, I've been awfully busy and I still am... hopefully by the end of the week.

L.

 

Cheers.

[Lorenzo Miniero]

Can you open an issue on github with the feature request? Otherwise I'll probably forget about this again...

Thanks,

Lorenzo

[Ancor Gonzalez Sosa]

El viernes, 10 de julio de 2015, 18:05:41 (UTC+2), Lorenzo Miniero escribió:

Can you open an issue on github with the feature request? Otherwise I'll probably forget about this again...

Don't worry, I will not let you forget it. ;-P
https://github.com/meetecho/janus-gateway/issues/277

Cheers.

[Ancor Gonzalez Sosa]

Resurrecting this thread just to let you know that we are already using the feature and for you to confirm a suspicion on my side.

We created a separate class (a factory in AngularJS jargon) to handle the status of the configuration flags and to keep in sync the remote and local value of those flags[1]. But I was checking again the videoroom plugin yesterday and I think I made an error.

Configure is an asynchronous call, so I assume that handling the reply here is wrong.
https://github.com/jangouts/jangouts/blob/master/src/app/components/feed/connection-config.factory.js#L101
Most likely that callback only means that the request was successfully sent. Nothing to do with the success of the configure operation.
I think I need to handle the result of the call (to update the local representation of the flags) here:
https://github.com/jangouts/jangouts/blob/master/src/app/components/room/room.service.js#L286
Am I right (i.e. is my code wrong)? :-)

Other than that, the feature works. By the way, it takes up to 2 seconds for the video to actually start playing again after receiving a {video: true}. Even if the server is in a lan. Even if it's localhost, I'd say. I assume there is some buffering involved or some similar issue. As soon as I fix my code, I'll be able to provide more trustworthy information about the reason.

Cheers.

[1] https://github.com/jangouts/jangouts/blob/master/src/app/components/feed/connection-config.factory.js

[Lorenzo Miniero]

Yes, configure is an asynchronous message, so what you said is right. The bug still needed to be fixed anyway, as people might be interested in when a response is received, whether it is definitive or temporary.

About the 2 seconds, not sure, but this might be ascribed to the need for a key frame which may or may not arrive right away. Definitely not buffering since buffering is never involved, it's all live.

L.

[Ancor Gonzalez Sosa]

El lunes, 24 de agosto de 2015, 12:27:54 (UTC+2), Lorenzo Miniero escribió:

Yes, configure is an asynchronous message, so what you said is right. The bug still needed to be fixed anyway, as people might be interested in when a response is received, whether it is definitive or temporary.

Ok. I'm implementing it in the right way now, listening to the "configured:ok" message.

When subscribing to a new feed, in the success callback of CreateAnswer(), I'm sending a "start" request followed by a "configure" request. As a result, I'm getting a "started:ok" event and TWO "configured:ok" events. I have checked and I'm completely sure I'm only sending one configure request. Looking at the source code I see no reason to get two events as a reply. Any clue?

[Ancor Gonzalez Sosa]

El martes, 8 de septiembre de 2015, 12:10:39 (UTC+2), Ancor Gonzalez Sosa escribió:

El lunes, 24 de agosto de 2015, 12:27:54 (UTC+2), Lorenzo Miniero escribió:

Yes, configure is an asynchronous message, so what you said is right. The bug still needed to be fixed anyway, as people might be interested in when a response is received, whether it is definitive or temporary.

Ok. I'm implementing it in the right way now, listening to the "configured:ok" message.

When subscribing to a new feed, in the success callback of CreateAnswer(), I'm sending a "start" request followed by a "configure" request. As a result, I'm getting a "started:ok" event and TWO "configured:ok" events. I have checked and I'm completely sure I'm only sending one configure request. Looking at the source code I see no reason to get two events as a reply. Any clue?

I have checked again and I'm also getting the "started:ok" twice. I'm completely sure the success callback is only being executed once. I'm also sure there is only one "start" and one "configure" in that callback. But I'm getting 2 messages as reply for each request. What can be the reason?

[Lorenzo Miniero]

Check the transaction ID of those notifications, as they'll match the original request. If they differ, it's two different requests.

L.

[Ancor Gonzalez Sosa]

El martes, 8 de septiembre de 2015, 13:01:29 (UTC+2), Lorenzo Miniero escribió:

Check the transaction ID of those notifications, as they'll match the original request. If they differ, it's two different requests.

Not sure if I got your explanation correctly, but I'd say it's the same request and I'd say the same behavior (duplicated replies) is present in your MCU demo. Steps to reproduce.

1) connect to http://janus.conf.meetecho.com/videomcutest.html with any browser and leave it connected (just to generate a pre-existing feed).
2) connect to the same url with a chromium browser (since chromium includes a quite convenient way of saving the js console log)
3) save the js console logs (right click on the log itself)
4) grep event whatever.log | grep transaction

You will see twice the event corresponding to "started:ok".

[Lorenzo Miniero]

Ok, found it:

Got event on session 4168335901

Object {janus: "event", session_id: 4168335901, sender: 4187426773, transaction: "M7lQtDLf91Bi", plugindata: Object}

  -- Event is coming from 4187426773 (janus.plugin.videoroom)

Object {videoroom: "event", room: 1234, started: "ok"}

Notifying application...

 ::: Got a message (listener) :::

{"videoroom":"event","room":1234,"started":"ok"}

Event: event

Long poll...

Got event on session 4168335901

Object {janus: "event", session_id: 4168335901, sender: 4187426773, transaction: "M7lQtDLf91Bi", plugindata: Object}

as I guessed the Janus API transaction ID is the same (M7lQtDLf91Bi) which means the event should be originated in response to the same request: that is, exactly the same transaction request, not a duplicated request, as that would have a different transaction ID. Unless the same request, including the generated transaction ID, is being sent twice by janus.js? (shouldn't happen).

I agree that looking at the plugin code that shouldn't be happening. I'll check if requests are being received twice or if it's something within the plugin that is handling the same request multiple times.

L.

[Lorenzo Miniero]

Just fixed this: https://github.com/meetecho/janus-gateway/commit/f1b79cd549c656b357dfd7c8e3d61661999fad20

We were not breaking the loop when receiving an answer. This definitely fixed the double "started" event. I couldn't replicate this for configured, instead: that's always sent once, just as it should.

Lorenzo

[Ancor Gonzalez Sosa]

El miércoles, 9 de septiembre de 2015, 11:28:16 (UTC+2), Lorenzo Miniero escribió:

Just fixed this: https://github.com/meetecho/janus-gateway/commit/f1b79cd549c656b357dfd7c8e3d61661999fad20

We were not breaking the loop when receiving an answer. This definitely fixed the double "started" event. I couldn't replicate this for configured, instead: that's always sent once, just as it should.

Yes. I confirm the commit fixes the problem. I was getting also duplicated replies for "configure" because my code uses the subscriber version of "configure", while the demo only sends "configure" for publisher (which was not affected).

Thanks. You are awesome!

Cheers.