New BBB streaming software (without a webbrowser!)

[Lukas Schauer]

Hi there,

just wanted to quickly present a little tool I've been working on for a few days now.

A lot of people are using BigBlueButton for public conferences and try to create livestreams so a bigger audience can watch. For now that's basically done using a (virtualized) webbrowser and a screen capture.

There isn't much control over what's shown on stream and it takes up a lot of cpu and memory that way.

I decided to write my own tool for this. The goals were to get rid of the browser completely, but still support audio, webcams, screenshares and even the whiteboard. I also wanted to make this compatible with completely unmodified servers.

I got it working.

For now it's still WIP but you can find the (actually working!) code and a link to a demo video here:

https://github.com/lukas2511/bbb-streaming

It's based around gstreamer. Audio/Webcams/Screenshare are received using WebRTC, the presentation is rendered internally using Cairo and RSVG and reusing my old slide-exporter code https://github.com/lukas2511/bbb-stuff with modifications for live-rendering. It's streaming to a user-defined rtmp endpoint but that's easily modified for e.g. SRT or whatever gstreamer supports.

Internally it takes a join_url variable with the BBB API join request. For testing it tries to enter a conference using greenlight but you could easily change that to provide the join request in any other way.

Screenshare/Whiteboard are switched automatically like in the browser, camera view is currently following active voice participants.

There currently are four supported scenes which can be switched via "!view <sbs|pip|pres|cam>" chat messages.

- Side-By-Side (making basically anything visible at the same time and allowing a nice background image with logo/conference name to be embedded)

- Picture-In-Picture (cam overlayed on presentation)

- Fullscreen presentation

- Fullscreen cam 

I'm planning to move those chat commands to a private chat and implement a little browser extension / userscript to show actual buttons for quick scene-switching on the BBB UI. 

I've tested it with the newest BBB 2.2 release and it seems to be working fine. I'd guess that most of the underlying stuff didn't change with 2.3 so it should probably be easy to support that too.

Hope you have some use for this, I'd love to have some beta testers and feedback on the project.

Lukas

[Victor Manuel Agudelo]

Nice work.

Thanks for sharing.

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/48957697-daac-4f13-99ba-c7abc7ba6fc0n%40googlegroups.com.

[Paulo Lanzarin]

Cool stuff. Clever usage of gst's webrtcbin.

Congrats.

Might I ask how's the CPU usage per decoded stream if you have measured that?

s,

prlanzarin

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/CAH2oU2ySxa8v2eCg8gmyaB3F2Oq1LmbsoTNzNv9fdQJ4b_GhpQ%40mail.gmail.com.

[Martin Thomas Schrott]

excellent! really appreciate this.

looking into it for sure.

could be combined with the janus streaming server and gstreamer switched to rtp for low latenzy ...

We just where thinking to try to get gstreamer into our streaming - but if webrtc can be used directly this is the approach to go I guess.

cheers

Martin

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/CADG0wOB_%3DG%2BwzNYJ5O5nD%3Dj2%2BbO_ZSGF4FEVQZd6PCa74G0s5w%40mail.gmail.com.

[Lukas Schauer]

I didn't measure it exactly but something like the test stream as being seen in my demo video running on a VM with four threads of an E5-2680 v4 now uses about 50% of the cpu (that's a lot less than I ever got with browser based setups).

Since currently all webcams are decoded load will slightly rise with every active webcam (depending on resolution/quality), but filters which webcams to decode should be easily implemented or you could simply use the locking feature of BBB to control who can activate their cams.

[Paulo Lanzarin]

> streamer switched to rtp for low latenzy ...

Tailing on what Martin said, made me think about something not completely related:

if you're struggling with webrtcbin quirks (which I've experienced myself in the past), you could also fetch the plain
unencrypted RTP streams from BBB with gst's rtpbin chain (which is rock solid), instead of using the DTLS-SRTP encrypted ones.

That can only be done internally, of course, since you need to use an (undocumented) internal bbb-webrtc-sfu API
different from the one you're using to do the offer-answer flux. It should happen transparently though once you

get the API tamed. If that's something you intend to explore in the future, please open an issue in your repo

and cc me there (@prlanzarin) so I can give more details. Not sure how much net benefit it would give, though.

s,

prlanzarin

[Lukas Schauer]

The main issue with the webrtc quirks seems to be packetloss, which should be gracefully handled by gstreamer (but isn't). I'm pretty sure I just didn't configure it correctly. Activating fec already improved image stability a lot.

I kinda want to keep it fully compatible with stock BBB setups without server access, so using an internal API is not really an option for me. Also running it on the same server gets rid of the issues anyway as packet-loss basically disappears completely.

[Martin Thomas Schrott]

Lukas,

I think Paulos idea was to impliment it exactly as you did - into the stock BBB - but an layer below, so you have better experience / less overlay

and if you get the rtp - gstreamer also should have less work to create an combined rtp for janus - all in all this sounds like a very slim and great way to realize a streaming that could be included in bbb generally.

correct me if I am wrong.

cheers

Martin

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/c731e54d-8a7a-4e32-a497-702fa49d5521n%40googlegroups.com.

[ICT Cloud]

Thumbs up.. Vote +1

[Brent W. Baccala]

Lukas, this looks great!

I've done my own experiments with streaming Big Blue Button conferences, so this seems like a good opportunity to share my thoughts.

Streaming viewers are very closely related to "Listen Only" viewers.  We can extend our two-tier user model to include at least a third tier.  Streamer -> attendee -> moderator.

Consider, for example, a dial-in talk show.  There might be thousands of people listening, then somebody can call in and the host says "Jeffrey from Wichita, you're on the air!".  This would be a nice capability to add to our system.  Just like a moderator can promote an attendee to moderator, they could promote a streamer to attendee.

Streaming services like twitch and youtube typically allow streamers to interact via the chat box, so it's not just totally one-way.  A twitch streamer is a lot more like a Listen Only viewer than somebody tuned in to a TV show.

A big UI issue that Lukas has addressed in a nice way is how to set the presentation layout for a viewer who is totally non-interactive.  You send chat box messages to a "bot" that controls the layout.

I have a very similar issue with my primary corporate client, except we're dealing with recordings instead of live streams.  Yet the issue is the same!  They want the recording in mp4, not using a custom playback application, so how do we set the presentation layout?

Now I'm thinking we've got a four tier system:

1. Non-interactive streamers/mp4 recordings.  We need to set the presentation layout, like Lukas has done.

2. Interactive "Listen Only" streamers/recordings played with our custom app.  User can't participate (except maybe chat box), but can set the presentation layout themselves.

3. Attendees

4. Moderators

I could go on, but I think I've said enough.  I'm just suggesting that this is a useful framework for thinking about how different users interact with our system, and Lukas has done a nice job addressing one of the biggest problems (presentation layout) that we've got with a "type 1" user.

    agape

    brent

[Mario Gasparoni Junior]

Very nice work, Lukas.

Congrats.

--

[Paulo Lanzarin]

I think Paulos idea was to impliment it exactly as you did - into the stock BBB - but an layer below, so you have better experience / less overlay

Not exactly. Lukas mentioned that he wants the application to work with a default installation and with or without server access.

I assumed the intention was to run the streamer alongside BBB, but it isn't. What I suggested implies one of two things: internal access
or a custom installation to expose the internal API I was talking about.

So it's right decision to stick with webrtcbin right now, even more when the RTP streams would have to be wrapped in SRTP if the streamer runs in a

separated instance.

Also don't think there's really that big of an overhead in handling the surplus DTLS-SRTP stuff. Really won't affect latency in a noticeable way.

As for Janus: same way the mixer is exposing the result to an RTMP endpoint it can be made to expose it to an RTP endpoint (the streaming plugin) in a relatively easy way.
It will be fairly easy to even wrap the mountpoint with SRTP in gstreamer as well which would be needed for real deployments.
That's what gstreamer is great for. So I do think this is very close to being able to integrate into the streaming plugin as a prototype. Cool stuff.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/CAKOUqquWnQyGvLmUxfDC0R0_8%2Bx4wYp_YXsOSR6kc2F6Ko8hDg%40mail.gmail.com.

[TeachReo]

Hi 

Very nice job.

Thanks for it 

Fabrice

[Martin Thomas Schrott]

> So it's right decision to stick with webrtcbin right now, even more
> when the RTP streams would have to be wrapped in SRTP if the streamer
> runs in a
> separated instance.
>
> Also don't think there's really that big of an overhead in handling
> the surplus DTLS-SRTP stuff. Really won't affect latency in a
> noticeable way.

ok thanks - great to hear this.

also think best would be if this can just be run / used with any
instance without having to customize it.


maybe in the end slightly integrated as installation option as
greenlight is ... the goal should be to get streaming to the endusers
eventually. Right now we have a few stable streaming options >/
projects. if we could bundle the knowhow and make "the one" streaming
that works best, this would be helpful in my opinion. Right now there
are at least a few people working on the same goal but different "wheels"

We do not need diversity here, just one good soolution is needed.

If one needs different interfaces or additional options he might fork
the "one" solution then.

But having more solutions with different strategies does not help in
this case.


Lukas attempt seems to be the most direct and thus will be the most
stable one.


Hope we can try to merge the streaming projects together to improove
even more.


I think it would be important to contact Tobias Gall  and Daniel
Schreiber  from TU Chemnitz,  to work together, if not already done. ;-)

> As for Janus: same way the mixer is exposing the result to an RTMP
> endpoint it can be made to expose it to an RTP endpoint (the streaming
> plugin) in a relatively easy way.
> It will be fairly easy to even wrap the mountpoint with SRTP in
> gstreamer as well which would be needed for real deployments.
> That's what gstreamer is great for. So I do think this is very close
> to being able to integrate into the streaming plugin as a prototype.
> Cool stuff.
>

Daniel and Thobias already use Janus - for now based on our capturing -
so this should be fairly easy to merge ...


cheers


Martin

To view this discussion on the web visit

https://groups.google.com/d/msgid/bigbluebutton-dev/CADG0wOD_VdZa0GTq5cAnnFpCzU7fbu6TfMTGqSgHJA9K91ncng%40mail.gmail.com.