Opencast clustering performance

[Sven Laudel]

Hello fellow Opencasters,

as the need for online and video learning rises at the moment, I just want to ask, how one could improve the performance of an opencast installation.

I was asked by our management, how many lectures we can provide, if our university switches to online courses this semester. 

Right now we have one admin, 3 worker and one presentation node.

All of our opencast nodes are in our private network, so to access both, the admin and the presentation node, we use a loadbalancer as reverse proxy in front of it. 

The admin node would be backed by one or more additional ingest nodes and much more worker nodes.

But could I place multiple presentation nodes behind the loadbalancer to improve delivery performance?

So what experiences do you have regarding opencast performance?

Best regards

Sven

[Lars Kiesow]

Multiple presentation nodes is basically nonsense. But you can use e.g.
Nginx to deliver the artifacts directly (serving /static). That you can
even do with a load balancer though I doubt it's actually necessary.
–Lars

[Greg Logan]

The advantage for multiple delivery nodes is higher availability, and (in theory) better search performance.  IIRC there are very few organizations who are running clustered delivery nodes, and those are mainly doing it for availability - you would have to be dealing with massive numbers of search requests before the search engine itself (Solr) becomes a problem.  Obviously if you're somehow running into a situation where your users are consuming more bandwidth than your single node has then a cluster helps, but IMO I would do what Lars suggested in that case and start spinning off nginx instances to serve the static content.

G

--
To unsubscribe from this group and stop receiving emails from it, send an email to users+un...@opencast.org.

[Kristof Keppens]

Hi Lars,

Could you briefly explain why this is nonsense.

We, as a lot of universities at the moment, are also looking into the options of scaling everything up to accommodate the extra demand of distance learning due to COVID-19. Before we upgraded to opencast 7 we had some issues with the presentation node going out-of-memory ( while having a generous 32Gb of memory ), so far this hasn’t occurred but it might be prudent to see how we can improve the presentation part of our installation before we have issues.

Kristof

[Sven Laudel]

Hello all,

I want to agree to Kristof, please let us know a bit more details about the presentation part of opencast.

Could one node deliver recordings to lets say 1000 simultaneous viewers if not using an additional streaming server? Or what would be the limiting factor, bandwith, cpu, memory?

Is there any documentation what to do, to enable the use of an additional webserver for content delivery?

I think I read over a configuration option in "etc" but I'm not sure, how to configure the whole thing.

Best regards

Sven

To unsubscribe from this group and stop receiving emails from it, send an email to us...@opencast.org.

[Lars Kiesow]

Hi,
very shortly, the presentation node basically does these things:

1. Create distribution channels (add single items in Solr)
2. OAI-PMH
3. Search through Engage publication (request things from Solr)
4. Deliver files

The load for one and two should not dramatically increase since it's no
hard work and you should only have a moderate amount of more work to do
when the amount of recordings increases. Essentially, I'm saying that
you have other problems before writing new recordings to a db becomes a
problem.

Three will increase but I've never seen Solr be a real problem even
under higher load. Usually other things become a problem much faster
than that. That said, you can run a dedicated Solr node instead if you
like and that's also what you would need anyway if you run multiple
presentation nodes. But, you likely even do not need that.

The hardest part for the presentation node is actually delivering files
and the most critical part here is probably getting and delivering the
files fast enough. A very easy way of improving performance here is
to let your HTTP proxy deliver the static files instead of having
Opencast internally deliver them. E.g. if you have Nginx running on
your presentation node, add a configuration like this:

location ^~ /static/ {
alias /path/to/opencast/downloads/;
sendfile on;
}

Nginx has a very high performance when it comes to delivering static
files and this should already help quite a bit. But then, you can also
push this part to another server or even have that behind a load
balancer which might be able to help you in case of network performance
problems.

Sorry for the short reply, we are pretty busy right now as well ;-)

–Lars

[Sven Laudel]

Hi Lars,

thanks for your reply. 

For me the delivery part was of most interest.
It's just what I needed to understand the presentation part and I will try configuring a webserver to serve the static content.

Best regards

Sven

> > send an email to us...@opencast.org.
>

[Rainer Rillke]

https://opencast.org/2015-05-01-university-of-manchester.html might be
somewhat of help when planning to upscale

> <javascript:>>, wrote:
> > > Multiple presentation nodes is basically nonsense. But you can use
> > > e.g. Nginx to deliver the artifacts directly (serving /static).
> > > That you can even do with a load balancer though I doubt it's
> > > actually necessary. –Lars
> > >
> > >
> > > On Wed, 11 Mar 2020 03:28:57 -0700 (PDT)
> > > "'Sven Laudel' via Opencast Users" <us...@opencast.org

> <javascript:>> wrote:
> > >  
> > > > Hello fellow Opencasters,
> > > >
> > > > as the need for online and video learning rises at the moment, I
> > > > just want to ask, how one could improve the performance of an
> > > > opencast installation. I was asked by our management, how many
> > > > lectures we can provide, if our university switches to online
> > > > courses this semester.
> > > >
> > > > Right now we have one admin, 3 worker and one presentation node.
> > > > All of our opencast nodes are in our private network, so to
> access
> > > > both, the admin and the presentation node, we use a loadbalancer
> > > > as reverse proxy in front of it.
> > > >
> > > > The admin node would be backed by one or more additional ingest
> > > > nodes and much more worker nodes.
> > > > But could I place multiple presentation nodes behind the
> > > > loadbalancer to improve delivery performance?
> > > >
> > > > So what experiences do you have regarding opencast performance?
> > > >
> > > > Best regards
> > > > Sven
> > > >  
> > >
> > > --
> > > To unsubscribe from this group and stop receiving emails from it,

> > > send an email to us...@opencast.org <javascript:>.

> >
>
> --
> To unsubscribe from this group and stop receiving emails from it, send

> an email to users+un...@opencast.org
> <mailto:users+un...@opencast.org>.

[Dietmar Zenker]

Hi all,

in addition to the sendfile option, there are two other options to optimize the delivery of static content by nginx: directio and asynchronous I/O (aio). As described here in the "Configuring NGINX I/O" section, directio is suitable for speeding up the delivery of large files (together with aio).  The three directives can also be mixed together to achieve different objectives.

Does anybody have experience with the directio and aio options, especially with CentOS? Are there any potential side effects and do they really significantly speed up delivery of video files?

Greetings,

Dietmar