Prosody 0.11.13 (lua5.2) vs 0.12.1 (lua 5.4) vs 0.12.1 (lua 5.2) performance difference

[Krebya]

Hello,

We are using docker-jitsi-meet prosody images with our custom plugins and cfg files.

With the same prosody.cfg.lua, jitsi.meet.cfg.lua, prosody-plugins and custom plugins folder (only difference are lua and prosody version) files I can join 400 participant in prosody 0.11.13 (7001 image) but while using prosody 0.12.1 (7868-3 image) I can only join 280 participant. After these numbers the room crashes.

I have built the image with prosody12.1 with lua5.2 .

Also the luajwtjitsi.lib.lua I used is from luajwtjitsi version 2.0.0 from luarocks because the one in latest prosody-plugins is not compatible to 5.2 only worked with 5.4. I had changed it previously while using 5.4 to one from jitsi (and the line in token/util.lib added the module:require line, I reversed it while building 5.2).

I have reached to 441 participant and couldn’t make it to crash the bots are finished. 

What makes the difference do you have an idea between lua 5.2 and lua 5.4 with prosody 12.1

Do you have any recommendation?

(jitsi thread-> https://community.jitsi.org/t/prosody-0-11-13-lua5-2-vs-0-12-1-lua-5-4-performance-difference/116785 )

Thank you in advance.
Best

[Matthew Wild]

Hi,

On Mon, 29 Aug 2022 at 12:08, Krebya <mkale...@gmail.com> wrote:
> What makes the difference do you have an idea between lua 5.2 and lua 5.4 with prosody 12.1
>
> Do you have any recommendation?
>

We are aware of the performance difference between Lua 5.2/5.3 and Lua 5.4.

Lua 5.2 and 5.3 have a garbage collector that struggles to handle
Prosody's memory allocation patterns efficiently.

Lua 5.4 is our recommendation for best performance, however many Lua
libraries in Debian 11 are not enabled for Lua 5.4. Most Prosody
deployments are relatively small, so Lua 5.2/5.3 is still fine for
these servers. We have contributed updates to the necessary packages
in Debian so that Debian 12 will be able to run Prosody on Lua 5.4
using only system packages.

Regards,
Matthew

[Krebya]

Thank you for your answer.

So because of these package issue as I understand from your answer, 

it is known that prosody 12.1 with lua 5.2 performs better than prosody 12.1 with lua 5.4.

But both my tests are in debian 11 inside podman container.

Is it possible that prosody 12.1 with lua 5.2 performs better than prosody 12.1 with lua 5.4 while both being in debian 11?

Thanks in advance

Best

[Matthew Wild]

On Mon, 29 Aug 2022 at 14:10, Krebya <mkale...@gmail.com> wrote:
>
> Thank you for your answer.
>
> So because of these package issue as I understand from your answer,
>
> it is known that prosody 12.1 with lua 5.2 performs better than prosody 12.1 with lua 5.4.

No, I was saying the opposite: Lua 5.4 performs better than Lua 5.2.

Are you saying that 5.2 is better for you?

> Is it possible that prosody 12.1 with lua 5.2 performs better than prosody 12.1 with lua 5.4 while both being in debian 11?

The Debian version only affects what versions of dependencies are
available from apt. Debian version alone won't affect performance.

Regards,
Matthew

[Krebya]

Yes As I have written in the first post I can reach 441 connections with prosody 12.1 with lua5.2 , however at  281 connections prosody 12.1 with lua 5.4 crashes . 

[Matthew Wild]

On Tue, 30 Aug 2022 at 10:16, Krebya <mkale...@gmail.com> wrote:
>
> Yes As I have written in the first post I can reach 441 connections with prosody 12.1 with lua5.2 , however at 281 connections prosody 12.1 with lua 5.4 crashes .

Can you provide more information about this "crash"? People use this
word to mean a lot of different things. Was Prosody still running, or
not? Was it consuming CPU? If it exited, was it a segfault, or did it
abort? Was it killed by the OOM killer?

Running `dmesg` on your system is often a good start to answering
these questions. If Prosody is still running, an strace capture would
be helpful (use strace -p to attach to a running process).

Regards,
Matthew

[Krebya]

In lua 5.4, thx lua process reaches %100 cpu usage then the meeting closes however the Prosody stil running , no Logs in error level but some error iq messages and unacked messages. In lua 5.2 stil reaches to %98 cpu usage when participants connecting but then drop after the entrance.  

Log:>

Destroying session with 107 unacked stanzas

Aug 25 11:46:50 c2s55df8420b9b0    debug    Sending[c2s]: <a xmlns='urn:xmpp:sm:3' h='5'> Aug 25 11:46:50 c2s55df8420b9b0    debug    Handled 6 incoming stanzas Aug 25 11:46:50 c2s55df8420b9b0    debug    Received[c2s]: <presence type='unavailable' xml:lang='en'> Aug 25 11:46:50 c2s55df8420b9b0    debug    Sending[c2s]: <presence type='unavailable' xml:lang='en' from='9a675492-6256-41f5...@guest.meet.jitsi.com/OwWiTDN1xLNb'> Aug 25 11:46:50 c2s55df8420b9b0    debug    Received </stream:stream> Aug 25 11:46:50 c2s55df8420b9b0    debug    Sending[c2s]: <a xmlns='urn:xmpp:sm:3' h='6'> Aug 25 11:46:50 c2s55df8420b9b0    debug    Sending[c2s]: <close xmlns='urn:ietf:params:xml:ns:xmpp-framing'> Aug 25 11:46:50 c2s55df8420b9b0    debug    c2s stream for 9a675492-6256-41f5...@guest.meet.jitsi.com/OwWiTDN1xLNb closed: session closed Aug 25 11:46:50 c2s55df8420b9b0    debug    Destroying session for 9a675492-6256-41f5...@guest.meet.jitsi.com/OwWiTDN1xLNb (9a675492-6256-41f5...@guest.meet.jitsi.com) Aug 25 11:46:50 c2s55df8420b9b0    debug    Destroying session with 107 unacked stanzas Aug 25 11:46:50 c2s55df8420b9b0    debug    Received[c2s]: <presence to='a...@conference.preprodtenant.meet.jitsi.com/focus' type='error' from='9a675492-6256-41f5...@guest.meet.jitsi.com/OwWiTDN1xLNb'> Aug 25 11:46:50 stanzarouter    debug    Discarding unhandled error presence (cancel, recipient-unavailable) from c2s: <presence to='[preprodtenant]a...@conference.meet.jitsi.com/focus' type='error' from='9a675492-6256-41f5...@guest.meet.jitsi.com/OwWiTDN1xLNb'>

[Matthew Wild]

On Tue, 30 Aug 2022 at 10:44, Krebya <mkale...@gmail.com> wrote:
>
> In lua 5.4, thx lua process reaches %100 cpu usage then the meeting closes however the Prosody stil running , no Logs in error level but some error iq messages and unacked messages. In lua 5.2 stil reaches to %98 cpu usage when participants connecting but then drop after the entrance.

So it sounds like a timeout was reached, rather than a crash.

As the garbage collector is one of the primary factors in performance
that also changed between Lua 5.2 and 5.4, you might want to try
tweaking the GC configuration. See this documentation page:
https://prosody.im/doc/advanced_gc

If you have plenty of RAM available, first experiment with increasing
the 'threshold' value - try 150, 200 or 250 to start with.

You can also experiment with increasing/decreasing the 'speed' and
'step_size' values (e.g. try step_size = 20).

It's hard to give generic advice, because optimal values depend on
your system, your traffic patterns, Lua version and Prosody
configuration.

Regards,
Matthew

[Krebya]

Hello,

Thank you for your suggestions I will try to tweak gc 

BTW I  was using same gc settings for both 5.2 and 5.4

gc = { mode = "incremental"; threshold = 105; speed = 250; }

[Matthew Wild]

On Tue, 30 Aug 2022 at 11:59, Krebya <mkale...@gmail.com> wrote:
> BTW I was using same gc settings for both 5.2 and 5.4
>
> gc = { mode = "incremental"; threshold = 105; speed = 250; }

The GC is different in every Lua version, so the behaviour is not
expected to be the same, even if the settings are the same. Prosody's
defaults try to strike a balance that works acceptably on a range of
Lua versions and systems, but if you are planning to place some load
on Prosody, it's always good to tweak the values to figure out what is
optimal for your deployment. Unfortunately there are so many factors
in GC performance, it's hard to give general advice that works for
everyone. That's why we provide the configuration and document what
guidance we can.

Regards,
Matthew

[Krebya]

Hello,

I have changed gc settings to gc =
{ mode = "incremental"; threshold = 150; speed = 250; step_size=13; }.

I have joined 441 participants and the room did not closed with prosody 12.1 and lua 5.4

I will tweak the gc more however my issue was definitely related with gc settings.

Thank you very much for your suggestions

Best