DDoS BigBlueButton

[GyR4uK]

Has anyone encountered DDoS attacks on udp ports bbb?

Our bbb servers are subject to DDoS attacks mainly on udp ports, if there are any ways to protect yourself?

Also, after DDoS for some time (5-10 minutes), it is impossible to pass the echo test and the webcam does not work.

bbb-webrtc--sfu.log

{"error":{"errno":"EPIPE","code":"EPIPE","syscall":"write"},"level":"error","message":"[mcs-freeswitch-esl-wrapper] FreeSWITCH ESL connection received error EPIPE","timestamp":"2020-11-17T03:26:23.740Z"}

{"error":{"errno":"EPIPE","code":"EPIPE","syscall":"write"},"level":"error","message":"[mcs-freeswitch-esl-wrapper] FreeSWITCH ESL connection received error EPIPE","timestamp":"2020-11-17T03:26:28.742Z"}

{"error":{"errno":"EPIPE","code":"EPIPE","syscall":"write"},"level":"error","message":"[mcs-freeswitch-esl-wrapper] FreeSWITCH ESL connection received error EPIPE","timestamp":"2020-11-17T03:26:33.744Z"}

[Paulo Lanzarin]

Which UDP ports are the target of the DDoS attack?

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/d4136a5d-0db5-4fc6-a06d-a60c3d58e788n%40googlegroups.com.

[GyR4uK]

Attacks are going all over pool 16384-32768. But mostly the attack is conducted on the freeswitch 16384-24576 pool. But they still don't understand why fs and kurrento don't work after the attack. I suspect while webrtc sfu or redis

вторник, 17 ноября 2020 г. в 19:59:30 UTC+3, plan...@gmail.com:

[basisbit]

What makes you think these issues are caused by DDoS? Did you get a threat from someone or do you see thousands of connect requests per second?

Does it happen on just one server or a bunch of servers? Is your server maybe just above its maximum amount of supported users? (is the nodejs/meteor process close to 80% of one CPU core or so?)

[GyR4uK]

CPU load is not significant and does not exceed 50-60%. We are seeing a sharp spike in the number of connections to our servers. The problem occurs on different servers and this is definitely DDoS

среда, 18 ноября 2020 г. в 01:43:29 UTC+3, basisbit:

[sd...@distancelearning.cloud]

Whats interesting.

 

] FreeSWITCH ESL connection   ß this is all internal network traffic as 8021 should not be exposed to outside and is TCP connection,

The bbb-webrtc-sfu MCS is talking internally to freeswitch.

 

 

I have seen the similar in production.  Where we had a cluster of 8 servers    and all the kurentos crashed the same evening, but that was while ago on older version.

Have also had a bunch of servers get corrupted core db in freeswitch.   Where will not start until the core.db is deleted.

 

Assume these my be related to DDOS attacks but not sure.

 

 

Regards,

Stephen

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/a45069c7-9b8a-4652-a1ce-41f7a29b16f3n%40googlegroups.com.

[GyR4uK]

The latest version of BBB 2.2.29 is used. in addition to fs, kurento does not work (the webcam and screen demonstration are not connected). After a while, everything starts working. It is also strange that when an attack occurs, the sound and webcam are immediately turned off, and when you connect the microphone, an error 1007 appears.

I still don't understand where the overflow occurs.

среда, 18 ноября 2020 г. в 14:58:13 UTC+3, DistanceLearning.cloud:

[Paulo Lanzarin]

That log message is symptomatic of freeswitch stalling, at best. And there's a reconnection procedure in there, so
" I suspect while webrtc sfu or redis" sounds incorrect. Why do you suspect redis? Why do you suspect bbb-webrtc-sfu?
Send more info.

I'm also having trouble understanding how a DDoS would be conducted in the whole UDP range if those ports are 
not bound by default. They're bound on demand. Trying to exploit ports that are not bound would get packets rejected
at the network level. Only thing that would happen is exhausting the bandwidth, not the applications themselves.

So either you have FreeSWITCH and Kurento's control plane ports open in your firewall and you're getting exploited through that,
or there is an underlying vulnerability I'm unaware of.

Whatever it is, we can't magically guess what's going on unless you provide detailed, useful information, as in:
  - are your firewall rules correctly set up?
  - what's the content being directed to that UDP port range?
  - send freeswitch and kurento logs in the time range where the supposed DDoS happened. Syslog would be even better
  - send the output of sudo netstat -putan from a moment where the supposed DDoS happens, if possible

Send any other info that you think would be useful to pinpoint whether this is in fact a DDoS and where the hole is.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/6ba55bf9-d526-4016-9f9d-61775489ae7en%40googlegroups.com.

[GyR4uK]

DDoS is coming with a big spike in simultaneous connections

Firewall using standard ufw

allowed: 80 / tcp 443/tcp 16384-32768/udp

I didn't change the port pool for WebRTC communication

16384-32768/udp

It is strange that after DDoS stops working simultaneously freeswitch and kurento. bbb-conf --restart (clean) does not solve the problem. Disabling the firewall also does not lead to a positive result.

среда, 18 ноября 2020 г. в 16:25:47 UTC+3, plan...@gmail.com:

[GyR4uK]

суббота, 21 ноября 2020 г. в 02:39:37 UTC+3, GyR4uK:

[GyR4uK]

During the next failure log netstat -putan 

[Paulo Lanzarin]

Yeah, nothing is bound to any UDP high range socket, so it must be getting blocked at network level.

Still weird that KMS and FS are getting stuck after it.

>   - send freeswitch and kurento logs in the time range where the supposed DDoS happened. Syslog would be even better

s,

prlanzarin

On Tue, Nov 24, 2020 at 7:16 AM GyR4uK <gyr...@gmail.com> wrote:

During the next failure log netstat -putan 

--
You received this message because you are subscribed to the Google Groups "BigBlueButton-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bigbluebutton-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bigbluebutton-dev/3d730677-232d-400c-8b8e-85375029a774n%40googlegroups.com.

[GyR4uK]

Determine the cause.

The problem was in the operator, when DDoS was active, it started dropping ip over udp.

вторник, 24 ноября 2020 г. в 23:22:14 UTC+3, plan...@gmail.com: