Context has not been initialized

[Konrad Piwowar]

I'm getting thousands apon thousands of lines like this spammed into our antmedia error log (2.3.0 RC1 Feb 24th build date)

[SWR @ 0x7fe01c08d6c0] Context has not been initialized

[SWR @ 0x7fe01c03da40] Context has not been initialized

[SWR @ 0x7fe01c078700] Context has not been initialized

[SWR @ 0x7fe01c08d6c0] Context has not been initialized

[SWR @ 0x7fe01c03da40] Context has not been initialized

[SWR @ 0x7fe01c078700] Context has not been initialized

[SWR @ 0x7fe01c08d6c0] Context has not been initialized

[SWR @ 0x7fe01c03da40] Context has not been initialized

[SWR @ 0x7fe01c078700] Context has not been initialized

[SWR @ 0x7fe01c08d6c0] Context has not been initialized

[SWR @ 0x7fe01c03da40] Context has not been initialized

[SWR @ 0x7fe01c078700] Context has not been initialized

[SWR @ 0x7fe01c08d6c0] Context has not been initialized

[SWR @ 0x7fe01c03da40] Context has not been initialized

[SWR @ 0x7fe01c078700] Context has not been initialized

[SWR @ 0x7fe01c08d6c0] Context has not been initialized

[SWR @ 0x7fe01c03da40] Context has not been initialized

[SWR @ 0x7fe01c078700] Context has not been initialized

[SWR @ 0x7fe01c08d6c0] Context has not been initialized

[SWR @ 0x7fe01c03da40] Context has not been initialized

[SWR @ 0x7fe01c078700] Context has not been initialized

[SWR @ 0x7fe01c08d6c0] Context has not been initialized

[SWR @ 0x7fe01c03da40] Context has not been initialized

[SWR @ 0x7fe01c078700] Context has not been initialized

[SWR @ 0x7fe01c08d6c0] Context has not been initialized

[SWR @ 0x7fe01c03da40] Context has not been initialized

[SWR @ 0x7fe01c078700] Context has not been initialized

[SWR @ 0x7fe01c08d6c0] Context has not been initialized

[SWR @ 0x7fe01c03da40] Context has not been initialized

[SWR @ 0x7fe01c078700] Context has not been initialized

[SWR @ 0x7fe01c08d6c0] Context has not been initialized

[SWR @ 0x7fe01c03da40] Context has not been initialized

Almost 5 gigs of this has been written in a few days. 

Any ideas?

[Konrad Piwowar]

Other errors we are getting

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/usr/local/antmedia/lib/spring-core-4.3.28.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)

WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

Mar 03, 2021 9:36:41 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

INFO: Loaded Apache Tomcat Native library [1.2.25] using APR version [1.6.3].

Mar 03, 2021 9:36:41 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].

Mar 03, 2021 9:36:41 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

INFO: APR/OpenSSL configuration: useAprConnector [true], useOpenSSL [true]

Mar 03, 2021 9:36:41 PM org.apache.catalina.core.AprLifecycleListener initializeSSL

INFO: OpenSSL successfully initialized [OpenSSL 1.1.1  11 Sep 2018]

Mar 03, 2021 9:36:41 PM org.apache.coyote.AbstractProtocol init

INFO: Initializing ProtocolHandler ["http-apr-0.0.0.0-5080"]

Mar 03, 2021 9:36:41 PM org.apache.coyote.AbstractProtocol init

INFO: Initializing ProtocolHandler ["https-openssl-apr-0.0.0.0-5443"]

Mar 03, 2021 9:36:41 PM org.apache.catalina.core.StandardService startInternal

INFO: Starting service [Tomcat]

Mar 03, 2021 9:36:41 PM org.apache.catalina.core.StandardEngine startInternal

INFO: Starting Servlet Engine: Apache Tomcat/8.5.58

Mar 03, 2021 9:36:41 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment

INFO: No global web.xml found

Mar 03, 2021 9:36:41 PM org.apache.catalina.core.ApplicationContext log

INFO: No Spring WebApplicationInitializer types detected on classpath

Mar 03, 2021 9:36:42 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize

SEVERE: Spring context lookup failed, skipping spring component provider initialization.

Mar 03, 2021 9:36:42 PM org.apache.catalina.core.ApplicationContext log

INFO: No Spring WebApplicationInitializer types detected on classpath

Mar 03, 2021 9:36:42 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize

SEVERE: Spring context lookup failed, skipping spring component provider initialization.

Mar 03, 2021 9:36:42 PM org.apache.catalina.core.ApplicationContext log

INFO: No Spring WebApplicationInitializer types detected on classpath

Mar 03, 2021 9:36:43 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize

SEVERE: Spring context lookup failed, skipping spring component provider initialization.

Mar 03, 2021 9:36:43 PM org.apache.coyote.AbstractProtocol start

INFO: Starting ProtocolHandler ["http-apr-0.0.0.0-5080"]

Mar 03, 2021 9:36:43 PM org.apache.coyote.AbstractProtocol start

INFO: Starting ProtocolHandler ["https-openssl-apr-0.0.0.0-5443"]

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

Antmedia has stopped responding 3 times today in the last few hours. We are doing WebRTC ingress and webrtc egress, no HLS or anythign else.

[Konrad Piwowar]

The context has not been initialized error was spamming shortly before the daemon stopped responding.

[Konrad Piwowar]

Sorry for the spam, trying to get all the info I can out for the devs to look at. The daemon has stopped responding quite a few times today it seems. Prior to today, nothing.

-rw-r--r-- 1 antmedia antmedia  185760 Mar  3 16:51 hs_err_pid10035.log

-rw-r--r-- 1 antmedia antmedia  222340 Mar  3 17:01 hs_err_pid10455.log

-rw-r--r-- 1 antmedia antmedia  256260 Mar  3 18:09 hs_err_pid12515.log

-rw-r--r-- 1 antmedia antmedia  311293 Mar  3 14:39 hs_err_pid1965.log

-rw-r--r-- 1 antmedia antmedia  162788 Mar  3 14:39 hs_err_pid24133.log

-rw-r--r-- 1 antmedia antmedia  161692 Mar  3 14:40 hs_err_pid24465.log

-rw-r--r-- 1 antmedia antmedia  177154 Mar  3 14:41 hs_err_pid24772.log

-rw-r--r-- 1 antmedia antmedia  196094 Mar  3 14:43 hs_err_pid25205.log

-rw-r--r-- 1 antmedia antmedia  229247 Mar  3 18:09 hs_err_pid25336.log

-rw-r--r-- 1 antmedia antmedia  192489 Mar  3 14:44 hs_err_pid25818.log

-rw-r--r-- 1 antmedia antmedia  199314 Mar  3 18:10 hs_err_pid26138.log

-rw-r--r-- 1 antmedia antmedia  149464 Mar  3 14:45 hs_err_pid26250.log

-rw-r--r-- 1 antmedia antmedia  145194 Mar  3 14:47 hs_err_pid26550.log

-rw-r--r-- 1 antmedia antmedia  158277 Mar  3 18:11 hs_err_pid26718.log

-rw-r--r-- 1 antmedia antmedia  147102 Mar  3 14:48 hs_err_pid26811.log

-rw-r--r-- 1 antmedia antmedia  239648 Mar  3 16:50 hs_err_pid27043.log

-rw-r--r-- 1 antmedia antmedia  201374 Mar  3 18:13 hs_err_pid27050.log

-rw-r--r-- 1 antmedia antmedia  188197 Mar  3 18:14 hs_err_pid27586.log

-rw-r--r-- 1 antmedia antmedia  185486 Mar  3 18:15 hs_err_pid28057.log

-rw-r--r-- 1 antmedia antmedia  197257 Mar  3 18:16 hs_err_pid28501.log

-rw-r--r-- 1 antmedia antmedia  184783 Mar  3 18:17 hs_err_pid29006.log

-rw-r--r-- 1 antmedia antmedia  177905 Mar  3 18:18 hs_err_pid29451.log

-rw-r--r-- 1 antmedia antmedia  169537 Mar  3 18:20 hs_err_pid29777.log

-rw-r--r-- 1 antmedia antmedia  169944 Mar  3 18:21 hs_err_pid30213.log

-rw-r--r-- 1 antmedia antmedia  161182 Mar  3 18:22 hs_err_pid30495.log

-rw-r--r-- 1 antmedia antmedia  172153 Mar  3 18:23 hs_err_pid30886.log

-rw-r--r-- 1 antmedia antmedia  164505 Mar  3 18:24 hs_err_pid31280.log

-rw-r--r-- 1 antmedia antmedia  148861 Mar  3 18:25 hs_err_pid31632.log

-rw-r--r-- 1 antmedia antmedia  158891 Mar  3 18:26 hs_err_pid31877.log

-rw-r--r-- 1 antmedia antmedia  156248 Mar  3 18:28 hs_err_pid32078.log

-rw-r--r-- 1 antmedia antmedia  157205 Mar  3 18:29 hs_err_pid32315.log

-rw-r--r-- 1 antmedia antmedia  250172 Mar  3 20:07 hs_err_pid32538.log

-rw-r--r-- 1 antmedia antmedia  206044 Mar  3 20:07 hs_err_pid46973.log

-rw-r--r-- 1 antmedia antmedia  226068 Mar  3 20:09 hs_err_pid47665.log

-rw-r--r-- 1 antmedia antmedia  207825 Mar  3 20:10 hs_err_pid48972.log

-rw-r--r-- 1 antmedia antmedia  212255 Mar  3 20:33 hs_err_pid5783.log

-rw-r--r-- 1 antmedia antmedia  242304 Mar  3 20:25 hs_err_pid777.log

-rw-r--r-- 1 antmedia antmedia  181275 Mar  3 20:33 hs_err_pid8738.log

-rw-r--r-- 1 antmedia antmedia  157187 Mar  3 20:34 hs_err_pid9180.log

-rw-r--r-- 1 antmedia antmedia  168797 Mar  3 20:35 hs_err_pid9485.log

-rw-r--r-- 1 antmedia antmedia  190964 Mar  3 16:50 hs_err_pid9609.log

-rw-r--r-- 1 antmedia antmedia  214222 Mar  3 20:39 hs_err_pid9765.log

The "Context has not been initialized" error seems to happen around the same time as the daemon stops responding. It's impossible to stop it via "service antmedia stop" - we must kill the PID.

The issue is very likely correlatated with someone trying to broadcast from Safari on an IPad (WebRTC Ingress) with the following user agent:

5.0 (iPad; CPU OS 12_4_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1.2 Mobile/15E148 Safari/604.1

The server in question was crashing like crazy, and we moved that user to another server. The first server stopped crashing, the second server started.

 

[Selim Emre]

Hi Konrad,

Thank you for reporting to us!

We need to check the beginning of Ant Media Server logs. Could you please send us a full of logs? You can find ant-media-server.log file under the /usr/local/antmedia/log folder. Also, please send hs_err_pid10035.log file. Are you using Adaptive bitrate?

Looking forward to hearing from you soon.

Best Regards,

Selim

[chandrashekar kola]

We also started noticing the same error for last few days. Do we know what can cause it. When ever we see this error, ant media server getting restarted. see below for the logs.

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/usr/local/antmedia/lib/spring-core-4.3.28.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)

WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

Mar 03, 2021 7:53:49 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

INFO: Loaded Apache Tomcat Native library [1.2.25] using APR version [1.6.3].

Mar 03, 2021 7:53:49 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].

Mar 03, 2021 7:53:49 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

INFO: APR/OpenSSL configuration: useAprConnector [true], useOpenSSL [true]

Mar 03, 2021 7:53:49 PM org.apache.catalina.core.AprLifecycleListener initializeSSL

INFO: OpenSSL successfully initialized [OpenSSL 1.1.1  11 Sep 2018]

Mar 03, 2021 7:53:49 PM org.apache.coyote.http11.AbstractHttp11Protocol configureUpgradeProtocol

INFO: The ["http-apr-0.0.0.0-5080"] connector has been configured to support HTTP upgrade to [h2c]

Mar 03, 2021 7:53:49 PM org.apache.coyote.AbstractProtocol init

INFO: Initializing ProtocolHandler ["http-apr-0.0.0.0-5080"]

Mar 03, 2021 7:53:49 PM org.apache.coyote.http11.AbstractHttp11Protocol configureUpgradeProtocol

INFO: The ["https-openssl-apr-0.0.0.0-5443"] connector has been configured to support negotiation to [h2] via ALPN

Mar 03, 2021 7:53:49 PM org.apache.coyote.AbstractProtocol init

INFO: Initializing ProtocolHandler ["https-openssl-apr-0.0.0.0-5443"]

Mar 03, 2021 7:53:49 PM org.apache.catalina.core.StandardService startInternal

INFO: Starting service [Tomcat]

Mar 03, 2021 7:53:49 PM org.apache.catalina.core.StandardEngine startInternal

INFO: Starting Servlet Engine: Apache Tomcat/8.5.58

Mar 03, 2021 7:53:49 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment

INFO: No global web.xml found

Mar 03, 2021 7:53:49 PM org.apache.catalina.core.ApplicationContext log

INFO: No Spring WebApplicationInitializer types detected on classpath

Mar 03, 2021 7:53:50 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize

SEVERE: Spring context lookup failed, skipping spring component provider initialization.

Mar 03, 2021 7:53:50 PM org.apache.catalina.core.ApplicationContext log

INFO: No Spring WebApplicationInitializer types detected on classpath

Mar 03, 2021 7:53:50 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize

SEVERE: Spring context lookup failed, skipping spring component provider initialization.

Mar 03, 2021 7:53:50 PM org.apache.catalina.core.ApplicationContext log

INFO: No Spring WebApplicationInitializer types detected on classpath

Mar 03, 2021 7:53:50 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize

SEVERE: Spring context lookup failed, skipping spring component provider initialization.

Mar 03, 2021 7:53:51 PM org.apache.coyote.AbstractProtocol start

INFO: Starting ProtocolHandler ["http-apr-0.0.0.0-5080"]

[Konrad Piwowar]

Selim, i've sent you the log file and a pid. We are still waiting for the user to reply with the exact device model they are using but it seems to be an old as hell iPad - the resolution we locked was 768x1024 so it's probably circa 2013-2014 at best. One of our staff noted we actually had an influx of crashes before as well when someone tried to stream from an old iPhone. It seems that these users are able to fairly consistently cause Antmedia daemon to stop responding just by streaming to it.

We have for now disabled this users account as  it was killing our production environment, and disabled streaming from iOS prior to v13 to prevent old devices from using the system.

Staff member tried streaming through 2.3.0 RC1 with his  IPhone XR (iOS 14) and the log is also flooding with these errors - albeit it didn't kill the daemon, so it seems to be recreatable by using Safari on an iPhone/iPad as WebRTC Ingress.

[Konrad Piwowar]

Sorry missed the Adaptive streaming question - Yes, we are using adaptive.

[Konrad Piwowar]

The user that was crashing our systems by streaming was using iPad A1475 which seems to be a first generation ipad air.

[Selim Emre]

Hi All,

Thanks for the details. I'm checking this issue in our test environment.

Best Regards,

Selim

[Konrad Piwowar]

And progress on this? We're still getting constant crashes, and it seems to trace back to iOS Safari WebRTC Ingress

[Selim Emre]

Hi Konrad,

Sorry for the delay. I checked your issue. As I remember we did some improvements for this case in v2.3(after the RC version). Could you please try latest version? 

Best Regards,

Selim

[Konrad Piwowar]

The latest snapshot available for download is from March 2nd, prior to me reporting this.

[Konrad Piwowar]

This issue is not resolved in the March 2nd build. We are also having substantial issues with audio not working for many broadcasters or cutting out.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

java.lang.NullPointerException

        at io.antmedia.enterprise.adaptive.AVFramePool.getAVFrame(AVFramePool.java:25)

        at io.antmedia.enterprise.adaptive.WebRTCEncoderAdaptor.onOpusPacket(WebRTCEncoderAdaptor.java:1007)

        at io.antmedia.enterprise.adaptive.WebRTCEncoderAdaptor.lambda$getAudioDecoderFactoryFactory$0(WebRTCEncoderAdaptor.java:960)

        at org.webrtc.BuiltinAudioDecoderFactoryFactory.onAudioPacket(BuiltinAudioDecoderFactoryFactory.java:41)

Exception in thread "network_thread - 27933"

# Fatal error in: gen/sdk/android/generated_builtin_audio_codes_jni/../../../../../../sdk/android/src/jni/jni_generator_helper.h, line 85

# last system error: 0

# Check failed: !env->ExceptionCheck()

# WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/usr/local/antmedia/lib/spring-core-4.3.28.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)

WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

Mar 08, 2021 12:48:16 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

INFO: Loaded Apache Tomcat Native library [1.2.25] using APR version [1.6.3].

Mar 08, 2021 12:48:16 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].

Mar 08, 2021 12:48:16 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

INFO: APR/OpenSSL configuration: useAprConnector [true], useOpenSSL [true]

Mar 08, 2021 12:48:16 PM org.apache.catalina.core.AprLifecycleListener initializeSSL

INFO: OpenSSL successfully initialized [OpenSSL 1.1.1  11 Sep 2018]

Mar 08, 2021 12:48:16 PM org.apache.coyote.AbstractProtocol init

INFO: Initializing ProtocolHandler ["http-apr-0.0.0.0-5080"]

Mar 08, 2021 12:48:16 PM org.apache.coyote.AbstractProtocol init

INFO: Initializing ProtocolHandler ["https-openssl-apr-0.0.0.0-5443"]

Mar 08, 2021 12:48:16 PM org.apache.catalina.core.StandardService startInternal

INFO: Starting service [Tomcat]

Mar 08, 2021 12:48:16 PM org.apache.catalina.core.StandardEngine startInternal

INFO: Starting Servlet Engine: Apache Tomcat/8.5.58

Mar 08, 2021 12:48:16 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment

INFO: No global web.xml found

Mar 08, 2021 12:48:16 PM org.apache.catalina.core.ApplicationContext log

INFO: No Spring WebApplicationInitializer types detected on classpath

Mar 08, 2021 12:48:17 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize

SEVERE: Spring context lookup failed, skipping spring component provider initialization.

Mar 08, 2021 12:48:18 PM org.apache.catalina.core.ApplicationContext log

INFO: No Spring WebApplicationInitializer types detected on classpath

Mar 08, 2021 12:48:18 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize

SEVERE: Spring context lookup failed, skipping spring component provider initialization.

Mar 08, 2021 12:48:18 PM org.apache.catalina.core.ApplicationContext log

INFO: No Spring WebApplicationInitializer types detected on classpath

Mar 08, 2021 12:48:18 PM org.glassfish.jersey.server.spring.SpringComponentProvider initialize

SEVERE: Spring context lookup failed, skipping spring component provider initialization.

Mar 08, 2021 12:48:18 PM org.apache.coyote.AbstractProtocol start

INFO: Starting ProtocolHandler ["http-apr-0.0.0.0-5080"]

Mar 08, 2021 12:48:18 PM org.apache.coyote.AbstractProtocol start

INFO: Starting ProtocolHandler ["https-openssl-apr-0.0.0.0-5443"]

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

(rtp_sender.cc:593): Tried to get DTMF sender from video sender.

[Selim Emre]

Hi Konrad,

Thanks for the details. I will discuss this issue with our technical team and make an effort to get it resolved quickly.

Best Regards,

Selim

[Konrad Piwowar]

Thanks - some more logs from times it crashed/stopped responding, or marked SEVERE:

java.lang.NullPointerException

        at io.antmedia.enterprise.adaptive.AVFramePool.getAVFrame(AVFramePool.java:25)

        at io.antmedia.enterprise.adaptive.WebRTCEncoderAdaptor.onOpusPacket(WebRTCEncoderAdaptor.java:1007)

        at io.antmedia.enterprise.adaptive.WebRTCEncoderAdaptor.lambda$getAudioDecoderFactoryFactory$0(WebRTCEncoderAdaptor.java:960)

        at org.webrtc.BuiltinAudioDecoderFactoryFactory.onAudioPacket(BuiltinAudioDecoderFactoryFactory.java:41)

Exception in thread "network_thread - 27933"

#

# Fatal error in: gen/sdk/android/generated_builtin_audio_codecs_jni/../../../../../../sdk/android/src/jni/jni_generator_helper.h, line 85

# last system error: 0

# Check failed: !env->ExceptionCheck()

# WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/usr/local/antmedia/lib/spring-core-4.3.28.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)

WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

-----------------

Tons of this being spammed in the logs (thousands of entries), after we tried switching to Mono yesterday to try see if that resolves it.

[opus @ 0x7f7a7012a1c0] Error decoding a SILK frame.

[opus @ 0x7f7a7012a1c0] Error decoding an Opus frame.

[opus @ 0x7f7a7012a1c0] Error decoding a SILK frame.

[opus @ 0x7f7a7012a1c0] Error decoding an Opus frame.

[Konrad Piwowar]

Another log from another unresponsiveness/crash today (2.3.0 March 2nd build)

Mar 09, 2021 6:07:18 PM org.apache.coyote.AbstractProtocol$ConnectionHandler process

SEVERE: Error reading request, ignored

java.lang.IllegalArgumentException: Reason Phrase cannot exceed 123 UTF-8 encoded bytes: Unexpected error [20,014] reading data from the APR/native socket [140,548,515,160,224] with wrapper [org.apache.tomcat.util.net.AprEndpoint$A$

        at javax.websocket.CloseReason.<init>(CloseReason.java:70)

        at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.upgradeDispatch(WsHttpUpgradeHandler.java:153)

        at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)

        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:59)

        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818)

        at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2075)

        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.base/java.lang.Thread.run(Thread.java:834)

Mar 09, 2021 6:07:18 PM org.apache.coyote.AbstractProtocol$ConnectionHandler process

SEVERE: Error reading request, ignored

java.lang.IllegalArgumentException: Reason Phrase cannot exceed 123 UTF-8 encoded bytes: Unexpected error [20,014] reading data from the APR/native socket [140,548,513,878,176] with wrapper [org.apache.tomcat.util.net.AprEndpoint$A$

        at javax.websocket.CloseReason.<init>(CloseReason.java:70)

        at org.apache.tomcat.websocket.server.WsHttpUpgradeHandler.upgradeDispatch(WsHttpUpgradeHandler.java:153)

        at org.apache.coyote.http11.upgrade.UpgradeProcessorInternal.dispatch(UpgradeProcessorInternal.java:54)

        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:59)

        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:818)

        at org.apache.tomcat.util.net.AprEndpoint$SocketProcessor.doRun(AprEndpoint.java:2075)

        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

        at java.base/java.lang.Thread.run(Thread.java:834)

[Konrad Piwowar]

And just to be clear as to what we consider unresponsive - the criteria for our script that catches and logs these issues is HTTP accessibility to Antmedia, so we are testing http://localhost:5080 and it takes longer then 5 seconds to respond we kill the pid and restart it. During timeouts like this, we are unable to do 'service antmedia stop' as it just hangs there.

[Selim Emre]

Hi Konrad,

Thanks for the great feedback!

Another log from another unresponsiveness/crash today (2.3.0 March 2nd build)

Is there any reproduce scenario for this issue? Reproduce scenario is important for the understand real cause.

Best Regards,

Selim

[Konrad Piwowar]

Unfortunately we haven't been able to identify anything in particular. The site just has performers streaming using WebRTC ingress and viewers using WebRTC Egress. It's random. Sometimes will happen 5x a day sometimes won't happen for 2 weeks. Sometimes there's 10 performers on a server and it's fine, sometimes with 1 it starts locking up.

The only crash we were able to identify was caused by Safari WebRTC Ingress on a  first gen Ipad Air but even when we blocked iOS < v13 we still have the daemon becoming unresponsive. The rest are using various devices, mostly chrome on pc, sometimes macs, sometimes android tablets or phones. Everything is done via browser.

This has been going effectively since we started using the software a year ago and the only way we can keep everything going is by running our timeout script on a cron every 2 minutes checking the status, killing the pid and starting the daemon again.  We've had alot of problems overall. Egress tends to be choppy for the first 15-20 seconds and then clears up - sometimes remains choppy but in some cases it is performer having terrible internet, 10 year old laptop or they are using wifi and apparently live in a Faraday cage. 

We suspected it might have to do with audio so we changed to Mono, which seemed to fix an issue with audio vanishing after a while but not the unresponsiveness. There's sometimes not even anything in the logs - it just becomes unresponsive and that's that. Most servers are 32 thread and 128-192gb of ram. We aren't hitting any resource limits. The amounts of viewers might be 50-100 on a server during peak. 

[Selim Emre]

Hi Konrad,

Thanks for all details. I will discuss this issue with our tech team.

Best Regards,

Selim

[Konrad Piwowar]

Has any progress on this been made?

Every time we crash we get this in the logs. Our error logs are several gigabytes in size...

WARNING: An illegal reflective access operation has occurred

WARNING: Illegal reflective access by org.springframework.cglib.core.ReflectUtils$1 (file:/usr/local/antmedia/lib/spring-core-4.3.28.RELEASE.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)

WARNING: Please consider reporting this to the maintainers of org.springframework.cglib.core.ReflectUtils$1

WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations

WARNING: All illegal access operations will be denied in a future release

I found this, maybe will help.

https://github.com/spring-projects/spring-framework/issues/22674

[Selim Emre]

Hi Konrad,

Have a good day.

Every time we crash we get this in the logs. Our error logs are several gigabytes in size...

I'm sorry for the late response. Actually, I discussed this issue with my colleagues, but we couldn't reproduce this issue. Is there any specific reproduce scenario for this issue? 

Thank you for reporting this warning log issue. I added an issue for you -> https://github.com/ant-media/Ant-Media-Server/issues/3184

Best Regards,

Selim

[Konrad Piwowar]

We are still experimenting with this. Thus far, we have seen one person be fairly consistently able to crash teh daemon by broadcasting from a very old macbook with Chrome 67. Not every time, but often. We have also seen this happen from some phones but also not always. Seems to be related to older apple devices we think. Right now we are playing wtih resolutions to see if this is maybe causing issue - basically going off 'what are we doing differently then everyone else'

[Selim Emre]

Hi Konrad,

I also reported this issue in our issue tracking system -> https://github.com/ant-media/Ant-Media-Server/issues/3186

We will schedule to fix this issue. 

Best Regards,

Selim