[Urgent] Streaming ending abruptly

Chai

unread,

May 12, 2021, 6:13:10 AM5/12/21

to Ant Media Server

Hi,

Our servers are consistently running into this issue which is causing major bottlenecks in our workflows.

Any ideas how we can resolve this issue?

May 11, 2021 8:33:26 PM io.vertx.core.impl.BlockedThreadChecker

WARNING: Thread Thread[vert.x-eventloop-thread-31,5,main]=Thread[vert.x-eventloop-thread-31,5,main] has been blocked for 6498801 ms, time limit is 2000 ms

io.vertx.core.VertxException: Thread blocked

at io.antmedia.enterprise.webrtc.WebRTCMuxer.unRegisterWebRTCClient(WebRTCMuxer.java:246)

at io.antmedia.enterprise.webrtc.WebRTCClient.stopPartial(WebRTCClient.java:1094)

at io.antmedia.enterprise.webrtc.WebRTCClient.lambda$stop$17(WebRTCClient.java:1137)

at io.antmedia.enterprise.webrtc.WebRTCClient$$Lambda$545/0x00000008407b3840.run(Unknown Source)

at io.antmedia.enterprise.webrtc.WebRTCClient.lambda$executeOnSignallingThread$12(WebRTCClient.java:965)

at io.antmedia.enterprise.webrtc.WebRTCClient$$Lambda$510/0x0000000840796c40.handle(Unknown Source)

at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366)

at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38)

at io.vertx.core.impl.EventLoopContext$$Lambda$485/0x000000084072f040.run(Unknown Source)

at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)

at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472)

at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)

at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)

at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)

at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)

at java.base@ 11.0.10/java.lang.Thread.run(Thread.java:834)

Tahir Gölge

unread,

May 12, 2021, 7:12:31 AM5/12/21

to Ant Media Server

Hi,

Thread deadlock problem is resolved now, it will be deployed in the next coming version. Thank you for your patience.

Cheers

Chai

unread,

May 12, 2021, 3:37:53 PM5/12/21

to Ant Media Server

Thanks, Tahir for the prompt response.

Is there something we can fix on our AWS instances interim while we wait for the next coming version?

Also - any insights on the root cause analysis of how this issue got introduced?

Tahir Gölge

unread,

May 13, 2021, 5:42:46 AM5/13/21

to Ant Media Server

Hi,

I fear it is not possible since it is related with the source code of the server. If you are using the license from Ant Media on AWS I can send you a SNAPSHOT maybe?

The issue was, in summary a synchronization problem: https://docs.oracle.com/javase/tutorial/essential/concurrency/syncmeth.html

Cheers

Chai

unread,

May 15, 2021, 5:36:09 PM5/15/21

to Ant Media Server

Yes,

Can you send us the snapshot?

Also, we are getting publish timeout errors. How can we solve this?

TCEncoderAdaptor - Encode video runner is returning because encoder adaptor is stopped for explore-the-australian-wilderness-1

2021-05-15 21:29:26,659 [vert.x-eventloop-thread-16] INFO i.a.e.adaptive.WebRTCEncoderAdaptor - leaving stop() for explore-the-australian-wilderness-1

2021-05-15 21:29:26,659 [vert.x-eventloop-thread-16] INFO i.a.AntMediaApplicationAdapter - Closing broadcast stream id: explore-the-australian-wilderness-1

2021-05-15 21:29:26,668 [vert.x-eventloop-thread-16] INFO i.a.AntMediaApplicationAdapter - Setting timer to call live stream ended hook for stream:explore-the-australian-wilderness-1

2021-05-15 21:29:26,708 [vert.x-eventloop-thread-27] INFO i.a.AntMediaApplicationAdapter - POST Response Status:: 301

2021-05-15 21:29:30,810 [https-openssl-nio2-0.0.0.0-5443-exec-7] INFO i.a.e.s.WebSocketLocalHandler - Web Socket opened

2021-05-15 21:29:32,545 [https-openssl-nio2-0.0.0.0-5443-exec-3] ERROR i.a.e.w.WebSocketEnterpriseHandler - Error: EOFException:

2021-05-15 21:29:36,572 [https-openssl-nio2-0.0.0.0-5443-exec-2] ERROR i.a.e.w.WebSocketEnterpriseHandler - Error: EOFException:

2021-05-15 21:29:36,781 [https-openssl-nio2-0.0.0.0-5443-exec-9] INFO i.a.e.s.WebSocketLocalHandler - Web Socket opened

2021-05-15 21:29:39,663 [https-openssl-nio2-0.0.0.0-5443-exec-4] ERROR i.a.e.w.WebSocketEnterpriseHandler - Error: EOFException:

2021-05-15 21:29:39,746 [https-openssl-nio2-0.0.0.0-5443-exec-7] INFO i.a.e.s.WebSocketLocalHandler - Web Socket opened

2021-05-15 21:29:44,039 [https-openssl-nio2-0.0.0.0-5443-exec-2] INFO i.a.e.s.WebSocketLocalHandler - Web Socket opened

2021-05-15 21:29:44,679 [https-openssl-nio2-0.0.0.0-5443-exec-6] ERROR i.a.e.w.WebSocketEnterpriseHandler - Error: EOFException:

2021-05-15 21:29:54,355 [vert.x-eventloop-thread-0] INFO i.antmedia.statistic.StatsCollector - System cpu load:0 process cpu load:0 available memory: 63138384 KB used memory(RSS): 984432 KB

2021-05-15 21:29:54,356 [vert.x-eventloop-thread-0] INFO i.antmedia.statistic.StatsCollector - Vertx worker queue size:0 WebRTCVertx worker queue size:0

2021-05-15 21:29:55,474 [https-openssl-nio2-0.0.0.0-5443-exec-10] INFO i.a.e.s.WebSocketLocalHandler - Web Socket opened

2021-05-15 21:34:50,938 [https-openssl-nio2-0.0.0.0-5443-exec-1] ERROR i.a.e.w.WebSocketEnterpriseHandler - Error: EOFException:

2021-05-15 21:34:51,704 [https-openssl-nio2-0.0.0.0-5443-exec-1] INFO i.a.e.s.WebSocketLocalHandler - Web Socket opened

2021-05-15 21:34:54,355 [vert.x-eventloop-thread-0] INFO i.antmedia.statistic.StatsCollector - System cpu load:0 process cpu load:0 available memory: 63132260 KB used memory(RSS): 989796 KB

2021-05-15 21:34:54,355 [vert.x-eventloop-thread-0] INFO i.antmedia.statistic.StatsCollector - Vertx worker queue size:0 WebRTCVertx worker queue size:0

2021-05-15 21:34:55,068 [https-openssl-nio2-0.0.0.0-5443-exec-4] ERROR i.a.e.w.WebSocketEnterpriseHandler - Error: EOFException:

Tahir Gölge

unread,

May 16, 2021, 5:18:10 AM5/16/21

to Ant Media Server

Please send an e-mail to sup...@antmedia.io for the snapshot.

We need to check your ICE connection process, can you elaborate your way of sending the broadcast? Also, you should check your firewall, for most of the time UDP ports are getting blocked for webRTC and people get publish time out errors. So please make sure that UDP ports between 5000-65000 are open for server and there are no blockage at the broadcaster.

Chaitanya Malla

unread,

May 16, 2021, 11:19:56 AM5/16/21

to Tahir Gölge, Ant Media Server

Tahir,

Thanks for the reply. We will check our UDP ports. We don’t have a firewall, but our web application is on Heroku and Ant Instance is on AWS.

Our process outline:

Our web application is hosted on Heroku. Our Ant Media servers are on AWS. A webpage with integrated WebRTC code will publish the stream which publishes media files to Ant Instance on AWS.

The viewer pages(on web application) hosted on Heroku will “fetch” for the streams and play them.

- Tahir

--
You received this message because you are subscribed to a topic in the Google Groups "Ant Media Server" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/ant-media-server/qb7T253AD54/unsubscribe.
To unsubscribe from this group and all its topics, send an email to ant-media-serv...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ant-media-server/5c84eef4-89e2-48bc-af35-06bc8fcd8f6bn%40googlegroups.com.