Is there any way to debug RTMPS

[solomax]

Hello All,

I'm currently facing weird RTMPS behavior (everything works as expected using RTMP and RTMPT)

Remote calls works as expected

Video not working (also described here http://markmail.org/thread/35cbwifpeme2loee and here: http://markmail.org/message/cjlf2nwwpiwawg2o)

Can anyone please suggest what can be done? (code modification, additional debug etc.)

Any help is really appreciated!

red5 version is 4727

[solomax]

just have tried to downgrade to 4463 (which should be 1.0.0)

RTMPS+SSL works somehow (lots of random disconnections) RTMP seems to work

Could you please suggest any stable version? (for RTMP/RTMPT/RTMPTS+SSL)

Thanks in advance

[Dominick Accattato]

Hello Maxim,

The latest version up on Red5.org is the most stable. RTMPS should work fine. What steps are you taking in order to implement it? Did you follow a tutorial?

--
 
---
You received this message because you are subscribed to the Google Groups "red5" group.
To unsubscribe from this group and stop receiving emails from it, send an email to red5interest...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Dominick Accattato

Red5 Co-Project Manager & Engineer

---

http://www.infrared5.com/
http://code.google.com/p/red5/ 
http://code.google.com/p/jedai/ 

[solomax]

Hello Dominick,

We are using Red5 as main component of openmeetings.apache.org.

I have configured RTMPS+SSL based on our instruction which was not changed for the long time and is available here: http://openmeetings.apache.org/RTMPSAndHTTPS.html

Latest version somehow doesn't work for us RTMP, RTMPT and RTMPS.

In our system each user has 2 connections (one RPC connection another Audio/Video+RPC connection), unfortunately Audio/Video doesn't work for more than 1 person in the room. Not sure how this can be debugged :(

Additionally latest client seems to be not working (using client 4393 with latest server)

Currently I'm trying to downgrade red5 (currently 4393 is being tested and seems to not stable)

Any hints/ideas are highly appreciated

[solomax]

Here is the stacktrace (r4731)

[INFO] [RTMPExecutor#FPOZQLR9VEZWO-1] org.red5.server.api.Red5 - Set connection: null with thread: RTMPExecutor#FPOZQLR9VEZWO-1

[WARN] [NioProcessor-7] org.red5.server.net.rtmp.RTMPMinaConnection - Incoming message handling failed on FPOZQLR9VEZWO

org.springframework.core.task.TaskRejectedException: Executor [java.util.concurrent.ThreadPoolExecutor@97a8f00[Running, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 582]] did not accept task: org.red5.server.net.rtmp.ReceivedMessageTask@4d5cd45e

at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:244) ~[spring-context-3.1.4.RELEASE.jar:3.1.4.RELEASE]

at org.red5.server.net.rtmp.RTMPMinaConnection.handleMessageReceived(RTMPMinaConnection.java:151) ~[red5-server.jar:na]

at org.red5.server.net.rtmp.RTMPMinaIoHandler.messageReceived(RTMPMinaIoHandler.java:162) [red5-server.jar:na]

at org.apache.mina.core.filterchain.DefaultIoFilterChain$TailFilter.messageReceived(DefaultIoFilterChain.java:690) [mina-core-2.0.7.jar:na]

at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417) [mina-core-2.0.7.jar:na]

at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47) [mina-core-2.0.7.jar:na]

at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765) [mina-core-2.0.7.jar:na]

at org.apache.mina.filter.codec.ProtocolCodecFilter$ProtocolDecoderOutputImpl.flush(ProtocolCodecFilter.java:407) [mina-core-2.0.7.jar:na]

at org.apache.mina.filter.codec.ProtocolCodecFilter.messageReceived(ProtocolCodecFilter.java:236) [mina-core-2.0.7.jar:na]

at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417) [mina-core-2.0.7.jar:na]

at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47) [mina-core-2.0.7.jar:na]

at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765) [mina-core-2.0.7.jar:na]

at org.red5.server.net.rtmpe.RTMPEIoFilter.messageReceived(RTMPEIoFilter.java:132) [red5-server.jar:na]

at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417) [mina-core-2.0.7.jar:na]

at org.apache.mina.core.filterchain.DefaultIoFilterChain.access$1200(DefaultIoFilterChain.java:47) [mina-core-2.0.7.jar:na]

at org.apache.mina.core.filterchain.DefaultIoFilterChain$EntryImpl$1.messageReceived(DefaultIoFilterChain.java:765) [mina-core-2.0.7.jar:na]

at org.apache.mina.core.filterchain.IoFilterAdapter.messageReceived(IoFilterAdapter.java:109) [mina-core-2.0.7.jar:na]

at org.apache.mina.core.filterchain.DefaultIoFilterChain.callNextMessageReceived(DefaultIoFilterChain.java:417) [mina-core-2.0.7.jar:na]

at org.apache.mina.core.filterchain.DefaultIoFilterChain.fireMessageReceived(DefaultIoFilterChain.java:410) [mina-core-2.0.7.jar:na]

at org.apache.mina.core.polling.AbstractPollingIoProcessor.read(AbstractPollingIoProcessor.java:710) [mina-core-2.0.7.jar:na]

at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:664) [mina-core-2.0.7.jar:na]

at org.apache.mina.core.polling.AbstractPollingIoProcessor.process(AbstractPollingIoProcessor.java:653) [mina-core-2.0.7.jar:na]

at org.apache.mina.core.polling.AbstractPollingIoProcessor.access$600(AbstractPollingIoProcessor.java:67) [mina-core-2.0.7.jar:na]

at org.apache.mina.core.polling.AbstractPollingIoProcessor$Processor.run(AbstractPollingIoProcessor.java:1124) [mina-core-2.0.7.jar:na]

at org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) [mina-core-2.0.7.jar:na]

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]

at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]

Caused by: java.util.concurrent.RejectedExecutionException: Task org.red5.server.net.rtmp.ReceivedMessageTask@4d5cd45e rejected from java.util.concurrent.ThreadPoolExecutor@97a8f00[Running, pool size = 1, active threads = 1, queued tasks = 7, completed tasks = 575]

at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2048) ~[na:1.7.0_45]

at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821) [na:1.7.0_45]

at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372) [na:1.7.0_45]

at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:241) ~[spring-context-3.1.4.RELEASE.jar:3.1.4.RELEASE]

... 27 common frames omitted

after ~30 seconds of video (RTMP)

[Dominick Accattato]

Interesting Maxim,

Have you tried with a more simplified vanilla application? Since I don't see any application specific code in the stack, it would lead me to believe that this is erroring before it gets to the application layer. A simple application should error the same way in this case and would more useful in debugging the issue.

[solomax]

Hello Dominick,

Here are my simplified steps:

1) create "dummy" red5 application:

     The only Application.java class extending ApplicationAdapter paerform no actions except for calling super.{appConnect,roomConnect,roomDisconnect}

2) created "dummy" AS3 flash application

     It connects to the server rtmp://localhost:1935/dummy/666 (provides the functionality of choosing camera and mic first), on "NetStream.Publish.Start" event user start seeing his/her own camera.

First connected client can see own video (connection successful)

Second receives " NetStream.Publish.BadName | 666"

[INFO] [NioProcessor-3] org.red5.server.net.rtmp.RTMPMinaIoHandler - Session opened: 2 id: APCLKZNHIAXX6

[INFO] [RTMPExecutor#APCLKZNHIAXX6-1] org.recorder.Application - W3C x-category:session x-event:connect c-ip:192.168.15.174 c-client-id:1

2013-11-11 11:50:07,805 [RTMPExecutor#APCLKZNHIAXX6-1] DEBUG org.recorder.Application - New appConnect attempt from 192.168.15.174, params: [clientcode, 1234567, hash]...

[WARN] [NioProcessor-2] org.red5.server.net.rtmp.RTMPMinaConnection - Incoming message handling failed on 3GYD8LEJNXVYD

org.springframework.core.task.TaskRejectedException: Executor [java.util.concurrent.ThreadPoolExecutor@16e1dd8] did not accept task: org.red5.server.net.rtmp.ReceivedMessageTask@696d5c

at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_35]

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_35]

at java.lang.Thread.run(Thread.java:662) [na:1.6.0_35]

Caused by: java.util.concurrent.RejectedExecutionException: null

at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:1768) ~[na:1.6.0_35]

at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767) ~[na:1.6.0_35]

at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658) ~[na:1.6.0_35]

at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:241) ~[spring-context-3.1.4.RELEASE.jar:3.1.4.RELEASE]

... 27 common frames omitted

[INFO] [NioProcessor-2] org.recorder.Application - W3C x-category:stream x-event:unpublish c-ip:192.168.15.229 cs-bytes:1100179 sc-bytes:3485 x-sname:3ae413bd-041c-4d2f-beae-c69e69e0be3e x-file-length:108 x-name:666

[INFO] [NioProcessor-2] org.recorder.Application - W3C x-category:session x-event:disconnect c-ip:192.168.15.229 c-client-id:0

[WARN] [NioProcessor-2] org.red5.server.net.rtmp.RTMPConnManager - Connection not found for 3GYD8LEJNXVYD

I can provide source code and log with any LOG level :)

Will really appreciate your help :)

We are going to release Openmeetings and seems to blocked by this :(

[solomax]

:( sorry for double posting

test was performed using 3 physical machines in one room:

1) Ubuntu 13.04 server: hosting red5 (r4731) and client code under Apache

2) iMac Safari: 1st client

3) Windows 7 Ent: 2nd client

The above behavior is reproducible no matter which client connects first (first connected is always successful)

Thanks in advance

[solomax]

Maybe I'm using red5/AS3 improperly?

In my client code:

nc.connect("rtmp://localhost:1935/dummy");

then, after connect was successful 

ns.attachCamera(cam);

ns.attachAudio(mic);

ns.publish("666", "live");

Maybe client code need to be modified somehow?

[Dominick Accattato]

Based on the event being returned:

" NetStream.Publish.BadName | 666"

it appears that you are trying to publish two NetStreams with the same name which isn't correct. 

The first client would use the following:

ns.attachCamera(cam);

ns.attachAudio(mic);

ns.publish("666", "live");

and the second would need a unique stream name like 777 or something else:

ns.attachCamera(cam);

ns.attachAudio(mic);

ns.publish("777", "live");

If you have the source code, you can send it to me directly.

[solomax]

Thanks Dominick,

I have updated my code

nc.connect("rtmp://localhost:1935/dummy/666");

ns.publish("" + new Date().time, "live");

So publish is started with different ID

second client get "NetConnection.Connect.Closed | undefined"

here is the part of server logs

[INFO] [RTMPExecutor#LGAFWKLWWSJAD-1] org.red5.server.stream.ClientBroadcastStream - Stream start: 1384318821510

[INFO] [RTMPExecutor#LGAFWKLWWSJAD-1] org.recorder.Application - W3C x-category:stream x-event:publish c-ip:192.168.15.174 x-sname:35c9e4dc-aa10-43e7-b4b0-d5e23726f291 x-name:1384318821510

[WARN] [NioProcessor-6] org.red5.server.net.rtmp.RTMPMinaConnection - Incoming message handling failed on LGAFWKLWWSJAD

org.springframework.core.task.TaskRejectedException: Executor [java.util.concurrent.ThreadPoolExecutor@5d75eb] did not accept task: org.red5.server.net.rtmp.ReceivedMessageTask@1e8b957

[INFO] [NioProcessor-6] org.recorder.Application - W3C x-category:stream x-event:unpublish c-ip:192.168.15.174 cs-bytes:151897 sc-bytes:3460 x-sname:35c9e4dc-aa10-43e7-b4b0-d5e23726f291 x-file-length:8 x-name:1384318821510

source code of client application is available here: https://github.com/solomax/recorder-flash/blob/master/src/Recorder.mxml

[Dominick Accattato]

Alright, 

we're getting a new error now. Do you want to send me the serverside and clientside apps directly to dominick at red5 dot org? 

[solomax]

Will write down my repo steps here (to have reference and maybe hint how to resolve this)

Set up:

Server: Ubuntu 13.04 red5: 1.0.2 RC3 (r4727)

Client 1: Safari+iMac

Client 2: Chrome+Ubuntu 13.10

3 different machines in one LAN (100Mbps)

scenario 1:

Client 1: perform connection to rtmp://192.168.15.178:135/recorder/room2 (at 12:20:15)

Client 1: publish("1", "live")

Client 2: perform connection to rtmp://192.168.15.178:135/recorder/room1 (at 12:20:01)

Client 2: publish("1", "live")

Result: 

Client 2 gets RejectedExecutionException and connection closed at 12:23:51

Client 1 gets RejectedExecutionException and connection closed at 12:27:59

scenario 2:

Client 1: perform connection to rtmp://192.168.15.178:135/recorder/666 (at 12:30:31)

Client 1: publish("2", "live")

Client 2: perform connection to rtmp://192.168.15.178:135/recorder/666 (at 12:29:50)

Client 2: publish("1", "live")

Result: 

Client 2 gets RejectedExecutionException and connection closed at 12:32:06

Client 1 gets RejectedExecutionException and connection closed at 12:32:06

please let me know what else can I do to help debug/fix this

Thanks in advance!

[solomax]

I have performed additional tests.

Server was updated to r4737 (reports 4730 after run)

both scenarios were reproducible with same result.

I have added 

scenario 3:

Client 2: perform connection to rtmp://192.168.15.178:135/recorder/room1 (at 14:16:01,518)

Client 2: publish("1", "live")

Client 1: perform connection to rtmp://192.168.15.178:135/recorder/room2 (at 14:16:11,966)

Client 1: publish("14", "live")

Result: 

Client 1 gets RejectedExecutionException and connection closed at 14:16:56,086

Client 2 Works as expected at least 10 minutes

scenario 4:

Client 2: perform connection to rtmp://192.168.15.178:135/recorder (at 14:27:56,388)

Client 2: publish("stream1", "live")

Client 1: perform connection to rtmp://192.168.15.178:135/recorder (at 14:28:25,077)

Client 1: publish("stream2", "live")

Result: 

Client 1 gets RejectedExecutionException and connection closed at 14:36:36,022

Client 2 Works as expected at least 10 minutes

So it seems like

1) all client IDs should be unique across the red5 instance.

2) There might be only one publishing(recording?) client in all rooms

Is it expected behavior?

Is there any workaround?

Additional question: 

Currently log is full of messages like

[INFO] [NioProcessor-3] org.red5.server.api.Red5 - Get connection on thread: NioProcessor-3

[INFO] [NioProcessor-3] org.red5.server.api.Red5 - Set connection: PERZAXDLFYHXB with thread: NioProcessor-3

[INFO] [NioProcessor-3] org.red5.server.api.Red5 - Set connection: null with thread: NioProcessor-3

[INFO] [RTMPExecutor#PERZAXDLFYHXB-1] org.red5.server.api.Red5 - Set connection: PERZAXDLFYHXB with thread: RTMPExecutor#PERZAXDLFYHXB-1

[INFO] [RTMPExecutor#PERZAXDLFYHXB-1] org.red5.server.api.Red5 - Set connection: null with thread: RTMPExecutor#PERZAXDLFYHXB-1

Maybe it is better to make this messages TRACE instead of INFO?

Thanks in advance for any help

[solomax]

I would like to bump the topic :)

Was anybody able to reproduce this error?

According to my tests only one person can publish video to red5 server, is it expected behavior?

[solomax]

Just tested r4738.

publishsing connections lives longer (all scenarious)

Unfortunately TaskRejected still happens :(

[Mondain]

I'm still not sure why this happens, it appears that on some connections the handling of incoming data is not completed fast enough.

http://gregoire.org/
http://code.google.com/p/red5/

[solomax]

According to my observations this exception happens more often if connections are made to the same room.

2 simultaneous connections to the top level rtmp://localhost:1935/myapp with different stream id were working 20+ minutes

2 simultaneous connections to the room rtmp://localhost:1935/myapp/room with different stream id were working ~5 minutes

Please let me know if there is anything I can do to help with this issue

[Mondain]

Thats a good clue to figure out the issue; I do know of an individual that has large numbers of people connecting to the same scope/room concurrently and he initially had a lot of issues with this. He has his task limits set to 48 I believe; the default now is 16.

[solomax]

I'm seen this issue with 2 connections only :(

Should I enlarge the task limit and recheck?

[Prabath Perera]

is this fixed?

[solomax]

up :)

please don't forget about this problem :)

[solomax]

Any progress on this?

Is this issue is reproducible to anyone else?

[solomax]

RC4 was tagged today, but no changes were made regarding this issue :(

Is there any chances this will be fixed?

[Mondain]

RTMPS had some modifications, but they were related to the TomcatLoader. I'll take a look at it before we get to 1.0.3.

--

 
---
You received this message because you are subscribed to the Google Groups "red5" group.
To unsubscribe from this group and stop receiving emails from it, send an email to red5interest...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[solomax]

Thanks a lot Paul!

The issue I have reported was reproduced using RTMP, so I'm afraid all protocols might be affectred

[solomax]

Just have tested r4756

2 clients were able to publish more than 1 hour

Will try to add red5 r4756 to Openmeetings and check

[Dominick Accattato]

Is this issue still open or does r4756 work as expected?

[solomax]

Hello Dominick,

just have tested using Openmeetings with 2 users in the room.

Still got TaskRejectedException

Clients were on the same machine (different browsers, different cameras)

Server was on separate machine

The issue seems to be still reproducible

[Dominick Accattato]

Alright, thanks.. going to look into this

[solomax]

Thanks a lot !

Please let me know if there is anything I can do