RTP Forwarding to FFmpeg - Missing Video

[m...@riff.tv]

Hi all,

Using the rtp_forward feature of the videoroom plugin, I'm trying to forward streams to FFmpeg.

Here's the Dockerfile I used to set up Janus / FFmpeg on a new Google Cloud Instance. For the record, I also tried to do this outside of a Docker container and ran into the same problem. https://gist.github.com/maxstoller/7058ef78e66fc046d1be6c57d1b02c12

The only change I made to Janus' configuration is I added a STUN server in janus.cfg.

I'm able to establish a connection and stream to Janus from my locally served videroomtest.html. Here's the Janus logs once I start streaming: https://gist.github.com/maxstoller/801efafc9ac8587f0454f8acb917bbce

I've modified videoroomtest.js to call Janus from the client and tell it to forward the stream in the success callback of publishOwnFeed: https://gist.github.com/maxstoller/af523ddc477ab9a405da4e29ad11f2a7

From the server console, I then try to receive the RTP stream with FFmpeg using the following command: 

./bin/ffmpeg -protocol_whitelist file,rtp,udp -i test.sdp -c:v libx264 test.mp4

Here's my test.sdp: https://gist.github.com/maxstoller/67814c77208ada321957affb147e4640

The issue I'm having is the .mp4 that FFmpeg is missing the video (but has the audio). Here's the output from FFmpeg: https://gist.github.com/maxstoller/40181ed866cfdab16b9727b5426701d9

Notice this message:

[sdp @ 0x2d22460] Could not find codec parameters for stream 1 (Video: vp8, yuv420p): unspecified size

Consider increasing the value for the 'analyzeduration' and 'probesize' options

I've tried setting analyzeduration and probesize to high values, but it doesn't help.

Anyone have any ideas? Thanks so much! 

[Lorenzo Miniero]

It's better to RTP forward in the webrtcup event: before that, Janus isn't getting any media to forward yet, and maybe the issue is that you're missing the keyframe (not sure, just speculating here). To check if it's an issue with the media forwarding or the FFmpeg script that receives it, you can try forwarding to a streaming mountpoint and check if you can see the video from the streaming demo page: if you can, then it's the FFmpeg part you need to fix somehow.

L.

[Lorenzo Miniero]

PS: make also sure the payload type you send is the same FFmpeg expects. There's are audio_pt and video_pt properties you can set when forwarding that force the plugin to overwrite it no matter what the browser is sending.

L.

[m...@riff.tv]

Thanks for the reply, Lorenzo! I'll give the webrtcup advice a shot first.

My plan is to run a Node app side-by-side with Janus that connects to it via WS and listens for webrtcup / hangup events. The Node app will create / delete SDP files and manage used ports based on these events.

I have my Node app running and connected to Janus via WS, but it's not receiving any events when someone connects to a videoroom. Do I have to subscribe to each session in order to receive webrtcup events, in which case my Node server will have to know all of the session ids?

Thanks so much.

[m...@riff.tv]

PS: I understand I can listen for webrtcup from the client, but hoping to do it from the server. I'd rather the server control what videos are rtp_forwarded. It'll have to manage ports / create SDP files anyway.

[Lorenzo Miniero]

If you want to be notified about join/leave events, you'll have to implement a complete client (minus the media part, of course). So creating a session, attaching a handle, joining as a publisher, receive events, etc. Once you do that, though, it might be easier to just use the nodejs application to handle all the users, instead of having them talk to Janus directly. If your nodejs application wraps the Janus API, then you'd have complete control of the signalling, and would be notified about everything. There are a couple of stacks in the Resources docs pages that might help there.

As an alternative, you might want to have a look at the Event Handler stuff: it's supposed to be mainly used for monitoring and troubleshooting, but if you configure it to only notify part of the events (e.g., session creation, handle creation, WebRTC setup state, VideoRoom related events to know who's who) then you can leverage that too. There's some stuff in the resources to help with that too (besides a couple of blog posts I wrote on the subject).

Both would require some work of course.

L.

[m...@riff.tv]

Hi Lorenzo,

Thank you again for the detailed reply. I opted for the 2nd method.

I've enabled sampleevh to POST all webrtc events to the Node app. 

e.g. Here's a webrtcup message: https://gist.github.com/maxstoller/3cc4ec894bc1aab2a379b5728e6ddce9

When the Node app receives a webrtcup event, it POSTs back to Janus to initiate the rtp_forward using the session_id and handle_id supplied in the webrtcup msg: https://gist.github.com/maxstoller/e64dc67ec9d85d6855276fb21987cf9f

You'll notice I'm getting a "No such feed (4976081272368347)" error. Any idea what that might mean? Here's the corresponding Janus logs: https://gist.github.com/maxstoller/badfd8d83c09b757b850ab8f5b562f3c

Thanks again!

[m...@riff.tv]

Sorry, I figured it out. Looks like I need to hit the listparticipants endpoint to get the participant-specific ID. I then POST to janus/sessionId/handleId with the participant id as the publisher_id to start forwarding. 

[m...@riff.tv]

For those that might find this thread later ---

I fixed it by forcing the video_pt to 100 in the rtp_forward API request (per Lorenzo's post above). AFAIK, 100 corresponds to VP8. Not sure what the consequences of this are, particularly if the streamer is not broadcasting VP8...but it's progress.

[Lorenzo Miniero]

You're confusing some concepts, here. 4976081272368347 in this context is the Janus core handle identifier. VideoRoom IDs are a different thing, and plugin specific. No need to notify about those, anyway, as they're something the web page knows about already (e.g., myid in the VideoRoom demo).

L.

[István Ferenc Tóth]

Hello

I get the same result - working audio, missing video (with videopt: 100).

I set the ffmpeg log to debug and find out:

Could not find codec parameters for stream 1 (Video: vp8, 1 reference frame, yuv420p): unspecified size

Consider increasing the value for the 'analyzeduration' and 'probesize' options

some details after ffmpeg finish:

Input file #0 (test.sdp):

  Input stream #0:0 (audio): 1596 packets read (128887 bytes); 1596 frames decoded (1532160 samples);

  Input stream #0:1 (video): 0 packets read (0 bytes);

  Total: 1596 packets (128887 bytes) demuxed

Output file #0 (rtmp://[RTMP RECEIVER]):

  Output stream #0:0 (audio): 499 frames encoded (510720 samples); 500 packets muxed (223897 bytes);

  Total: 500 packets (223897 bytes) muxed

1596 frames successfully decoded, 0 decoding errors

[Aleksey Ivanov]

Hello, I have faced with the similar problem.

Did you find any solution for this?

[Aleksey Ivanov]

Hello Lorenzo,

> you can try forwarding to a streaming mountpoint

Could you please explain how can I do that? What is the mountpoint in this case?

Currently I use videoroom plugin.

After joining a room I send rtp_forward request via browser console like:

var message = { "request": "rtp_forward", publisher_id: publisherId, room: 1234, "audio_port": 10033, "audiopt": 111, "video_port": 10038, "videopt": 100, "host": "127.0.0.1", "secret": "adminpwd" };
sfutest.send({"message": message});

Unfortunately, I have the same issue: only audio stream.

Am I doing anything wrong?

Thank you.

[Lorenzo Miniero]

There's nothing I can add to what I already said in the previous replies. You have to check if Janus is not sending those packets, or if your receiver is discarding them. This is something you have to investigate yourself. Wireshark/tcpdump might help; forwarding to a Streaming mountpoint can also help.

L.

[Aleksey Ivanov]

Thank you Lorenzo,

It turns out I have wrong SDP. It describes video as RTP payload type 100, though actually it is 96. 

[Xiaowu]

Hi all, met the same problem again...

Besides, the docs about  rtp_forward is wrong:

  the doc describe para 

"video_port" : <port to forward the video RTP packets to>,"

  but in the code, it use "video_pt",  code: 

json_t *pt = json_object_get(root, "video_pt");

I directed the video_pt to 100 as above, and set it in sdp file to 100 too. 

(I'm sure because I change the code to add video_pt to response)

code:

json_object_set_new(rtp_stream, "video_pt", json_integer(video_pt[0]));

and get response:

"video_pt": 100,

But the ffmpeg still error:

[sdp @ 0x3ee04a0] Keyframe missing
[sdp @ 0x3ee04a0] Could not find codec parameters for stream 1 (Video: vp8, yuv420p): unspecified size

Consider increasing the value for the 'analyzeduration' and 'probesize' options

the sdp about video is below, it similar to others who success= =

m=video 2093 RTP/AVP 100
a=rtpmap:100 VP8/90000
a=fmtp:100 packetization-mode=1

I tried to solve it for several days but still got nothing.

And now I don't know where is still wrong...

Thank You

[Lorenzo Miniero]

video_port and video_pt are completely different things...

Lorenzo

[Xiaowu]

Sorry, it's just a copy and paste error...

I mean, in the doc:

"video_ptype" : <video payload type to use when streaming; optional>,

in the code:

json_t *pt = json_object_get(root, "video_pt");

Other same as above

[Lorenzo Miniero]

Ah you're right! Just fixed, thanks: https://janus.conf.meetecho.com/docs/videoroom

Lorenzo

[Xiaowu]

So is there any suggestion about this ffmpeg error?

It's can't be sdp payload error like others now. But I didn't find any other similar problems....

[Steve Hardt]

That's why I could not get that to work. I guess I should have looked at the source.

[Steve Hardt]

It is actually an issue with the ffmpeg version you are using. I had the same issue. Make sure you use the latest ffmpeg build.

[Xiaowu]

I'm using ffmpeg-4.0.2-64bit, the latest version in official website...

In fact, I have suspected that this version was too new....so I tried the 3.4.4 but still failed = =

If you can get that to work, can you tell me what version you are using? I will just try it.

Or I should try much older version of ffmpeg and janus? I don't know.

I also suspected the "Keyframe missing" warning, but didn't get any useful answer on Google too. QAQ

[Lorenzo Miniero]

Check the suggestion it gives:

Consider increasing the value for the 'analyzeduration' and 'probesize' options

I think somebody proposed a solution in older posts, so just do a search here.

Lorenzo

[Xiaowu]

If i do search using "Consider increasing the value for the 'analyzeduration' and 'probesize' options" and "janus" as keywords on Google, the only two post in Group I can find is this post and "https://groups.google.com/forum/#!topic/meetecho-janus/fiSBx9QyeAA"...

Obviously there is no solution except about payload type in sdp...

Even out of the group, the only useful link I can find is a github issue: https://github.com/atyenoria/janus-webrtc-gateway-docker/issues/2

Their problem is also the wrong payload type in sdp.....

I also search "Could not find codec parameters for stream 1 (Video: vp8, yuv420p): unspecified size" or "Keyframe missing" or "ffmpeg pull udp streams using sdp", etc. But got noting.

It just seems that all problems are about wrong payload type= =

But I tried to read the sdp of stream publisher in janus message to get the payload type, this is always 96 for vp8 and 100/102 for h264 (I tried both vp8 and h264 but got same error...).

I even tried every payload type occurs in the sdp of publisher, but....

Then I tried to direct it in message (Thank you about the reply to this post early) and still failed...

To avoid message error, I change the janus code to return the result of payload type after received my message, but still failed.

I don't want to be like a free rider, but If you remember that there is somebody proposed a solution in older posts, can you help me to find it ? Even a new keyword is very helpful. Thanks

[王勃]

Hi,

In my experience, setting 'analyzeduration' to 5k for ffmpeg to streaming janus
rtp data to rtmp data is ok in most cases. You can have a try.

Payload type in your 'rtp_forward' message should match the couterparts in ths
sdps. Maybe you should check it.

>--
>You received this message because you are subscribed to the Google Groups "meetecho-janus" group.
>To unsubscribe from this group and stop receiving emails from it, send an email to meetecho-janu...@googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.


--
Best wishes from 王勃 <wangbo12 AT 58ganji DOT com>
Mobile : (+86) 17610272985

[Xiaowu]

I have tried to set analyzeduration and probesize both to 999999 and the effect was waiting more time but still failed...

I remembered that if only set analyzeduration to a big number, ffmpeg will give error message about probesize. I can try it again tomorrow and copy the error message here.

Have you successful used this feature recently? If you do, can you tell me the version of janus and ffmpeg which you are using?

On Monday, September 3, 2018 at 9:21:09 PM UTC+8, 王勃 wrote:

Hi,

In my experience, setting 'analyzeduration' to 5k for ffmpeg to streaming janus
rtp data to rtmp data is ok in most cases.  You can have a try.

r

[izemize]

Hi all! Same problem here. I just use avconv instead of ffmepg. Analyzeduration increasing not solved the problem, and avconv send only audio to rtmp server.
avproble reports i have opus audio and vp8 video in rtp stream. When i force mapping -map 0:0, -map 0:1 i get "Too many packets buffered for output stream 0:0"

Input #0, sdp, from 'test.sdp':

  Metadata:

    title           : RTP Video

  Duration: N/A, start: 0.000000, bitrate: N/A

    Stream #0:0: Audio: opus

      48000 Hz, stereo, fltp

    Stream #0:1: Video: vp8

      yuv420p

      90k tbn

# avprobe output

I try change "videocodec" to h264 in publish request, and set VP8 to H264 in my sdp file. Can't get video stream. Something wrong.

Anyone solve these?

[izemize]

Input #0, sdp, from 'test.sdp':

  Metadata:

    title           : RTP Video

  Duration: N/A, bitrate: N/A

    Stream #0:0: Audio: opus

      48000 Hz, stereo, fltp

    Stream #0:1: Video: h264

      none

      90k tbn

# avprobe output

[izemize]

rtp_forward "request" only works, when forward ports is listening right?

so first need start ffmpeg/avconv to "listen" and then start sending rtp_forward "request" to work?