RecorderEndPoint debugging...

[Andres Tello]

• Operating system (Ubuntu 16.04, 18.04, etc.) --> Centos 7 
  
• Installation method (apt-get, Docker, AWS, build from sources) --> docker
• Kurento version --> nigthly 

For the docker startup I use the option --network host 

I know I can reach the http endpoing to storage the data, but KMS seems not to be able to contact it..  any ideas of what to debug?

I suppose is a network/routing issue.

I have tested that the Recording Enpoint is reacheable from within the docker host.. 

 

It migth be the issue with the --network host options maybe? 

0:04:46.320345629     1 0x7faf44042000 ERROR           curlbasesink gstcurlbasesink.c:400:gst_curl_base_sink_render:<curlhttpsink0> poll timed out after 0:00:30.000000000
0:04:46.320472236     1 0x7faf44042000 WARN            curlbasesink gstcurlbasesink.c:401:gst_curl_base_sink_render:<curlhttpsink0> error: poll timed out after 0:00:30.000000000
0:04:46.320652328     1 0x7faf44042000 WARN               structure gststructure.c:1935:priv_gst_structure_append_to_gstring: No value transform to serialize field 'gerror' of type 'GError'
0:04:46.320639140     1 0x7faf44042000 ERROR       recorderendpoint kmsrecorderendpoint.c:1972:bus_sync_signal_handler:<kmsrecorderendpoint0> Message error message: 0x7fae70001a00, time 99:99:99.999999999, seq-num 16208, e
lement 'curlhttpsink0', GstMessageError, gerror=(GError)NULL, debug=(string)"gstcurlbasesink.c\(401\):\ gst_curl_base_sink_render\ \(\):\ /GstPipeline:pipeline1/GstCurlHttpSink:curlhttpsink0";
0:04:46.320732487     1 0x7faf44042000 WARN               structure gststructure.c:1935:priv_gst_structure_append_to_gstring: No value transform to serialize field 'gerror' of type 'GError'
0:04:46.320726360     1 0x7faf44042000 ERROR       recorderendpoint kmsrecorderendpoint.c:1976:bus_sync_signal_handler:<kmsrecorderendpoint0> Error: error message: 0x7fae70001a00, time 99:99:99.999999999, seq-num 16208, el
ement 'curlhttpsink0', GstMessageError, gerror=(GError)NULL, debug=(string)"gstcurlbasesink.c\(401\):\ gst_curl_base_sink_render\ \(\):\ /GstPipeline:pipeline1/GstCurlHttpSink:curlhttpsink0";
0:04:46.321131299     1 0x7faf44042000 WARN                 basesrc gstbasesrc.c:2948:gst_base_src_loop:<videoSrc> error: Internal data flow error.
0:04:46.321194986     1 0x7faf44042000 WARN                 basesrc gstbasesrc.c:2948:gst_base_src_loop:<videoSrc> error: streaming task paused, reason error (-5)

[andres tello]

Definitely, isn't recording at all. 

I have configured uri endpoint with 

http or https, ip or  fqdn, modified dns entry, hosts entry and nothing. 

RecordEndpoint is only recording to local files...but not sending the record to anywhere.  

 
I even used curl -X PUT <host> from inside the container to test and it was successful, my webservice created the as the put was done 

KMS isn't sending any data to my http recording endpoint... 

 

Any other thing to verify? 

--
You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/212c1deb-7192-475f-a00d-5db61ab3ebc9o%40googlegroups.com.

[Juan Navarro]

Extend the default logging level (https://doc-kurento.readthedocs.io/en/latest/features/logging.html), to add these other categories:

GST_URI:6,basemediamuxer:5

which should help to understand what is going on. You should look for messages such as

No sink for URI '%s'

No URI handler for the %s protocol found

didn't accept URI '%s'

created sink for URL '%s'

filesink location=%s

configuring location=%s

Invalid URI: '%s'

No URI handler implemented for '%s'

(%s is a template that will be written with the actual URI you pass)

Look for them in the logs and if you find any of these (or any other message that might be relevant) copy them here.

[Andres Tello]

Found No sink for URI

 GST_URI gsturi.c:498:gst_uri_get_location: extracted location xxxxxxx.online/xxxxxx/store/testing-work-lvt5efe5' from URI 'http://xxxxxxx.online/xxxxxx/store/testing-work-lvt5efe5'
0:00:16.597316776     1 0x7fdb10003d30 DEBUG                GST_URI gsturi.c:644:gst_element_make_from_uri: type:1, uri:http://xxxxxxx.online/xxxxxx/store/testing-work-lvt5efe5, elementname:(null)
0:00:16.597426334     1 0x7fdb10003d30 DEBUG                GST_URI gsturi.c:651:gst_element_make_from_uri: No sink for URI 'xxxxxxx.online/xxxxxx/store/testing-work-lvt5efe5'

This means is disabled? 

[Juan Navarro]

Not necessarily.

Notice how in Kurento code the function "gst_element_make_from_uri" is called first, to try and automatically found a plugin that is able to handle the type of URL you gave:
https://github.com/Kurento/kms-elements/blob/master/src/gst-plugins/recorderendpoint/kmsbasemediamuxer.c#L173

if it fails to find a sink, as in your case, it will return NULL, thus then the code continues and ends up creating a "curlhttpsink":
https://github.com/Kurento/kms-elements/blob/master/src/gst-plugins/recorderendpoint/kmsbasemediamuxer.c#L138

This is done because some GStreamer plugins, do not declare being able to handle any URL, even if they can, so we have to try first the automatic method and only then fall back to try by hand.

It is this "curlhttpsink" that had been disabled for a long time, but it should now be available again if you install from the nightly repositories (but make sure to either do a full upgrade, or a clean installation, as explained in Local Upgrade).

So, you should NOT see the messages "CURL HTTP plugin not available: curlhttpsink" and "No URI handler implemented for 'https'"

You can also explicitly check that the GstCurl plugin is there. Previous to enabling it, for example with repos from the release 6.13.0 or 6.14.0, this command would not show any results:

$ dpkg -L gstreamer1.5-plugins-bad | grep -i curl

However, with the current packages in the nightly repo:

$ dpkg -L gstreamer1.5-plugins-bad | grep -i curl
/usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstcurl.so

You can add this to the logging categories, in order to see the messages from the GstCurl plugin:

curl*:5

this will show all the debug logs from HTTP operations inside GStreamer (which is what Kurento really does, it doesn't really perform any HTTP operation by itself, instead it just delegates it to the Curl plugin)

--
Juan Navarro
Kurento maintainer & developer
j1elo @ GitHub and Twitter

--
You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/8c7604bd-17a5-4b49-9cd2-3e56ded3a152n%40googlegroups.com.

[Juan Navarro]

I think I might have found the cause of the issue. Contrary to what the docs say, you need a POST server, not a PUT server.

This has been frustrating because I spent several hours debugging the cause of the errors, which I could also reproduce, but in the end all the issues came from the fact that "curlhttpsink" performs a POST request.

I don't know why, at some point in 2016 the then-developers of Kurento decided to change HTTP library providers, from SOUP to CURL. So they changed from GStreamer's "souphttpclientsink" (which does a PUT request) to "curlhttpsink" (which does a POST request).

Change was done here:
https://github.com/Kurento/kms-elements/commit/800bf60a32860fab1334547709b1845e44ac3683

I think this was a big mistake, as it encapsulates silently a breaking change, given that the server requirements changed. To make things worse, they didn't even bother to update the RecorderEndpoint docs, which still to this day mentions the PUT request. What a mess.

Normally I would react reverting the offending commit, but it's being 4 years already so that might introduce _even_ more breakage. So I think the best course of action right now is to leave it as it is, and update the docs.

You will need an HTTP server that is able to accept chunked mode POST requests (with HTTP header "Transfer-Encoding: chunked"). Just for completeness, I happened to find this StackOverflow question without answer so I went ahead and wrote a sample server that I could use for my tests, it can be changed to use POST and then it works perfectly with Kurento RecorderEndpoint:

How to handle chunked encoding in Python BaseHTTPRequestHandler?


Hopefully you can make it work too!

On 2/7/20 23:55, Andres Tello wrote:

[andres tello]

Holy *hit!!! You are my hero, I think everyones hero in this list! XD 

 

Ill try and report back. 

--
You received this message because you are subscribed to the Google Groups "kurento" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kurento+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/7817c101-736e-3e34-a95d-5fd13d27245a%40gmx.es.

[Tito Cumpen]

Hello,

I am currently having a similar issue when using the nightly kurento build and issuing the record command to an https sink. I am using Ubuntu

Description: Ubuntu 16.04.7 LTS

Release: 16.04 xenial

Here is is the error I am seeing:

2020-10-07T23:11:56,725609 3901 0x00007efc649c0700   error basemediamuxer            kmsbasemediamuxer.c:144 kms_base_media_muxer_get_sink_fallback() <KmsAVMuxer@0x7efc503a2700>  CURL HTTP plugin not available: curlhttpsink

2020-10-07T23:11:56,725693 3901 0x00007efc649c0700   error basemediamuxer            kmsbasemediamuxer.c:228 kms_base_media_muxer_get_sink() <KmsAVMuxer@0x7efc503a2700>  No URI handler implemented for "https".

2020-10-07T23:11:56,725723 3901 0x00007efc649c0700   error basemediamuxer            kmsbasemediamuxer.c:252 kms_base_media_muxer_create_sink_impl() <KmsAVMuxer@0x7efc503a2700>  No available sink for uri https

Are there any additional steps to install curlhttpsink? or would installing kurento-media-server suffice?

Thanks,

Tito

[Juan Navarro]

Make sure you use the nightly build of Kurento. 6.15.0 will get released very soon, next week(s), but meanwhile the needed changes are only available in nightly. Check the docs for instructions on how to update correctly from release to nightly builds.

--
Juan Navarro
Kurento developer
Connect: @j1elo at GitHub & Twitter

To view this discussion on the web visit https://groups.google.com/d/msgid/kurento/594ab3e7-0a11-49a1-a8d3-6f3d093663d5n%40googlegroups.com.

[Neil Young]

Having build from 6.16. release. No curlhttpssink either. 

[Neil Young]

Also not in 6.18 source build...

[Neil Young]

I found that I was missing "libcurl-dev", which comes in different versions on Ubuntu:

Package libcurl-dev is a virtual package provided by:
  libcurl4-openssl-dev 7.68.0-1ubuntu2.14
  libcurl4-nss-dev 7.68.0-1ubuntu2.14
  libcurl4-gnutls-dev 7.68.0-1ubuntu2.14
You should explicitly select one to install.

I have chosen libcurl4-openssl-dev. Could recompile gst-plugins-bad and had the "curlhttpsink". But the entire KMS crashed when attempting to use it:

[1]+  Stopped                 kurento-media-server
Segmentation fault (thread 281469842859664, pid 60418)
Stack trace:
ubuntu@webrtc:~/kms-logs$ [strcmp]
/lib/aarch64-linux-gnu/libc.so.6:0x7D098
[getrn]
lhash.c:?
[lh_insert]
/lib/libcrypto.so.1.0.0:0xE1130
[OBJ_NAME_add]
/lib/libcrypto.so.1.0.0:0x6DD7C
[ERR_load_SSL_strings]
/lib/aarch64-linux-gnu/libssl.so.1.1:0x32740
[__pthread_once_slow]
/build/glibc-70LW9c/glibc-2.31/nptl/pthread_once.c:118
[CRYPTO_THREAD_run_once]
/lib/aarch64-linux-gnu/libcrypto.so.1.1:0x18B6C4
[OPENSSL_init_ssl]
/lib/aarch64-linux-gnu/libssl.so.1.1:0x328F8
[curl_url_set]
/lib/aarch64-linux-gnu/libcurl.so.4:0x5C7FC
[curl_strnequal]
/lib/aarch64-linux-gnu/libcurl.so.4:0x2BB34
[curl_easy_init]
/lib/aarch64-linux-gnu/libcurl.so.4:0x2BDE4
[gst_curl_base_sink_transfer_setup_unlocked]
kms-build/gst-plugins-bad/ext/curl/gstcurlbasesink.c:1238
[g_test_get_filename]
/lib/aarch64-linux-gnu/libglib-2.0.so.0:0x7B47C
[start_thread]
/build/glibc-70LW9c/glibc-2.31/nptl/pthread_create.c:477
[clone]
/lib/aarch64-linux-gnu/libc.so.6:0xD122C
^C
[1]+  Aborted                 (core dumped) kurento-media-server

Since it crashed whit libcrypto behind I supposed, the OpenSSL version wasn't the best choice. Uninstalled it and installed  libcurl4-gnutls-dev instead. Clean, recompile plugins-bad. And... It works :)

[Neil Young]

Just authentication is not going to work

0:00:07.545649943 106823 0xffff7400be10 DEBUG                GST_URI gsturi.c:644:gst_element_make_from_uri: type:1, uri:https://username:pass...@127.0.0.1:8080/api/record/abcde/1, elementname:(null)

0:00:07.545751735 106823 0xffff7400be10 DEBUG                GST_URI gsturi.c:650:gst_element_make_from_uri: No sink for URI 'https://username:pass...@127.0.0.1:8080/api/record/abcde/1'

0:00:07.545791485 106823 0xffff7400be10 ERROR         basemediamuxer kmsbasemediamuxer.c:147:kms_base_media_muxer_get_sink_fallback:<KmsAVMuxer@0xffff54037490> URL not valid

0:00:07.545797985 106823 0xffff7400be10 ERROR         basemediamuxer kmsbasemediamuxer.c:228:kms_base_media_muxer_get_sink:<KmsAVMuxer@0xffff54037490> No URI handler implemented for 'https'

0:00:07.545800485 106823 0xffff7400be10 ERROR         basemediamuxer kmsbasemediamuxer.c:252:kms_base_media_muxer_create_sink_impl:<KmsAVMuxer@0xffff54037490> No available sink for uri https://username:password@1