segfault at ngx_shmtx_lock(subscription->channel->mutex) in ngx_http_push_stream_worker_subscriber_cleanup by timeout

69 views
Skip to first unread message

Dmitry Vasilyev

unread,
Sep 21, 2018, 5:59:27 AM9/21/18
to nginxpushstream
Good day everyone!

We have a segfault problem under heavy load in SUBJ, I created an issue on github with details (https://github.com/wandenberg/nginx-push-stream-module/issues/280)

Are there any recommendations and workarounds to avoid this problem?

Wandenberg Peixoto

unread,
Sep 21, 2018, 8:52:08 AM9/21/18
to nginxpu...@googlegroups.com
Hi Dmitry,

I saw your issue on github, but will not have time to investigate it in the next days.
Whould be good if you can generate a stacktrace and a coredump of the error.
You can send to me directly.
Also describe a bit your environment, how the application works and what exactly you call a heavy load.
If you have any other third party modules.
If you are using this nginx server to something else or just to do streaming.
Any extra information you can provide will help to find the cause.

I have not tested the module with this nginx version, so don't know if there is any substantial change on it that might be causing the issue.
In case you want to, you can use an "older" version of nginx, like 1.10.1 and see if the issue also appears.

Kind regards,
Wandenberg

--
You received this message because you are subscribed to the Google Groups "nginxpushstream" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nginxpushstre...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Dmitry Vasilyev

unread,
Sep 24, 2018, 4:34:06 AM9/24/18
to nginxpu...@googlegroups.com

Hi Wandenberg!

I've added new information to the issue that may be useful to you!

Also looked at the source code and ask to see if patch will help in this case?

 

 

diff --git a/src/ngx_http_push_stream_module_utils.c b/src/ngx_http_push_stream_module_utils.c

index a87c76b..510ab89 100644

--- a/src/ngx_http_push_stream_module_utils.c

+++ b/src/ngx_http_push_stream_module_utils.c

@@ -1020,6 +1020,7 @@ ngx_http_push_stream_collect_expired_messages_and_empty_channels_data(ngx_http_p

         channel = ngx_queue_data(q, ngx_http_push_stream_channel_t, queue);

         q = ngx_queue_next(q);

+        ngx_shmtx_lock(channel->mutex);

        if ((channel->stored_messages == 0) && (channel->subscribers == 0) && (channel->expires < ngx_time()) && !channel->for_events) {

             channel->deleted = 1;

             channel->expires = ngx_time() + NGX_HTTP_PUSH_STREAM_DEFAULT_SHM_MEMORY_CLEANUP_OBJECTS_TTL;

@@ -1035,6 +1036,7 @@ ngx_http_push_stream_collect_expired_messages_and_empty_channels_data(ngx_http_p

             ngx_http_push_stream_send_event(mcf, ngx_cycle->log, channel, &NGX_HTTP_PUSH_STREAM_EVENT_TYPE_CHANNEL_DESTROYED, temp_pool);

         }

+        ngx_shmtx_unlock(channel->mutex);

     }

     ngx_shmtx_unlock(&data->channels_queue_mutex);

 

 

От: Wandenberg
Отправлено: 21 сентября 2018 г. в 15:52
Кому: nginxpu...@googlegroups.com
Тема: Re: segfault at ngx_shmtx_lock(subscription->channel->mutex) inngx_http_push_stream_worker_subscriber_cleanup by timeout

Wandenberg Peixoto

unread,
Oct 1, 2018, 6:51:38 PM10/1/18
to nginxpu...@googlegroups.com
Hi Dmitry,

thanks for the information you added to the issue, and sorry for not replying before.

Just today I was able to check the trace and your comments.
I don't think that this patch you suggested will solve the issue.
But using your trace and configuration, I imagine that the problem is somewhere between a channel being deleted and a long-polling connection timed out, and this subscriber that was "pending" being removed from all channels before closing the connection.
The subscriptions should be removed in this case, but maybe there is some race condition that I did not check.

Can you confirm if your subscribers are connecting only to one channel or to multiple channels?
Also, how hard would be to add some logs and apply a patch to your servers to help debug the issue?

Kind regards,
Wandenberg

Reply all
Reply to author
Forward
0 new messages