nginx+pushstream module hangs under high load ...

[john...@gmail.com]

Hello.
My Environment:
------------
OS: CentOs 5.5;
uname -a :Linux server4 2.6.18-194.32.1.el5 #1 SMP Wed Jan 5 17:52:25
EST 2011 x86_64 x86_64 x86_64 GNU/Linux;

nginx -V
nginx: nginx version: nginx/1.0.2
nginx: built by gcc 4.1.2 20080704 (Red Hat 4.1.2-48)
nginx: TLS SNI support disabled
nginx: configure arguments: --user=nginx --group=nginx --prefix=/usr/
share/nginx --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/
nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/
var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/
tmp/client_body --http-proxy-temp-path=/var/lib/nginx/tmp/proxy --http-
fastcgi-temp-path=/var/lib/nginx/tmp/fastcgi --pid-path=/var/run/
nginx.pid --lock-path=/var/lock/subsys/nginx --with-
http_secure_link_module --with-http_random_index_module --with-
http_ssl_module --with-http_realip_module --with-http_addition_module
--with-http_sub_module --with-http_dav_module --with-http_flv_module --
with-http_gzip_static_module --with-http_stub_status_module --with-
http_perl_module --with-debug --with-mail --with-mail_ssl_module --
with-cc-opt='-O2 -g -m64 -mtune=generic' --with-ipv6 --add-module=/usr/
src/redhat/BUILD/nginx-1.0.2/nginx-upstream-fair --add-module=/usr/src/
redhat/BUILD/nginx-1.0.2/nginx-upload-progress-module --add-module=/
usr/src/redhat/BUILD/nginx-1.0.2/mod_zip-1.1.6 --add-module=/usr/src/
redhat/BUILD/nginx-1.0.2/nginx_upload_module-2.2.0 --add-module=/usr/
src/redhat/BUILD/nginx-1.0.2/nginx_mod_h264_streaming-2.2.7 --add-
module=/usr/src/redhat/BUILD/nginx-1.0.2/nginx-push-stream-module --
add-module=/usr/src/redhat/BUILD/nginx-1.0.2/nginx_upstream_hash-0.3.1
------------
Now about my problem.
I try publish test message to 25000 non exist channels.
Nginx begin to eat CPU and then it ate all 100 % CPU.
Q: Why batch publish messages to not exist channels bring to 100 % CPU
utilization by nginx?

[Wanden]

Hi John,

We detect this problem last week and we are working on the solution.
In fact, the branch called log_locks has our fix to this problem. It
is under test and I will release a new version very soon. I hope to
release until the end of week.

If you can, do a checkout of this branch and do your tests again to
see if it work for you, and let me know the results.

When I release the new version I send another message.

Wandenberg

On May 17, 11:08 am, "JohnBa...@gmail.com" <johnba...@gmail.com>
wrote:

[john...@gmail.com]

Hello.
I have built nginx-1.0.2 with push_stream_pluging from  log_locks branch, and problem disappears.
Thanks for you work!
When you released next version ?

[Wanden]

Hi John,

I finish the release 0.2.2 with this and another fixes.

If you can, do another test and send me a feedback.

Regards,
Wandenberg

[Alexey Vdovin]

Wanden,

Thank you for new release!
It seems is nothing changed with X-Accel-Redirect issue. I still have
connection close after lingering_timeout.
And one more issue appears: Nginx now shows error message if I put
chunked_transfer_encoding off in subscriber location.
nginx: [emerg] "chunked_transfer_encoding" directive is duplicate in
nginx.conf:123

---
Best Regards
Alexey

--
---
Best Regards
Alexey Vdovin

[Wanden]

Hi Alexey,

Yes, you have to remove the directive "chunked_transfer_encoding" for
the locations used for the module.
This control will be done inside the module.

I will start some tests for X-Accel-Redirect issue.
Sorry, I spend some time finishing this release and didn't start the
tests yet.

[Alexey Vdovin]

Wanden,

I did some research on my issue and found when we set discard_body to
1, nginx replaces r->read_event_handler to
ngx_http_discarded_request_body_handler and closes connection.
r->discard_body = 0; - connection is not closed but it could cause
some side effects.
You know push module architecture better then me, probably it will
help you to find right solution.

---
Best Regards
Alexey

[Wandenberg Peixoto]

I will check this, but is strange. Why connections done with curl or directly for the browser are not closed?

Where did you see the replacement of r->read_event_handler, in which code?

[Alexey Vdovin]

Wanden,

See ngx_http_finalize_connection() it sets it own read_event_handler
if r->main->count != 1

I did debug trace and found that.

Here is piece of debug log:
[debug] 13079#0: *1 http finalize request: -4,
"/listen?id=my_channel_1" a:1, c:2
[debug] 13079#0: *1 event timer add: 3: 5000:518281623
[debug] 13079#0: *1 http request count:2 blk:0
[debug] 13079#0: *1 http finalize request: -4,
"/listen?id=my_channel_1" a:1, c:1
[debug] 13079#0: *1 set http keepalive handler

Take look at c:2 that means r->main->count.

---
Best Regards
Alexey

--

[Wandenberg Peixoto]

Yes, I see that.

The problem is not the read_event_handler.

The problem is "if r->main->count != 1" it always call the method ngx_http_close_request. Or directly if discard_body is not set, or indirectly using a timer.

With this, is impossible to do an internal request with streaming.

When you call the subscriber location the request pass on the other if
         if (!ngx_terminate
         && !ngx_exiting
         && r->keepalive
         && clcf->keepalive_timeout > 0)
and everything work ok.

I'm not sure if it's a bug on nginx or a behavior by design :)
We can ask on nginx developer list.

In my opition this method should check first if the request is has keepalive setted and after decide to close or not the subrequests.

What do you think?

[john...@gmail.com]

Wanden, thank you very match for 0.2.2 release!
Today I start to test version-0.2.2.
I'll write if there are any problems.

[Alexey Vdovin]

Wanden,

I've found workaround by using ngx_http_auth_request_module to make
authentication before subscribe client to the channel.

You are right, it seems nginx has architecture limitation in
subrequest processing.

---
Best Regards
Alexey

[Wandenberg Peixoto]

Hi Alexey,

I don't know if you are using the directive push_stream_subscriber_connection_timeout (I recommend you to use :) ).
If yes, another alternative is set the lingering_timeout with the same value or bigger than of push_stream_subscriber_connection_timeout, only on the subscriber location.

With this the internal redirect seems to work properly.

Regards,
Wandenberg

[Alexey Vdovin]

Wanden,

I found one more curios think.
If I put auth_request /auth; in subscriber location nginx stops
closing connection even after X-Accel-Redirect

PS: Thank you for solution!

---
Best Regards
Alexey

On Wed, May 25, 2011 at 4:45 PM, Wandenberg Peixoto

[Wandenberg Peixoto]

Only to finish the thread and say this problem was solved and now is
possible to use X-Accel-Redirect.

Regards,
Wanden

[Alexey Kopytko]

Hi Wanden,

Would you mind if I ask you in which commits did you fix the issue?
Currently I am looking at c4bb9c7a2, but the commit's message is a bit vague on the purpose of removal.

Thanks.

-Alexey

[Alexey Kopytko]

Nevermind. It was in 8c855057.

On Monday, September 26, 2011 1:03:56 AM UTC+9, Wanden wrote:

[Wandenberg Peixoto]

Hi Alexey,

you are using a very old version, I suggest use a newer version.

Regards,

Wandenberg

--
You received this message because you are subscribed to the Google Groups "nginxpushstream" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nginxpushstre...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.