Keepalive not working with upstream tcp server

[Jai]

Hi

I am trying to use nginx to proxy my requests to a custom tcp backend server that I have. I am following the same model as the default memcached module with in the nginx code base (1.5.10) , the relevant config file attached.  Tried with most of the keepalive options but I still see new connections getting created to upstream server. (rather I do see the connection getting closed by nginx in strace)

Do I need to specifically compile nginx with http upstream keepalive module, thought it was default enabled but the code doesn't seem to go through it looking in gdb.  Is there any other specific module  or setting I need to use for the upstream tcp persistent connection usecase.

Thanks in advance, Jai

--- nginx.conf------

http {
    default_type  application/octet-stream;
    keepalive_timeout  100;
    proxy_http_version 1.1;

    upstream my_backend {
            server 127.0.0.1:1111;
            keepalive 100;
    }

    server {
         listen       4080;
         keepalive_timeout  100;
         keepalive_requests 100000;
         proxy_http_version 1.1;
         proxy_set_header Connection keepalive;

         location / {
                 root   html;
                 index  index.html index.htm;
                 my_pass my_backend;

         }
    }

[Yichun Zhang (agentzh)]

Hello!

On Wed, Apr 2, 2014 at 1:03 AM, Jai wrote:
> I am trying to use nginx to proxy my requests to a custom tcp backend server
> that I have. I am following the same model as the default memcached module
> with in the nginx code base (1.5.10) , the relevant config file attached.
> Tried with most of the keepalive options but I still see new connections
> getting created to upstream server. (rather I do see the connection getting
> closed by nginx in strace)
>

You need to check

1. if your remote server closes the connection first, and
2. if your remote server returns the "Connection: close" response
header or alike.

> Do I need to specifically compile nginx with http upstream keepalive module

No.

> location / {
> root html;
> index index.html index.htm;
> my_pass my_backend;
>

What is the "my_pass" directive? You're not using the standard ngx_proxy module?

Regards,
-agentzh

[Jai]

Thanks for the reply .

You need to check

1. if your remote server closes the connection first, and
2. if your remote server returns the "Connection: close" response
header or alike.

 Will check carefully again, but currently it doesn't seem like the server is closing the connection but it indeed is closed by the nginx looking through strace. 

>                  my_pass my_backend;
>

What is the "my_pass" directive? You're not using the standard ngx_proxy module?

 Think of it like a memcache module set up, so we have our proxy handler (based on memcache handler) which hooks into create_request/process_header... phases.  The connections are managed by ngx upstream module since it is a normal TCP connection and hopefully keep alive module.

Regards,
-agentzh

[Yichun Zhang (agentzh)]

Hello!

On Wed, Apr 2, 2014 at 12:27 PM, Jai wrote:
> Will check carefully again, but currently it doesn't seem like the server
> is closing the connection but it indeed is closed by the nginx looking
> through strace.
>

nginx will close the connection when the remote already closes the
connection or there is unread data in the system receive buffer.

>
> Think of it like a memcache module set up, so we have our proxy handler
> (based on memcache handler) which hooks into
> create_request/process_header... phases. The connections are managed by ngx
> upstream module since it is a normal TCP connection and hopefully keep alive
> module.
>

You can test if the standard ngx_memcached module works for you. If
yes, then it's more likely to be a bug in your own nginx module or
your remote server.

Regards,
-agentzh