NoRouteToHostException

[mudphone]

We're currently running Redis 1.01 on Ubuntu.

We frequently see the following error when reading one of our sets in
Redis. It doesn't happen on every read, but it does happen enough to
cause problems.

java.lang.RuntimeException: java.net.NoRouteToHostException: Cannot
assign requested address

Is this a known problem?

Thanks,
Kyle

[Joubin Houshyar]

Hi,

Never seen this before. Appears to be OS / setup specific.

This is dated (2004/jdk1.4) but perhaps relevant:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6206527

/R

[mudphone]

Hi Joubin,

Thanks for the reply.

At this point we believe that the NoRouteToHostException was due to
our opening and closing thousands of connections to Redis in a very
short period of time. Our theory is that, although we were opening
AND CLOSING the ports, the file handles (or something related) we not
recovered immediately by the OS. So, we were using up all of our
ports and receiving this error. When it occurred the exception showed
up on the client, but the Redis logs didn't show any connection. This
seems to correlate with a failure to connect at all.

Will update if we learn more.

Thanks,
Kyle

[Stuart Halloway]

Hi Kyle,

I have duplicated the problem you are seeing, from both Clojure and
Ruby clients. Slightly different errors, but in both cases it appears
that some resource is not being closed by close, and if you create and
close sockets fast enough bad things will happen.

Did you decide on a workaround?

Stu

[Jeremy Hinegardner]

I saw something similar when doing massive amounts of connections to a tokyo
tyrant server.

run 'netstat -an'

I'm betting you are seeing thousands and thousands of TIME_WAIT sockets, and the
reason you are seeing NoRouteToHoseException is that all of the available
ephemeral outbound ports are taken.

Or, I should say, all available outbound ports that could connect to the redis
server are taken. Since each TCP 4-tuple( src_ip, src_ort, dst_ip, dst_port)
defines a unique connection on the network, you have probably used up the entire
ephemeral port range on the client machine. They are all probably in a
TIME_WAIT state and no more are available.

So, if this is the case, there are a couple of things you can do, all of these
are linux examles, they can be updated in the /proc file system and made
permanent across reboots with appropriate updates to /etc/sysctl.conf:

(1) make more ephemeral ports available
% cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000
# if you need more
% echo "15000 65535" > /proc/sys/net/ipv4/ip_local_port_range
# make it permanent by putting this in /etc/sysctl.conf
net.ipv4.ip_local_port_range = 15000 65535

(2) lower the TCP_FIN_TIMEOUT parameter, allowing the resource to be free'd up
if you are the one closing the socket.

# Lower the fin_wait timeout
net.ipv4.tcp_fin_timeout = 30
# or echo "30" > /proc/sys/net/ipv4/tcp_fin_timeout

(3) cycle TIME_WAIT sockets faster. These 2 options generally come
with the big red warning saying 'only do this if you know what
you are doing' and to try the tcp_tw_reuse one before tcp_tw_recycle

# reuse sockets in TIME_WAIT state for new connections when it
# is safe from protocol viewpoint
net.ipv4.tcp_tw_reuse = 1
# and/or echo "1" > /proc/sys/net/ipv4/tcp_tw_reuse

# enables fast recycling of TIME_WAIT sockets
net.ipv4.tcp_tw_recycle = 1
# and/or echo "1" > /proc/sys/net/ipv4/tcp_tw_recycle

enjoy,

-jeremy

On Tue, Oct 20, 2009 at 07:11:30PM -0700, Stuart Halloway wrote:
>
> Hi Kyle,
>
> I have duplicated the problem you are seeing, from both Clojure and
> Ruby clients. Slightly different errors, but in both cases it appears
> that some resource is not being closed by close, and if you create and
> close sockets fast enough bad things will happen.
>
> Did you decide on a workaround?
>
> Stu
>

> On Sep 25, 2:19?am, mudphone <kyle...@gmail.com> wrote:
> > Hi Joubin,
> >
> > Thanks for the reply.
> >
> > At this point we believe that the NoRouteToHostException was due to
> > our opening and closing thousands of connections to Redis in a very

> > short period of time. ?Our theory is that, although we were opening

> > AND CLOSING the ports, the file handles (or something related) we not

> > recovered immediately by the OS. ?So, we were using up all of our

> > ports and receiving this error. When it occurred the exception showed

> > up on the client, but the Redis logs didn't show any connection. ?This

> > seems to correlate with a failure to connect at all.
> >
> > Will update if we learn more.
> >
> > Thanks,
> > Kyle
> >
> > On Sep 24, 9:01?pm, Joubin Houshyar <suno...@gmail.com> wrote:
> >
> > > Hi,
> >

> > > Never seen this before. ?Appears to be OS / setup specific.

> >
> > > This is dated (2004/jdk1.4) but perhaps relevant:http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6206527
> >
> > > /R
> >
> > > On Sep 24, 6:15?pm, mudphone <kyle...@gmail.com> wrote:
> >
> > > > We're currently running Redis 1.01 on Ubuntu.
> >
> > > > We frequently see the following error when reading one of our sets in

> > > > Redis. ?It doesn't happen on every read, but it does happen enough to

> > > > cause problems.
> >
> > > > java.lang.RuntimeException: java.net.NoRouteToHostException: Cannot
> > > > assign requested address
> >
> > > > Is this a known problem?
> >
> > > > Thanks,
> > > > Kyle
> >
> >
> >

--
========================================================================
Jeremy Hinegardner jer...@hinegardner.org

[Stuart Halloway]

Yes, that's it. Lucky for me some combination of (1) and (2) looks
sufficient.

Thanks!
Stu