Open file descriptor limits being ignored?

[Jonathan Bugeja]

Hi,

I am running Crossbar under an unprivileged user:

ubuntu@ip-XX-XX-XX-XX:~$ ps aux | grep crossbar
ubuntu    3782  0.2  4.4 238288 180540 ?       Ss   13:54   0:04 crossbar-controller
ubuntu    3824  0.4  4.8 254740 198004 ?       S    13:54   0:07 crossbar-worker [router]
ubuntu    7272  0.0  0.0  10460   940 pts/4    S+   14:25   0:00 grep --color=auto crossbar

These are the limits at the filesystem level for that use:

ubuntu@ip-XX-XX-XX-XX:~$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 31538
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 10000
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

as soon as the file descriptors opened by the application go over 1024, I am getting "Could not accept new connection" like the below and the server stops accepting new connections:

2016-01-11T14:19:22+0000 [Router       3824 log_legacy] Could not accept new connection (EMFILE)

Is there a possibility that there is some other (hardcoded?) limit that is being hit?

Here is my version output

ubuntu@ip-XX-XX-XX-XX:~$ /opt/crossbar/bin/crossbar version
Automatically choosing optimal Twisted reactor
Running on Linux and optimal reactor (epoll) was installed.
     __  __  __  __  __  __      __     __
    /  `|__)/  \/__`/__`|__) /\ |__)  |/  \
    \__,|  \\__/.__/.__/|__)/~~\|  \. |\__/


 Crossbar.io        : 0.11.1
   Autobahn         : 0.10.9
     UTF8 Validator : autobahn
     XOR Masker     : autobahn
     JSON Codec     : stdlib
     MsgPack Codec  : msgpack-python-0.4.6
   Twisted          : 15.4.0-EPollReactor
   Python           : 2.7.10/PyPy-2.6.1
 OS                 : Linux-3.13.0-48-generic-x86_64-with-debian-jessie-sid
 Machine            : x86_64

Any help would be greatly appreciated

[Jonathan Bugeja]

This was in actual fact more of a systems issue, since I was using upstart and it ignores limit.conf, I have added the below to the upstart job to get it fixed:

limit nofile 1048576 1048576

BR

Jonathan

[Elvis Stansvik]

2016-01-11 15:54 GMT+01:00 Jonathan Bugeja <xad...@gmail.com>:
> This was in actual fact more of a systems issue, since I was using upstart
> and it ignores limit.conf, I have added the below to the upstart job to get
> it fixed:
>
> limit nofile 1048576 1048576

Good that you solved it. Be aware however that there's a system-wide
limit as well, which applies to all processes combined:

root@c4b278f184da:/# sysctl fs.file-max
fs.file-max = 797774

That was from an Ubuntu 14.04 docker container. So you may want to
increase that one as well.

Elvis

> --
> You received this message because you are subscribed to the Google Groups
> "Crossbar" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to crossbario+...@googlegroups.com.
> To post to this group, send email to cross...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/crossbario/8fa5ec36-922c-4a4a-a981-f3bb8522c390%40googlegroups.com.
>
> For more options, visit https://groups.google.com/d/optout.

[Tobias Oberstein]

Hi Jonathan,

http://crossbar.io/docs/Network-Tuning/

Cheers,
/Tobias

> newconnection (EMFILE)

> |
>
> Is there a possibility that there is some other (hardcoded?) limit that
> is being hit?
>
> Here is my version output
>
> |
> ubuntu@ip-XX-XX-XX-XX:~$ /opt/crossbar/bin/crossbar version
> Automaticallychoosing optimal Twistedreactor

> Runningon Linuxandoptimal reactor (epoll)was installed.

> __ __ __ __ __ __ __ __
> /`|__)/ \/__`/__`|__) /\ |__) |/ \
> \__,| \\__/.__/.__/|__)/~~\| \. |\__/
>
>
> Crossbar.io : 0.11.1
> Autobahn : 0.10.9
> UTF8 Validator : autobahn
> XOR Masker : autobahn
> JSON Codec : stdlib
> MsgPack Codec : msgpack-python-0.4.6
> Twisted : 15.4.0-EPollReactor
> Python : 2.7.10/PyPy-2.6.1
> OS : Linux-3.13.0-48-generic-x86_64-with-debian-jessie-sid
> Machine : x86_64
> |
>
> Any help would be greatly appreciated
>

> --
> You received this message because you are subscribed to the Google
> Groups "Crossbar" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to crossbario+...@googlegroups.com

> <mailto:crossbario+...@googlegroups.com>.

> To post to this group, send email to cross...@googlegroups.com

> <mailto:cross...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/crossbario/f71bd674-854d-4acf-8cb7-0441befa602e%40googlegroups.com
> <https://groups.google.com/d/msgid/crossbario/f71bd674-854d-4acf-8cb7-0441befa602e%40googlegroups.com?utm_medium=email&utm_source=footer>.

[xad...@gmail.com]

Hi Tobias,

Yes I did configure limits correctly. They are however ignored by upstart so thought it was worth pointing that out for anyone who's using upstart to manage their crossbar setup.

Regards,
Jonathan

> You received this message because you are subscribed to a topic in the Google Groups "Crossbar" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/crossbario/EqScWNAKEbQ/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to crossbario+...@googlegroups.com.
> To post to this group, send email to cross...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/crossbario/5696BF30.2090109%40gmail.com.

[Tobias Oberstein]

Hi Jonathan,

do you have

Modify /etc/security/limits.conf for the following

# wildcard does not work for root, but for all other users
* soft nofile 1048576
* hard nofile 1048576

for running under non-root?

also rgd running under unpriv. user:
http://crossbar.io/docs/Running-on-privileged-ports/

Cheers,
/Tobias