[erlang-questions] Increase the handle limit
Martin Dimitrov
I am developing on a Windows 7 machine, Erlang R14B04. The limit of open
handles seems to be 1200. But this is not imposed by the operating
system (Windows has a hard coded limit of 16 million handles). So my
question is how can I increase the handle limit in Erlang?
Thanks,
Martin
_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://erlang.org/mailman/listinfo/erlang-questions
Joel Reymont
I bump against this all the time on the Mac.
Erlang uses the FD_SETSIZE setting from the system header files when compiled, normally 1024.
Editing this value and re-compiling Erlang causes epmd to stop working.
My solution is to save the old epmd and recompile after editing the system header files.
On Oct 21, 2011, at 7:25 AM, Martin Dimitrov wrote:
> Hi all,
>
> I am developing on a Windows 7 machine, Erlang R14B04. The limit of open
> handles seems to be 1200. But this is not imposed by the operating
> system (Windows has a hard coded limit of 16 million handles). So my
> question is how can I increase the handle limit in Erlang?
--------------------------------------------------------------------------
- for hire: mac osx device driver ninja, kernel extensions and usb drivers
---------------------+------------+---------------------------------------
http://wagerlabs.com | @wagerlabs | http://www.linkedin.com/in/joelreymont
---------------------+------------+---------------------------------------
Joel Reymont
On Oct 21, 2011, at 8:58 AM, Antoine Koener wrote:
> FD_SETSIZE is used by the select() call, and I would be suprised if erlang is still using select() instead of kqueue or epoll or completion port.
>
> But I've not checked epmd source at the moment...
Martin Dimitrov
Martin
Peer Stritzinger
> I don't think you can.
>
> I bump against this all the time on the Mac.
>
> Erlang uses the FD_SETSIZE setting from the system header files when compiled, normally 1024.
I took the time and wrote a little test because this sounded too
unbelievable, but nobody disagreed.
TL;DR Version: This is not true. Tested on MacOSX 10.5.8 on PowerPC
with Erlang R14B03
(used the oldest version of Mac OSX anybody sane would run and the
Erlang is also not cutting edge version):
Could open about 56000 files the I hit some OS system limit. Didn't
try to increase further since this already 50 times the amount of
FD_SETSIZE which is indeed set to 1024 on my system.
In order to be able to do this I had to do this:
Set ERL_MAX_PORTS (see erlang manpage) to increase the number of ports.
Set ulimit -n to allow more open files in the process
Start Erlang with +P to allow enough processes
With this settings I get up to about 10000 open files since I can't
set the ulimit above the system limit. In order to get more I had to
sysctl -w kern.maxfiles=120000
sysctl -w kern.maxfilesperproc=100000
The increase ulimits -n and rerun the test:
{enfile,56016}
Error enfile means from open(2):
[ENFILE] The system file table is full.
Since I set maxfiles to 120000 there should have been more files
possible, don't know what limit I hit, maybe number of files in a
directory?
Definitely not a Erlang limit though.
The program I used is below.
>
> Editing this value and re-compiling Erlang causes epmd to stop working.
This is no wonder. Editing the header doesn't change the system call
internals, probably epmd uses select() which fails because the system
call won't handle sets with increased set size.
Editing the system headers is definitely not to recommend. It may
break all kinds of libraries in a subtle way.
> My solution is to save the old epmd and recompile after editing the system header files.
>
> On Oct 21, 2011, at 7:25 AM, Martin Dimitrov wrote:
>
>> Hi all,
>>
>> I am developing on a Windows 7 machine, Erlang R14B04. The limit of open
>> handles seems to be 1200. But this is not imposed by the operating
>> system (Windows has a hard coded limit of 16 million handles). So my
>> question is how can I increase the handle limit in Erlang?
All this probably won't help you solving the Windows problem in a
systematic way, unfortunately (actually fortunately I think ;-) I'm no
Windows expert.
Looking the the corresponding source files of ERTS might help:
* find out how Erlang finds out about file descriptor changes
* find out where the limit comes from
Also what might lead you to the reason of the limit is the exact error
message you get when you hit the limit.
Cheers
-- Peer
Little program used to find limit and reason:
-module(test_fds).
-compile([export_all]).
count() ->
count(1, []).
count(N, Fds) ->
case file:open(integer_to_list(N), [write]) of
{ok, F} ->
count(N+1, [F| Fds]);
{error, Err} ->
[ file:close(F) || F <- Fds ],
{Err, N}
end.
Joel Reymont
On Oct 21, 2011, at 3:49 PM, Peer Stritzinger wrote:
> TL;DR Version: This is not true. Tested on MacOSX 10.5.8 on PowerPC
> with Erlang R14B03
> (used the oldest version of Mac OSX anybody sane would run and the
> Erlang is also not cutting edge version):
Try to open that many sockets.
Let us know what happens.
Peer Stritzinger
>
> On Oct 21, 2011, at 3:49 PM, Peer Stritzinger wrote:
>
>> TL;DR Version: This is not true. Tested on MacOSX 10.5.8 on PowerPC
>> with Erlang R14B03
>> (used the oldest version of Mac OSX anybody sane would run and the
>> Erlang is also not cutting edge version):
>
>
> Try to open that many sockets.
>
> Let us know what happens.
Sorry no time for more experiments at the moment.
OTOH: sockets are also just file descriptors so no other limits apply
for sockets, except you might run out of ephemeral port numbers.
Cheers,
-- Peer
Joel Reymont
On Oct 21, 2011, at 4:23 PM, Peer Stritzinger wrote:
> Sorry no time for more experiments at the moment.
>
> OTOH: sockets are also just file descriptors so no other limits apply
> for sockets, except you might run out of ephemeral port numbers.
I agree with you in theory. In practice, it doesn't work as you describe.
I know because I actually tried.
I don't have Mac OSX 10.5.8 around but did use 10.6 and 10.7.
I reported my findings in response to the original poster and stand behind them.
The hard-coded limit is enforced even with kernel poll enabled.
Martin Dimitrov
Regards,
Martin
Dave Cottlehuber
On 21 October 2011 16:49, Peer Stritzinger <pee...@gmail.com> wrote:
[snip]
> Little program used to find limit and reason:
>
> -module(test_fds).
> -compile([export_all]).
>
> count() ->
> count(1, []).
>
> count(N, Fds) ->
> case file:open(integer_to_list(N), [write]) of
> {ok, F} ->
> count(N+1, [F| Fds]);
> {error, Err} ->
> [ file:close(F) || F <- Fds ],
> {Err, N}
> end.
Erlang R14B03 (erts-5.8.4) [source] [smp:2:2] [rq:2] [async-threads:0]
Eshell V5.8.4 (abort with ^G)
1> pwd().
C:/erlang/scripts
ok
2> c(limit).
{ok,limit}
3> limit:count().
{system_limit,1021}
Tim Watson
Joel Reymont
On Oct 21, 2011, at 8:48 PM, Tim Watson wrote:
> I think Erlang still uses select on a number of platforms and IIRC kqueue on OS-X is somehow broken so Erlang is still using select() on the mac.
Are you user kqueue is broken on the Mac?
--------------------------------------------------------------------------
- for hire: mac osx device driver ninja, kernel extensions and usb drivers
---------------------+------------+---------------------------------------
http://wagerlabs.com | @wagerlabs | http://www.linkedin.com/in/joelreymont
---------------------+------------+---------------------------------------
_______________________________________________
Joel Reymont
On Oct 21, 2011, at 8:48 PM, Tim Watson wrote:
> Is the gist of this is that select() has a size limit (1024) which has nothing to do with the OS limit on how many file handles you can open at a time? I think Erlang still uses select on a number of platforms and IIRC kqueue on OS-X is somehow broken so Erlang is still using select() on the mac.
It seems that ERTS_POLL_USE_SELECT is defined on the Mac, although +K true is also accepted.
This causes the following block in erts_poll_init() of erts/emulator/sys/common/erl_poll.c to be triggered
#if ERTS_POLL_USE_SELECT && defined(FD_SETSIZE)
if (max_fds > FD_SETSIZE)
max_fds = FD_SETSIZE;
#endif
This really should NOT be a compile-time setting.
It should check to see if kernell poll, etc. are enabled at runtime and only clip max_fds if they are not.
--------------------------------------------------------------------------
- for hire: mac osx device driver ninja, kernel extensions and usb drivers
---------------------+------------+---------------------------------------
http://wagerlabs.com | @wagerlabs | http://www.linkedin.com/in/joelreymont
---------------------+------------+---------------------------------------
_______________________________________________
Dave Cottlehuber
>
> -module(test_fds).
> -compile([export_all]).
>
> count() ->
> count(1, []).
>
> count(N, Fds) ->
> case file:open(integer_to_list(N), [write]) of
> {ok, F} ->
> count(N+1, [F| Fds]);
> {error, Err} ->
> [ file:close(F) || F <- Fds ],
> {Err, N}
> end.
On my built-from-source Windows 7 x64 (32bit erlang of course):
Erlang R14B03 (erts-5.8.4) [source] [smp:2:2] [rq:2] [async-threads:0]
Eshell V5.8.4 (abort with ^G)
1> pwd().
C:/erlang/scripts
ok
2> c(limit).
{ok,limit}
3> limit:count().
{system_limit,1021}
HTH,
Dave
Jon Watte
--
Americans might object: there is no way we would sacrifice our living standards for the benefit of people in the rest of the world. Nevertheless, whether we get there willingly or not, we shall soon have lower consumption rates, because our present rates are unsustainable.
Rickard Green
Joel Reymont wrote:
> On Oct 21, 2011, at 8:48 PM, Tim Watson wrote:
>
>> Is the gist of this is that select() has a size limit (1024) which has nothing to do with the OS limit on how many file handles you can open at a time? I think Erlang still uses select on a number of platforms and IIRC kqueue on OS-X is somehow broken so Erlang is still using select() on the mac.
>
> It seems that ERTS_POLL_USE_SELECT is defined on the Mac, although +K true is also accepted.
>
> This causes the following block in erts_poll_init() of erts/emulator/sys/common/erl_poll.c to be triggered
>
> #if ERTS_POLL_USE_SELECT && defined(FD_SETSIZE)
> if (max_fds > FD_SETSIZE)
> max_fds = FD_SETSIZE;
> #endif
>
> This really should NOT be a compile-time setting.
>
> It should check to see if kernell poll, etc. are enabled at runtime and only clip max_fds if they are not.
>
This has been up on the list before...
The "clipping" is there for a reason. Kernel-poll on MacOSX is
implemented using kqueue(). Neither kqueue() nor poll() can handle all
types of file-descriptors on MacOSX. The runtime system needs to be able
to handle any type of file-descriptor that may appear. We are therefore
forced to fall back on select() when a file-descriptor that cannot be
handled by kqueue() appears. A file-descriptor that cannot be handled by
kqueue() may appear at any time, i.e. we need to be able to ensure that
all file-descriptors that may appear in the runtime-system can be
handled by select().
Regards,
Rickard
--
Rickard Green, Erlang/OTP, Ericsson AB.
Joel Reymont
I see no basis for your statement below.
On Oct 24, 2011, at 12:52 PM, Rickard Green wrote:
> The "clipping" is there for a reason. Kernel-poll on MacOSX is implemented using kqueue(). Neither kqueue() nor poll() can handle all types of file-descriptors on MacOSX. The runtime system needs to be able to handle any type of file-descriptor that may appear.
Could this be old information that's no longer valid?
I just looked at the documentation for kqueue and poll and saw no restrictions on the type of file descriptor being handled.
Poll has no restrictions whatsoever and kqueue has slightly different notification behavior depending on descriptor type.
Thanks, Joel
--------------------------------------------------------------------------
- for hire: mac osx device driver ninja, kernel extensions and usb drivers
---------------------+------------+---------------------------------------
http://wagerlabs.com | @wagerlabs | http://www.linkedin.com/in/joelreymont
---------------------+------------+---------------------------------------
_______________________________________________
Rickard Green
> Rickard,
>
> I see no basis for your statement below.
>
> On Oct 24, 2011, at 12:52 PM, Rickard Green wrote:
>
>> The "clipping" is there for a reason. Kernel-poll on MacOSX is implemented using kqueue(). Neither kqueue() nor poll() can handle all types of file-descriptors on MacOSX. The runtime system needs to be able to handle any type of file-descriptor that may appear.
>
> Could this be old information that's no longer valid?
>
> I just looked at the documentation for kqueue and poll and saw no restrictions on the type of file descriptor being handled.
>
> Poll has no restrictions whatsoever and kqueue has slightly different notification behavior depending on descriptor type.
>
> Thanks, Joel
On our Lion machine:
$ uname -a
Darwin xxx 11.2.0 Darwin Kernel Version 11.2.0: Tue Aug 9 20:54:00 PDT
2011; root:xnu-1699.24.8~1/RELEASE_X86_64 x86_64
$ man kqueue
...
BUGS
Not all filesystem types support kqueue-style notifications. And
even some that do, like some remote filesystems, may only support a
subset of the notification semantics described here.
$ man poll
...
BUGS
The poll() system call currently does not support devices.
There may of course always be improvements in future releases. We fall
back on select() on MacOSX since our configure test for a poll()
implementation that can handle devices fails. When poll() on MacOSX can
handle devices, it will automatically be used as fallback instead of
select().
--
Rickard Green, Erlang/OTP, Ericsson AB.
Joel Reymont
On Oct 24, 2011, at 1:31 PM, Rickard Green wrote:
> We fall back on select() on MacOSX since our configure test for a poll() implementation that can handle devices fails. When poll() on MacOSX can handle devices, it will automatically be used as fallback instead of select().
Fair enough.
Still, it peeves me to no end that regular file descriptors on Mac OSX (sockets and files) are crippled by possible corner cases.
I would enable clipping at runtime and give the user/developer the choice of whether to clip or not.
--------------------------------------------------------------------------
- for hire: mac osx device driver ninja, kernel extensions and usb drivers
---------------------+------------+---------------------------------------
http://wagerlabs.com | @wagerlabs | http://www.linkedin.com/in/joelreymont
---------------------+------------+---------------------------------------
_______________________________________________
Rickard Green
>
>>
>> Set ERL_MAX_PORTS (see erlang manpage) to increase the number of ports.
>>
Without actually testing it, I am pretty sure that it is enough to raise
the max ports limit (see above) on windows. Have you done that? It needs
to be set before starting the runtime system. Verify that it is set by
calling os:getenv("ERL_MAX_PORTS").
1200 also seems strange, since max ports defaults to 1024.
Regards,
Rickard
--
Rickard Green, Erlang/OTP, Ericsson AB.
Joel Reymont
On Oct 24, 2011, at 1:31 PM, Rickard Green wrote:
> $ man kqueue
> ...
> BUGS
> Not all filesystem types support kqueue-style notifications. And even some that do, like some remote filesystems, may only support a subset of the notification semantics described here.
As far as I've been told on the darwin-dev list, that refers to vnode-watching which select() doesn't support anyway.
> $ man poll
> ...
> BUGS
> The poll() system call currently does not support devices.
This is, likely, a legacy consideration as kqueue was not complete in earlier versions of Mac OSX.
select and poll are subsets of kqueue functionality-wise.
--------------------------------------------------------------------------
- for hire: mac osx device driver ninja, kernel extensions and usb drivers
---------------------+------------+---------------------------------------
http://wagerlabs.com | @wagerlabs | http://www.linkedin.com/in/joelreymont
---------------------+------------+---------------------------------------
_______________________________________________
Joel Reymont
On Oct 24, 2011, at 1:31 PM, Rickard Green wrote:
> There may of course always be improvements in future releases. We fall back on select() on MacOSX since our configure test for a poll() implementation that can handle devices fails. When poll() on MacOSX can handle devices, it will automatically be used as fallback instead of select().
--------------------------------------------------------------------------
- for hire: mac osx device driver ninja, kernel extensions and usb drivers
---------------------+------------+---------------------------------------
http://wagerlabs.com | @wagerlabs | http://www.linkedin.com/in/joelreymont
---------------------+------------+---------------------------------------
_______________________________________________
Martin Dimitrov
it to 5000, I would be able to make 8188 allocations, still 4 less than
the power of 2 number. os:getenv("ERL_MAX_PORTS") reports 5000. So,
thanks for the help :)
Can you bring some light to the meaning of ERL_MAX_PORTS? It is
mentioned in the doc under open_port. Does open_port participate in the
opening of files? And also, what is the purpose of ulimit? What does it
configure?
Thanks.
Martin
On 10/24/2011 4:58 PM, Rickard Green wrote:
>>>
>>> Set ERL_MAX_PORTS (see erlang manpage) to increase the number of ports.
>>>
>
> Without actually testing it, I am pretty sure that it is enough to
> raise the max ports limit (see above) on windows. Have you done that?
> It needs to be set before starting the runtime system. Verify that it
> is set by calling os:getenv("ERL_MAX_PORTS").
>
> 1200 also seems strange, since max ports defaults to 1024.
>
> Regards,
> Rickard
_______________________________________________
Dave Cottlehuber
> FD_SETSIZE is 64 on Windows, probably because that is the limit for
> efficient implementation of select() using MsgWaitForMultipleObjectsEx().
> However, all efficient I/O on Windows uses I/O completion ports, which are a
> lot more efficient than select() (more like epoll on Linux). I imagine
> Erlang uses IOCP as well on Windows, but I don't know for sure.
> The erl environment has various "raise these limits" options. We start ours
> with something like:
> -env ERL_MAX_PORTS 200500
> +P 1001001
> This lets us spawn over a million processes and open two hundred thousand
> sockets and files. (The machine itself also has increased the ulimit for the
> number of file descriptors)
> Hope this helps!
> Sincerely,
> jw
Thanks Jon,
Re-running Peer's test module with raised higher ERL_MAX_PORTS yielded
over 250000 open files before I decided the point was proven.
For reference the most docs I found on this are in open_port/2 in erts
[1] and some comments in the archives [2].
A+
Dave
[1]: http://www.erlang.org/doc/apps/erts/erts.pdf
[2]: http://erlang.2086793.n4.nabble.com/error-emfile-on-windows-td3064840.html
Martin Dimitrov
>
> For reference the most docs I found on this are in open_port/2 in erts
> [1] and some comments in the archives [2].
>
> A+
> Dave
>
> [1]: http://www.erlang.org/doc/apps/erts/erts.pdf
> [2]: http://erlang.2086793.n4.nabble.com/error-emfile-on-windows-td3064840.html
>
interesting unanswered questions. Can anyone provide some info on them?
Thanks,
Martin
Rickard Green
> Apparently, the main complaint has been about the lack of kqueue support for TTYs, which was resolved in Lion.
>
> On Oct 24, 2011, at 1:31 PM, Rickard Green wrote:
>
>> There may of course always be improvements in future releases. We fall back on select() on MacOSX since our configure test for a poll() implementation that can handle devices fails. When poll() on MacOSX can handle devices, it will automatically be used as fallback instead of select().
Joel Reymont wrote:
> On Oct 24, 2011, at 1:31 PM, Rickard Green wrote:
>
>> > $ man kqueue
>> > ...
>> > BUGS
>> > Not all filesystem types support kqueue-style notifications. And even some that do, like some remote filesystems, may only support a subset of the notification semantics described here.
> As far as I've been told on the darwin-dev list, that refers to vnode-watching which select() doesn't support anyway.
>
>> > $ man poll
>> > ...
>> > BUGS
>> > The poll() system call currently does not support devices.
>
> This is, likely, a legacy consideration as kqueue was not complete in earlier versions of Mac OSX.
>
> select and poll are subsets of kqueue functionality-wise.
I tried to access the darwin-dev web archive, but I'm not able to access
it, so I don't know what have been said to you there.
If/when kqueue() can handle all types of file-descriptors that select()
can handle with *no* exceptions, no fallback is needed. We need to know
that this is the case for *certain* before a no-fallback solution can be
used. This is needed for backward compatibility.
If/when poll() can handle all types of file-descriptors that select()
can handle with *no* exceptions, poll() can be used as fallback instead
of select().
I did a quick test on our MacOSX Lion machine which suggests that
kqueue() cannot be used for devices. I may have made something wrong,
though, but I don't have more time to put on this. I'm quite sure we
need select() as fallback, but I could be wrong.
If you think kqueue() on MacOSX can be used without fallback, you have
to dig that information up and present it for us. Also provide us with
test cases that show that it for example can handle devices, etc...
Regards,
Rickard
--
Rickard Green, Erlang/OTP, Ericsson AB.
Joel Reymont
On Oct 25, 2011, at 1:27 PM, Rickard Green wrote:
> I tried to access the darwin-dev web archive, but I'm not able to access it, so I don't know what have been said to you there.
Mirrored here:
http://groups.google.com/group/darwin-dev/browse_thread/thread/a7e2e4bec12d4472
> I did a quick test on our MacOSX Lion machine which suggests that kqueue() cannot be used for devices. I may have made something wrong, though, but I don't have more time to put on this.
Can you please send me the code?
> If you think kqueue() on MacOSX can be used without fallback, you have to dig that information up and present it for us. Also provide us with test cases that show that it for example can handle devices, etc...
Digging into your quick test would be a good start.
--------------------------------------------------------------------------
- for hire: mac osx device driver ninja, kernel extensions and usb drivers
---------------------+------------+---------------------------------------
http://wagerlabs.com | @wagerlabs | http://www.linkedin.com/in/joelreymont
---------------------+------------+---------------------------------------
_______________________________________________