[erlang-questions] I/O performance when reading from stdin

[Hynek Schlawack]

Hi,

I'm starting to learn and love Erlang and wanted as a small exercise
implement a word occurrence counter for http://ptrace.fefe.de/wp/ .

To say it kindly, Erlang isn't built for string operations. :) My
(mostly approved by #erlang) version[1] is twice as slow as the slowest
already submitted[2].

Anyway, I encountered a rather strange effect I'd like to ask about:
When I read from stdin, the program takes twice the time and memory
compared to the case when I open the file myself. Any hints? Is it a
bug? Isn't it just the "intended use" of Erlang and nobody cares? The
effect occurs under Linux, haven't tested it elsewhere.

Thanks!
Hynek
--
[1] http://github.com/hynek/wp-erlang/tree/master/wp.erl
[2] http://ptrace.fefe.de/wp/timings.txt

_______________________________________________
erlang-questions mailing list
erlang-q...@erlang.org
http://www.erlang.org/mailman/listinfo/erlang-questions

[Edward Stow]

2009/1/11 Hynek Schlawack <h...@ox.cx>:

> Hi,
>
> I'm starting to learn and love Erlang and wanted as a small exercise
> implement a word occurrence counter for http://ptrace.fefe.de/wp/ .
>
> To say it kindly, Erlang isn't built for string operations. :) My
> (mostly approved by #erlang) version[1] is twice as slow as the slowest
> already submitted[2].
>

I am only an Erlang / FP noob but I would not be surprised that the
Erlang code is slower as it is not really equivalent to the other
implementations : for example the python implemenation
http://ptrace.fefe.de/wp/wp.py is streaming over stdin and only
accumulating the word counts in a dictionary.

Whereas the erlang implementation reads the entire file into memory,
with the overheads that are needed for storage and processing by ets.

--

Edward Stow

[Fredrik Svahn]

It is a known problem that reading from stdin is slow. I believe it
has been discussed several times on the mailing list.

Most erlang applications do not read from stdin, at least not huge
amounts of data. Erlang was originally built and optimized for
applications that are supposed to run for years with thousands of
concurrent processes, not for shortlived scripts running just a single
process.

However, there has been some small improvements in later releases. You
do not mention what release you are running, but my guess would be
R12B-4 or earlier? I get pretty much the same results as you get when
running R12B-3. With R12B-5, however, the stdio variant is just as
fast as file io with get_line (but still many times slower than a
corresponding c/c++ program). There is no improvement in memory
consumption in R12B-5, though.

By the way, if the file is small enough to fit in memory then
file:read_file/1 is a lot faster!

BR /Fredrik
PS, How would you like to improve the string handling?

[Hynek Schlawack]

Edward Stow schrieb:

> I am only an Erlang / FP noob but I would not be surprised that the
> Erlang code is slower as it is not really equivalent to the other
> implementations : for example the python implemenation
> http://ptrace.fefe.de/wp/wp.py is streaming over stdin and only
> accumulating the word counts in a dictionary.
>
> Whereas the erlang implementation reads the entire file into memory,
> with the overheads that are needed for storage and processing by ets.

Pardon my ignorance, but I though, I've been also streaming? Or is it
how reading from stdin in Erlang works: It's first completely read into
memory? That would explain bad performance as well as memory usage
compared to reading directly from a file.

Cheers,
Hynek