Hi all,

I'm thinking about to start a new project which has some rather
critical requirements. It will be compiled with the native compiler
initially targeting Linux/Unix but eventually also windoze.

The app consists of a main engine which spawns many short lived
child threads or processes. The children go away, do their work and
then pass their results back to the main engine. Many of the children
will spawn another process and read the child process output via a
pipe and many of the children will block on I/O. A small portion of
the children may be I/O bound, but there is not way of telling which
beforehand.

Since I would like to maximize the throughput on multi-core and
multi-processor machines I am thinking of using a mix of forking and
threading. For communications, I was thinking of using the Event
module for communication between threads, but I don't think that
works for forked process (pipes maybe?).

Anybody have any advice for this project? Any war stories from similar
projects? Any readings they can recommend?

Thanks,
Erik
--
+-----------------------------------------------------------+
  Erik de Castro Lopo
+-----------------------------------------------------------+
"Data is not information, Information is not knowledge, Knowledge is
not understanding, Understanding is not wisdom."
-- Clifford Stoll

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

Am Dienstag, den 10.10.2006, 22:16 +1000 schrieb Erik de Castro Lopo:

You know that there is no fork on Windows?

If there wasn't the Windows requirement I could recommend this:

You may have a look at ocamlnet2. It includes the netplex library that
manages parent/children relationships between either forked processes or
threads. You can use SunRPC for communication (or whatever you like, but
SunRPC support is included). The bad news is that it does not run on the
native Windows port, not even in the threaded variant (because there's
no socketpair...).

Ah yes, I have a war story, but cannot tell it now. I can only say it is
used for a really big commercial project.

Download it here:

http://www.ocaml-programming.de/packages/ocamlnet-2.2test13.tar.gz

An online manual is here:

http://ocamlnet.sourceforge.net/manual-2.2/

Gerd
--
------------------------------------------------------------
Gerd Stolpmann * Viktoriastr. 45 * 64293 Darmstadt * Germany
g...@gerd-stolpmann.de          http://www.gerd-stolpmann.de
Phone: +49-6151-153855                  Fax: +49-6151-997714
------------------------------------------------------------

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

I'm pushing for Ocaml on this project. If that doesn't fit
the bill there are others in the team pushing for Erlang.

I'm trying to optimize performance of lots of small sub-processes
which are likely to block on I/O. I'm therefore trying to start
as many as possible simultaneously rather than run them one by
one serially.

I have been led to believe that Erlang also fits the bill.

I call C++ a curse on programmers everywhere; the language
that has enabled and encouraged more stupidity and bad software
design than any other programming language ever.

Erik
--
+-----------------------------------------------------------+
  Erik de Castro Lopo
+-----------------------------------------------------------+
"It is forbidden for a Muslim to be a loyal friend to someone who does
not believe in God and His Prophet, or someone who fights the religion
of Islam." -- Fifth grade text book from Saudi Arabia
http://www.washingtonpost.com/wp-dyn/content/article/2006/05/19/AR200...

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

Yeah, but I was thinking of using the Cygwin version which I
think does have fork (and select which is also missing).

I hate windows. There are so many good, reliable, well designed
OSes out there and people on those good OSes still have to work
around the fact the the vast majority of people are on a broken
piece of crap like windows.

Erik
--
+-----------------------------------------------------------+
  Erik de Castro Lopo
+-----------------------------------------------------------+
The main confusion about C++ is that its practitioners think
it is simultaneously a  high and low level language when in
reality it is good at neither.

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs