Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

[Caml-list] [ANN] coThreads 0.10

9 views
Skip to first unread message

Zheng Li

unread,
Sep 17, 2007, 12:33:33 PM9/17/07
to caml...@inria.fr

Hi,

I'd like to announce the first public release of coThreads, a
concurrent programming library for OCaml.

It originated as STMlib [1], and was accepted as a Google Summer
of Code 2007 project [2] mentored by Yoriyuki Yamagata from the
Free Software Initiative of Japan [3]. The official website of
coThreads is http://cothreads.sourceforge.net where you can find
the source code and some preliminary documents. It's a wiki-based
website, so feel free to register and contribute. A byproduct
of the project, the Vprint [4] module, was already announced
independently a few weeks ago on this list. Enjoy. [1]
http://www.pps.jussieu.fr/~li/software/index.html#stmlib [2]
http://code.google.com/soc/2007/ [3] http://fsij.org [4]
http://www.pps.jussieu.fr/~li/software/index.html#vprint From
the README:
==============================================================================
= Description = coThreads is a concurrent programming library for
OCaml. It enhances the Threads library of the standard OCaml
distribution in two dimensions: * coThreads implements the same
API of the standard Threads library on different execution engines
(process, netwoker(todo)), so that a single copy of source code
can be compiled and deployed to different environments without
modification * coThreads is also a super set of the standard
Threads library, with extra components (STM etc.), functions
(spawn etc.) and features (object-level compatibility etc.) =
Features = The design of coThreads brings several advantages: ==
Powerfulness == * The process engine can give you real speedup on
multi-core and multi-processor machines, the networker engine
(todo) will give you both speedup and scalability. * Combining
the original components from the Threads library and the newly
added ones, coThreads is a full-fledged toolbox covering two main
concurrent programming paradigms, namely shared-memory and message
passing, with both high-level and low-level constructors. * All
constructors (e.g. thread, mutex, condition, event, channel, tvar,
stm etc.) are first class values that can be communicated and
shared between independent threads. == Compatibility == * Full
compatibility with the original Threads library (systhreads and
vmthreads), so that you can now deploy your legacy code to new
environment for free — without modifying source code, without
learning something new. * Both source-level and object-level
compatibility among different execution engines, so that you can
keep a single copy of source code as well as a single copy of
objects files. == Convenience == * The full library is
implemented in user-space, no modification to the OCaml compiler
and runtime, so that you can use it as a set of plain modules. *
Switching engines is as easy as changing the include paths of
compilation, and it's quite easy to automate the building process
over a set of engines (e.g. with lines of pattern rules in your
Makefile) --
Zheng Li
http://www.pps.jussieu.fr/~li

_______________________________________________
Caml-list mailing list. Subscription management:
http://yquem.inria.fr/cgi-bin/mailman/listinfo/caml-list
Archives: http://caml.inria.fr
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

Zheng Li

unread,
Sep 17, 2007, 12:52:08 PM9/17/07
to caml...@inria.fr

Hi,

Really sorry for the format in the previous posts. Something must be wrong with
my Emacs setting. Anyway, here is the reformatted one, hope it works.

skaller

unread,
Sep 17, 2007, 1:44:17 PM9/17/07
to Zheng Li, caml...@inria.fr
On Mon, 2007-09-17 at 18:48 +0200, Zheng Li wrote:

> * The process engine can give you real speedup on multi-core and
> multi-processor machines, the networker engine (todo) will give you both
> speedup and scalability.

I'm curious how this can be possible**.. is this only with the message
passing model?

** Since Ocaml can't multi-process and neither the compiler
nor library are modified ..

--
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

Zheng Li

unread,
Sep 17, 2007, 2:02:38 PM9/17/07
to caml...@inria.fr
skaller <ska...@users.sourceforge.net> writes:
> On Mon, 2007-09-17 at 18:48 +0200, Zheng Li wrote:
>
>> * The process engine can give you real speedup on multi-core and
>> multi-processor machines, the networker engine (todo) will give you both
>> speedup and scalability.
> I'm curious how this can be possible**.. is this only with the message
> passing model?
> ** Since Ocaml can't multi-process and neither the compiler
> nor library are modified ..

Well, in order to have semantic consistency among different engines without
modifying standard OCaml itself, shard-memory style concurrency should only be
achieved through the STM module. It's documented in the pitfall pages:

http://cothreads.sourceforge.net/doc/pitfall

_______________________________________________

skaller

unread,
Sep 17, 2007, 5:37:06 PM9/17/07
to Zheng Li, caml...@inria.fr
On Mon, 2007-09-17 at 19:51 +0200, Zheng Li wrote:
> skaller <ska...@users.sourceforge.net> writes:
> > On Mon, 2007-09-17 at 18:48 +0200, Zheng Li wrote:
> >
> >> * The process engine can give you real speedup on multi-core and
> >> multi-processor machines, the networker engine (todo) will give you both
> >> speedup and scalability.
> > I'm curious how this can be possible**.. is this only with the message
> > passing model?
> > ** Since Ocaml can't multi-process and neither the compiler
> > nor library are modified ..
>
> Well, in order to have semantic consistency among different engines without
> modifying standard OCaml itself, shard-memory style concurrency should only be
> achieved through the STM module. It's documented in the pitfall pages:
>
> http://cothreads.sourceforge.net/doc/pitfall

Yes but what I'm getting at is the claim of multi-processing,
which Ocaml 3.10 at least cannot do. If you run Ocaml on a multi-core
machine, even if you have 8 cores and 8 threads, one on each core,
only one will ever run at once, so there is no performance gain.
In fact, it will be slower than a single core.

If you are *enforcing* message passing and using separate processes
instead of threads, then 8 processes will run in parallel on 8 cores,
and you'll get roughly 8 times speedup.

So I'm not asking about how to ensure the code is consistent
over the various models, but rather how you can get ANY genuine
concurrency WITHOUT using message passing and processes
(in which case the networking model should be easy to implement)

Whether you explicitly send messages or use transactional memory
or whatever to wrap the message passing isn't the question:
I can see how that can work. However note, the message passing
has to be used for immutable values too, if the threads are
represented by processes in separate address spaces.


--
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

_______________________________________________

Zheng Li

unread,
Sep 17, 2007, 6:39:12 PM9/17/07
to caml...@inria.fr

Hi,

skaller <ska...@users.sourceforge.net> writes:
> Yes but what I'm getting at is the claim of multi-processing,
> which Ocaml 3.10 at least cannot do. If you run Ocaml on a multi-core
> machine, even if you have 8 cores and 8 threads, one on each core,
> only one will ever run at once, so there is no performance gain.
> In fact, it will be slower than a single core.
>
> If you are *enforcing* message passing and using separate processes
> instead of threads, then 8 processes will run in parallel on 8 cores,
> and you'll get roughly 8 times speedup.

Yes, you are right. The process engine is implemented with process. That's how
it can speedup and that's why it's called ``process engine''.

> So I'm not asking about how to ensure the code is consistent
> over the various models, but rather how you can get ANY genuine
> concurrency WITHOUT using message passing and processes
> (in which case the networking model should be easy to implement)

I'm sorry if the document misleadingly made you think it get speedup without
using process and message passing. Threads and related tools is just the
abstract interface on which all engines agree, they implement it with various
facilities from VM threads, system threads to process or independent programs.

> Whether you explicitly send messages or use transactional memory
> or whatever to wrap the message passing isn't the question:
> I can see how that can work. However note, the message passing
> has to be used for immutable values too, if the threads are
> represented by processes in separate address spaces.

For immutable value, it's not different from the traditional threads, at least
from the library users' point of view: if a variable is already inside the
scope of a threads (i.e. it's global and created before launching the thread),
there is no necessity to message passing it; if not, you should anyway do it
explicitly even with the traditional threads.

Regards

_______________________________________________

skaller

unread,
Sep 17, 2007, 7:30:22 PM9/17/07
to Zheng Li, caml...@inria.fr
On Tue, 2007-09-18 at 00:37 +0200, Zheng Li wrote:

> For immutable value, it's not different from the traditional threads, at least
> from the library users' point of view: if a variable is already inside the
> scope of a threads (i.e. it's global and created before launching the thread),
> there is no necessity to message passing it; if not, you should anyway do it
> explicitly even with the traditional threads.

Yes, however such a value can't be shared by parallel control paths
(in ocaml) because the collector won't allow it. Of course
you can *copy* it eg with fork() and run separate collectors.
Then the functional value is the same, but the object identity
(machine address) may differ.

BTW: an interesting application would be to adapt GLR parser
to actually use multi-core CPU to speed up parsing by spawning
alternate parses as parallel control streams.

--
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

_______________________________________________

Zheng Li

unread,
Sep 17, 2007, 8:39:02 PM9/17/07
to caml...@inria.fr
skaller <ska...@users.sourceforge.net> writes:
> On Tue, 2007-09-18 at 00:37 +0200, Zheng Li wrote:
>> For immutable value, it's not different from the traditional threads, at least
>> from the library users' point of view: if a variable is already inside the
>> scope of a threads (i.e. it's global and created before launching the thread),
>> there is no necessity to message passing it; if not, you should anyway do it
>> explicitly even with the traditional threads.
>
> Yes, however such a value can't be shared by parallel control paths
> (in ocaml) because the collector won't allow it. Of course
> you can *copy* it eg with fork() and run separate collectors.
Given that the copy-on-write strategy is widely adopted in the *nix family, I
hope it's not a big problem.

> Then the functional value is the same, but the object identity
> (machine address) may differ.
Yes. For immutable value, physical equivalence is not preserved across
*multiple* rounds of message passing. This is another slight semantic
difference I should document on the pitfall page. As mutable value, if shared
through STM, their inter-references are actually preserved.

_______________________________________________

Erik de Castro Lopo

unread,
Sep 17, 2007, 8:55:33 PM9/17/07
to caml...@inria.fr
Zheng Li wrote:

> The official website of coThreads is http://cothreads.sourceforge.net where you

Is anybody else having difficulty downloading this? I have tried
a large number of different sourceforge mirrors and none of them
seem to have this file whic means I re-directed back and forth
across the net til one of these things redirects me to this:

http://.dl.sourceforge.net/sourceforge/cothreads/cothreads_0.10.tar.bz2

which isn't even a legal host name.

I've also tried wget with no success.

Erik
--
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------
"I want to make sure (a user) can't get through ... an online
experience without hitting a Microsoft ad."
- Microsoft CEO, Steve Ballmer on the Microsoft search engine.

Erik de Castro Lopo

unread,
Sep 17, 2007, 9:28:18 PM9/17/07
to caml...@inria.fr
Erik de Castro Lopo wrote:

> Is anybody else having difficulty downloading this? I have tried

This works:

svn co https://cothreads.svn.sourceforge.net/svnroot/cothreads/trunk cothreads

Erik
--
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------

"I run Linux on pretty much everything except the microwave and
washing machine. Those are tempting targets but would probably
make Telsa extremely cross." -- Alan Cox

Erik de Castro Lopo

unread,
Sep 17, 2007, 10:13:43 PM9/17/07
to caml...@inria.fr
Zheng Li wrote:

>
> Hi,
>
> I'd like to announce the first public release of coThreads, a
> concurrent programming library for OCaml.

I've compiled this with Ocaml 3.09.1 on x86 Linux and x86-64 Linux.

Looking at the example program ray_nocol.ml, it works correctly
on x86 Linux but fails on x86-64 [0]:

# > ./ray_nocol.proc.opt 9 1024 1 ray_nocol.proc.opt.pgm
Fatal error: exception Unix.Unix_error(12, "lseek", "")

I've also tried the non-opt version and the nath version. All fail
in the same way.

The errno value of 12 means "Out of memory". I did a little debugging
but couldn't find anything obvious.

Erik

[0] This is the standard 3.09.1 compiler shipped with Ubuntu Feisty.


--
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------

The Earth is around 70% water. Fish rule the seas.
Humans are over 90% water. It's only a matter of time.

skaller

unread,
Sep 18, 2007, 12:33:37 AM9/18/07
to caml...@inria.fr
On Tue, 2007-09-18 at 10:53 +1000, Erik de Castro Lopo wrote:
> Zheng Li wrote:
>
> > The official website of coThreads is http://cothreads.sourceforge.net where you
>
> Is anybody else having difficulty downloading this?

Confirmed .. but then we live next door :)

--
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

_______________________________________________

skaller

unread,
Sep 18, 2007, 2:05:01 AM9/18/07
to caml...@inria.fr
On Tue, 2007-09-18 at 12:10 +1000, Erik de Castro Lopo wrote:
> Zheng Li wrote:
>
> >
> > Hi,
> >
> > I'd like to announce the first public release of coThreads, a
> > concurrent programming library for OCaml.
>
> I've compiled this with Ocaml 3.09.1 on x86 Linux and x86-64 Linux.
>
> Looking at the example program ray_nocol.ml, it works correctly
> on x86 Linux but fails on x86-64 [0]:
>
> # > ./ray_nocol.proc.opt 9 1024 1 ray_nocol.proc.opt.pgm
> Fatal error: exception Unix.Unix_error(12, "lseek", "")
>
> I've also tried the non-opt version and the nath version. All fail
> in the same way.
>
> The errno value of 12 means "Out of memory". I did a little debugging
> but couldn't find anything obvious.

But it isn't an errno value that is reported, it is the index
of the variant that is 12, and that is "Invalid Argument".

I think I found it:

let fresh_number =
let usable_size = Sys.word_size -2 in
let bits_of_id = 16 in (* Should be sufficient in most OS *)
let bits_of_num = usable_size - bits_of_id in
let counter = ref 0 in
fun () ->
let self_id = id (self ()) in
let id_part = bit_chop_to_n bits_of_id self_id in
let num_part =
counter := bit_chop_to_n bits_of_num (!counter + 1);
!counter in
(id_part lsl bits_of_num) + num_part

produces:

1695112678495748100

which seems a little big for lseek on a zero size file ..

My man page says:

lseek - reposition read/write file offset

RETURN VALUE
Upon successful completion, lseek() returns the resulting offset
location as
measured in bytes from the beginning of the file.
Otherwise, a value of
(off_t)-1 is returned and errno is set to indicate the error.

It is not clear that a seek to an invalid position in the file
is going to succeed. It's also not clear to me that the seek
argument isn't 32 bits (depends on complex ugly GNU macro
hackery what type off_t is .. my Caml got built with
the LARGE_FILE macro thing so it should be 64 bits).


--
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

_______________________________________________

Erik de Castro Lopo

unread,
Sep 18, 2007, 2:26:16 AM9/18/07
to caml...@inria.fr
skaller wrote:

> But it isn't an errno value that is reported, it is the index
> of the variant that is 12, and that is "Invalid Argument".

Ok.

> It is not clear that a seek to an invalid position in the file
> is going to succeed. It's also not clear to me that the seek
> argument isn't 32 bits (depends on complex ugly GNU macro
> hackery what type off_t is .. my Caml got built with
> the LARGE_FILE macro thing so it should be 64 bits).

Shouldn't off_t always be 64 bits on a 64 bit CPU? I only see
this problem on x86-64 and Sys.word_size is 64.

Erik


--
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------

"I run Linux on pretty much everything except the microwave and
washing machine. Those are tempting targets but would probably
make Telsa extremely cross." -- Alan Cox

_______________________________________________

Zheng Li

unread,
Sep 18, 2007, 5:05:15 AM9/18/07
to caml...@inria.fr

Hi,


Erik de Castro Lopo <mle+...@mega-nerd.com> writes:
> skaller wrote:
>
>> But it isn't an errno value that is reported, it is the index
>> of the variant that is 12, and that is "Invalid Argument".
>
> Ok.
>
>> It is not clear that a seek to an invalid position in the file
>> is going to succeed. It's also not clear to me that the seek
>> argument isn't 32 bits (depends on complex ugly GNU macro
>> hackery what type off_t is .. my Caml got built with
>> the LARGE_FILE macro thing so it should be 64 bits).
>
> Shouldn't off_t always be 64 bits on a 64 bit CPU? I only see
> this problem on x86-64 and Sys.word_size is 64.

Thank you for the bug report.

Unfortunately I don't have a x64 machine to test, and know quite little about
64bit machine (that's why I made the mistake).

But the bug reason seems obvious: on a x64 machine, one have to use
Unix.LargeFile.lseek to seek the address in the extra space.

On the other hand, I think 2^14 locks (the x86 case) are fairly enough in
most situation. So we don't have to bother with Int64 and LargeFile. Could you
(or anyone else with a x64 machine) help to test the simple solution: change
"Sys.word_size - 2" in the definition of "fresh_number" to
"(min Sys.word_size 32) - 2" as follows

let fresh_number =
let usable_size = (min Sys.word_size 32) -2 in


let bits_of_id = 16 in (* Should be sufficient in most OS *)

........

and report whether it works? Thanks

_______________________________________________

skaller

unread,
Sep 18, 2007, 5:14:49 AM9/18/07
to caml...@inria.fr, Zheng Li
On Tue, 2007-09-18 at 16:23 +1000, Erik de Castro Lopo wrote:
> skaller wrote:

> > It is not clear that a seek to an invalid position in the file
> > is going to succeed. It's also not clear to me that the seek
> > argument isn't 32 bits (depends on complex ugly GNU macro
> > hackery what type off_t is .. my Caml got built with
> > the LARGE_FILE macro thing so it should be 64 bits).
>
> Shouldn't off_t always be 64 bits on a 64 bit CPU?

I don't know. I traced through the code and did some debugging,
and the problem is that system wide mutex is emulated by using
locking on an lseek on a (deleted) lock file. The seek address
is a magic number which is rather large, and this appears to
cause the EINVAL errno on the lseek call.

let lock_fd =
print_endline "CREATING MUTEX";
let lock_name = fresh_name "_mutex" in
remove_exists lock_name;
let fd = openfile lock_name [O_WRONLY; O_CREAT] file_perm in
remove_exists lock_name;
fd

type t = int (* The offset *)

let create = fresh_number

let rec lock lk =
print_endline ("LOCK " ^ string_of_int lk);
if lk <> lseek lock_fd lk SEEK_SET then assert false;
lockf lock_fd F_LOCK 1

So basically a mutex is modelled by an offset,
which is created by fresh_number routine, and using
Unix discretionary locks with lockf function, which
is based on the current position in the file.

Well, I don't see how the lseek can fail unless there
is a filesize constraint somewhere detecting that

1695112678495748100

is an invalid offset for lseek.

A possible patch is to take this in process.coordinator

(* fresh_number fresh_name ensure that there won't exist number/name
confliction between running processes.
*)


let fresh_number =
let usable_size = Sys.word_size -2 in
let bits_of_id = 16 in (* Should be sufficient in most OS *)
let bits_of_num = usable_size - bits_of_id in
let counter = ref 0 in
fun () ->
let self_id = id (self ()) in
let id_part = bit_chop_to_n bits_of_id self_id in
let num_part =
counter := bit_chop_to_n bits_of_num (!counter + 1);
!counter in
(id_part lsl bits_of_num) + num_part


and use the line 2:

let usable_size = 30 in

instead (though I didn't try it, some other places may need
to replace Sys.word_size the same way for it to work
properly).

--
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

_______________________________________________

Markus E L

unread,
Sep 18, 2007, 9:02:15 AM9/18/07
to caml...@inria.fr

Erik de Castro Lopo wrote:

> skaller wrote:
>
>> But it isn't an errno value that is reported, it is the index
>> of the variant that is 12, and that is "Invalid Argument".
>
> Ok.
>
>> It is not clear that a seek to an invalid position in the file
>> is going to succeed. It's also not clear to me that the seek
>> argument isn't 32 bits (depends on complex ugly GNU macro
>> hackery what type off_t is .. my Caml got built with
>> the LARGE_FILE macro thing so it should be 64 bits).
>
> Shouldn't off_t always be 64 bits on a 64 bit CPU? I only see
> this problem on x86-64 and Sys.word_size is 64.

This might be, or might be not. "The standard has nothing to say about
this, but usually it's at least long int whatever that is at the
corresponing platform. In contrast, I find in unix.ml

external lseek : file_descr -> int -> seek_command -> int = "unix_lseek"

but also in otherlibs/unix/lseek.c

ret = lseek(Int_val(fd), Long_val(ofs),
seek_command_table[Int_val(cmd)]);

(see the Long_Val).

Am I only imagining potential problems with variable width here?

Regards -- Markus

Zheng Li

unread,
Sep 18, 2007, 9:44:16 AM9/18/07
to caml...@inria.fr

Hi,

Thanks skaller for testing the fix on his machine. Here is a few updates:

- I chose the simple fix: just restrict the upper bound of fresh_number
based on the word_size of 32bit machine. It's only used to generate fresh
number, name and offset, so I suspect the range is fairly enough. On the
other hand, it won't be difficult to make use int64 and LargeFile for 64bit
machine if necessary, but I'd better to get some x64 machine around for
testing first.

- A README for /example is added (also attached to the end of this post)

- Some minor updates to a few examples

- I have no idea what's wrong with the sourceforge file download system. It
did work when I uploaded the file a few days ago. Anyway, I re-uploaded the
package (with the above modifications), hope it works.

Best regards.


README for examples
===================

== BUILD ==

Just ''make all'' after you've got coThreads installed on your system.
Type ''make clean'' to remove all imtermediate and final building results.

== LIST ==

* coth (use: Thread (or Cothread), Mutex)
Simple test of mutex. A set of threads tries to grab a single mutex and
release it for random times

* evt (use: Thread (or Cothread), Event)
Simple test of event. Most examples are directly from the OCaml OReilly
book. The execution won't exist, this is intentional.

* lock (use: Thread (or Cothread), Mutex)
Simple test of mutex. A set of threads try to grab two mutex. Each thread
first must grab the first mutex before the second mutex, then release the
second mutex and the first one.

* mcast (use: Thread (or Cothread), Stm)
STM example from [1], contributed by Yoriyuki Yamagata

* merge (use: Thread (or Cothread), Stm)
STM example from [1], contributed by Yoriyuki Yamagata

* mvar (use: Thread (or Cothread), Stm)
STM example from [1], contributed by Yoriyuki Yamagata

* phil (use: Thread (or Cothread), Stm)
Classical philosophers dinning problem written in STM. Launch it with
[./phil n], where n is the number of philosophers and chopsticks.

* ray_col, ray_nocol (use: Cothread, Event)
Replanted versions of Jon Harrop's ray tracer [3]. ray.ml is the module
containing common computation functions, ray_xxx.ml are parallel engines.
In ray_nocol.ml, the workers don't send the results back to master, instead
they write them directly to the output file; in ray_col.ml, the workers send
results back to the master, and the master write them to the output file.
Launch it with [./ray_xxx level size degree outputfile], where [level] and
[size] are about the quality of output image, and [degree] is the parallel
degree which should equal or greater than the cores or cpus of your machine
if you'd like to get the most speedup. Or you may just lanch it with
[./ray_xxx] which takes the default setting [./ray_xxx 9 512 2 ray_xxx.pgm]

* santa (use: Thread (or Cothread), Stm)
The Santa Clause problem documented in [2]. The haskell version is attached
as comment at the end of the file.

* sing (use: Thread (or Cothread), Stm)
Simple test of Stm. Two threads constantly update a single tvar.

* test (use: Thread (or Cothread), Stm)
Simple test of Stm to calculate the sum of [0..n-1] with n threads. The i_th
thread is responsible for adding i to the sum. Its action is not allowed to
take place until the current sum exceed sum (i/10).

* The Makefile itself is an example. It shows that how you can build your
applications against a set of execution engines with just a few lines of
pattern rules.

[1] http://research.microsoft.com/users/simonpj/papers/stm/index.htm#composble
[2] https://research.microsoft.com/users/simonpj/papers/stm/index.htm#beautiful
[3] http://www.ffconsultancy.com/languages/ray_tracer/index.html


Zheng Li <l...@pps.jussieu.fr> writes:
> Thank you for the bug report.
>
> Unfortunately I don't have a x64 machine to test, and know quite little about
> 64bit machine (that's why I made the mistake).
>
> But the bug reason seems obvious: on a x64 machine, one have to use
> Unix.LargeFile.lseek to seek the address in the extra space.
>
> On the other hand, I think 2^14 locks (the x86 case) are fairly enough in
> most situation. So we don't have to bother with Int64 and LargeFile. Could you
> (or anyone else with a x64 machine) help to test the simple solution: change
> "Sys.word_size - 2" in the definition of "fresh_number" to
> "(min Sys.word_size 32) - 2" as follows
>
> let fresh_number =
> let usable_size = (min Sys.word_size 32) -2 in
> let bits_of_id = 16 in (* Should be sufficient in most OS *)
> ........
>
> and report whether it works? Thanks

_______________________________________________

Erik de Castro Lopo

unread,
Sep 18, 2007, 7:55:59 PM9/18/07
to caml...@inria.fr
Zheng Li wrote:

> Thanks skaller for testing the fix on his machine.

Fix confirmed for me as well. Thanks.

> - A README for /example is added (also attached to the end of this post)

Ahh, good, I was going to ask for that :-).

Cheers,


Erik
--
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------

"Every time microshaft's stock price drops again, I rejoice. I
want to see that bunch of criminals brought to their knees.
Preferably at the chopping block."
-- rixt in http://linuxtoday.com/stories/20659_flat.html

Erik de Castro Lopo

unread,
Sep 19, 2007, 6:15:08 AM9/19/07
to caml...@inria.fr
Zheng Li wrote:

> I'd like to announce the first public release of coThreads, a concurrent
> programming library for OCaml.

What wasn't mentioned in this announcement was the existance of a
mailing list. I posted to the mailing list:

http://sourceforge.net/mailarchive/forum.php?forum_name=cothreads-discuss

but I suspect that I may be the only person on the list :-).

Cheers,
Erik
--
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------

Microsoft VISTA : Virus Infection Spyware Trojans and Adware!

Jan Kybic

unread,
Sep 19, 2007, 7:03:48 AM9/19/07
to caml...@inria.fr
>> I'd like to announce the first public release of coThreads, a concurrent
>> programming library for OCaml.

I have had only a quick look at the web page but I did not understand
whether and how can coThreads be used on a cluster of independent
computers (Beowulf-style) that are only connected by network (no
shared memory) for which I normally use MPI (message passing).
Is coThreads usable in such a scenario? What is the backend - does it
run on the top of PVM, MPI, or similar?

Thanks for clarification.

Jan

--
-------------------------------------------------------------------------
Jan Kybic <ky...@fel.cvut.cz> tel. +420 2 2435 5721
http://cmp.felk.cvut.cz/~kybic ICQ 200569450

Erik de Castro Lopo

unread,
Sep 19, 2007, 7:16:20 AM9/19/07
to caml...@inria.fr
Jan Kybic wrote:

> I have had only a quick look at the web page but I did not understand
> whether and how can coThreads be used on a cluster of independent
> computers (Beowulf-style) that are only connected by network (no
> shared memory) for which I normally use MPI (message passing).

>From the TODO file, it seems that the networker engine is not yet
complete which means the answer to your question is no (at least
for now).

I can however report that on a single machine with multiple cores,
message passing between CoThreads using the Event module is an absolute
pleasure to use.

Erik
--
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------

"I have found that Big Design Up Front environments attract people who
want to talk about software development, while iterative environments
attract people who want to develop software. Is it any surpise that in
companies where BDUF is predominant, nobody wants to stoop to coding?
Everyone wants to be an 'architect' or a 'Business Analyst' or a
'Product Manager'." -- Reg Braithwaite

Zheng Li

unread,
Sep 19, 2007, 9:23:19 AM9/19/07
to caml...@inria.fr

Hi,

Erik de Castro Lopo <mle+...@mega-nerd.com> writes:

> Jan Kybic wrote:
>> I have had only a quick look at the web page but I did not understand
>> whether and how can coThreads be used on a cluster of independent
>> computers (Beowulf-style) that are only connected by network (no
>> shared memory) for which I normally use MPI (message passing).
>
>>From the TODO file, it seems that the networker engine is not yet
> complete which means the answer to your question is no (at least
> for now).

Yes, he is right. What you're thinking about is the networker engine. It is not
available for now but planned for the future. You'll still be able to have
shared memory through STM, though the communication cost will be larger.

The implementation is no more difficult than the process engine, on the
contrary it can reuse most of the techniques developed with process
engine. But I'm kind of busy in this period, and won't be able to start the
implementation right now.

_______________________________________________

Vu Ngoc San

unread,
Sep 19, 2007, 3:18:36 PM9/19/07
to Zheng Li, caml...@inria.fr
Hi

this looks like something I'd be very happy to use. Have you tried
compiling lablgtk2 with it ?

(so that I could get rid of the following:)
[
The files /usr/lib/ocaml/3.10.0/process/thread.cmi
and /usr/lib/ocaml/3.10.0/lablgtk2/gtkThread.cmi
make inconsistent assumptions over interface Thread
]

San

Zheng Li

unread,
Sep 19, 2007, 4:14:07 PM9/19/07
to caml...@inria.fr

Hi,

Vu Ngoc San <san.v...@ujf-grenoble.fr> writes:
> this looks like something I'd be very happy to use. Have you tried compiling
> lablgtk2 with it ?
>
> (so that I could get rid of the following:)
> [
> The files /usr/lib/ocaml/3.10.0/process/thread.cmi
> and /usr/lib/ocaml/3.10.0/lablgtk2/gtkThread.cmi
> make inconsistent assumptions over interface Thread
> ]

I'm not sure what you're trying to do, but the reason of error seems obvious:
lablgtk2 seems to have been compiled with the Thread module of system threads
or VM thread (both come with OCaml's standard distribution), while you're
linking them with other modules compiled against the Thread module of process
engine.

The Thread module interfaces from the three engines differ with each
other! Note that this is not the problem from coThreads, it's a problem of
standard Threads library:

- The Thread interfaces of system threads and VM threads coming with the
standard distribution *already* differ with each other, so that the objects
compiled with system Thread won't be able to mix with objects compiled with
VM thread, vice versa. (The problem is the same as in your case)

- We have no way to unify the three of them without modifying the files from
original distribution, which is exactly what we manage to avoid.

On the other hand, the Cothread module (a super set of the Thread interface) is
provided to achieve object-level compatibility, so that the objects compiled
with different engines can mix. All other modules from Threads and coThreads,
e.g. Mutex, Event, don't have this problem.

You'd like to read the compatibility page for more explanation:
http://cothreads.sourceforge.net/doc/compatibility

Regards

_______________________________________________

skaller

unread,
Sep 19, 2007, 8:54:36 PM9/19/07
to Zheng Li, caml...@inria.fr
On Wed, 2007-09-19 at 22:10 +0200, Zheng Li wrote:

> The Thread module interfaces from the three engines differ with each
> other! Note that this is not the problem from coThreads, it's a problem of
> standard Threads library:

Why don't you just use a different module name?


--
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

_______________________________________________

Jon Harrop

unread,
Sep 20, 2007, 12:30:38 AM9/20/07
to caml...@inria.fr
On Wednesday 19 September 2007 12:13:46 Erik de Castro Lopo wrote:
> I can however report that on a single machine with multiple cores,
> message passing between CoThreads using the Event module is an absolute
> pleasure to use.

Carry on... :-)

Any examples?

--
Dr Jon D Harrop, Flying Frog Consultancy Ltd.
OCaml for Scientists
http://www.ffconsultancy.com/products/ocaml_for_scientists/?e

Erik de Castro Lopo

unread,
Sep 20, 2007, 12:32:50 AM9/20/07
to caml...@inria.fr
skaller wrote:

> Why don't you just use a different module name?

I agree. Although it is nice the have the Cothreads API be so similar
to the native Threads module, I can't possibly imagine ever wanting
to go from Cothreads back to Threads.

Erik
--
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------

"That being done, all you have to do next is call free() slightly
less often than malloc(). You may want to examine the Solaris
system libraries for a particularly ambitious implementation of
this technique."
-- Eric O'Dell (comp.lang.dylan)

Erik de Castro Lopo

unread,
Sep 20, 2007, 2:14:38 AM9/20/07
to caml...@inria.fr
Jon Harrop wrote:

> Carry on... :-)
>
> Any examples?

All my examples so far are trivial, like the following which creates a
Cothread and then polls for an event sent by the child.

-------------------------------------------------------------------
(*
** Needs Cothreads installed :
** http://cothreads.sourceforge.net/
**
** Run with :
** ocaml -I +process unix.cma cothreads.cma poll_event.ml
**
** Native compile with :
** ocamlopt -I +process unix.cmxa cothreads.cmxa poll_event.ml -o poll_event
*)

let child_thread chan =
(* Print thread id (actually process id) and flush stdout. *)
Printf.printf "child_thread : %d\n%!" (Cothread.id (Cothread.self ())) ;
(* Hang about for a bit. *)
Cothread.delay 2.5 ;
(* Send an event. *)
Event.sync (Event.send chan "Event from child.") ;
(* Hang about a bit more. *)
Cothread.delay 1.5 ;
Cothread.exit ()

let () =
(* Create an event channel for sending stuff between main and child. *)
let chan = Event.new_channel () in
(* Create the child thread and pass it the channel. *)
let t1 = Cothread.create child_thread chan in
(* Set the exit condition to false. *)
let fini = ref false in
while not !fini do
(* Now we can poll on the incoming event. *)
match (Event.poll (Event.receive chan)) with
| None -> Printf.printf "Nothing.\n%!" ; Cothread.delay 0.3
| Some s -> fini := true ; print_endline s
done ;
(* Wait for child thread to exit. *)
Printf.printf "Waiting for child thread.\n%!" ;
Cothread.join t1 ;
print_endline "Done."

-------------------------------------------------------------------

I intend to blog about this over the weekend. I'll post a URL when
its done.

Erik
--
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------

"I believe C++ instills fear in programmers, fear that the
interaction of some details causes unpredictable results. Its
unmanageable complexity has spawned more fear-preventing tools
than any other language, but the solution _should_ have been
to create and use a language that does not overload the
whole goddamn human brain with irrelevant details."
-- Erik Naggum

skaller

unread,
Sep 20, 2007, 3:15:15 AM9/20/07
to caml...@inria.fr
On Thu, 2007-09-20 at 14:29 +1000, Erik de Castro Lopo wrote:
> skaller wrote:
>
> > Why don't you just use a different module name?
>
> I agree. Although it is nice the have the Cothreads API be so similar
> to the native Threads module, I can't possibly imagine ever wanting
> to go from Cothreads back to Threads.

Undesirable, but there is an issue which might lead to that,
namely the licence.

CoThreads is GPL, not even LGPL, let alone LGPL with linking exception,
whereas my project, for example, is FFAU/BSD, and only tolerates
non FFAU/BSD licences in Ocaml standard distro (not 3PLs).

GPL isn't so bad for executables, but it isn't very good for
libraries.


--
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

_______________________________________________

Erik de Castro Lopo

unread,
Sep 20, 2007, 3:55:10 AM9/20/07
to caml...@inria.fr
skaller wrote:

> CoThreads is GPL, not even LGPL, let alone LGPL with linking exception,

I posted a request to the CoThreads mailing list for a licence change
to LGPL with linking exception.

Erik
--
-----------------------------------------------------------------
Erik de Castro Lopo
-----------------------------------------------------------------

Linux : Think of it as 'free' as in 'free speech' not 'free beer'.

Zheng Li

unread,
Sep 20, 2007, 4:20:33 AM9/20/07
to caml...@inria.fr

Hi,

First note that the naming convention follows the standard threads library in
OCaml: Threads and coThreads (both with 's') refer to the library (*.cm(x)a);
Thread and Cothread (neither with 's') are modules (*.cm[ox]) inside.

skaller <ska...@users.sourceforge.net> writes:
>> The Thread module interfaces from the three engines differ with each
>> other! Note that this is not the problem from coThreads, it's a problem of
>> standard Threads library:
>
> Why don't you just use a different module name?

The different module name *is* Cothread, a compatible super set of Thread
module. Use this module instead if you want to have object-level
compatibility. In short, all the three engines have the two libraries with
isomorphic structures (quite simple) :

<code>
threads.cm(x)a = threads.cm[ox], mutex.cm[ox], condition.cm[ox], event.cm[ox]
cothreads.cm(x)a = threads.cm[ox], mutex.cm[ox], condition.cm[ox], event.cm[ox], cothread.cm[ox], stm.cm[ox]
</code>

See, we also provide the compatible "threads.cm(x)a" for the process
engine. Think about the following scenario:

- If you're working on legacy code, you don't care STM and don't care
object-level compatibility, the only thing you're interested is to running
your code with process to speedup, then the only thing to change in your
Makefile is the include path e.g. "-I +threads" -> "-I +process", you can
still using Thread module and "threads.cma", they are present in process
engine.

- You have several projects, some of them using traditional Threads, some
using coThreads, you don't want to bother to remember that. So simply
changing all linking library from threads.cma to cothreads.cma is fine,
because coThreads library contains every modules in Threads library.

- You have some legacy code written in standard Threads, and you'd like to
have it run with processes. Though you don't care about the newly introduced
modules of coThreads, you do want to have a single copy of object files for
each engines. In such case, you need to add one line to any source code
which makes use of the Thread module:

module Thread = Cothread

because Thread module (the only module in Threads and coThreads) don't have
object-level compatibility, and change any occurrence of “threads.cm(x)a” to
cothreads.cm(x)a” in your Makefile.

For more explanation, see http://cothreads.sourceforge.net/doc/compatibility

_______________________________________________

Zheng Li

unread,
Sep 20, 2007, 4:32:13 AM9/20/07
to caml...@inria.fr

Hi,

Erik de Castro Lopo <mle+...@mega-nerd.com> writes:

> skaller wrote:
>> Why don't you just use a different module name?
> I agree. Although it is nice the have the Cothreads API be so similar
> to the native Threads module, I can't possibly imagine ever wanting
> to go from Cothreads back to Threads.

It's easy to neglect that coThreads actually comes with a compatible
threads.cm(x)a for the process engine. So if you're working with legacy code
but wish to try the process engine, you can still using Thread module and
threads library, and don't have to change anything except the include path.

On the other hand, if you're using shared-memory concurrency in your legacy
code, you'll have to port it with STM --- the only compatible way to have
shared-memory across different engines.

_______________________________________________

Zheng Li

unread,
Sep 20, 2007, 4:39:04 AM9/20/07
to caml...@inria.fr

Hi,

Erik de Castro Lopo <mle+...@mega-nerd.com> writes:

> skaller wrote:
>> CoThreads is GPL, not even LGPL, let alone LGPL with linking exception,
> I posted a request to the CoThreads mailing list for a licence change
> to LGPL with linking exception.

Yes. I have no problem to change the licence to another one if desired by the
community. I can change it since next release. Is LGPL+linking exception the
right one?

_______________________________________________

Zheng Li

unread,
Sep 20, 2007, 4:50:56 AM9/20/07
to caml...@inria.fr

Hi Jon,

Jon Harrop <j...@ffconsultancy.com> writes:
> On Wednesday 19 September 2007 12:13:46 Erik de Castro Lopo wrote:
>> I can however report that on a single machine with multiple cores,
>> message passing between CoThreads using the Event module is an absolute
>> pleasure to use.
> Carry on... :-)
> Any examples?

There is a replanted version of your ray tracer in the /example directory:
ray.ml is the module containing common computation functions, ray_col.ml and
ray_nocol.ml are two different parallel engines. There's nothing new in this
example, it's traditional Thread and Event, but running on process.

_______________________________________________

Zheng Li

unread,
Sep 20, 2007, 5:13:04 AM9/20/07
to caml...@inria.fr

Hi,

Nice try!

Erik de Castro Lopo <mle+...@mega-nerd.com> writes:

> All my examples so far are trivial, like the following which creates a
> Cothread and then polls for an event sent by the child.
>
> -------------------------------------------------------------------
> (*
> ** Needs Cothreads installed :
> ** http://cothreads.sourceforge.net/
> **
> ** Run with :
> ** ocaml -I +process unix.cma cothreads.cma poll_event.ml
> **
> ** Native compile with :
> ** ocamlopt -I +process unix.cmxa cothreads.cmxa poll_event.ml -o poll_event
> *)

One comment: you don't have to hard-wired "Cothread.xxx" into source, since all
the functions you're using are compatible between Thread and Cothread modules,
and moreover, coThreads does provide a compatible version of threads.cm(x)a and
Thread module being able to work with process.

So by "open Cothread" or "open Thread" or "module Thread=Cothread", you'll be
given more flexibility to change easily back and forth between coThreads and
standard Threads (image sometime later you need to deploy the same code to a
standard-OCaml-only environment ...)

--
Zheng Li
http://www.pps.jussieu.fr/~li

_______________________________________________

skaller

unread,
Sep 20, 2007, 6:43:51 AM9/20/07
to Zheng Li, caml...@inria.fr
On Thu, 2007-09-20 at 10:37 +0200, Zheng Li wrote:
> Hi,
>
> Erik de Castro Lopo <mle+...@mega-nerd.com> writes:
> > skaller wrote:
> >> CoThreads is GPL, not even LGPL, let alone LGPL with linking exception,
> > I posted a request to the CoThreads mailing list for a licence change
> > to LGPL with linking exception.
>
> Yes. I have no problem to change the licence to another one if desired by the
> community. I can change it since next release. Is LGPL+linking exception the
> right one?

It depends how restrictive you *intend* to be. BSD, MIT, Boost,
Creative Commons, etc are for genuinely free software.
Free as in FFAU: Free for Any Use. Do what you like with it!

GPL is if you want to prevent anyone using it who hides any
source code (most commercial use).

LGPL+linking X is when you don't mind if people hide
their own source, as long as they don't hide yours.

My opinion is: for a MAJOR public executable like gcc,
GPL is OK. For major libraries like glibc, LGPL (with
linking permitted) would be ok.

For code which has hardly any developers or users, if you
wish to encourage the widest possible use, the most
permissive licence (eg BSD) is probably the best:
why exclude potential users who may help with bug reports
or patches? IMHO *especially* commercial users who put funds
into it and make a commitment, are even more likely to help
with maintenance and development.

Unless or until you expect to actually make money out of the
code itself, restrictive licences are counter-productive ..
in my opinion of course.

EG: GHC Haskell is currently trying to dump GMP because
of the licence. GMP is LGPL, GHC is BSD: dependence on GMP
is therefore a serious barrier to people using Haskell in
industry.. not what the authors want. (Plain LGPL doesn't
have a static linking exception as Ocaml's libraries do).

Another *famous* example is 'readline()', the code that
does line editing for bash. Why is Ocaml's top level
so crappy, with no editing? Because of readline()'s GPL
licence it can't be used, because GPL is a virus, it
propagates to anything it touches.

--
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

_______________________________________________

Matthew Hannigan

unread,
Sep 20, 2007, 7:04:51 AM9/20/07
to skaller, Zheng Li, caml...@inria.fr
On Thu, Sep 20, 2007 at 08:43:09PM +1000, skaller wrote:
> Another *famous* example is 'readline()', the code that
> does line editing for bash. Why is Ocaml's top level
> so crappy, with no editing? Because of readline()'s GPL

Surely that's not a good reason anymore; there are non-GPL
workalikes to readline().

> licence it can't be used, because GPL is a virus, it
> propagates to anything it touches.

_______________________________________________

Florian Weimer

unread,
Sep 20, 2007, 7:58:06 AM9/20/07
to skaller, Zheng Li, caml...@inria.fr
> It depends how restrictive you *intend* to be. BSD, MIT, Boost,
> Creative Commons, etc are for genuinely free software.
> Free as in FFAU: Free for Any Use. Do what you like with it!

The majority of Creative Commons license is definitely not "FFAU".

> EG: GHC Haskell is currently trying to dump GMP because
> of the licence. GMP is LGPL, GHC is BSD: dependence on GMP
> is therefore a serious barrier to people using Haskell in
> industry.. not what the authors want. (Plain LGPL doesn't
> have a static linking exception as Ocaml's libraries do).

It does, and it's quite straightforward to fulfill. I don't think
this is a significant issue, and the "get rid of GMP" project doesn't
appear to be terribly successful.

skaller

unread,
Sep 20, 2007, 11:03:25 AM9/20/07
to Matthew Hannigan, Zheng Li, caml...@inria.fr
On Thu, 2007-09-20 at 20:44 +1000, Matthew Hannigan wrote:
> On Thu, Sep 20, 2007 at 08:43:09PM +1000, skaller wrote:
> > Another *famous* example is 'readline()', the code that
> > does line editing for bash. Why is Ocaml's top level
> > so crappy, with no editing? Because of readline()'s GPL
>
> Surely that's not a good reason anymore; there are non-GPL
> workalikes to readline().

Well one might ask Inria why they don't use it?

[BTW: Python does, yet some maintains FFAU licence]

--
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

_______________________________________________

Christophe Raffalli

unread,
Sep 20, 2007, 11:08:38 AM9/20/07
to skaller, Matthew Hannigan, Zheng Li, caml...@inria.fr
skaller a écrit :

> On Thu, 2007-09-20 at 20:44 +1000, Matthew Hannigan wrote:
>
>> On Thu, Sep 20, 2007 at 08:43:09PM +1000, skaller wrote:
>>
>>> Another *famous* example is 'readline()', the code that
>>> does line editing for bash. Why is Ocaml's top level
>>> so crappy, with no editing? Because of readline()'s GPL
>>>
>> Surely that's not a good reason anymore; there are non-GPL
>> workalikes to readline().
>>
>
> Well one might ask Inria why they don't use it?
>
> [BTW: Python does, yet some maintains FFAU licence]
>
>
There is even one (http://pauillac.inria.fr/~ddr/ledit) written in
OCaml ... and easy to add to any OCaml
program ...

--
Christophe Raffalli
Universite de Savoie
Batiment Le Chablais, bureau 21
73376 Le Bourget-du-Lac Cedex

tel: (33) 4 79 75 81 03
fax: (33) 4 79 75 87 42
mail: Christoph...@univ-savoie.fr
www: http://www.lama.univ-savoie.fr/~RAFFALLI
---------------------------------------------
IMPORTANT: this mail is signed using PGP/MIME
At least Enigmail/Mozilla, mutt or evolution
can check this signature. The public key is
stored on www.keyserver.net
---------------------------------------------

Christophe.Raffalli.vcf
signature.asc

skaller

unread,
Sep 20, 2007, 11:47:22 AM9/20/07
to Florian Weimer, Zheng Li, caml...@inria.fr
On Thu, 2007-09-20 at 13:39 +0200, Florian Weimer wrote:

> (Plain LGPL doesn't
> > have a static linking exception as Ocaml's libraries do).
>
> It does, and it's quite straightforward to fulfill.

A derivative work has conditions imposed on it which are contrary
to normal (closed source/proprietary) commercial practice.

As I read the licence, the provider must make it possible for the
user to link an alternate or modified library. Static linked executables
provided by commercial vendors very rarely provide any way to do that.

Whether or not the conditions are straightforward to fulfil
depends on the producers goals. If you were making an embedded
application it would almost certainly not be, and it would
be a legal impediment. Do you know any vendors of console
games, for example, that tell you how to modify a part of the
game? The software build processes used are almost certainly
trade secrets.

Even in my own FFAU product, whilst the whole of the sources
is available for inspection and modification, I certainly
don't document all of it -- if you want to modify a library
you'll have to go read all the source and find out yourself.

It isn't clear that satisfies the "make it possible for the
client to replace the library" requirement.


--
John Skaller <skaller at users dot sf dot net>
Felix, successor to C++: http://felix.sf.net

_______________________________________________