[Caml-list] OCaml maintenance status / community fork (again)

[Benedikt Meurer]

Dear caml-list,

I'd like to get back to the original topic, which BTW had nothing to do with Web 2.0, documentation, books, teaching, Batteries, PR, or whatever other topics came up. Of course those are important topics too, but hijacking other threads won't help with either.

There were already a few useful comments on the topic, but no statement from the current INRIA officials. Opening up the development of OCaml is a great suggestion, for example. Personally I'd even suggest to disconnect OCaml and INRIA, with an independent team of core maintainers (with appropriate spare time and knowledge). INRIA would still contribute to OCaml, but no longer control OCaml.

And to respond to those who think that the current development process is a "good thing" and leads to stability: By no way. It leads to stagnation and ignorance (most probably). For example, have a look at PR/3746, a great example. It took almost 4 years(!) to update the ARM port to softfp (and EABI). At the time the issue was finally fixed, most ARM application boards were already shipping with VFP, so the port is lacking behind several years. And even after all that time, the ARM port is not implemented properly, i.e. it's of the IP for argument passing does not only cause trouble with position independent code, but is forbidden in general because IP is reserved for linker stubs, both static and dynamic. The relevant bug report PR/5404, which includes a backward compatible patch, is already waiting for a sign of life for 3 weeks now (maybe wait another 4 years to get the port fixed). And what about the ARMv7-a / armhf port? I almost got it working, but looking at the past !
of the ARM port, I'm already pissed off by the fact that it will probably take ages for someone to even respond to the patch, not to mention that it will take forever to get it out to the users (well maybe Debian will include the patch for armhf, but that means the Debian developers have to do upstream work...).

Granted, ARM is not (yet) the main target platform, it's just to illustrate the inherent weakness of closed development combined with lack of time/interest. Sure there will always be areas where projects cannot keep up because of lack of knowledge/time, but it is unnecessarily frustrating and harmful for an open source project to lack behind in areas with active contributors with both knowledge and time. That's why I'd suggest to set OCaml free, either with the help of INRIA and by leaving INRIA behind if there's no other way to move on.

In either case, it'd be great if the official core team would at least take the time to comment on this issue.

best regards,
Benedikt

PS: Please avoid hijacking this thread.

--
Caml-list mailing list. Subscription management and archives:
https://sympa-roc.inria.fr/wws/info/caml-list
Beginner's list: http://groups.yahoo.com/group/ocaml_beginners
Bug reports: http://caml.inria.fr/bin/caml-bugs

[Alain Frisch]

On 12/08/2011 10:10 AM, Benedikt Meurer wrote:
> There were already a few useful comments on the topic, but no statement from the current INRIA officials. Opening up the development of OCaml is a great suggestion, for example. Personally I'd even suggest to disconnect OCaml and INRIA, with an independent team of core maintainers (with appropriate spare time and knowledge). INRIA would still contribute to OCaml, but no longer control OCaml.

Honestly, opening up the development of OCaml would be terrific; but
trying to disconnect it from INRIA sounds like a very bad idea to me.

Concerning the issues with the ARM port, there is no chance to get a
good support for this architecture (which includes accepting patches)
without someone in the core team who feels responsible for the port and
commits to maintaining it. You might want to create a new "core team"
completely disconnected from INRIA, but the problem would remain the
same (I don't believe in a completely decentralized development model
for something like OCaml). Instead, I'd suggest contacting the existing
core team (ca...@inria.fr) in order to find a solution.


Best regards,

Alain

[Gabriel Scherer]

Benedikt, just a methodology remark

> There were already a few useful comments on the topic, but no statement from the current INRIA officials.

> [..]

> In either case, it'd be great if the official core team would at least take the time to comment on this issue.

This is not the right way to ask feedback. You launch a polite but
quite critical fork of some people's codebase, without giving them
previous notice, and expect an immediate public response. It would
have been better to contact the INRIA people privately before raising
the issue on the list, just to give them notice and gather feedback.

I'm confident this awkwardness is not a result of bad will on your
part and won't be held against you. Still, please keep in mind that
there are human issues in play here, and try to be a bit smoother
here. "Be strict in what you send, but generous in what you receive",
the saying applies to human interactions as well.

Note: I have no idea what the "INRIA officials" opinion is, and of
course you could have contacted them first without making it explicit
in your mail. Apologies for being wrong in that case.


PS.: I'm sorry if you felt discussions about other aspects of the
OCaml community were prompted by your initial mail -- and I'm partly
responsible for this. I just wanted to say that, contrarily to the
message you sent, compiler hacking is not the only field welcoming
participation. I think everyone agrees that it is a legitimate topic,
and that each should participate according to its own interest, with
no value judgment of "better" contributions area. Your hacking feats
on the compiler are greatly appreciated.

On Thu, Dec 8, 2011 at 10:10 AM, Benedikt Meurer
<benedik...@googlemail.com> wrote:
> Dear caml-list,
>
> I'd like to get back to the original topic, which BTW had nothing to do with Web 2.0, documentation, books, teaching, Batteries, PR, or whatever other topics came up. Of course those are important topics too, but hijacking other threads won't help with either.
>
> There were already a few useful comments on the topic, but no statement from the current INRIA officials. Opening up the development of OCaml is a great suggestion, for example. Personally I'd even suggest to disconnect OCaml and INRIA, with an independent team of core maintainers (with appropriate spare time and knowledge). INRIA would still contribute to OCaml, but no longer control OCaml.
>

> And to respond to those who think that the current development process is a "good thing" and leads to stability: By no way. It leads to stagnation and ignorance (most probably). For example, have a look at PR/3746, a great example. It took almost 4 years(!) to update the ARM port to softfp (and EABI). At the time the issue was finally fixed, most ARM application boards were already shipping with VFP, so the port is lacking behind several years. And even after all that time, the ARM port is not implemented properly, i.e. it's of the IP for argument passing does not only cause trouble with position independent code, but is forbidden in general because IP is reserved for linker stubs, both static and dynamic. The relevant bug report PR/5404, which includes a backward compatible patch, is already waiting for a sign of life for 3 weeks now (maybe wait another 4 years to get the port fixed). And what about the ARMv7-a / armhf port? I almost got it working, but looking at the pas!

[Alain Frisch]

On 12/08/2011 11:28 AM, Benedikt Meurer wrote:
> The problem is IMHO that there is no one at INRIA caring about ARM.

I'm pretty sure you mean "core development team", not INRIA. INRIA is a
large research institute, you know. And as I said, the "core development
team" is not limited to members of INRIA. It may very well be the case
that no one in this team care about ARM (honestly, I don't know), and if
this is the case, I agree this should be fixed.

> In an open model we would have maintainers for the ARM port(s).

Yes, I argue for opening the current model a little, by enlarging the
existing core team. Creating a new "core team" would create more
frustration and problems than it would solve, in my opinion. We have
enough fragmentation in this community.

>> Instead, I'd suggest contacting the existing core team (ca...@inria.fr) in order to find a solution.
>

> I wasn't aware that there is a separate communication channel for the core team. I was under the impression that the caml-list was the best way to reach both the core team and the community.

I confirm that the best way to contact the core team is ca...@inria.fr
(see http://caml.inria.fr/contact.en.html).

[ivan chollet]

It's nice to see this thread coming back to the original issue after having
been hijacked. You can notice that contributers to this thread have
opinions but not many facts and arguments to support them.
Contacting the OCaml maintainers privately is definitely not the way to let
the development process happen in the open, but bad habits are hard to get
rid of.
I think we all understand that the core team is in an uncomfortable and
conflicting situation and that the questions asked in this thread won't get
an answer. In french this is called "avoir le cul entre deux chaises".
Benedikt did not suggest at any point to set OCaml free from the original
developers, he suggested to set it free from the INRIA.


On Thu, Dec 8, 2011 at 10:28 AM, Benedikt Meurer <
benedik...@googlemail.com> wrote:

>
> On Dec 8, 2011, at 10:54 , Alain Frisch wrote:
>
> > On 12/08/2011 10:10 AM, Benedikt Meurer wrote:
> >> There were already a few useful comments on the topic, but no statement
> from the current INRIA officials. Opening up the development of OCaml is a
> great suggestion, for example. Personally I'd even suggest to disconnect
> OCaml and INRIA, with an independent team of core maintainers (with
> appropriate spare time and knowledge). INRIA would still contribute to
> OCaml, but no longer control OCaml.
> >
> > Honestly, opening up the development of OCaml would be terrific; but
> trying to disconnect it from INRIA sounds like a very bad idea to me.
> >
> > Concerning the issues with the ARM port, there is no chance to get a
> good support for this architecture (which includes accepting patches)
> without someone in the core team who feels responsible for the port and
> commits to maintaining it. You might want to create a new "core team"
> completely disconnected from INRIA, but the problem would remain the same
> (I don't believe in a completely decentralized development model for
> something like OCaml).
>

> The problem is IMHO that there is no one at INRIA caring about ARM. In an

> open model we would have maintainers for the ARM port(s).
>

> > Instead, I'd suggest contacting the existing core team (ca...@inria.fr)
> in order to find a solution.
>

> I wasn't aware that there is a separate communication channel for the core
> team. I was under the impression that the caml-list was the best way to
> reach both the core team and the community.
>

> > Best regards,
> > Alain
>
> greets,
> Benedikt

[Stéphane Glondu]

On 12/08/2011 10:10 AM, Benedikt Meurer wrote:

> [...] I'm already pissed off by the fact that it will probably take ages for someone to even respond to the patch, not to mention that it will take forever to get it out to the users (well maybe Debian will include the patch for armhf, but that means the Debian developers have to do upstream work...).

As a maintainer of OCaml (among other things) in Debian, I must agree
with you that slow reaction, opacity in its development process, etc.
make OCaml a very frustrating upstream (one of the worst I have to deal
with). See for example:

http://caml.inria.fr/mantis/view.php?id=4963
http://caml.inria.fr/mantis/view.php?id=5131
http://caml.inria.fr/mantis/view.php?id=5254
http://caml.inria.fr/mantis/view.php?id=5279
http://caml.inria.fr/mantis/view.php?id=5393

All very precise bugreports with rather trivial patches or suggestions.
Yet, no satisfactory reaction.

I once proposed a patch to "fix" the "-custom" behaviour that made
resulting executables not strippable, to avoid all sort of
special-casing that had to be done in Debian (and Fedora) because of
this behaviour. It has been rejected (but I applied it nonetheless to
the Debian package). See the discussion at:

http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=256900

(in particular, starting from message #42).

All this often leads me to just not trying to solve bugs in OCaml
myself, even for things that do affect Debian. See for example:

http://caml.inria.fr/mantis/view.php?id=1710
http://caml.inria.fr/mantis/view.php?id=1711
http://caml.inria.fr/mantis/view.php?id=1836
http://caml.inria.fr/mantis/view.php?id=4812
http://caml.inria.fr/mantis/view.php?id=5371

Concerning your work on ARM, PR#5049 completely deterred me from working
on it myself, or submitting it to more competent people (and I do know
some thanks to Debian and Ubuntu).

I really think that OCaml could benefit from a more open development
model, and should leverage external contributors (that are not members
of the consortium) instead of plainly ignoring them. There is for
example no need for someone in the "core team" to feel responsible for
the ARM port. Have a look at:

https://bugs.launchpad.net/ubuntu/+source/ocaml/+bug/810402

People who care about ARM solved the bug themselves. Having ocamlopt
working on all Debian architectures wouldn't look so unreasonable if the
"core team" were willing to integrate necessary changes as part of
official releases (which is clearly not the case).

I'm not advocating for faster development, endless addition of new
features, or a better standard library, I just wish better reactivity to
the community concerning the compiler or the language itself. The
general lack of "official" (whatever that means) participation in this
mailing-list is another example of this problem.


Cheers,

--
Stéphane

[Benedikt Meurer]

On Dec 8, 2011, at 11:46 , Alain Frisch wrote:

> On 12/08/2011 11:28 AM, Benedikt Meurer wrote:
>> The problem is IMHO that there is no one at INRIA caring about ARM.
>
> I'm pretty sure you mean "core development team", not INRIA. INRIA is a large research institute, you know. And as I said, the "core development team" is not limited to members of INRIA.

Right, of course, wrong wording.

>> In an open model we would have maintainers for the ARM port(s).
>
> Yes, I argue for opening the current model a little, by enlarging the existing core team. Creating a new "core team" would create more frustration and problems than it would solve, in my opinion. We have enough fragmentation in this community.

I don't know what's the best to move on. But whatever it will be, the entry barriers for contributors and new maintainers should be lowered and both the process and the outcome of decisions should be opened up, IMHO. Otherwise the overall situation will stay the same; OCaml may have an ARM maintainer then, but will continue to lack an XYZ maintainer, even though there are patches and contributors. Enlarging the existing core won't magically improve communication / contact with the community; as Gabriel noted there are human issues in play here.

>> I wasn't aware that there is a separate communication channel for the core team. I was under the impression that the caml-list was the best way to reach both the core team and the community.
>
> I confirm that the best way to contact the core team is ca...@inria.fr (see http://caml.inria.fr/contact.en.html).

Ok, I'll keep them CC'd. I don't want to start a new private discussion with them as that certainly won't help with openness either.

> Alain

best regards,
Benedikt

[Pierre-Alexandre Voye]

2011/12/8 Benedikt Meurer <benedik...@googlemail.com>

>
>
>
> The problem is IMHO that there is no one at INRIA caring about ARM. In an

> open model we would have maintainers for the ARM port(s).
>

Note that if Ocaml compiler would have a C backend, all these problems or
architecture port would disappear...
Ocaml would have more than 30 target[1]
In my Opinion, trying to generate assembler is a bad idea because modern
CPU require a lot of work to generate good assembler.
Only the GCC and LLVM team are big enough to be able to make a good job.

In the Lisaac project, we were able to compete with C[2]. Lisaac is a
compiler for a Smalltalk like language : the if/then/else is unknown to the
compiler, it is defined in the true/false object. So it is a proof that a
very high level language can reach C performance. Ocaml can do this,
because the compiler is able to know a lot of type informations.
The Lisaac compiler use strong flow analysis and, more importantly generate
C code. To reach performance, Lisaac tailor C code to help GCC to generate
very optimized code.
For instance, GCC is able to produce MMX/SSE/AVX code when you write code
like this :
http://gcc.gnu.org/projects/tree-ssa/vectorization.html#vectorizab

AutoVectorization is just an example of what you can do with GCC (or LLVM
soon certainly) and which would require a lot of work with an own asm
generator.

[1] http://en.wikipedia.org/wiki/GNU_Compiler_Collection#Architectures
[2]
http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=lisaac&lang2=gcc


--
---------------------
https://twitter.com/#!/ontologiae/
http://linuxfr.org/users/montaigne

[oliver]

On Thu, Dec 08, 2011 at 10:47:58AM +0000, ivan chollet wrote:
> It's nice to see this thread coming back to the original issue after having
> been hijacked.

"having been hijacked" sounds black-hearted.

"Different interpretation of the orig post" would match it better.

> You can notice that contributers to this thread have
> opinions but not many facts and arguments to support them.
> Contacting the OCaml maintainers privately is definitely not the way to let
> the development process happen in the open, but bad habits are hard to get
> rid of.

Doing things openly has advantages like "transparency" and seeing,
what and how it is done (lurking on activity).
But this also can be annoying to the developers.
Just when looking at those many discussions that popped up again and again here
during the years.

> I think we all understand that the core team is in an uncomfortable and
> conflicting situation and that the questions asked in this thread won't get
> an answer.


> In french this is called "avoir le cul entre deux chaises".
> Benedikt did not suggest at any point to set OCaml free from the original
> developers, he suggested to set it free from the INRIA.

1) Is this realistic?
2) And what would be the advantages of it?

1:
Who would know INRIA, if there were no OCaml at INRIA?
It even attracted some big companies, which also do bring in some mones:
http://caml.inria.fr/consortium/

So often universities are seen as ivory towers, which do not produce anything
meaningful to zthe industry (and industry is thinking they do a good kob, using crap langauges).
But OCaml could attract those companies...

...so OCaml, even it was from the ivory tower, it could attract idustry.

It's not the masses, but some interesting companies are listed there.

Of course I would be happy to have more projects using OCaml,
but the reason why Ocaml is not used too often in industry, is, because there
are not enough people who know OCaml.
In one project I used OCaml for some tasks; but later it was decided only to
use langauegs, that most people of the team knew. And I was the only OCaml
programmer there. So it was decided to use C and Python. In other projects.
OCaml was no option from the beginning.

So, that OCaml is at INRIA is not the reason for OCaml not being widely used.
And that it's driven by the ivory tower is rather an advantage, because what
we like, is that it is not another crap language that every average programmer
wants to use, driven by a hype.

A while ago, when talking about a job opportunity, mentioning OCaml, the
answer was, that C++ is preferred to C, because it offers OO.
People just don't know it. And even some people who used it, but did not
looked at it more closely, misunderstand some concepts.
And learning Ocaml, in the beginning is painful, because the type system is
always painful, until you get used to it.

That's why OCaml is living in a niche.

I doubt that it is, because the core team develops in secured intimacy,
instead in the hectical world of this or that server of the community.



2)
INRIA will not only be an institution which has constrictions for the team,
but also possibilities. As an institution, it offers infrastructure which can
be relied on. And maybe other people who might join.


If people want to have more influence on the Caml team it might make sense,
launching an open institution, which looks out for donations and then
enters the Caml Consortium.

Maybe such a group/institution could be an interface from Ocaml Core Team
to the public of unpatient OCaml users.


Ciao,
Oliver

[Fabrice Le Fessant]

As Alain said, the core team is important, it is the label of quality of
OCaml, but there is a real need to enlarge the existing core team, who
cannot cope anymore with reviewing and integrating all the
contributions. However, it cannot be done by just opening the doors to
all volunteers: people have to prove that they can not only write very
good, very useful code, but also commit to support it for years.

This last point is a long-term process, and OCamlPro has been created
with this idea in mind. The company has been founded in April this year,
and is growing slowly, but steadily, thanks to the trust and support of
its first customers. Of course, once OCamlPro joins the core team, the
control will still be at INRIA, but there will be more manpower to
review contributions, improve them and integrate them in the official
distribution, so things are going to move.

Of course, people who want to make this process complete faster, are
welcome to help it, for example by making their companies and
universities officially support OCamlPro (con...@ocamlpro.com).

--Fabrice

On 12/08/2011 11:46 AM, Alain Frisch wrote:
> Yes, I argue for opening the current model a little, by enlarging the
> existing core team. Creating a new "core team" would create more
> frustration and problems than it would solve, in my opinion. We have
> enough fragmentation in this community.

[Török Edwin]

On 12/08/2011 01:11 PM, Pierre-Alexandre Voye wrote:
> 2011/12/8 Benedikt Meurer <benedik...@googlemail.com>
>
>>
>>
>>
>> The problem is IMHO that there is no one at INRIA caring about ARM. In an
>> open model we would have maintainers for the ARM port(s).
>>
> Note that if Ocaml compiler would have a C backend, all these problems or
> architecture port would disappear...
> Ocaml would have more than 30 target[1]
> In my Opinion, trying to generate assembler is a bad idea because modern
> CPU require a lot of work to generate good assembler.
> Only the GCC and LLVM team are big enough to be able to make a good job.
>
> In the Lisaac project, we were able to compete with C[2]. Lisaac is a
> compiler for a Smalltalk like language : the if/then/else is unknown to the
> compiler, it is defined in the true/false object. So it is a proof that a
> very high level language can reach C performance. Ocaml can do this,
> because the compiler is able to know a lot of type informations.
> The Lisaac compiler use strong flow analysis and, more importantly generate
> C code. To reach performance, Lisaac tailor C code to help GCC to generate
> very optimized code.

Wouldn't it be more reliable to write a GCC OCaml frontend, or an LLVM OCaml frontend?
(or generate GCC Gimple, or LLVM IR).
I think GCC already has a frontend which is written in the language itself (Ada).

There are things that cannot be easily expressed in C, such as tail calls,
type hierarchies, etc. Also there is a lot of undefined behaviour in C that you can quite easily hit,
especially if you automatically generate code.

Best regards,
--Edwin

[Jacques Garrigue]

On 2011/12/08, at 18:10, Benedikt Meurer wrote:

> Dear caml-list,
>
> I'd like to get back to the original topic, which BTW had nothing to do with Web 2.0, documentation, books, teaching, Batteries, PR, or whatever other topics came up. Of course those are important topics too, but hijacking other threads won't help with either.
>
> There were already a few useful comments on the topic, but no statement from the current INRIA officials. Opening up the development of OCaml is a great suggestion, for example. Personally I'd even suggest to disconnect OCaml and INRIA, with an independent team of core maintainers (with appropriate spare time and knowledge). INRIA would still contribute to OCaml, but no longer control OCaml.
>

> And to respond to those who think that the current development process is a "good thing" and leads to stability: By no way. It leads to stagnation and ignorance (most probably). For example, have a look at PR/3746, a great example. It took almost 4 years(!) to update the ARM port to softfp (and EABI). At the time the issue was finally fixed, most ARM application boards were already shipping with VFP, so the port is lacking behind several years. And even after all that time, the ARM port is not implemented properly, i.e. it's of the IP for argument passing does not only cause trouble with position independent code, but is forbidden in general because IP is reserved for linker stubs, both static and dynamic. The relevant bug report PR/5404, which includes a backward compatible patch, is already waiting for a sign of life for 3 weeks now (maybe wait another 4 years to get the port fixed). And what about the ARMv7-a / armhf port? I almost got it working, but looking at the pas!

t !
> of the ARM port, I'm already pissed off by the fact that it will probably take ages for someone to even respond to the patch, not to mention that it will take forever to get it out to the users (well maybe Debian will include the patch for armhf, but that means the Debian developers have to do upstream work...).
>
> Granted, ARM is not (yet) the main target platform, it's just to illustrate the inherent weakness of closed development combined with lack of time/interest. Sure there will always be areas where projects cannot keep up because of lack of knowledge/time, but it is unnecessarily frustrating and harmful for an open source project to lack behind in areas with active contributors with both knowledge and time. That's why I'd suggest to set OCaml free, either with the help of INRIA and by leaving INRIA behind if there's no other way to move on.
>
> In either case, it'd be great if the official core team would at least take the time to comment on this issue.
>
> best regards,
> Benedikt
>
> PS: Please avoid hijacking this thread.

Dear Benedikt,

I do agree that the problem with ARM reflect some problem in the current development organization, but I don't think that you need to fork to solve it. (And note by the way that a real fork could be in contradiction with the QPL.)

Before attacking the current model, you should understand how it works, and why it actually works mostly well considering that OCaml, without being huge, is still a rather complex piece of software.
My understanding is that somebody is responsible for every line of code in the distribution.
So when a bug report comes, it is usually clear who should do the job, and generally this is done pretty fast.
Not always in the next 24 hours, but everybody tries no to be too late.
Considering other obligations this can mean a few weeks.
(When it is more than that, this may actually mean an oversight; there is a huge number of bug reports, so this can happen)
The main point here is that the person who fixes the problem usually is the most competent person available for this piece of code.
This seems a requisite for quality.

Another consequence of this approach is that usually patches are not applied directly.
When you know that you are going to be responsible for the code, you look carefully at a patch before applying it,
and often you find a better way to do it in the process :-)

Now, your specific problem occurs in a port that probably only few people in the core team even use.
If you look at the log, you will see that Xavier is maintaining it, in his spare time.
I cannot answer for him, but he might be more than willing to find somebody else to do this specific job,
as maintaining a port requires checking lots of information sources for a small number of lines of code.
This kind of code could be a good candidate for "outsourcing" as maintaining it requires more knowledge of the specific architecture than of the inner workings of OCaml.

However, this is not the case for most of the rest of OCaml.
As the problem is specific, the solution should probably be specific, and forking seems an overkill.
On the other hand, there is no restriction on cloning the repository and distributing patches in the meantime.
Lots of people are already doing that for their own reasons (I have done that in the past), and I don't think that having all of them working with the same repository would be useful.

Jacques Garrigue (personal viewpoint)

[Jérémie Dimino]

Le vendredi 09 décembre 2011 à 11:11 +0900, Jacques Garrigue a écrit :
> I do agree that the problem with ARM reflect some problem in the current development
> organization, but I don't think that you need to fork to solve it.

> *(And note by the way that a real fork could be in contradiction with the QPL.)*

I thought OCaml was a free software...

Seriously, i don't understand why a fork would be a bad thing. There are
lots of parts for which one don't need to be superman to contribute. And
right now there are lots of trivial patches on the tracker that remain
without response after several years, so there is clearly a need for
improvement.

Also the development of OCaml seems a bit opaque, we don't know where
the discutions of the core team happen. Maybe it is on ca...@inria.fr but
it is not public. I think people are interested (i am) about technical
discutions on the compiler.

Cheers,

--
Jérémie

[Gabriel Scherer]

> Also the development of OCaml seems a bit opaque, we don't know where
> the discutions of the core team happen. Maybe it is on ca...@inria.fr but
> it is not public. I think people are interested (i am) about technical
> discutions on the compiler.

Just a remark on that: I agree that it's frustrating to be kept out
the discussions of the core caml team.
I would be interested in learning more about it as well as you are.

However, I also understand the reason why this choice is made: when
discussion programming language matters, there is usually an
extraordinary amount of bike-shedding. Camlers are quite disciplined
but still the caml-list often come backs on the same topic with few
additional content added, starts endless discussions about thing that
don't move so much in the end (because discussing is fun and
interesting, etc.).
I have heard from people that do take part in internal discussions
that the debates are already long and exhausting. I do understand (yet
does not support) the choice to not open them to public discussion. I
would love, for example, a kind of read-only mode where we hear about
the discussion, without adding noise to it; but could we restrain
ourselves (... and others of the list and on the web) to silence? I
can very well imagine a discussion on the public lists "on some
suggestions about the debate about ...", filled with sneaky remarks
such that "I'm disappointed no core-team member has taken the time to
comment on my proposal".

Just a little story: between the 3.11 and 3.12 transition I followed
the ocaml SVN branches (which are publicly accessible ways of having
information about the current evolution of Caml) and was excited about
the new stuff. When I stumbled about Alain Frisch's work on rigid
names for type variables (
http://caml.inria.fr/pub/docs/manual-ocaml/manual021.html#toc80 ), I
was immediately appalled by the syntax.
I immediately started writing a mail to the list discussing the issue,
making several suggestions, etc. Yet I decided to contact Alain
privately first, as it seemed fittingą. His immediate answer was
around the lines "please don't; we already had a troll about it, other
options were mentioned, but we really need to settle on a compromise
here, no another, longer troll". I respected his opinion, and in
retrospect I think that was the right choice.

ą: It seemed fitting to contact him privately first, and would also
have been in Benedikt's case by the way, contrarily to what some
people seem to think. Giving people notice does not hurt transparency
or open discussion. If I wrote a scientific paper discussing someone's
work, especially if it's a bit negative about it, I would try to
contact him privately before sending it to the public or reviewers.

On Fri, Dec 9, 2011 at 11:37 AM, Jérémie Dimino <jer...@dimino.org> wrote:

[Stefano Zacchiroli]

On Fri, Dec 09, 2011 at 12:03:30PM +0100, Gabriel Scherer wrote:
> when discussion programming language matters, there is usually an
> extraordinary amount of bike-shedding

<snip>

> I would love, for example, a kind of read-only mode where we hear
> about the discussion, without adding noise to it

Well, *if* this is the problem, then the solutions to it are well known
and adopted by other communities that maintain programming languages:
just add a "core-dev" mailing list, where only list members could post,
and someone who moderates subscription requests. Such a think will also
help defining who the core team are and give some sort of public reward
to who makes into it. Given that (as discussed in this threader) there
already OCaml committers who are not affiliated to INRIA, that would
also dispel the feeling that OCaml is an INRIA pet.

( Now probably Benedikt will probably want to kill both me and Gabriel
for thread hijack, *again* :-))

--
Stefano Zacchiroli zack@{upsilon.cc,pps.jussieu.fr,debian.org} . o .
Maître de conférences ...... http://upsilon.cc/zack ...... . . o
Debian Project Leader ....... @zack on identi.ca ....... o o o
« the first rule of tautology club is the first rule of tautology club »

[Jonathan Protzenko]

Just for the record, ca...@inria.fr also happens to be list where the
members of the Caml Consortium discuss their issues. There's potentially
private/sensitive information in there, and it's not always clear what
relates to the consortium member's interests, and what is more about
language design. So having a third list just to discuss language design
issues is not as easy as it seems.

Cheers,

jonathan

On 12/09/2011 12:17 PM, Stefano Zacchiroli wrote:
> On Fri, Dec 09, 2011 at 12:03:30PM +0100, Gabriel Scherer wrote:
>> when discussion programming language matters, there is usually an
>> extraordinary amount of bike-shedding
> <snip>
>> I would love, for example, a kind of read-only mode where we hear
>> about the discussion, without adding noise to it
> Well, *if* this is the problem, then the solutions to it are well known
> and adopted by other communities that maintain programming languages:
> just add a "core-dev" mailing list, where only list members could post,
> and someone who moderates subscription requests. Such a think will also
> help defining who the core team are and give some sort of public reward
> to who makes into it. Given that (as discussed in this threader) there
> already OCaml committers who are not affiliated to INRIA, that would
> also dispel the feeling that OCaml is an INRIA pet.
>
> ( Now probably Benedikt will probably want to kill both me and Gabriel
> for thread hijack, *again* :-))
>

--

[Benedikt Meurer]

On Dec 9, 2011, at 11:37 , Jérémie Dimino wrote:

> Le vendredi 09 décembre 2011 à 11:11 +0900, Jacques Garrigue a écrit :
>> I do agree that the problem with ARM reflect some problem in the current development
>> organization, but I don't think that you need to fork to solve it.
>> *(And note by the way that a real fork could be in contradiction with the QPL.)*
>
> I thought OCaml was a free software...

I'm not a lawyer, but the QPL is not necessarily compatible with free software licenses, as already noted in another thread.

> Seriously, i don't understand why a fork would be a bad thing. There are
> lots of parts for which one don't need to be superman to contribute. And
> right now there are lots of trivial patches on the tracker that remain
> without response after several years, so there is clearly a need for
> improvement.

Right. I would like to place focus on discussing this point, as it seems to be the root of the evil. It would be so easy to fix, IMHO, and you don't need to give up control by the core team. Why not accept a model similar to i.e. the NetBSD project, with a lot of committers (experts in their own areas) and 2-3 people to keep an eye on the overall direction?

Benedikt

[Mehdi Dogguy]

On 12/09/2011 03:24 PM, Benedikt Meurer wrote:
>
> Right. I would like to place focus on discussing this point, as it
> seems to be the root of the evil. It would be so easy to fix, IMHO,
> and you don't need to give up control by the core team. Why not
> accept a model similar to i.e. the NetBSD project, with a lot of
> committers (experts in their own areas) and 2-3 people to keep an
> eye on the overall direction?
>

or instead of changing how OCaml is currently handled by INRIA's team,
we could consider some development model similar to Eglibc's one. Eglibc
is not a fork of glibc, but a glibc distribution.

http://lwn.net/Articles/333755/ could be an interesting read (for those
who care).

Regards,

--
Mehdi Dogguy مهدي الدڤي
http://dogguy.org/

[Benedikt Meurer]

On Dec 9, 2011, at 18:00 , Mehdi Dogguy wrote:

> On 12/09/2011 03:24 PM, Benedikt Meurer wrote:
>>
>> Right. I would like to place focus on discussing this point, as it
>> seems to be the root of the evil. It would be so easy to fix, IMHO,
>> and you don't need to give up control by the core team. Why not
>> accept a model similar to i.e. the NetBSD project, with a lot of
>> committers (experts in their own areas) and 2-3 people to keep an
>> eye on the overall direction?
>
> or instead of changing how OCaml is currently handled by INRIA's team,
> we could consider some development model similar to Eglibc's one. Eglibc
> is not a fork of glibc, but a glibc distribution.

Whether you call it a "fork" or a "distribution" doesn't matter. It's basically what I described in my initial proposal for a "community fork" (or a "community distribution"). Simply a place where contributors are welcome, patches are reviewed and processed (in a reasonable time frame), with focus on cooperation and communication.

If people fear the word "fork", can we agree on a "community distribution" (in the eglibc sense) instead?

> http://lwn.net/Articles/333755/ could be an interesting read (for those
> who care).

While we don't have a Drepper here in OCaml land (yet?), there are certainly a few common problems, Stephane already mentioned OCaml as an unpleasant upstream.

> Regards,
> Mehdi Dogguy مهدي الدڤي

Benedikt

[Mehdi Dogguy]

On 12/09/2011 06:36 PM, Benedikt Meurer wrote:
>
> Whether you call it a "fork" or a "distribution" doesn't matter.
>

It does, IMHO. Forking a project allows you to integrate more intrusive
changes, and doesn't force you to stay compatible (in some way) with the
original work. That's, in my understanding, the biggest difference
between the two approaches.

Regards,

--
Mehdi Dogguy مهدي الدڤي
http://dogguy.org/

[oliver]

On Thu, Dec 08, 2011 at 08:18:58PM +0200, Török Edwin wrote:
> On 12/08/2011 01:11 PM, Pierre-Alexandre Voye wrote:
> > 2011/12/8 Benedikt Meurer <benedik...@googlemail.com>
> >
> >>
> >>
> >>
> >> The problem is IMHO that there is no one at INRIA caring about ARM. In an
> >> open model we would have maintainers for the ARM port(s).
> >>
> > Note that if Ocaml compiler would have a C backend, all these problems or
> > architecture port would disappear...
> > Ocaml would have more than 30 target[1]
> > In my Opinion, trying to generate assembler is a bad idea because modern
> > CPU require a lot of work to generate good assembler.
> > Only the GCC and LLVM team are big enough to be able to make a good job.
> >
> > In the Lisaac project, we were able to compete with C[2]. Lisaac is a
> > compiler for a Smalltalk like language : the if/then/else is unknown to the
> > compiler, it is defined in the true/false object. So it is a proof that a
> > very high level language can reach C performance. Ocaml can do this,
> > because the compiler is able to know a lot of type informations.
> > The Lisaac compiler use strong flow analysis and, more importantly generate
> > C code. To reach performance, Lisaac tailor C code to help GCC to generate
> > very optimized code.
>
> Wouldn't it be more reliable to write a GCC OCaml frontend, or an LLVM OCaml frontend?

[...]

The LLVM issue already was discussed months or years ago.
There also was a testlanguage implementation with some features of OCaml,
performing good, but AFAIK it was far away from competing with OCaml in respect
to what the language offered as features. (Performance alone is not enough.)

Don't know if this project exists today.
But my impression is, that this was one of many attempts
that started with good ideas but did not come to a stage
that could be interfaced with OCaml.

One reason, why I'm sceptical on the many suggestions done
on here on the list these days....


Ciao,
Oliver

[oliver]

On Fri, Dec 09, 2011 at 12:03:30PM +0100, Gabriel Scherer wrote:

> > Also the development of OCaml seems a bit opaque, we don't know where
> > the discutions of the core team happen. Maybe it is on ca...@inria.fr but
> > it is not public. I think people are interested (i am) about technical
> > discutions on the compiler.
>
> Just a remark on that: I agree that it's frustrating to be kept out
> the discussions of the core caml team.
> I would be interested in learning more about it as well as you are.
>
> However, I also understand the reason why this choice is made: when
> discussion programming language matters, there is usually an
> extraordinary amount of bike-shedding. Camlers are quite disciplined
> but still the caml-list often come backs on the same topic with few
> additional content added, starts endless discussions about thing that
> don't move so much in the end (because discussing is fun and
> interesting, etc.).
> I have heard from people that do take part in internal discussions
> that the debates are already long and exhausting. I do understand (yet
> does not support) the choice to not open them to public discussion. I
> would love, for example, a kind of read-only mode where we hear about
> the discussion, without adding noise to it;

[...]

Yes, a read-only list was, what today came to my mind also.

> but could we restrain
> ourselves (... and others of the list and on the web) to silence?

Instead of commanded silence, the Caml-list could be a place,
where these issues could be discussed.

So, the core-team could sometimes look at the Caml list and get some
ideas from there, without being forced to answer.

[...]

> Just a little story: between the 3.11 and 3.12 transition I followed
> the ocaml SVN branches (which are publicly accessible ways of having
> information about the current evolution of Caml) and was excited about
> the new stuff. When I stumbled about Alain Frisch's work on rigid
> names for type variables (
> http://caml.inria.fr/pub/docs/manual-ocaml/manual021.html#toc80 ), I
> was immediately appalled by the syntax.
> I immediately started writing a mail to the list discussing the issue,
> making several suggestions, etc. Yet I decided to contact Alain

> privately first, as it seemed fitting¹. His immediate answer was

> around the lines "please don't; we already had a troll about it, other
> options were mentioned, but we really need to settle on a compromise
> here, no another, longer troll". I respected his opinion, and in
> retrospect I think that was the right choice.

[...]

This is what I think: there might be some reasons for certain decisions,
which might not be clear to those who are new in a certain field.

I had experienced simliar things last year on a hardware project,
when a new colleage added some (flyback) diodes to a current source, and
some days later he saw that the backward current brings problems that make
the circuit unusable. :-) LOL
I had good reasons, NOT to insert those diodes... which the colleague
was not aware of. Later he knew it. ;)

That's very similar to what you say here on the point of the type
annotation syntax. :-)

Regarding the discussions on this list, I think many times
it was similar. First big proposals and announcements,
later stumbling...


But the read-only option on some discussions from the core team
of course would be interesting.

It could gain two points: more openly sharing of the work of the
core team together with the protective barrier that stops
discussions becoming too much bloated by too many posts,
that are not directly accessing the discussion of the core team
(which of course has best insights into OCaml).

It also would be fine, to know if there is something like a
priority list of the Ocaml team.
The points mentioned by the one Debian OCaml maintainer
seem not to be very satisfactory.
Not sure if some parts of the codes were already planned to change...

So, maybe the inner circle / core team has just different priorities?

I think it might also make sense to ask in a different direction:
instead on insisting how the core team has to be(have), to make
the development more open, it maybe could be better to ask,
how the core team could be helped, so that the goals of this
team and the OCaml users can be brought together.

Maybe they already have ideas or wishes on changing the process,
but being confronted just with a bunch of request on how they have to change.

Maybe asking on how to help is better than insisting that they have to be as
peoplefrom the outside want it.


Ciao,
Oliver

[Goswin von Brederlow]

Stefano Zacchiroli <za...@upsilon.cc> writes:

> On Fri, Dec 09, 2011 at 12:03:30PM +0100, Gabriel Scherer wrote:
>> when discussion programming language matters, there is usually an
>> extraordinary amount of bike-shedding
> <snip>
>> I would love, for example, a kind of read-only mode where we hear
>> about the discussion, without adding noise to it
>
> Well, *if* this is the problem, then the solutions to it are well known
> and adopted by other communities that maintain programming languages:
> just add a "core-dev" mailing list, where only list members could post,
> and someone who moderates subscription requests. Such a think will also
> help defining who the core team are and give some sort of public reward
> to who makes into it. Given that (as discussed in this threader) there
> already OCaml committers who are not affiliated to INRIA, that would
> also dispel the feeling that OCaml is an INRIA pet.
>
> ( Now probably Benedikt will probably want to kill both me and Gabriel
> for thread hijack, *again* :-))

Or just have someone post minutes of such discussions. A short summary
of what was discussed and what is planed as a result. We don't need to
see all the fighting and back and forth. The conclusion is usualy
enough.

MfG
Goswin

[Goswin von Brederlow]

Mehdi Dogguy <me...@dogguy.org> writes:

> On 12/09/2011 06:36 PM, Benedikt Meurer wrote:
>>
>> Whether you call it a "fork" or a "distribution" doesn't matter.
>>
>
> It does, IMHO. Forking a project allows you to integrate more intrusive
> changes, and doesn't force you to stay compatible (in some way) with the
> original work. That's, in my understanding, the biggest difference
> between the two approaches.
>
> Regards,

Lets call it a half-way house for patches. Half way between the author
of the patch and the ocaml core team. A place to share patches with
other distributions prior to the ocaml core team including them.

MfG
Goswin

[Benedikt Meurer]

On Dec 10, 2011, at 00:24 , Goswin von Brederlow wrote:

>>> Whether you call it a "fork" or a "distribution" doesn't matter.
>>
>> It does, IMHO. Forking a project allows you to integrate more intrusive
>> changes, and doesn't force you to stay compatible (in some way) with the
>> original work. That's, in my understanding, the biggest difference
>> between the two approaches.
>>
>> Regards,
>
> Lets call it a half-way house for patches. Half way between the author
> of the patch and the ocaml core team. A place to share patches with
> other distributions prior to the ocaml core team including them.

I don't care for the name. I'm fine with "community distribution".

> MfG
> Goswin

greets,
Benedikt

[Xavier Leroy]

On 12/08/2011 10:10 AM, Benedikt Meurer wrote:

> Opening up the development of OCaml is a great suggestion, for
> example. Personally I'd even suggest to disconnect OCaml and INRIA,
> with an independent team of core maintainers (with appropriate spare
> time and knowledge). INRIA would still contribute to OCaml, but no
> longer control OCaml.

Oh, my! I go underwater for one week, first preparing a two-days training
course then delivering it deep inside an Airbus factory (without Wifi of
course so I had to read all that angst on my iPhone), and voila it's Arab
spring on the caml-list :-)

This discussion started on the wrong foot, and I don't see how I could
seriously consider Benedikt's plans given the amount of flaming and trolling
that surround it. Benedikt, you should realize that your first message was
aggressive, even taking German style into account, and that your second is
offensive. You barking louder isn't going to make me listen more. For
instance, here you're aiming for the jugular (and missing):

> For example, have a look at PR/3746, a great example. It took almost
> 4 years(!) to update the ARM port to softfp (and EABI). At the time
> the issue was finally fixed, most ARM application boards were
> already shipping with VFP, so the port is lacking behind several
> years.

It is a great example indeed, but of my dedication to supporting OCaml in
the long run. Yes, it took a long time, first because adapting ocamlopt to
a soft-FP model was absolutely not obvious (the code generator really wants
FP regs to hold FP values), second because I had to wait a long time before
an appropriate development and testing environment was available (namely,
Debian ARM EABI running inside QEMU), third because nobody really cared.
Your average maintainer would have given up at that point and canceled the
ARM port of OCaml because ARM's floating-point is a mess. I didn't and
spent one full week of my summer vacations recoding the ARM port instead.
I'm quite happy with the result, thank you very much, and disagree with your
"lacking behind" claim -- the only way that this ARM port can live in the
long run (and I definitely care, as my CompCert project needs OCaml on ARM)
is to stick with the lowest common denominator, namely soft FP.

> The relevant bug report PR/5404, which includes a backward
> compatible patch, is already waiting for a sign of life for 3 weeks
> now (maybe wait another 4 years to get the port fixed).

More bile. What's so urgent about it? The next release of OCaml is 3-6
months in the future; your suggestion will be examined by then. You know,
some people have busy professional lives. I can only devote 1 to 2 days per
months to OCaml maintenance, and would prefer to spend them on technical
work rather than on massaging egos or putting out the fires that you light
up on this list.

Having missed the jugular, you then aim for the groin:

> While we don't have a Drepper here in OCaml land (yet?) [...]

It's funny you mention Ulrich Drepper: I worked with him in the late 90's
integrating LinuxThreads version 1 into glibc, and he was courteous, to the
point, and frighteningly efficient. Draw your own conclusions.

Enough of that: my physician is already concerned about my blood pressure,
and this discussion is making me grow a stomach ulcer on top. So, let me
close with a few general messages to the esteemed Caml community.

1- Count your blessings and don't be ungrateful. With the exception of the
Caml consortium members, no one is paying for the work of the Caml core
development team, yet several of you have demands that are ridiculous even
if you were paying customers of a commercial product. (Minutes of the
internal discussions of the development team?? What's next, a live video
feed from my bathroom?)

2- As pointed out already in this discussion, it's not on the Caml compiler
that community efforts are most needed. For example, the most impactful
action that his community could take, in my opinion, is to adopt and embrace
a common packaging system like Cabal for Haskell.

3- I am not impressed by the ability of the Caml community to do actual
community work. We have N >= 2 (often N >> 2) independent implementations
of many things but a definite lack of convergence. For instance, I admire
Gerd Stolpmann's and Sylvain Le Gall's efforts on the packaging front, but
see very few people actually joining these efforts. For another example,
the core Caml development team sometimes asks for testing and opinions on
new directions (e.g. I on the x86-32bits-SSE2 code generator, or Jacques
Garrigue on GADT prototypes) and response rate is very low. It feels like
herding cats.

4- Yes, we obviously have problems with PR triaging, in part because Mantis
makes this task more bureaucratic than strictly necessary, but more
importantly because it is often hard to guess who cares about this or that
suggestion, or even what problem it is supposed to address. Volunteers
could greatly help by simply commenting on the PRs in Mantis, to express
support or disagreement, or to ask for clarifications.

5- Before embarking on patching the core Caml distribution, it wouldn't hurt
to ask (privately) where the priorities are. For instance, I don't see the
point for a linear-scan allocator (Benedikt Meurer) or more efficient
compilation of value let-rec (Fabrice le Fessant), but anyone who would come
up with a GHC-quality function inliner would be welcome like the Messiah.
Likewise, for many years I've been looking for developers to work on the
Windows port(s) of OCaml and never found any. Finally, at the latest OCaml
consortium meeting, the idea of splitting Camlp4 off the core distribution
was floated around; volunteers to take over its maintenance would be most
welcome.

All right. Let me stop here and pray for constructive, non-knee-jerk reactions.

- Xavier Leroy

[Benedikt Meurer]

On Dec 10, 2011, at 15:45 , Xavier Leroy wrote:

> This discussion started on the wrong foot, and I don't see how I could
> seriously consider Benedikt's plans given the amount of flaming and trolling
> that surround it. Benedikt, you should realize that your first message was
> aggressive, even taking German style into account, and that your second is
> offensive.

You are right, and I'm sorry for that. My first message wasn't meant to be aggressive.
Maybe we can get back to my original proposal and restart on the right foot this time?

Benedikt

[Török Edwin]

On 12/10/2011 04:45 PM, Xavier Leroy wrote:

> All right. Let me stop here and pray for constructive, non-knee-jerk reactions.

I am not an active member of the OCaml community, but I'll try to suggest
some (constructive I hope) solutions.

> see very few people actually joining these efforts. For another example,
> the core Caml development team sometimes asks for testing and opinions on
> new directions (e.g. I on the x86-32bits-SSE2 code generator, or Jacques
> Garrigue on GADT prototypes) and response rate is very low. It feels like
> herding cats.

>

> 5- Before embarking on patching the core Caml distribution, it wouldn't hurt
> to ask (privately) where the priorities are.
> For instance, I don't see the
> point for a linear-scan allocator (Benedikt Meurer) or more efficient
> compilation of value let-rec (Fabrice le Fessant), but anyone who would come
> up with a GHC-quality function inliner would be welcome like the Messiah.
> Likewise, for many years I've been looking for developers to work on the
> Windows port(s) of OCaml and never found any. Finally, at the latest OCaml
> consortium meeting, the idea of splitting Camlp4 off the core distribution
> was floated around; volunteers to take over its maintenance would be most
> welcome.

IMHO it would be useful if there was a single place that collects this information.
It could be something as simple as a webpage (or a wiki).
It'd contain a link to the relevant ML thread, and a (sort of) up-to-date status (for example:
testing on platform X is adequate, but we still need testing on platform Y).
- Testing Needed: URL of SVN branch/tag that needs to be tested, and list of platforms
- New language feature feedback: URL of SVN branch/tag that implements it, and what kind of feedback is expected
- Open Projects for improving the compiler (i.e. the GHC-quality inliner above)
- Open Projects for improving the standard library
- Low-Priority projects: i.e. things that were suggested by the community, but are not needed right now,
perhaps with a one-line motivation (i.e. the regalloc you mention)
- Roadmap for next OCaml version: a glimpse of what can be expected for the next major/minor version
- Pings: link to ML thread that didn't receive enough feedback, and it would be important to get it soon.
Or topics that go ignored for a long time (i.e. help with Windows port you mention above)

People could check that from time to time, and see where they could help, and more importantly
see where the core team needs help.

It could be something as simple as a webpage (linked from caml.inria.fr)
that contains a link to the relevant mailing list thread, or a wiki page that can only be edited by the core team.

If maintaining that webpage would take up too much time, then perhaps someone from the community could
volunteer to maintain it. The core team would only send an email with a specific subject tag to the regular ML,
and post updated the same way (testing on platform X is adequate, we still need more on platform Y)
Subject: [ Ping ] [ Testing Needed | Language feature feedback .... ] <.....>
CC: communitymaintainer@....

>
> 4- Yes, we obviously have problems with PR triaging, in part because Mantis
> makes this task more bureaucratic than strictly necessary, but more
> importantly because it is often hard to guess who cares about this or that
> suggestion, or even what problem it is supposed to address. Volunteers
> could greatly help by simply commenting on the PRs in Mantis, to express
> support or disagreement, or to ask for clarifications.

Would it help if people wrote small patches for the bugreports they care about
(at least for bugreports that are not about the core parts of the compiler)?
For example fixes/improvements to ocamlbuild, or I've just noticed a French message from ocamldoc yesterday, etc.
Or does reviewing these patches take up almost as much time as writing it yourself?

Best regards,
--Edwin

[Jérémie Dimino]

Le samedi 10 décembre 2011 à 15:45 +0100, Xavier Leroy a écrit :
> 5- Before embarking on patching the core Caml distribution, it wouldn't hurt
> to ask (privately) where the priorities are. For instance, I don't see the
> point for a linear-scan allocator (Benedikt Meurer) or more efficient
> compilation of value let-rec (Fabrice le Fessant), but anyone who would come
> up with a GHC-quality function inliner would be welcome like the Messiah.
> Likewise, for many years I've been looking for developers to work on the
> Windows port(s) of OCaml and never found any. Finally, at the latest OCaml
> consortium meeting, the idea of splitting Camlp4 off the core distribution
> was floated around; volunteers to take over its maintenance would be most
> welcome.

I am volunteer for the maintenance of Camlp4.

But there is something i don't understand here. Why is there camlp4 and
camlp5 ? These two projects do exactly the same thing and are
incompatible. So i don't see the point of maintaining them both. We
should at least deprecate one.

Cheers,

--
Jérémie

[Wojciech Meyer]

Hi,

Jérémie Dimino <jer...@dimino.org> writes:

> But there is something i don't understand here. Why is there camlp4 and
> camlp5 ? These two projects do exactly the same thing and are
> incompatible. So i don't see the point of maintaining them both. We
> should at least deprecate one.

BTW: Are there any plans to add type reflection to Camlp4 (or Camlp5)?

>
> Cheers,
>
> --
> Jérémie

Cheers;
Wojciech

[Wojciech Meyer]

Wojciech Meyer <wojciec...@googlemail.com> writes:

> Hi,
>
> Jérémie Dimino <jer...@dimino.org> writes:
>
>> But there is something i don't understand here. Why is there camlp4 and
>> camlp5 ? These two projects do exactly the same thing and are
>> incompatible. So i don't see the point of maintaining them both. We
>> should at least deprecate one.
>
> BTW: Are there any plans to add type reflection to Camlp4 (or Camlp5)?

Hi Jérémie,

Sorry about OT.

I'm asking, because certainly it would be a very wanted feature. I can
see two major limitations of the current Camlp4/p5 system:

- no way of recursively expand syntax, generate some code and then
re-generate again using same syntax (staging, it's partially there as
you can compose syntax extensions, but there is no way to keep
expanding until fixpoint)

- no way of making an action based on inferred type during previous stage
of macro expansions. This is biggie. It needs either using `ocaml -i' hack
or kind of type reflection support in the compiler.

Also, a packrat lexless parsers with a way to compose them would be a
huge thing too.

I'm aware that these are huge changes to Camlp4, but it would make
meta programming more powerful and push Camlp4 to the next level.

People will mention about MetaOCaml (which is an excellent way for meta
programming in OCaml), but here I'm more interested in syntactical
abstraction than partial evaluation.

[Jérémie Dimino]

Le samedi 10 décembre 2011 à 19:10 +0000, Wojciech Meyer a écrit :
> I'm asking, because certainly it would be a very wanted feature. I can
> see two major limitations of the current Camlp4/p5 system:
>
> - no way of recursively expand syntax, generate some code and then
> re-generate again using same syntax (staging, it's partially there as
> you can compose syntax extensions, but there is no way to keep
> expanding until fixpoint)
>
> - no way of making an action based on inferred type during previous stage
> of macro expansions. This is biggie. It needs either using `ocaml -i' hack
> or kind of type reflection support in the compiler.
>
> Also, a packrat lexless parsers with a way to compose them would be a
> huge thing too.
>
> I'm aware that these are huge changes to Camlp4, but it would make
> meta programming more powerful and push Camlp4 to the next level.

Sure. But it seems that the next version of OCaml will have runtime
types, see http://www.lexifi.com/blog/runtime-types , so maybe it is not
needed to add this to camlp4.

Also they are problems that i don't know how to solve with the camlp4
approach. For example consider:

let x = 1
type int = A
let y = A

The typer knows that x has the type (int, 1) and y has the type (int,
42). But what you send to ocaml is a parse tree, and you cannot make
this difference in the parse tree.

Cheers,

--
Jérémie

[Wojciech Meyer]

Jérémie Dimino <jer...@dimino.org> writes:

> Le samedi 10 décembre 2011 à 19:10 +0000, Wojciech Meyer a écrit :
>> I'm aware that these are huge changes to Camlp4, but it would make
>> meta programming more powerful and push Camlp4 to the next level.
>
> Sure. But it seems that the next version of OCaml will have runtime
> types, see http://www.lexifi.com/blog/runtime-types , so maybe it is not
> needed to add this to camlp4.

It's interesting and I didn't know about it. However, the problem is
slightly different, I would like to know the typing of a freshly
generated piece of code by Camlp4 in the previous phase. Then, have
pattern matching against these meta types in annotated AST and produce
another AST, which in turn have most likely another typing and pass to
the next phase etc. I would say that Camlp4 is fine for the simpler
syntax extensions and majority of small DSLs but when you start composing
syntax extensions and macros it quickly becomes a problem.

>
> Also they are problems that i don't know how to solve with the camlp4
> approach. For example consider:
>
> let x = 1
> type int = A
> let y = A
>
> The typer knows that x has the type (int, 1) and y has the type (int,
> 42). But what you send to ocaml is a parse tree, and you cannot make
> this difference in the parse tree.

Yes, you would need a type information in the parse tree as mentioned
before, so you want to feed up the compiler with AST start unrolling
macros top-down and then follow up with the inferred types bottom-up.

Cheers;
Wojciech

[Jesper Louis Andersen]

On Sat, Dec 10, 2011 at 15:45, Xavier Leroy <Xavier...@inria.fr> wrote:

> 2- As pointed out already in this discussion, it's not on the Caml compiler
> that community efforts are most needed.  For example, the most impactful
> action that his community could take, in my opinion, is to adopt and embrace
> a common packaging system like Cabal for Haskell.

This one piqued my interest somewhat. I should refrain from
commenting, but I can't help myself.

What makes the Cabal infrastructure successful is that it is three
different types of systems in one. First, there is hackage which
provides a package repository of "blessed" packages - online search of
the repository and general dumping ground of rotting packages. In
other words, a perl CPAN for Haskell. Quality varies a lot, but at
least there is a go-to place when searching for packages.

Second, Cabal provides a system for automatically building packages.
That is, if you have cabal, chances are that you can just blindly ask
it to build the source code and you are done.... when it works.

Thirdly, Cabal provides dependency tracking among packages. That is,
given a package, you can easily fetch its dependencies and build
another version of the package yourself.

That said, my personal opinion, which probably not means much as I
don't currently write scores of Ocaml or Haskell source, is still that
Cabal is somewhat overengineered. Usually you end up with trouble when
you try to build new packages. If you forget profiling flags, you have
to rebuild essentially everything. Multiple versions of packages
creates headaches when trying to use them (think the diamond-problem
in a subtyping setting with multiple inheritance or functors without
sharing constraints). And when I build Haskell source, I usually end
up cursing the hell out of Cabal :)

Erlang has a tool, doing parts two and three above: rebar. This tool
is fundamentally simpler than Cabal. Its dependency tracking is
essentially download instructions from software repositories and
that's it. Packaging is done by creating the equivalent of static
binaries .. dependencies are shoved into a release of an erlang
runtime with all the necessary modules ready into a selfhosting,
selfcontaining system. The approach is *much* simpler, but it works
tremendously well. Even if it doesn't handle the problem of the
diamond. It also allows different software to use different versions
of libraries, but it doesn't really solve dependency convergence (that
is yet another name for the diamond problem: A needs B and A needs C -
but B needs D in version 1 and C needs D in version 2):

digraph {
D -> B -> A;
D -> C -> A
}

In my experience, it is the simple solutions that works best, even if
they can't handle certain scenarios. In some sense a certain amount of
limitation is more a virtue than a curse. I'd also suggest splitting
the search and indexing part from the dependency tracking and building
part. They are far enough apart that they can be solved each by
themselves.

Finally there are two other things I would recommend one investigates:
FreeBSD ports or Mac OSX homebrew for the indexing part. And pythons
virtualenv (Ian Bicking was the original author). Virtualenv allows
you to build a "virtual environment" in which you install python
packages, which are distinct from the system installation. Of course
you can symlink system libraries into the virtual environment, but it
also means that programs can live next to each other and the
sandboxing going on solves a lot of dependency problems in a simple
way.

In other words, to solve this, I'd much rather try to rig the
excellent Ocamlbuild and Ocamlfind infrastructure such that it
incorporates some of the above things, ... that is if it doesn't do
that already...

--
J.

[Gabriel Scherer]

A summary to this lengthy mail:
(1) Why type-enriched Camlp4 is an unreasonable idea
(2) We should extract the typedtree; why it's hard
(3) A fictional narrative of the camlp4/camlp5 history
(4) Why you don't want to become Camlp4 maintainer
(5) How we could try not to use Camlp4 in the future
(6) Syntax extension survival advices

# (1) Why type-enriched Camlp4 is an unreasonable idea

Wojciech, your idea of having type information at the Camlp4 level is
absolutely unreasonable. You are not speaking about a "minor change"
here, but a major rewrite that would affect the compiler internals as
well. It would really be a new (and interesting) project.

Camlp4 is, and I guess will remain, a syntax-level preprocessing
tool. You have to accept the fact that you can't use type information
at this level (but you can certainly "interact" in some way with the
type system by producing/transforming pieces of code in a way that you
know will have interesting typing effects; for example, you may want
to generate code that is purposedly ill-typed in some cases, to
disallow certain uses of your syntax extension). I'm not even sure
what it would mean to access type information at the camlp4 level, as
you're producing and transforming untyped AST; would you want
partially typed ASTs? How is the typer supposed to work on the part
that you haven't transformed yet, and therefore are not valid OCaml
syntax? I suppose you could have a "preprocessing and transformation"
tool at the typedtree level, but that would be a different tool with
different uses, distinct from the syntactic preprocessing part (though
you may develop "extensions" that act on both fronts).

I'm not aware of so much Camlp4 situations that would really require
typing information. I would be interested in good examples if you have
some. One problem that I have had with Camlp4 is that you don't have
identifier resolution information (eg. you don't know if the
identifier "(@)" you're seeing is really list concatenation, or has
been redefined/shadowed in the context); this makes uses of Camlp4 for
inlining, for example, quirky and fragile. That's still a simpler
problem than type information.



# (2) We should extract the typedtree; why it's hard

If you really want to play with type information and scope-resolved
identifiers, AST-manipulating tools is probably not the way to go: you
indeed want full access to the typedtree. Currently this is only
possible by hacking the compiler, and this is what for example Jun
Furuse's Ocamlspotter project does. Those kind of tools could be made
less intrusive if it was possible to pass typedtree-like information
in and out of the compiler.

I remember reading that some people (OcamlPro, I suppose) have this on
their target list. The problem however is that the current internal
compiler's typedtree representation is not at all adapted for external
communication. If you want a kind of tool that is robust and
future-proof in any sense (you could probably get something working by
just marshalling the current typedtree, but then it could break
awfully after minor language changes, make the compiler choke, etc.;
I certainly wouldn't want to use that), you have to design a clean and
efficient representation for OCaml programs after the type inference
phase. Having a solid proposal on this topic wil be an awful lot of work.



(3) A fictional narrative of the camlp4/camlp5 history

Jérémie Dimino wrote:
> But there is something I don't understand here. Why is there camlp4 and

> camlp5 ? These two projects do exactly the same thing and are
> incompatible. So i don't see the point of maintaining them both. We
> should at least deprecate one.

Let me repeat the story as I know it (with possible mistakes, I was
still a caml baby in the <3.10 times) in a hopefully compact form for
those on the list who have no idea about it. DISCLAIMER: this is only
a fictional storytelling, meant to give a reasonable idea (or at least
my vision) of the situation. I may be wrong about the events
chronology, people name, hard facts, and of course english spelling
and grammar. The story is complicated and I don't know the gory
details.
If you know a better story, feel free to add important precisions,
correct the obvious mistakes, etc. I also welcome suggestions to make
it a funny, entertaining read; finally, a few romantic details could
clearly turn it into a blockbuster.


The original Camlp4 tool was mostly developped by Daniel de
Rauglaudre. Apparently, personal relations between Daniel and the
OCaml team were not easy, and Camlp4 was gradually becoming more and
more external to the OCaml distribution (in the past, the stream
syntax was available as part of the core language, but it was moved to
Camlp4; the Oreilly book was written before that move) and its
maintainance status incertain.

In the 3.09/3.10 transition, Nicolas Pouillard started working on
a refactoring of the Camlp4 codebase (which was mostly a silent,
non-moving animal at that time) to make it more easy to evolve and
maintain. The refactoring maybe went "a bit too far", in that it
brought a number of changes to the external interface and, in
particular, broke existing Camlp4 extension, as well as the (quite
good) Camlp4 documentation. Of course they also were advantages, in
that the new design was modular and, for example, the bootstrapping
process was made easier, and Nicolas Pouillard could maintain the tool
as offered by the distribution. The 3.10 transition was however very
painful for people using existing Camlp4 extensions (I'm thinking of
eg. Martin Jambon, which had extensive Camlp4 extensions, and the Coq
team which has user-defined notations using Camlp4 and, huh, I really
don't want to know the details); basically they didn't upgrade to
3.10 -- instead of porting the extensions, as was originally hoped.

Personal note: I learned Camlp4 in this period, just after the release
of OCaml 3.10. I'm honestly unaware of how pre-3.10 Camlp4 was (though
I guess it is not too difficult to move from one to the other) so my
interpretations of previous times are all based on reading the
mailing-list.

Daniel, which apparently did not agree with some of the changes made,
relatively suddenly restarted developpment of "his" branch of Camlp4,
taken from the old sources, before refactoring. This was done as
a separate project, outside the OCaml distribution (apparently Daniel
and the OCaml team prefer not to work together). In a quite creative
move, he named his version "camlp5" so that it could be easily
distinguished from the "upstream camlp4", the incompatible new version
being distributed with OCaml. Camlp5 is therefore the continuation of
the *old* camlp4. Development, however, continued (while still
preserving or mostly preserving compatibility with pre-3.10
extensions, ensuring grateful thanks from the users) with
non-neglectible changes (eg. addition of a library of non-destructive
streams), and is currently ongoing (as is maintainance of pre-3.10
camlp4 in the OCaml distribution).
http://cristal.inria.fr/~ddr/camlp5/CHANGES

Of the projects that relied on Camlp4 before 3.10, some of them ported
they extension to >=3.10 Camlp4, and lived happily ever after, and
some of them jumped on camlp5 as a lower-cost migration opportunity
and, I suppose, also lived happily ever after; or at least, as happily
as you can knowing that your codebase depends on camlp{4,5}.

I make no technical judgments of which version (Camlp4 or Camlp5) is
"better". I know and (try not to) use Camlp4. I also understand the
position of Camlp5 users for compatibility reasons. I would advise
newcomers to try Camlp4 first as it has a larger user base (being
distributed with the OCaml distribution). Anyway, see at the end of
this mail why you may not want to use camlp* anyway.


# (4) Why you don't want to become Camlp4 maintainer

> I am volunteer for the maintenance of Camlp4.

Jérémie, I am deeply impressed by your sense of sacrifice.

Camlp4 is a piece of devil beauty. It does incredibly clever things,
and is incredibly complex inside: Daniel is clearly a remarkable
hacker, but his code is not easy to understand. I know that
maintaining the whole thing is very hard; and that is the reason why
Camlp4 tends to have problems to bump from one version to another,
when non-neglectible syntaxic changes are made to the language.

The core internals of Camlp4 are quite imperative: parsing is done by
erasing tokens from the input stream (which makes for lovely Stream
debugging), but, more importantly, grammar extensions are destructive
mutations of the grammar. It is sometimes painful when using Camlp4,
and I suppose it must be hell maintaining it.


# (5) How we could try not to use Camlp4 in the future

I think Alain's idea of moving out of Camlp4 is actually a good
thing. After having used Camlp4 extensively, I came out with the
general feeling that allowing general extension of the grammar is not
a good thing, because it is too complex and the solutions are too
fragile. On the contrary, the quotation mechanism of Camlp4 is
a wonderful tool, that acts as a *controlled* extension point for the
grammar, and is therefore much more robust. The other excellent use of
Camlp4 is generating OCaml code out of "annotations" on certain AST
nodes, as the 'type-conv' extension does; this is also a form of
controlled extensibility that could, I think, be taken into account in
the base language or a simple tool, and doesn't require full-fledged
grammar mutation.

I think we should isolate such restricted uses of syntax extensibility
and allow them through simpler, more robust tools. Alain's idea, and
the reactions from Camlp4 users (Martin Jambon, Nicolas Pouillard),
can be found on his blog:
http://www.lexifi.com/blog/syntax-extensions-without-camlp4

I'm personally not sure his solution is quite ready to replace Camlp4
yet; or at least the state in which it was at this time. He has an
annotation mechanism, but it is somehow not restricted enough to
guarantee a reasonable behaviour; as long as people are tempted to use
"annotations" to define try..finally, we will have fragile extensions
floating around.

(And I'm not sure it's a good idea to move Camlp4 out of the
distribution as long as we don't have a viable alternative proposal
and some users have started moving to it. Camlp4 will still need to be
supported by someone anyway, and it needs to evolve in lockstep with
OCaml language changes.)

Yet I do agree that "without Camlp4" is the future. It may take some
time, but what I would personally like to see in a few years is an
OCaml world where we don't need Camlp4 anymore, because we have other,
simpler tools to do the reasonable things, and have learned to live
without the rest. The whole "maintain Camlp4" entreprise will still be
useful, necessary until that time, but it won't stay, I hope, at the
center of attentions.


# (6) Syntax extension survival advices

To the reader considering use of a new syntax extension in his next project:

- don't

- if you really must, try to make sure that your code is also
reasonable to use *without* a syntax extension (eg. by producing
a library with a clean interface, making your extension desugar to
uses of it, but also making sure that it can be used by the human
user)

- if you really must, try to get it in the form of a quotation; the
rest is fragile

- alternatively, try to branch yourself on existing "flexible" syntax
extensions such as Markus Mottl's 'type-conv' and Jeremy Yallop's
'patterns': you are relatively safe if you don't write any line of
code modifying OCaml's syntax yourself.

http://hg.ocaml.info/release/type-conv
http://code.google.com/p/ocaml-patterns/

- do not hesitate to send for code review and/or ask for help on the
list

- don't

[Wojciech Meyer]

Gabriel Scherer <gabriel...@gmail.com> writes:

> A summary to this lengthy mail:
> (1) Why type-enriched Camlp4 is an unreasonable idea
> (2) We should extract the typedtree; why it's hard
> (3) A fictional narrative of the camlp4/camlp5 history
> (4) Why you don't want to become Camlp4 maintainer
> (5) How we could try not to use Camlp4 in the future
> (6) Syntax extension survival advices
>
> # (1) Why type-enriched Camlp4 is an unreasonable idea
>
> Wojciech, your idea of having type information at the Camlp4 level is
> absolutely unreasonable. You are not speaking about a "minor change"
> here, but a major rewrite that would affect the compiler internals as
> well. It would really be a new (and interesting) project.

Hello Gabriel,

I agree would be a serious changes, and I was thinking even of experimenting
a bit with this kind of strictly typed meta programming. It's perfectly
viable, as I've seen some good examples in my life. (and Template
Haskell does it AFAIR).

>
> Camlp4 is, and I guess will remain, a syntax-level preprocessing
> tool. You have to accept the fact that you can't use type information
> at this level (but you can certainly "interact" in some way with the
> type system by producing/transforming pieces of code in a way that you
> know will have interesting typing effects; for example, you may want
> to generate code that is purposedly ill-typed in some cases, to
> disallow certain uses of your syntax extension). I'm not even sure
> what it would mean to access type information at the camlp4 level, as
> you're producing and transforming untyped AST; would you want
> partially typed ASTs? How is the typer supposed to work on the part
> that you haven't transformed yet, and therefore are not valid OCaml
> syntax? I suppose you could have a "preprocessing and transformation"
> tool at the typedtree level, but that would be a different tool with
> different uses, distinct from the syntactic preprocessing part (though
> you may develop "extensions" that act on both fronts).
>
> I'm not aware of so much Camlp4 situations that would really require
> typing information. I would be interested in good examples if you have
> some. One problem that I have had with Camlp4 is that you don't have
> identifier resolution information (eg. you don't know if the
> identifier "(@)" you're seeing is really list concatenation, or has
> been redefined/shadowed in the context); this makes uses of Camlp4 for
> inlining, for example, quirky and fragile. That's still a simpler
> problem than type information.

Of course worth to point out MetaOCaml and MetaML. They do runtime
type safe staged meta programming.

Good example would be:
let t = (1,2,3,4) in
map_tuple t (fun x -> x*2)

(at this point I deliberately chosen closure and not additional syntax
extension because you can pass closures and you can't pass code easily
in Camlp4, because of the mentioned single stage meta programming).

of course it's possible with dependent types and lists, but with meta
programming you could just expand map_tuple to the code you want, *but*
only when the type information is available. (or you can infer the type
yourself, in the simplest case if the tuple was expression passed
directly to map_tuple).

Currently I believe that DSLs are the best way of achieving higher level
abstractions, because you are no longer bound to the language
syntax. And you can stay distant from the semantics of your target
language.

> # (6) Syntax extension survival advices
>
> To the reader considering use of a new syntax extension in his next project:
>
> - don't

Personally I disagree with this statement, because:

- syntax extension allows you to do certain things that are not available
in a normal way, to enumerate: list comprehensions, pa_lwt, pa_where,
pa_monad, pa_js. So they are clearly very useful in some cases.

- combinators in particular monads allow certain syntactical
abstractions, however there are runtime costs because of constructing
closures and evaluating them later. These runtime costs can easily
trimmed to some level with some compiler optimisations for instance
good inlining (that we don't have as mentioned before in the thread) and
efficiency of runtime (we do have a very good gc).

- compilation is much easier than interpretation. You see the code
generated and the code generated is restricted semantically by the
target language. Handling environment is way lower level than just
generating some let bindings in a quotation.

-

>
> - if you really must, try to make sure that your code is also
> reasonable to use *without* a syntax extension (eg. by producing
> a library with a clean interface, making your extension desugar to
> uses of it, but also making sure that it can be used by the human
> user)
>
> - if you really must, try to get it in the form of a quotation; the
> rest is fragile

I agree that quotations are the best part of Camlp4. Right now I use
them to generate a lot of ML code (more than hundreds of KLOC) out of data
oriented DSL that parsing is detached from the official Camlp4 parser (I
use Menhir for that task). It works pretty well and the code generator
is pretty compact.

>
> - alternatively, try to branch yourself on existing "flexible" syntax
> extensions such as Markus Mottl's 'type-conv' and Jeremy Yallop's
> 'patterns': you are relatively safe if you don't write any line of
> code modifying OCaml's syntax yourself.
>
> http://hg.ocaml.info/release/type-conv
> http://code.google.com/p/ocaml-patterns/
>
> - do not hesitate to send for code review and/or ask for help on the
> list

Additionally in case of if you need any help, I would volunteer to help.

[Stéphane Glondu]

Le 11/12/2011 00:34, Gabriel Scherer a écrit :
> The original Camlp4 tool was mostly developped by Daniel de
> Rauglaudre.

> [...]

> (I'm thinking of
> eg. Martin Jambon, which had extensive Camlp4 extensions, and the Coq
> team which has user-defined notations using Camlp4 and, huh, I really
> don't want to know the details); basically they didn't upgrade to
> 3.10 -- instead of porting the extensions, as was originally hoped.

> [...]

> Daniel, which apparently did not agree with some of the changes made,
> relatively suddenly restarted developpment of "his" branch of Camlp4,
> taken from the old sources, before refactoring. This was done as
> a separate project, outside the OCaml distribution (apparently Daniel
> and the OCaml team prefer not to work together).

> [...]

> Camlp4 is a piece of devil beauty. It does incredibly clever things,
> and is incredibly complex inside: Daniel is clearly a remarkable
> hacker, but his code is not easy to understand. I know that
> maintaining the whole thing is very hard; and that is the reason why
> Camlp4 tends to have problems to bump from one version to another,
> when non-neglectible syntaxic changes are made to the language.

> [...]

> (And I'm not sure it's a good idea to move Camlp4 out of the
> distribution as long as we don't have a viable alternative proposal
> and some users have started moving to it. Camlp4 will still need to be
> supported by someone anyway, and it needs to evolve in lockstep with
> OCaml language changes.)

Coq has been adapted to work with both camlp4 and camlp5. In my
experience, Daniel has been much more responsive on camlp5 matters than
the OCaml team on camlp4 matters. And Xavier's mail suggests that camlp4
is a maintenance burden for the OCaml team.

Why is it such a bad idea to drop camlp4 out of the distribution, and
just let camlp5 live? My feeling is that people using camlp5 care about
it, but people using camlp4 do so just because it's in the official
distribution and wouldn't care switching to camlp5 (or stop using this
kind of syntax extension) if camlp4 was officially deprecated in favor
of camlp5. At worst, someone will start maintaining camlp4 independently.

Daniel already maintains and develops camlp5. I guess Jérémie would also
volunteer to join him (but he already offered to maintain camlp4
independently). In Debian, there are currently 49 packages depending on
camlp4, and 7 depending on camlp5. That also increases incentive /
possible help for maintaining it.


Cheers,

--
Stéphane

[Török Edwin]

There's another use of camlp4 besides syntax extensions: better
syntax errors, and the alternate syntax.

While learning OCaml I've used camlp4o quite a few times to figure
out what is wrong with the code, as ocamlc just gives you a location and "Syntax error",

In fact learning the alternate syntax, and using camlp4r was easier for me,
because the syntax errors were closer to the actual place of the error.
With the original syntax the errors are often reported late, for example a typo of ";" instead
of ";;" can cause syntax errors to be reported in what you'd consider to be another function.
Eventually I learned how the original syntax works, but I still find camlp4o to be a time saver
sometimes.

If camlp4 is dropped from the core distribution can these improved syntax error messages
be merged into the core parser? Also what will happen with the alternate syntax?

Best regards,
--Edwin

[Fabrice Le Fessant]

On 12/11/2011 12:34 AM, Gabriel Scherer wrote:
> A summary to this lengthy mail:
> (1) Why type-enriched Camlp4 is an unreasonable idea
> (2) We should extract the typedtree; why it's hard
> (3) A fictional narrative of the camlp4/camlp5 history
> (4) Why you don't want to become Camlp4 maintainer
> (5) How we could try not to use Camlp4 in the future
> (6) Syntax extension survival advices

Sorry, it's too long to read entirely for a sunday morning, so I reply
just after reading points (1) and (2):

(1) many extensions in Camlp4 are used to manipulate types (most of them
based on type-conv). So, having the compiler knowledge of types at a
point where you want to expanse a macro is actually very interesting for
them. I wrote a patch to 3.12.0 a long time ago called "ocaml-template"
that would exactly do that: it would call a plugin function at some
points in the program, giving the function both the string to parse and
the typing environment, so that it is possible (to some extend, since
typing might only be partial at the point where it is called) to know
the type of an identifier, and the complete description of the type.

Here is the link to the patch:

http://www.ocamlpro.com/files/ocaml-template-0.1-for-3.12.0.tar.gz

It is not documented much, but there are some simple examples of plugins
and uses.

(2) exporting the typedtree is not really a problem. The main problem is
to add enough information in the typedtree for all the tools that could
use it (so that ocamlspotter and refactoring tools can use it). We are
currently in the process of polishing our final patch to submit it for
integration, now that we have tested it with enough tools.

--Fabrice

[Gabriel Scherer]

> And Xavier's mail suggests that camlp4 is a maintenance burden for the OCaml team.
> Why is it such a bad idea to drop camlp4 out of the distribution, and
> just let camlp5 live?

First of all, I don't have a strong opinion here: I just voiced
doubt. My reasoning for going so goes along two lines of argument:

1. I'm not exactly sure how Camlp4 being in or out of the distribution
will change the maintainance burden. The main maintainance
difficulty with camlp{4,5} is that it needs to evolve its own
parser in parallel with the 'official' one (that's by design), with
changes to the language syntax. You need to change Camlp{4,5} when
Ocaml 3.N+1 introduces a new syntax, or it won't be usable on OCaml
3.N+1 code.

There are already users relying on Camlp{4,5}, and those user
generally wish to use the new, exciting features of the next
version. If they can't, they will complex, regardless of whether
their preprocessor is in or out the distribution. That means that
when the OCaml team is about to release a new version with
syntactic changes, they have to worry about the preprocessor
anyway, or make users unhappy.

So there is a "preprocessor burden" on the OCaml team, independently
of where the code is maintained and located. If the change means
"make it easier to distribute camlp4 fixes without bumping OCaml's
version number", why not. If it means "now we won't care about
Camlp4 state before releasing a new versions", this may mean
a degradation in the life of Camlp4 users. I doubt that's the idea.

Being in or out the distribution also wouldn't change much, I think,
the possibility of external contributions. In my experience Nicolas
Pouillard and now Xavier Clerc have a good track record of
integrating external contributions (I have sent one or two bugfixes
on the tracker), and I'm confident they would be able to work with
Jérémie Dimino if he wished to contribute to camlp4's evolution more
frequently.

2. I suppose -- purely personal guess -- the intention being the
consortium's suggestion is to try to move away slowly from
Camlp{4,5}, towards alternatives such as Alain Frisch's
"annotations" proposal. As I said previously, I would personally
welcome such a move, but I don't see said alternatives released
yet. I haven't had the opportunity to play with alternate tools,
have an idea of how a transition would work out, see if the
documentation is reasonably complete, etc.

It would make more sense, in my opinion, to downplay camlp{4,5}
*once* we have played with alternatives and are confident that they
are mature enough to make a transition.

Please also remember that the consortium members represent
relatively large, well-educated, experienced players in the OCaml
community. It probably wouldn't bother them much if, say,
ocamlbuild, ocamldoc, or ocamldbg where taken out of the official
distribution. They have important tooling in place and would adapt
relatively easily. The end user or OCaml beginner may not adapt to
such changes that easily. Now this is a matter where distributions,
such as Debian, can be of great help, by providing "complete"
packages regardless of what is or isn't in the official
distribution. However, I have handled just enough users reports
wondering why they didn't have "camlp4o" available, or
graphics.cma, or whatever, to know that this can also be a barrier
to use and adoption.


Again, no strong opinion. I will welcome any change that strenghten
the OCaml language. If you think distributing camlp4 out of the
distribution would ease the live of OCaml developpers and maintainers,
at no cost to the users, nor complicating the distribution side, then
all is good. I just feel that it may be a bit too soon.

[Gabriel Scherer]

> (1) many extensions in Camlp4 are used to manipulate types (most of them
> based on type-conv). So, having the compiler knowledge of types at a
> point where you want to expanse a macro is actually very interesting for
> them.

That's not my understanding of how type-conv works. From what I know
of it, type-conv uses syntactic definition of types to generate
additional phrases (functions in particular) derived from those
definitions. It doesn't rely at preprocessing-time on the type of
program expressions.

Now type-conv may have evolved since the last time I looked at
it. I would welcome any pointer to additional type-environment-aware
features.

Thanks for your patch. It looks interesting, but I wasn't able to read
the diff directly (it's 16MB and unreadable without applying). Maybe
you should publish the examples separately, to make it easier to look
at.

How far are the expected applications from Alain Frisch's "dynamic
type representation" proposal? The way I understand it, he proposes an
in-language construct to inject a dynamic type representation based on
type-level information at some point of the program. You can then use
OCaml code directly to manipulate the representation and use
type-generic functions. I suppose there are cases where working
outside the language is easier (ie. you don't have your transformation
type-checked).

> (2) exporting the typedtree is not really a problem. The main problem is
> to add enough information in the typedtree for all the tools that could
> use it (so that ocamlspotter and refactoring tools can use it).

I was afraid there could be sharing issue, and more importantly that
the internal invariant of the compiler (levels, unique names) could be
difficult to manipulate and preserve. It's good to know those can be
avoided in practice.

[Gerd Stolpmann]

Am Sonntag, den 11.12.2011, 00:28 +0100 schrieb Jesper Louis Andersen:
> On Sat, Dec 10, 2011 at 15:45, Xavier Leroy <Xavier...@inria.fr> wrote:
>
> > 2- As pointed out already in this discussion, it's not on the Caml compiler
> > that community efforts are most needed. For example, the most impactful
> > action that his community could take, in my opinion, is to adopt and embrace
> > a common packaging system like Cabal for Haskell.
>
> This one piqued my interest somewhat. I should refrain from
> commenting, but I can't help myself.

It's interesting to hear these comments. In short, in ocaml we have
these options:

- Full featured package management: we have this (GODI), but the
community is not too large (package maintainers are always
welcome!) As every full-featured system, GODI has complicated
parts, and you need to invest 1-2 days before you fully
understand it and can build packages.

- Simplified package management: This is what OASIS-DB tries to do.
Fewer features, but it is easier to create new packages. Hopefully,
it will attract more developers. OASIS-DB can be integrated into
full-featured systems like GODI or Debian, so the idea is also
to avoid duplicate work.

- Bottom end: Findlib and the build tools (ocamlbuild and omake).
This already includes dependency management on the "compiler
switch level"

This means, the situation is not too bad. OASIS-DB is the missing link
between the build tools and the high-level packaging systems.

> What makes the Cabal infrastructure successful is that it is three
> different types of systems in one. First, there is hackage which
> provides a package repository of "blessed" packages - online search of
> the repository and general dumping ground of rotting packages. In
> other words, a perl CPAN for Haskell. Quality varies a lot, but at
> least there is a go-to place when searching for packages.

For GODI we have docs.camlcity.org. Once there is OASIS-DB, these
packages will also be covered (automatically).

> Second, Cabal provides a system for automatically building packages.
> That is, if you have cabal, chances are that you can just blindly ask
> it to build the source code and you are done.... when it works.
>
> Thirdly, Cabal provides dependency tracking among packages. That is,
> given a package, you can easily fetch its dependencies and build
> another version of the package yourself.
>
> That said, my personal opinion, which probably not means much as I
> don't currently write scores of Ocaml or Haskell source, is still that
> Cabal is somewhat overengineered. Usually you end up with trouble when
> you try to build new packages. If you forget profiling flags, you have
> to rebuild essentially everything.

Sounds more like it is underengineered - or leaves too many options for
the users to create errors.

However, there is certainly always a complexity problem in full-featured
package managers. We see this problem in GODI, too, and this is probably
why it is hard to convince people to help creating packages. Once a
package exists, GODI is very simple to use, but getting there is not
completely easy.

> Multiple versions of packages
> creates headaches when trying to use them

Well, I do not know any package manager that supports this feature well.
Finally, you have to decide to use a certain version.

> (think the diamond-problem
> in a subtyping setting with multiple inheritance or functors without
> sharing constraints). And when I build Haskell source, I usually end
> up cursing the hell out of Cabal :)
>
> Erlang has a tool, doing parts two and three above: rebar. This tool
> is fundamentally simpler than Cabal. Its dependency tracking is
> essentially download instructions from software repositories and
> that's it. Packaging is done by creating the equivalent of static
> binaries .. dependencies are shoved into a release of an erlang
> runtime with all the necessary modules ready into a selfhosting,
> selfcontaining system. The approach is *much* simpler, but it works
> tremendously well.

Sounds like this is more on the level of OASIS-DB.

> Even if it doesn't handle the problem of the
> diamond. It also allows different software to use different versions
> of libraries, but it doesn't really solve dependency convergence (that
> is yet another name for the diamond problem: A needs B and A needs C -
> but B needs D in version 1 and C needs D in version 2):
>
> digraph {
> D -> B -> A;
> D -> C -> A
> }

Right, this is what makes it difficult to support several versions in
the same repo.

> In my experience, it is the simple solutions that works best, even if
> they can't handle certain scenarios. In some sense a certain amount of
> limitation is more a virtue than a curse. I'd also suggest splitting
> the search and indexing part from the dependency tracking and building
> part. They are far enough apart that they can be solved each by
> themselves.

Well, this is not completely independent (I can say this as developer of
docs.camlcity.org). Indexing profits a lot from having a compiled
version of the packages, because you have a strict name lookup then
(i.e. when a name comes from a predecessor package, you can resolve this
dependency easily, and you do not have to resort to fuzzy methods of
lookup).

Also, in docs.camlcity.org we can present the user the resulting file
tree after a package is installed. This is at least interesting
information.

> Finally there are two other things I would recommend one investigates:
> FreeBSD ports or Mac OSX homebrew for the indexing part. And pythons
> virtualenv (Ian Bicking was the original author). Virtualenv allows
> you to build a "virtual environment" in which you install python
> packages, which are distinct from the system installation. Of course
> you can symlink system libraries into the virtual environment, but it
> also means that programs can live next to each other and the
> sandboxing going on solves a lot of dependency problems in a simple
> way.
>
> In other words, to solve this, I'd much rather try to rig the
> excellent Ocamlbuild and Ocamlfind infrastructure such that it
> incorporates some of the above things, ... that is if it doesn't do
> that already...

Well, we are a bit farther than you knew, obviously :-)

Gerd

[Gabriel Scherer]

> I agree would be a serious changes, and I was thinking even of experimenting
> a bit with this kind of strictly typed meta programming. It's perfectly
> viable, as I've seen some good examples in my life. (and Template
> Haskell does it AFAIR).

I'm not familiar with Template Haskell at all, but I don't think it
does what you say. From what I understood of it, it was quite close
from Camlp4, if maybe integrated to GHC in a tighter way (Camlp4 is
really a separate source-to-source preprocessing phase). In
particular, you can write and use an extension even if is produces
ill-typed code in some circumstances (I mean type-checking the
extension doesn't give you that the expansions will themselves be
well-typed), and I haven't heard of getting type feedback at
extension-writing time. I would welcome any pointer about this.

> Of course worth to point out MetaOCaml and MetaML. They do runtime
> type safe staged meta programming.

Staged metaprogramming is indeed interesting but it is also very
different from Camlp4. It is not a separate processing pass but really
program generation at runtime. I don't think it would make much sense
to try to integrate it into Camlp4, but I would indeed support such an
addition to the OCaml language.

Btw., there is an olg bugtracker entry about it, which I follow as a
sign of support (maybe there would be a market for "I support feature
request #0004608" stickers). It hasn't evolved much as I suppose
nobody has time to work on it, and getting this to work reliably in
both ocamlc and ocamlopt would certainly be an awful lot of work, but
that's one case where "community" has helped bug triaging by
expressing interest.
http://caml.inria.fr/mantis/print_bug_page.php?bug_id=4608

> Good example would be:
> let t = (1,2,3,4) in
> map_tuple t (fun x -> x*2)

I'm not sure this example works in MetaOCaml, or at least that
map_tuple could be used on tuples of any width (of course you could
have a "map_tuple" function only taking care of 4-uplets).

I understand that you want a type-dependent metaprogramming layer to
expression transformations that couldn't be typed in the source
language. I think Alain Frisch dynamic type information injection
proposal, plus in-language runtime computation, could be simpler for
about the same expressive power. If you throw in MetaOCaml quotations,
you can even remove the runtime interpretive costs. But who's going to
work on that? I think that's orthogonal to syntactic processing
anyway.

> - syntax extension allows you to do certain things that are not available
> in a normal way, to enumerate: list comprehensions, pa_lwt, pa_where,
> pa_monad, pa_js. So they are clearly very useful in some cases

If we had no Camlp4, we should push for some of these things to be
integrated in the language.
A reasonable but solid mixfix syntax could replace pa_monad, the
"##"-syntax of pa_js, and some aspects of pa_lwt.
"jsnew" and list comprehensions could perfectly be handled as quotations.

I'm honored you mention mention pa_where (being one of the original
authors, I'm happy to know that it brings happiness to your home in
those Christmas times). I think that's also a good example of the
dangers of Camlp4. The precedence rules for such a syntax extension
are absolutely horrible to define and to get right; we have been
bitten by countless bug writing it, and I still don't think it's safe
for all uses. We would be better with either a solid version of it
entering the language (... but it's hard to convince people that it's
worth it, especially if you introduce new ambiguities opportunities),
or maybe just forgetting the idea.

> - combinators in particular monads allow certain syntactical
> abstractions, however there are runtime costs because of constructing
> closures and evaluating them later

I personally dislike uses of Camlp4 to optimize code by rewriting some
known expressions (eg. (... >>= (fun x -> ...)) into a more efficient
form. It is fragile precisely because you can't get accurate points-to
information so are unsure the operators you're manipulating really
have the semantics corresponding to your rewrite. I think those things
should be done in a different layer (not Camlp4), or not at all.
Evidently other people have a different opinion, and that's fine.

[Gerd Stolpmann]

Many people are still frustrated with the camlp4/p5 situation. IMHO, we
should give up on camlp4 inside the distribution, and only implement a
few of its features in the regular parser:

- Antiquotation syntax (i.e. << >> expressions) because this makes it
very easy to incorporate foreign syntax elements. Of course, we
would also need the ability to recursively invoke the ocaml parser
for parts of the antiquoted expression (this would also open the
door for transformations of the AST).

- A simple macro language (ifdef at least), for the cases antiquotations
would be too complex.

That means we would give up on arbitrary syntax extensions in the core
distribution. There could be still camlp4 or p5 as a separate add-on,
but it will be a second-class citizen, and it would be clear that it is
a bit behind with the newest syntactical features. Also, it could be
developed outside the core team.

Gerd

[Gabriel Scherer]

Gerd, you are summing up in a few paragraphs what I tried to say in a few pages.

There are other parts of Camlp4 that I would also welcome:
- the OCaml quotation parsers that reads quoted OCaml expression (and
patterns) and translate them to their ASTs (as an OCaml expression);
this makes generating OCaml code easy
- controlled extension points such as 'type-conv' and 'deriving',
hopefully generalized to most AST nodes; this goes in the direction of
Alain's annotation proposal

> There could be still camlp4 or p5 as a separate add-on,
> but it will be a second-class citizen, and it would be clear that it is
> a bit behind with the newest syntactical features.

It's not a big problem if the second-class extensions have trouble
processing and producing pieces of OCaml AST using the newest
syntactical features (eg. the OCaml quotations don't have sugar for
them). But it's a problem if they simply can't be run on files using
those features, forcing their users to keep older versions of OCaml.
It's not a *new* problem as it already appears at version transitions,
but it also won't go away as long as people use a preprocessor that
insists on understanding the whole AST, whether those tools are first-
or second-class.

[Alain Frisch]

On 12/11/2011 12:34 AM, Gabriel Scherer wrote:

> the Coq
> team which has user-defined notations using Camlp4 and, huh, I really
> don't want to know the details

My understanding (please correct me if I'm wrong) is that Coq uses
camlp{4,5} only as an extensible parser library in order to parse its
own language (which can be extended with user-defined notations). In
particular, Coq does not use the following camlp{4,5} features:

- the revised OCaml syntax
- the alternative representation of OCaml AST
- the Camlp4 grammar definitions for OCaml syntax(es)
- quotations/antiquotations to produce fragments of the OCaml AST
- OCaml syntax extensions to define grammar entries
- custom OCaml syntax extension for the Coq source code itself
(or maybe only very simple one, like macro/conditional compilation?)

I wonder how much energy it would take to create a stand-alone
extensible parser library, implemented in pure OCaml (normal syntax),
and following a similar API and semantics as camlp{4,5}, on which Coq
parsing could be built. The same library could be used as a foundation
for future versions of camlp{4,5}. It would be a simple library, with
no external dependency (in particular, no dependency to the OCaml
internals), and very little maintenance burden.

My guess is: this would not take so much energy. After all, the
representation of extensible grammars and the top-down parsing
technology are not so complex. But I would be interested to hear from
people who know Coq and camlp{4,5} better.


-- Alain

[Goswin von Brederlow]

Xavier Leroy <Xavier...@inria.fr> writes:

> On 12/08/2011 10:10 AM, Benedikt Meurer wrote:
>> The relevant bug report PR/5404, which includes a backward
>> compatible patch, is already waiting for a sign of life for 3 weeks
>> now (maybe wait another 4 years to get the port fixed).
>
> More bile. What's so urgent about it? The next release of OCaml is 3-6
> months in the future; your suggestion will be examined by then. You know,
> some people have busy professional lives. I can only devote 1 to 2 days per
> months to OCaml maintenance, and would prefer to spend them on technical
> work rather than on massaging egos or putting out the fires that you light
> up on this list.

I don't want to be aggressive or offensive and if it comes across as
that then please forgive me. No need to massage my ego also. I would
rather be told to piss off or that a patch will never be added because
it is stupid than have you play nice and have patches be left unatended
in mantis. Because how else will I ever learn? That said ...


So maybe PR/5404 was a bad example. How about these?

[1] 1 month to acknowledge, 24 month silence since then
[2] 1 month to acknowledge, 24 month silence since then
[3] 11 month to acknowledge, 19 month silence since then
[4] 16 month to acknowledge, 18 month silence since then
[5] 15 month to acknowledge, 6 month silence since then

[1] http://caml.inria.fr/mantis/view.php?id=4919
[2] http://caml.inria.fr/mantis/view.php?id=4909
[3] http://caml.inria.fr/mantis/view.php?id=4812
[4] http://caml.inria.fr/mantis/view.php?id=4987
[5] http://caml.inria.fr/mantis/view.php?id=5005

All of these have patches (or just needs a function declaration copied to
the .h file). They don't need huge work consuming changes to the
compiler and they don't add anything fundamentally new to it. They just
improve/extend the existing stuff a little.

For those to lazy to follow the links here is one of the patches that
has been sitting there for 2 years now:

----------------------------------------------------------------------
--- ocaml-3.11.0.orig/otherlibs/unix/unixsupport.h
+++ ocaml-3.11.0/otherlibs/unix/unixsupport.h
@@ -20,6 +20,7 @@
#define Nothing ((value) 0)

extern value unix_error_of_code (int errcode);
+extern int code_of_unix_error (value error);
extern void unix_error (int errcode, char * cmdname, value arg) Noreturn;
extern void uerror (char * cmdname, value arg) Noreturn;

--- ocaml-3.11.0.orig/otherlibs/unix/unixsupport.c
+++ ocaml-3.11.0/otherlibs/unix/unixsupport.c
@@ -263,6 +263,15 @@
return err;
}

+extern int code_of_unix_error (value error)
+{
+ if (Is_block(error)) {
+ return Int_val(Field(error, 0));
+ } else {
+ return error_table[Int_val(error)];
+ }
+}
+
void unix_error(int errcode, char *cmdname, value cmdarg)
{
value res;
----------------------------------------------------------------------

Adding this does not take a week of your summer vacation but still it
gets ignored.

Sure it is more fun to write code for first-class modules and the core
team has done a great job there. On the other hand trvial things are not
being done and that is verry frustrating to the submitter. Frankly I've
given up writing patches for ocaml because what is the point. They are
not getting looked at. Through things like that you are loosing a
workforce that could contribute at lot.


On the other hand if the model of glibc / eglibc for example where used
for ocaml the frustration would be reduced, the patches would get
audited by others in the community and added to the one community "fork"
before reaching the core team. You would get less patches, patches that
are better tested and cleaner. "Bad" patches would get filtered out
before they reach you.

This is not a condemnation of you or the core team. It is probably not
even something you can do anything about no matter how hard you try. It
is probably just a matter of scale. As a project growes at some point
there just seem to be a point where a single central core doesn't work
anymore. Things pile up, things fall through the cracks, priotization
leaves things starving. There plain aren't enough hours in the day to
cover every single bug or reply to every mail on the mainlinglist.

So let the community help and offload some work on them, like vetting a
simple patch like the above, so that you have more time to concentrate on the
difficult things like arm support or fist class modules (which I love by
the way) or to take a deserved vacation.

MfG
Goswin

[Gabriel Scherer]

> I wonder how much energy it would take to create a stand-alone extensible
> parser library, implemented in pure OCaml (normal syntax), and following a
> similar API and semantics as camlp{4,5}, on which Coq parsing could be
> built.

You are suggesting a dynamic parser tool. This is already available as
the dypgen tool:
http://dypgen.free.fr/

I suspect adapting Coq to such a tool would require considerable work,
because the devil is in the details and they really need backward
compatibility. I'm already impressed they managed to support both
camlp4 and camlp5.

[Alain Frisch]

On 12/11/2011 2:35 PM, Gabriel Scherer wrote:
>> I wonder how much energy it would take to create a stand-alone extensible
>> parser library, implemented in pure OCaml (normal syntax), and following a
>> similar API and semantics as camlp{4,5}, on which Coq parsing could be
>> built.
>
> You are suggesting a dynamic parser tool. This is already available as
> the dypgen tool:
> http://dypgen.free.fr/
>
> I suspect adapting Coq to such a tool would require considerable work,
> because the devil is in the details and they really need backward
> compatibility. I'm already impressed they managed to support both
> camlp4 and camlp5.

Agreed. It would probably be a lot easier to adapt Coq to work with a
(new) parsing library that uses the same parsing algorithms as Camlp4.

[Stéphane Glondu]

Le 11/12/2011 14:27, Alain Frisch a écrit :
> My understanding (please correct me if I'm wrong) is that Coq uses
> camlp{4,5} only as an extensible parser library in order to parse its
> own language (which can be extended with user-defined notations). In
> particular, Coq does not use the following camlp{4,5} features:
>

> [1] the revised OCaml syntax
> [2] the alternative representation of OCaml AST
> [3] the Camlp4 grammar definitions for OCaml syntax(es)
> [4] quotations/antiquotations to produce fragments of the OCaml AST
> [5] OCaml syntax extensions to define grammar entries
> [6] custom OCaml syntax extension for the Coq source code itself

> (or maybe only very simple one, like macro/conditional compilation?)

As far as I know, Coq uses:

[1,2,3] not at all
[4] a bit (e.g. tactics/tauto.ml4)
[5] a lot (e.g. most of *.ml4 files)
[6] for macros and conditional compilation indeed

> I wonder how much energy it would take to create a stand-alone
> extensible parser library, implemented in pure OCaml (normal syntax),
> and following a similar API and semantics as camlp{4,5}, on which Coq
> parsing could be built. The same library could be used as a foundation
> for future versions of camlp{4,5}. It would be a simple library, with
> no external dependency (in particular, no dependency to the OCaml
> internals), and very little maintenance burden.

I've heard of some attempts to port Coq to dypgen, but since no public
announcement was made, I guess this is quite difficult...

Are there other such libraries? It's not really Coq's business to create
a new one...


Cheers,

--
Stéphane

[Benedikt Meurer]

I fully agree, and as mentioned earlier, I'll help to maintain such a community distribution (or whatever we will call that glibc/eglibc thing for OCaml). Adding to the list above, a community distribution would provide an additional benefit for the users/community: We could release patch sets often and early while regular OCaml releases appear only every year or two. As Goswin already mentioned, big eye-catchers like first class modules are nice, but fixing the bugs (i.e. PR/4863, trivial patch from a core member(!) waiting to be applied for 3 months) is even more important (although probably less interesting from the core hackers point of view).

> MfG
> Goswin

greets,
Benedikt

[Arnaud Spiwack]

> My understanding (please correct me if I'm wrong) is that Coq uses
> camlp{4,5} only as an extensible parser library in order to parse its own
> language (which can be extended with user-defined notations). In
> particular, Coq does not use the following camlp{4,5} features:
>

This is not entirely correct. Coq uses camlp4's extensible ocaml syntax in
quite a bit of its code. (though less than it use to, I hear). The
following features are used:

- the alternative representation of OCaml AST
> - the Camlp4 grammar definitions for OCaml syntax(es)
> - quotations/antiquotations to produce fragments of the OCaml AST
> - OCaml syntax extensions to define grammar entries
> - custom OCaml syntax extension for the Coq source code itself
>

Though quotations are used *very* sparsely (mostly in legacy code, they are
not maintained).







Arnaud Spiwack

[Jérémie Dimino]

Le dimanche 11 décembre 2011 à 12:19 +0100, Gabriel Scherer a écrit :
> If we had no Camlp4, we should push for some of these things to be
> integrated in the language.
> A reasonable but solid mixfix syntax could replace pa_monad, the
> "##"-syntax of pa_js, and some aspects of pa_lwt.
> "jsnew" and list comprehensions could perfectly be handled as quotations.

I don't think it is possible to do pa_lwt via mixfix syntax. For example
pa_lwt does this kind of transformation:

lwt x = f () and y = g () and z = h () in
return (x + y + z)

-->

let t1 = f () and t2 = g () and t3 = h () in
Lwt.bind t1 (fun x -> Lwt.bind t2 (fun y -> Lwt.bind t3 (fun z -> return (x + y + z))))

The first form is much more readable. And it has another big advantage:
it adds backtrace support. Because as soon as you use monads in ocaml,
backtraces are unusable. And you just can't seriously ask the user to
write that kind of code:

Lwt.fail (try raise End_of_file with exn -> exn)

So unless we have better support for monads in OCaml itself, i think
Camlp4 is still required.

That said, i am not either a big fan of extending the syntax since it
makes code harder to understand for those who don't know the extensions.

--
Jérémie

[Xavier Leroy]

On 12/10/2011 04:58 PM, Benedikt Meurer wrote:

> Maybe we can get back to my original proposal and restart on the
> right foot this time?

All right. I have a number of concerns about your proposal, most of
which have already been mentioned by others. Whether you call it a
fork or not, the mere fact of having two separate code bases for
exactly the same software components raises more issues than it solves:

- It complicates the lives of OCaml users, packagers, and 3rd-party
library developers: what version should they use? what will be the
basis for the packagers's distribution-specific patches? what
happens if a library is compatible with one version but not the
other? what if the community effort runs out of steam in a few years?

- It means additional efforts with some duplication. The
synchronization between the "community" and the "reference" code
bases will take work. From the core team's standpoint, that's about
as much work as dealing with individual suggestions and
contributions. At the same time, the community team will spend
non-negligible time organizing and administering itself, at least
initially.

- It would be a PR disaster. I struggled (and still have to struggle)
with my INRIA management to get a little bit of support for OCaml's
development, e.g. Xavier Clerc's part-time assignment to work on
OCaml. This support is fragile and can easily disappear if there's
a perception that "the community" will take care of that anyway.
Likewise, I don't quite see how to explain the situation to the
members of the Caml consortium.

For these reasons, I feel that the cure is worse than the illness.
I am, however, much more sympathetic to your other proposal, namely:

> Why not accept a model similar to i.e. the NetBSD project, with a
> lot of committers (experts in their own areas) and 2-3 people to
> keep an eye on the overall direction?

Except for the "lot of committers" part, this is pretty much how the
core Caml team has been working, and there is definitely room for more
contributors. As I mentioned earlier, there are a number of areas
where we're lacking manpower, e.g. Windows-related stuff and tools
such as the debugger or Camlp4, but many other areas of the system
would benefit from more contributors too. Smaller contributions
are most welcome as well, such as commenting on the PRs, testing
changes, new features and release candidates, PR triaging, helping to
identify the top little things to be resolved by the next release,
etc.

That's a much more low-key approach, but one that is more likely to
succeed. Volunteers are welcome to contact us as ca...@inria.fr.

- Xavier Leroy

[Benedikt Meurer]

On Dec 12, 2011, at 11:21 , Xavier Leroy wrote:

> - It complicates the lives of OCaml users, packagers, and 3rd-party
> library developers: what version should they use? what will be the
> basis for the packagers's distribution-specific patches? what
> happens if a library is compatible with one version but not the
> other? what if the community effort runs out of steam in a few years?

If we can adopt the eglibc model, then the community thing will be the version shipped by distributions, i.e. the community thing is the OCaml for distributions/packagers, not an alternative to the official version. That way we do no longer need to maintain specific patches / versions for Debian, Red Hat, MacPorts, etc. This ensures that versions are compatible across different distributions (because they do no longer need to maintain their own set of patches).

I think of the community thing as:

(a) A single place for patches, while waiting for the next official OCaml release.
(b) A common ground for experimenting with new features. As you already noted, feedback and adoption for experimental stuff hidden within the SVN repository wasn't all that great, so maybe we can change it this way (esp. since it would be coordinated with the distributions/packagers to some degree).

> - It means additional efforts with some duplication. The
> synchronization between the "community" and the "reference" code
> bases will take work. From the core team's standpoint, that's about
> as much work as dealing with individual suggestions and
> contributions. At the same time, the community team will spend
> non-negligible time organizing and administering itself, at least
> initially.

The community team (namely the packagers in this case) already spend this time, as already noted by others (i.e. Stephane). And even if that's the same effort for the core team (I doubt that, since if nothing else, you'll at least have a single tested patch set to review, instead of a bunch of independent, incompatible, and probably untested patches), then there's at least the win for the community.

I shouldn't have used the term "fork" initially, this is misleading. Don't get me wrong. Don't think of my proposal as a "OpenBSD vs. NetBSD" thing with a whole new project.

>> Why not accept a model similar to i.e. the NetBSD project, with a
>> lot of committers (experts in their own areas) and 2-3 people to
>> keep an eye on the overall direction?
>
> Except for the "lot of committers" part, this is pretty much how the
> core Caml team has been working, and there is definitely room for more
> contributors. As I mentioned earlier, there are a number of areas
> where we're lacking manpower, e.g. Windows-related stuff and tools
> such as the debugger or Camlp4, but many other areas of the system
> would benefit from more contributors too. Smaller contributions
> are most welcome as well, such as commenting on the PRs, testing
> changes, new features and release candidates, PR triaging, helping to
> identify the top little things to be resolved by the next release,
> etc.
>
> That's a much more low-key approach, but one that is more likely to
> succeed. Volunteers are welcome to contact us as ca...@inria.fr.

I can help with the low-level compiler and runtime stuff, others have already expressed their intention to help with Camlp4, etc. You'll receive an email.

> - Xavier Leroy


Benedikt

[Mehdi Dogguy]

On 12/12/2011 11:59 AM, Benedikt Meurer wrote:
>
> On Dec 12, 2011, at 11:21 , Xavier Leroy wrote:
>
>> - It complicates the lives of OCaml users, packagers, and
>> 3rd-party library developers: what version should they use? what
>> will be the basis for the packagers's distribution-specific
>> patches? what happens if a library is compatible with one version
>> but not the other? what if the community effort runs out of steam
>> in a few years?
>
> If we can adopt the eglibc model, then the community thing will be
> the version shipped by distributions, i.e. the community thing is the
> OCaml for distributions/packagers, not an alternative to the official
> version. That way we do no longer need to maintain specific patches /
> versions for Debian, Red Hat, MacPorts, etc. This ensures that
> versions are compatible across different distributions (because they
> do no longer need to maintain their own set of patches).
>

No, distributions won't start shipping the community distribution just
because it exists. I cannot speak for Fedora (and others), but
Debian/Ubuntu won't switch to the community distribution that easily. In
fact, it may appear as a seperate source package at some point but won't
replace INRIA's ocaml in Debian.

If you look at Debian's ocaml package, you'll notice that very few
custom patches are applied.

http://patch-tracker.debian.org/package/ocaml/3.12.1-2

Patches from 1 to 6 are just (let's say) build configurations to cope
with Debian's standards. All the rest has been submitted to upstream,
except patch 11. Some of the submitted patches were even applied in
trunk. I guess the rest will be integrated as well… Most of the patches
(from 7 to 15) are there to avoid build failures on some architectures.

The problem with the community distribution is that:
1) it doesn't exist yet. So, we cannot commit on something more than vague.
2) we don't even know its release schedule, etc…
3) we don't know the kind of changes that will be integrated to the
community distribution
4) as of today, we don't even know if it is a good idea ; if it will be
a success ; etc…

> I think of the community thing as:
>
> (a) A single place for patches, while waiting for the next official
> OCaml release.
>
> (b) A common ground for experimenting with new features. As you
> already noted, feedback and adoption for experimental stuff hidden
> within the SVN repository wasn't all that great, so maybe we can
> change it this way (esp. since it would be coordinated with the
> distributions/packagers to some degree).
>

(a) seems very reasonable but (b) is unclear to me. You don't need to
create a community distribution of ocaml to experiment new features
sitting in ocaml's svn. Just checkout the relevant branch, and start
playing. The idea of the community distribution is to have an ocaml with
some patches on top of it (patches for bugs those sitting in mantis for
long time, a few features/experiments that are not present in ocaml's
svn). The idea is also to not diverge too much to keep things compatible.

my 2cents,

--
Mehdi Dogguy مهدي الدڤي
http://dogguy.org/

[Gerd Stolpmann]

>
> On Dec 12, 2011, at 11:21 , Xavier Leroy wrote:
>
>> - It complicates the lives of OCaml users, packagers, and 3rd-party
>> library developers: what version should they use? what will be the
>> basis for the packagers's distribution-specific patches? what
>> happens if a library is compatible with one version but not the
>> other? what if the community effort runs out of steam in a few years?
>
> If we can adopt the eglibc model, then the community thing will be the
> version shipped by distributions, i.e. the community thing is the OCaml
> for distributions/packagers, not an alternative to the official version.
> That way we do no longer need to maintain specific patches / versions for
> Debian, Red Hat, MacPorts, etc. This ensures that versions are compatible
> across different distributions (because they do no longer need to maintain
> their own set of patches).

Most distribution patches are really distribution-specific. I don't think
that there will be any savings.

IMHO, you have to convince the distributions whether they want to ship a
community version of ocaml or keep the original version.

> I think of the community thing as:
>
> (a) A single place for patches, while waiting for the next official OCaml
> release.

IF the community version is just a place for testing new patches this
would be for sure a good thing. The ocaml distributions could ship this in
special "testing" repositories. This way the patches would see broader
testing before the inclusion into the core repo. There would be certainly
some users who are willing to accept the additional risks of using a
testing distro.

IF this is just meant for smuggling in patches the core team does not
like, I don't see the value of it. The community should not work against
them.

> (b) A common ground for experimenting with new features. As you already
> noted, feedback and adoption for experimental stuff hidden within the SVN
> repository wasn't all that great, so maybe we can change it this way (esp.
> since it would be coordinated with the distributions/packagers to some
> degree).

I think the problem is that you can only easily build core ocaml this way,
but you normally need more packages for testing (e.g. run your program).
GODI has support for building ocaml from any SVN branch, but there is
close to zero documentation how this works.

>> - It means additional efforts with some duplication. The
>> synchronization between the "community" and the "reference" code
>> bases will take work. From the core team's standpoint, that's about
>> as much work as dealing with individual suggestions and
>> contributions. At the same time, the community team will spend
>> non-negligible time organizing and administering itself, at least
>> initially.

Xavier, I would not see this as negatively (but I am also not fully
convinced). If the community makes good suggestions, it could save a lot
of development work. Granted, the time needed for communication will be
higher - e.g. for setting quality standards, for explaining future
directions etc, but in total this could nevertheless be a win. The biggest
plus could be to attract new members for the core team, and to generally
increase the expertise about ocaml internals in the community.

Of course, if the community does not care about quality etc., and just
includes everything, there is no win.

Gerd

--
Gerd Stolpmann, Darmstadt, Germany ge...@gerd-stolpmann.de
Creator of GODI and camlcity.org.
Contact details: http://www.camlcity.org/contact.html
Company homepage: http://www.gerd-stolpmann.de
*** Searching for new projects! Need consulting for system
*** programming in Ocaml? Gerd Stolpmann can help you.

[Goswin von Brederlow]

Mehdi Dogguy <me...@dogguy.org> writes:

> On 12/12/2011 11:59 AM, Benedikt Meurer wrote:
>>
>> On Dec 12, 2011, at 11:21 , Xavier Leroy wrote:
>>
>>> - It complicates the lives of OCaml users, packagers, and
>>> 3rd-party library developers: what version should they use? what
>>> will be the basis for the packagers's distribution-specific
>>> patches? what happens if a library is compatible with one version
>>> but not the other? what if the community effort runs out of steam
>>> in a few years?
>>
>> If we can adopt the eglibc model, then the community thing will be
>> the version shipped by distributions, i.e. the community thing is the
>> OCaml for distributions/packagers, not an alternative to the official
>> version. That way we do no longer need to maintain specific patches /
>> versions for Debian, Red Hat, MacPorts, etc. This ensures that
>> versions are compatible across different distributions (because they
>> do no longer need to maintain their own set of patches).
>>
>
> No, distributions won't start shipping the community distribution just
> because it exists. I cannot speak for Fedora (and others), but
> Debian/Ubuntu won't switch to the community distribution that easily. In
> fact, it may appear as a seperate source package at some point but won't
> replace INRIA's ocaml in Debian.

That is probably what they said about eglibc too in the beginning. Now
eglibc is used in Debian/Ubuntu and is (or because it is) a great
success.

Obviously nobody is going to say: Yes, lets start this project, we will
ship it right away. It first has to proof itself. And first adopters
will be those that have the most problems with the standard ocaml.

MfG
Goswin

[oliver]

Hello,


installing packages in R also is quite easy.
In a running R you just type "install.packages("RMySQL")"
get a window to select a server and things then happen
automagically: source download, compilation, installation.
either in a systemwide manner or if permissions of your session
do not allow it, somewhere beyond your local $HOME.
It's even easier than with Perl / CPAN.

(And using not the official CRAN packages is easy: just
change the setting of the used repository and you could
install the newest/freshest stuff easily too.)


Or when installing at the OS level not officially supported packages
on Arch linux (for official ones you could just use
the installer pacman), then makepgg will help you.
For installation as well as creating packages by your own.

On Sun, Dec 11, 2011 at 12:02:57PM +0100, Gerd Stolpmann wrote:
> Am Sonntag, den 11.12.2011, 00:28 +0100 schrieb Jesper Louis Andersen:
> > On Sat, Dec 10, 2011 at 15:45, Xavier Leroy <Xavier...@inria.fr> wrote:
> >
> > > 2- As pointed out already in this discussion, it's not on the Caml compiler
> > > that community efforts are most needed. For example, the most impactful
> > > action that his community could take, in my opinion, is to adopt and embrace
> > > a common packaging system like Cabal for Haskell.
> >
> > This one piqued my interest somewhat. I should refrain from
> > commenting, but I can't help myself.
>
> It's interesting to hear these comments. In short, in ocaml we have
> these options:
>
> - Full featured package management: we have this (GODI), but the
> community is not too large (package maintainers are always
> welcome!) As every full-featured system, GODI has complicated
> parts, and you need to invest 1-2 days before you fully
> understand it and can build packages.

[...]

And why to use GODI and create OCaml/GODI packages and not
a *.deb or *.rpm or PKGBUILD/tgz?

Or does GODI also allows creation of such OS-related packages?


Ciao,
Oliver

[Benedikt Meurer]

On Dec 12, 2011, at 18:48 , Stéphane Glondu wrote:

>>> The relevant bug report PR/5404, which includes a backward
>>> compatible patch, is already waiting for a sign of life for 3 weeks
>>> now (maybe wait another 4 years to get the port fixed).
>>
>> More bile. What's so urgent about it? The next release of OCaml is 3-6

>> months in the future; your suggestion will be examined by then. [...]
>
> It's still unclear to me whether this kind of patches are welcome. I've
> been recently approached by an armhf porter (other than Benedikt)
> interested in having an ocamlopt running there, but I told him to back
> off because I don't know whether needed changes would be merged. Once
> again, I don't say we would have come up with a working solution before
> the next release of OCaml, but the result (if any) would have already
> been tested and deployed in Debian (and/or Ubuntu) before an upstream
> submission.

You can grab my current patch for the new OCaml ARM backend at:

http://ps.informatik.uni-siegen.de/~meurer/tmp/ocaml-arm-20111213.diff

Compared to the old backend, this one does the following:

- Support for both softfp and VFPv3-D16 (if present).
- Properly supports interworking with Thumb/Thumb-2 code (for both OCaml and C code!)
- Supports dynamic linking and large memory models.
- Optional support for position-independent code via -fPIC, disabled by default and not required for natdynlink.
- Can emit both ARM and Thumb-2 code (currently Thumb-2 is used for ARMv7+ and ARM is used for everything else), with avg. code size savings of 27% for Thumb-2 (quite close the optimal 30% advertised by ARM Ltd.). I may also add support to emit small functions using Thumb-1 (for pre-ARMv7 / armel) in the future to reduce code size.
- Supports both AAPCS (armel) as well as extended VFP calling conventions (armhf).
- Properly supports backtraces.
- Recognizes several special ARM instructions (=> reduced code size and latency).
- Does not rely on GCC internals, but uses the standard ARM EABI runtime.

Feel free to test the patch, but be aware that it is

(a) highly experimental, it's tested and verified to work with Debian armel on ARMv5T and Debian armhf on a Cortex-A8, but YMMV,
(b) it only supports ARMv4T+ using the armel ABI w/o VFPv3, Thumb, etc. or ARMv7+ using the armhf ABI w/ VFPv3, Thumb-2, etc., there's no way to choose i.e. armel ABI w/ VFPv3, I don't know whether there's anyone except Ubuntu using such configurations currently,
(c) incompatible with .cmx/.cmxa/.cmxs files generated by the old ARM backend, and
(d) it's still a bit rough around the edges, i.e. the generated code is usually better (shorter and runs faster), but there's still room for improvement (i.e. instruction selection and scheduling need some more love).

Comments and suggestions are welcome.

> Cheers,
> Stéphane

HTH,
Benedikt

[Mark Shinwell]

On Tue, Dec 13, 2011 at 09:39:03PM +0100, Benedikt Meurer wrote:
> You can grab my current patch for the new OCaml ARM backend at:
>
> http://ps.informatik.uni-siegen.de/~meurer/tmp/ocaml-arm-20111213.diff
>
> Compared to the old backend, this one does the following:
>
> - Support for both softfp and VFPv3-D16 (if present).
> - Properly supports interworking with Thumb/Thumb-2 code (for both OCaml and C code!)
> - Supports dynamic linking and large memory models.
> - Optional support for position-independent code via -fPIC, disabled by default and not required for natdynlink.
> - Can emit both ARM and Thumb-2 code (currently Thumb-2 is used for ARMv7+ and ARM is used for everything else), with avg. code size savings of 27% for Thumb-2 (quite close the optimal 30% advertised by ARM Ltd.). I may also add support to emit small functions using Thumb-1 (for pre-ARMv7 / armel) in the future to reduce code size.
> - Supports both AAPCS (armel) as well as extended VFP calling conventions (armhf).
> - Properly supports backtraces.
> - Recognizes several special ARM instructions (=> reduced code size and latency).
> - Does not rely on GCC internals, but uses the standard ARM EABI runtime.

Dear Benedikt,

This looks like an impressive amount of work. I'd like to suggest that a good
approach to getting some/all of this merged might be to divide the patch up
into smaller units each addressing a particular deficiency or implementing
a particular feature. In that way it should be more straightforward to
understand the effects on the whole system of each particular patch and
maintain stability and compatibility. I understand that this might cause some
work and that some of the patches might be intertwined. (Another approach
might be to introduce this backend as a second ARM backend with the
understanding that it may be experimental to a certain extent---but I think
that's less satisfactory, and less likely to find favour with everyone.)

Which items from the above list would you most like to see in the core
distribution? I would have thought the basic VFP support was fairly high
up the list.

Mark

[Gerd Stolpmann]

Am Dienstag, den 13.12.2011, 20:36 +0100 schrieb oliver:
> Hello,
>
>
> installing packages in R also is quite easy.

Well, package management for scripting languages is way easier. You can
especially do some of the management at load time. If, for example, a
symbol is missing because a predecessor is too old, there is generally
the possibility to work around by providing an alternate definition, and
to resolve this within the scripting environment.

A statically typed compiler language like Ocaml is more difficult,
because you can do these kinds of fixups only at build time. Part of
this is that you need to recompile all successors of a package if you
upgrade the package.

The question includes the answer. Do you really want to support five
different package types?

> Or does GODI also allows creation of such OS-related packages?

Not yet, but this is certainly not impossible with a to-be-written
package converter.

So far, there has been nobody who investigated this in detail.
(Actually, I'm a bit surprised that none of the OS packagers had the
idea yet - this could save a lot of time for them.)


Gerd

[Benedikt Meurer]

On Dec 14, 2011, at 10:18 , Mark Shinwell wrote:

>> - Support for both softfp and VFPv3-D16 (if present).
>> - Properly supports interworking with Thumb/Thumb-2 code (for both OCaml and C code!)
>> - Supports dynamic linking and large memory models.
>> - Optional support for position-independent code via -fPIC, disabled by default and not required for natdynlink.
>> - Can emit both ARM and Thumb-2 code (currently Thumb-2 is used for ARMv7+ and ARM is used for everything else), with avg. code size savings of 27% for Thumb-2 (quite close the optimal 30% advertised by ARM Ltd.). I may also add support to emit small functions using Thumb-1 (for pre-ARMv7 / armel) in the future to reduce code size.
>> - Supports both AAPCS (armel) as well as extended VFP calling conventions (armhf).
>> - Properly supports backtraces.
>> - Recognizes several special ARM instructions (=> reduced code size and latency).
>> - Does not rely on GCC internals, but uses the standard ARM EABI runtime.
>

> This looks like an impressive amount of work. I'd like to suggest that a good
> approach to getting some/all of this merged might be to divide the patch up
> into smaller units each addressing a particular deficiency or implementing
> a particular feature. In that way it should be more straightforward to
> understand the effects on the whole system of each particular patch and
> maintain stability and compatibility. I understand that this might cause some
> work and that some of the patches might be intertwined. (Another approach
> might be to introduce this backend as a second ARM backend with the
> understanding that it may be experimental to a certain extent---but I think
> that's less satisfactory, and less likely to find favour with everyone.)
>
> Which items from the above list would you most like to see in the core
> distribution? I would have thought the basic VFP support was fairly high
> up the list.

The port is not in a clean state yet. It's still a bit messy, and I'm going to separate various things first (i.e. select FPU / CPU independent of platform ABI, for example use VFPv3 with softfp EABI, etc. or use Thumb-2 on armel, etc.). I hope to get some more time to work on it during the week. You can follow the work at:

http://github.com/bmeurer/ocaml-arm

Testing and feedback is always welcome, of course.

Concerning your proposal to split the patch into multiple smaller patches:

(a) This would probably result in roughly two patches, the first one to fix the existing armel port, and the second one to add support for FPUs, armhf EABI, Thumb, Thumb-2, etc. (these depend on each other in various ways).
(b) It would be a lot of work for almost no benefit, because it is an almost completely new backend and not simply a set of bug fixes for the existing ARM backend, and must be treated this way. And even if I'd spend my days splitting up the patch, there's still that unresolved OCaml contributions issue (i.e. high probability of wasting my time because that patch set will just idle in Mantis).

> Mark

Benedikt

[Stéphane Glondu]

Le 14/12/2011 13:13, Gerd Stolpmann a écrit :
> Not yet, but this is certainly not impossible with a to-be-written
> package converter.
>
> So far, there has been nobody who investigated this in detail.
> (Actually, I'm a bit surprised that none of the OS packagers had the
> idea yet - this could save a lot of time for them.)

There is alien [1]. However, I don't really see the point when you
already have access to quality native packages, and I usually prefer to
run directly upstream's "./configure && make && make install" for stuff
that are not already packaged (and, sometimes, I package it myself :).
For system-wide installation, I usually use checkinstall [2] to register
it to my package manager. Alien and checkinstall cannot be used to make
redistributable packages, though, and they are quite Linux-centric.

[1] http://kitenet.net/~joey/code/alien/
[2] http://checkinstall.izto.org/

My humble opinion about all these packaging issues is that the OCaml
community is just too small (and/or lacks dedicated people) to create
and maintain a quality package set that can be advertised to attract
more people. Moreover, most OCaml stuff are for the OCaml community,
which is quite a restricted audience. In my experience (in Debian), the
"written in OCaml" argument works only for people who know and program
in OCaml already, and tends to make others flee. On the other way, there
are many applications written in e.g. Python that are popular, and you
know that they are written in Python only by looking at the source code.

Maybe people should write more end-user applications in OCaml (but
neither scientific nor OCaml-centric) to promote OCaml better. Like
unison, ledit, mldonkey, etc. And be reactive to external contributions
(be it bugreports, feature wishes or patches), so that people get
progressively contaminated by the OCaml virus :-)


Cheers,

--
Stéphane

[Stéphane Glondu]

Le 10/12/2011 15:45, Xavier Leroy a écrit :
> 4- Yes, we obviously have problems with PR triaging, in part because Mantis
> makes this task more bureaucratic than strictly necessary, but more
> importantly because it is often hard to guess who cares about this or that
> suggestion, or even what problem it is supposed to address. Volunteers
> could greatly help by simply commenting on the PRs in Mantis, to express
> support or disagreement, or to ask for clarifications.

By the way, is there a way to "subscribe" to all ocaml bugs? In my
account preferences, all "E-mail on ..." are checked, but I don't
receive mails for all activities on the bugtracker (I only receive mails
for bug I'm monitoring).

[Till Varoquaux]

There's an RSS link in the mantis. I think
http://caml.inria.fr/mantis/issues_rss.php?project_id=0
would do the trick. I used the link successfully in my reader but the
address I is different (it has the login info).

Till

[Benedikt Meurer]

On Dec 13, 2011, at 21:39 , Benedikt Meurer wrote:

> - Support for both softfp and VFPv3-D16 (if present).
> - Properly supports interworking with Thumb/Thumb-2 code (for both OCaml and C code!)
> - Supports dynamic linking and large memory models.
> - Optional support for position-independent code via -fPIC, disabled by default and not required for natdynlink.
> - Can emit both ARM and Thumb-2 code (currently Thumb-2 is used for ARMv7+ and ARM is used for everything else), with avg. code size savings of 27% for Thumb-2 (quite close the optimal 30% advertised by ARM Ltd.). I may also add support to emit small functions using Thumb-1 (for pre-ARMv7 / armel) in the future to reduce code size.
> - Supports both AAPCS (armel) as well as extended VFP calling conventions (armhf).
> - Properly supports backtraces.
> - Recognizes several special ARM instructions (=> reduced code size and latency).
> - Does not rely on GCC internals, but uses the standard ARM EABI runtime.

I've updated the backend, and a new patch is available for testing at [1], you can also test it by cloning the ocaml-arm repository at [2], some documentation is available on the Wiki at [3].

The following new features are included:

- Support for both VFPv3-D16 and VFPv3(-D32), using ocamlopt switch -ffpu (armhf only).
- Support for architecture selection using -farch.
- Support for profiling using gprof.
- Code cleanups.
- Confirmed to work with Debian/armel on ARM9 / Cortex-A8 and Debian/armhf on Cortex-A8.

Testers and feedback welcome.

greets,
Benedikt

[1] https://github.com/downloads/bmeurer/ocaml-arm/ocaml-arm-3.12.1+20111218.diff.bz2
[2] https://github.com/bmeurer/ocaml-arm
[3] https://github.com/bmeurer/ocaml-arm/wiki

[Benedikt Meurer]

On Dec 18, 2011, at 12:57 , Benedikt Meurer wrote:

>> - Support for both softfp and VFPv3-D16 (if present).
>> - Properly supports interworking with Thumb/Thumb-2 code (for both OCaml and C code!)
>> - Supports dynamic linking and large memory models.
>> - Optional support for position-independent code via -fPIC, disabled by default and not required for natdynlink.
>> - Can emit both ARM and Thumb-2 code (currently Thumb-2 is used for ARMv7+ and ARM is used for everything else), with avg. code size savings of 27% for Thumb-2 (quite close the optimal 30% advertised by ARM Ltd.). I may also add support to emit small functions using Thumb-1 (for pre-ARMv7 / armel) in the future to reduce code size.
>> - Supports both AAPCS (armel) as well as extended VFP calling conventions (armhf).
>> - Properly supports backtraces.
>> - Recognizes several special ARM instructions (=> reduced code size and latency).
>> - Does not rely on GCC internals, but uses the standard ARM EABI runtime.
>

> - Support for both VFPv3-D16 and VFPv3(-D32), using ocamlopt switch -ffpu (armhf only).
> - Support for architecture selection using -farch.
> - Support for profiling using gprof.
> - Code cleanups.
> - Confirmed to work with Debian/armel on ARM9 / Cortex-A8 and Debian/armhf on Cortex-A8.

Some notes about performance of the new backend: It is up to 4x faster (Cortex-A8 / armhf) and code size is reduced by up to 28% compared to the 3.12.1 backend (Cortex-A8 / armel). See [1] for details.

> Testers and feedback welcome.

While we are at it: Can we have this patch applied to the Debian OCaml package? This would (a) get OCaml running native on armhf and (b) get some testing / feedback for the patch.

HTH,
Benedikt

[1] https://github.com/downloads/bmeurer/ocaml-arm/ocaml-arm-3.12.1+20111218-benchmark.pdf

[Benedikt Meurer]

On Dec 18, 2011, at 12:57 , Benedikt Meurer wrote:

>

> On Dec 13, 2011, at 21:39 , Benedikt Meurer wrote:
>
>> - Support for both softfp and VFPv3-D16 (if present).
>> - Properly supports interworking with Thumb/Thumb-2 code (for both OCaml and C code!)
>> - Supports dynamic linking and large memory models.
>> - Optional support for position-independent code via -fPIC, disabled by default and not required for natdynlink.
>> - Can emit both ARM and Thumb-2 code (currently Thumb-2 is used for ARMv7+ and ARM is used for everything else), with avg. code size savings of 27% for Thumb-2 (quite close the optimal 30% advertised by ARM Ltd.). I may also add support to emit small functions using Thumb-1 (for pre-ARMv7 / armel) in the future to reduce code size.
>> - Supports both AAPCS (armel) as well as extended VFP calling conventions (armhf).
>> - Properly supports backtraces.
>> - Recognizes several special ARM instructions (=> reduced code size and latency).
>> - Does not rely on GCC internals, but uses the standard ARM EABI runtime.
>

> - Support for both VFPv3-D16 and VFPv3(-D32), using ocamlopt switch -ffpu (armhf only).
> - Support for architecture selection using -farch.
> - Support for profiling using gprof.
> - Code cleanups.
> - Confirmed to work with Debian/armel on ARM9 / Cortex-A8 and Debian/armhf on Cortex-A8.

Reported as PR#5433 at http://caml.inria.fr/mantis/view.php?id=5433.

Benedikt

[Alexandre Pilkiewicz]

Hi Benedikt

On Sun, Dec 18, 2011 at 1:08 PM, Benedikt Meurer
<benedik...@googlemail.com> wrote:
>
>
> [1] https://github.com/downloads/bmeurer/ocaml-arm/ocaml-arm-3.12.1+20111218-benchmark.pdf


Could you by any chance give the execution time (one should be enough
since we have the ratio) and measurement technique of your benchmarks?

Thanks in advance

Alexandre

[Benedikt Meurer]

On Dec 18, 2011, at 15:50 , Alexandre Pilkiewicz wrote:

> Hi Benedikt

Hey Alexandre,

>> [1] https://github.com/downloads/bmeurer/ocaml-arm/ocaml-arm-3.12.1+20111218-benchmark.pdf
>
> Could you by any chance give the execution time (one should be enough
> since we have the ratio) and measurement technique of your benchmarks?

Sure. Measured as combined user + system time with best result of 5 subsequent runs (Linux kernel was 2.6.31.14.25-efikamx in all tests), here are the execution times (in seconds):

3.12.1 armel:
-------------
almabench 455.020
bdd 6.560
boyer 7.310
fft 21.770
nucleic 74.720
quicksort 1.640
sieve 0.120
soli 0.060
sorts 20.190

ocaml-arm armel:
----------------
almabench 455.500
bdd 6.650
boyer 7.230
fft 21.910
nucleic 74.110
quicksort 1.570
sieve 0.110
soli 0.060
sorts 20.950

ocaml-arm armhf:
----------------
almabench 111.810
bdd 6.590
boyer 7.120
fft 6.680
nucleic 19.430
quicksort 1.570
sieve 0.110
soli 0.070
sorts 20.900

> Thanks in advance
> Alexandre

HTH,
Benedikt

[Stéphane Glondu]

Le 18/12/2011 14:08, Benedikt Meurer a écrit :
> While we are at it: Can we have this patch applied to the Debian OCaml package? This would (a) get OCaml running native on armhf and (b) get some testing / feedback for the patch.

Your patch is way too monolithic. You cannot seriously expect anyone to
apply it as it is.

The git repository looks more sensible, but the history is kind of messy
(e.g. one big initial import removing/duplicating arm.S, then later
merge). My advice is to maintain a clean patch series (e.g. one patch
for each new feature you advertise, or whatever logical split you see
fit), in addition to publishing your working git repository. Please read:

http://halobates.de/on-submitting-patches.pdf

An acceptable workflow for me would be two git branches, one with your
ongoing development, and another one starting from 3.12.1 release with a
cleaned up and detailed history. Both would continuously have the same
tree, but the cleaned-up one would be rebased if needed to keep the
history clean. Then each commit of the cleaned-up branch would be
reviewed and discussed separately as a candidate patch for upstream
inclusion. All this can be done efficiently with git.

What tests did you perform, by the way?


Cheers,

--
Stéphane

[Romain Beauxis]

Hi all,

2011/12/12 Mehdi Dogguy <me...@dogguy.org>:

I have been busy with several fires and only came back to list
recently. I might be late for the party but I would like to voice my
opinion as this idea of "forking" OCaml is a pretty sensitive one.

I, for one, am convinced that "forking" OCaml's core would be a
terrible idea. As mentioned above by Mehdi as well as others, a
"community" driven development process would not address the
fundamental issue of taking core decisions. Those familiar with the
Debian project surely know how these processes can be time-consuming
and bring another form of frustration anyway.

I would also like to mention that OCaml core developers probably do
not get the proper recognition they deserve for the work they've put
in the past years. Surely, some aspects could be criticized, but
overall I have seen many changes which also preserved backward
compatibility and minimized the amount of bugs introduced. I think
this is overall a great job.

OCaml's core is a sensitive part of the language and I can see how it
can get frustrating to get one's work included into it. But,
considering the importance of this part of the language, frustration
is probably somehow a good thing. At least, it does not seem to me to
justify any kind of forking.

My real concern for the future of OCaml's development is more about
side tasks that could very well be addressed by the community without
completely breaking down the stability of its core. For instance, I
have been maintaining for some time a version of Richard's
cross-compiler in Debian. Most of those changes concern OCaml's build
system which, admittedly, could be very well improved.

On this point, I believe that it would be very nice to have, indeed, a
clearer integration process and more communication from the core
development team. For instance, if I would be to propose a complete
rewrite of OCaml's build system, I'd like to know beforehand if my
changes have any chance to be integrated and, if so, how should I
tackle the issue in order to facilitates this integration (split up
the changes in several patches, post them for review somewhere etc..).

This also stand for the patches required to build the cross-compiler,
which I think would be a great thing to have out of the box and would
not require much changes in the core.

On another topic, I also agree that there should be a real effort put
into the visibility of the language and, more importantly, available
libraries. Contrary to what others have mentioned, functional
programming is not, I believe, a paradigm that would be so difficult
to take on. Things like coffeescript [1], a "syntactic sugar" for
javascript, becoming more and more used by JS programmers shows that
the basics of functional programming syntactic have a great potential
for adoption by a wider basis of developers.

As it has been mentioned earlier, I, too, think that having more
visibility on places where other languages and projects are also
available is a key here. Github, indeed, seems like unavoidable at
this point and, on the contrary, hosting our own gitorious would be
yet another failure as it would set up an isolated platform for the
language.

Ultimately, I do not believe that academics should be taken
responsible for the adoption of OCaml by young programmers. If one
looks at ruby/rails or nodejs or, until recently, python, the success
of those languages have little to do with academic programs but more
with the dynamic, visibility and communication of their early
adopters.

Romain


[1]: http://jashkenas.github.com/coffee-script/

[Alain Frisch]

On 12/19/2011 5:09 AM, Romain Beauxis wrote:
> On this point, I believe that it would be very nice to have, indeed, a
> clearer integration process and more communication from the core
> development team. For instance, if I would be to propose a complete
> rewrite of OCaml's build system, I'd like to know beforehand if my
> changes have any chance to be integrated and, if so, how should I
> tackle the issue in order to facilitates this integration (split up
> the changes in several patches, post them for review somewhere etc..).

Jonathan Protzenko has started to write down some guidelines how to
contribute most effectively to OCaml. These ideas are being discussed,
and should be published soon.

For a big project like the one you mention (a complete rewrite of
OCaml's build system), it is always a good idea to contact the
development team first with a detailed proposal and wait for some
feedback before doing the heavy work. Contacting the development team
can be done either via the bug tracker or by email (ca...@inria.fr)



Alain

[Benedikt Meurer]

On Dec 18, 2011, at 18:23 , Stéphane Glondu wrote:

> Le 18/12/2011 14:08, Benedikt Meurer a écrit :
>> While we are at it: Can we have this patch applied to the Debian OCaml package? This would (a) get OCaml running native on armhf and (b) get some testing / feedback for the patch.
>

> The git repository looks more sensible, but the history is kind of messy
> (e.g. one big initial import removing/duplicating arm.S, then later
> merge). My advice is to maintain a clean patch series (e.g. one patch
> for each new feature you advertise, or whatever logical split you see
> fit), in addition to publishing your working git repository.

I did some initial work on this in the development branch. This is for the trivial stuff.

> What tests did you perform, by the way?

The test suite that ships with OCaml plus some custom test programs to test specific cases in the instruction selection and scheduling.

> Cheers,
> Stéphane

Benedikt