I just got back from QCon 2008 in London.
Lots of interesting presentations, and among them
one by Joe Armstrong about Erlang.
One talk was about the Eclipse team and how they had
gone from a closed development style to a style where
all their internal processes was completely open to public(*).
It is now 10 years (I think...) since Erlang became Open Source.
The last year, the interest in Erlang has exploded and I think
it is time to take the next step; open up the development of Erlang.
Now, I know that Erlang is buried in Clearcase somewhere in the
dungeons of Ericsson and that it probably will be a major undertaking
to change the "corporate policy". I think however that the matter
is worth discussing.
So, suggestions:
- Put Erlang into a modern (distributed) VCS like Mercurial or Git.
- Make it possible to browse the full repository.
- Open up the bug-tracker to become fully public.
- Publish the testserver result, e.g using CruiseControl.
- Make the whole testsuite available for download and deployment.
- Open up any other (as of today) internal dev.tools, wikis etc.
This way, it would be easy for the public to provide well tested
patches that easily can be tracked. It would make it easier both
for the OTP team to cherry-pick patches as well as make it easier
to maintain non-accepted patches in ones own respositories.
Cheers, Tobbe
(*) www.jazz.net
+1
--
paul
> _______________________________________________
> erlang-questions mailing list
> erlang-q...@erlang.org
> http://www.erlang.org/mailman/listinfo/erlang-questions
>
--
Gleb Peregud
http://gleber.pl/
Every minute is to be grasped.
Time waits for nobody.
-- Inscription on a Zen Gong
Did the Eclipse team mention who the legal owner of the Eclipse
code base is?
I ask because one of the problems for any company involved in
Open Source development is the fact that it's legally responsible for
all the code in the repository. This is perhaps a bigger problem for
a large company than a small one, int that large companies make
for juicy targets in IPR lawsuits.
> - Open up the bug-tracker to become fully public.
...under the proviso, of course, that proprietary information given
in bug reports are not revealed to the public.
> - Publish the testserver result, e.g using CruiseControl.
> - Make the whole testsuite available for download and deployment.
Yes.
> - Open up any other (as of today) internal dev.tools, wikis etc.
Same thing here. OTP would have to find a way to keep confidential
information confidential.
BR,
Ulf W
Using git (or mercurial) is more about enabling the acceptance of
community (distributed) development into the core distribution. The
official tree would still reside where it is, just that the tools enable
community contributions more simply and effectively. Having internal
development done via git would promote a workflow where community
contributions could be on equal footing (given that they have merit.)
IPR is a separate issue, and the same efforts would need to be made to
contribution integrated into upstream tree. What is done with patches
submitted today?
--
paul
[snip]
>
> > - Open up any other (as of today) internal dev.tools, wikis etc.
>
> Same thing here. OTP would have to find a way to keep
> confidential information confidential.
One solution I've seen is to have two repositories. One repository contains the open-source core product, which can be packaged and installed on its own.
The other repository has the proprietary component. To build the proprietary component, you must have built the open source component and perhaps also have installed it.
This solution isn't without its problems, but it's probably the cleanest solution I've seen to the problem it was meant to solve.
- Christian
>
It could be even simpler: the proprietary "component" could "simply"
be a fork of the open-source one, additions to the open-source product
would be merged into the proprietary one as needed but not the other
way around. Mercurial and Git both provide facilities for that kind of
workflow.
---------- Forwarded message ----------
From: Elliot Murphy <elliot...@gmail.com>
Date: Mon, Mar 17, 2008 at 9:44 AM
Subject: Re: [erlang-questions] Erlang 10 years of Open Source; it is
time for the next step
To: Torbjorn Tornkvist <to...@tornkvist.org>
Hi!
On Mon, Mar 17, 2008 at 6:38 AM, Torbjorn Tornkvist
<to...@tornkvist.org> wrote:
> Now, I know that Erlang is buried in Clearcase somewhere in the
> dungeons of Ericsson and that it probably will be a major undertaking
> to change the "corporate policy". I think however that the matter
> is worth discussing.
Corporate policy or not, changing version control systems is a lot of work,
and should not be underestimated. I've helped several large companies
change their
version control system, and I'm happy that I can say it is a
reasonable thing to do.
Much of the work can be done up front by a handful of people - the most
important thing to get right is selecting a tool and doing the history
conversion.
>
> So, suggestions:
>
> - Put Erlang into a modern (distributed) VCS like Mercurial or Git.
> - Make it possible to browse the full repository.
I've recently imported the source tarballs into Bazaar, which is an
adaptive/distributed version control system that we use at Canonical:
https://code.launchpad.net/~erlang-dev/erlang/trunk
I was working from the source drops, without access to ClearCase, so
the history is naturally limited to seeing the changes between releases. Still,
it's a good step in the right direction. I am currently working with other
companies to convert major projects from proprietary VCS to Bazaar, while
preserving all their important history, and I'd be delighted to do the same
for the Erlang project.
> - Open up the bug-tracker to become fully public.
> - Publish the testserver result, e.g using CruiseControl.
> - Make the whole testsuite available for download and deployment.
> - Open up any other (as of today) internal dev.tools, wikis etc.
>
> This way, it would be easy for the public to provide well tested
> patches that easily can be tracked. It would make it easier both
> for the OTP team to cherry-pick patches as well as make it easier
> to maintain non-accepted patches in ones own respositories.
I think that is a great idea. My day job is working on https://launchpad.net
(click on the tour button for more info),
which is a web based development/collaboration suite with mailing lists,
code hosting, distributed bug tracker, support tools, I18N tools, and much
much more. One of the core ideas behind Launchpad is to deal with
dependencies in software - you get a lot of those when building a linux
distribution for example. The place where the bug is reported is not
necessarily
the place where it needs to be fixed.
The other place where we have seen the ideas we've built in Launchpad work
really really well is in communities based on a programming language - we've
seen significant numbers of python projects moving to Launchpad, for example.
All this rambling is to say that I think this is a great idea, and we at
Launchpad would be happy to provide as much support for the Erlang project
as the community would like. Two recent projects that moved are Zope and
Inkscape. I was really happy to see how the community reacted to the Inkscape
move: http://bryceharrington.org/drupal/node/18
One of the tricky bits with opening up a bug tracker is that sometimes you
have gotten customer information mixed into the bug tracker that absolutely
needs to be kept private, while the majority of the bugs are actually about
technical bugs and could easily be available to the public. We've dealt with
that in Launchpad by building a private bug capability, so if something like
migrating the bugs from the internal tracker to a public bug tracker
on Launchpad
was desired we could flag bugs as private as needed.
--
Elliot Murphy
--
Elliot Murphy
It is now 10 years (I think...) since Erlang became Open Source.
The last year, the interest in Erlang has exploded and I think
it is time to take the next step; open up the development of Erlang.
*chuckle*
I wish I had a nickel (i.e. 5% of a US Dollar) every time this happened to people.
> I wish I had a nickel (i.e. 5% of a US Dollar) every time this
> happened to people.
It's often a whole lot more entertaining when the reverse happens.
Matt
I would second this question and argue against opening the repository
unless a clear advantage over the existing system, besides the "I want
to contribute", can be demonstrated. Contrary to what we open source
advocates may believe, OS is not the oasis of bug free code, useable,
tested code.
> On Mon, Mar 17, 2008 at 11:38 AM, Torbjorn Tornkvist <to...@tornkvist.org
> >
> wrote:
>
>>
>> It is now 10 years (I think...) since Erlang became Open Source.
>> The last year, the interest in Erlang has exploded and I think
>> it is time to take the next step; open up the development of Erlang.
>>
>
> Please help me to understand: What are the real benefits in doing
> this?
Note that the following are my guesses, or the benefits I would see
>
> Erlang and OTP are a good example of conceptual integrity, mostly
> because
> they are designed to solve real world problems; what are the
> benefits in
> allowing, for example, the forking of the build tree?
Having a full history, allowing people out of Ericsson to create
patches (that could then be forwarded to the core team for inclusion)
or to create temporary or permanent forks to test out features (that
would probably be much rarer and harder).
The first suggestion is one that is extremely well supported by DVCS:
any user can clone the repository (which is the equivalent of checking
it out in e.g. Subversion), then hack around (patch bugs, try to find
things to cleanup, or even just create documentation patches which are
already invaluable), save everything in his local repository and when
needed "patchbomb" an erlang dev list (basically send revision as
inline patches) so that they can be reviewed and considered for
inclusion.
The patchbomb step could be replaced by "attach patches to bugs" if a
bug tracker is opened (the former is the way Mercurial development
works, the latter is the way Django's works -- in the Python world)
Conceptual integrity would be preserved because only the people who
already have commit rights now would still have it, and these people
would therefore have veto power over patches and suggestions, being
able to enforce design decisions or style issues (and possibly ignore
patches altogether if they don't have the time).
DVCS would just make contributions easier by allowing everybody to
hack around locally (with history) and have an history of the Erlang
development for documentation as well as bug-finding (bisection
search) purposes.
>
> The language and its libraries are not static, they are maintained by
> Ericsson in a win-win situation for the end user. Until now, the
> Erlang
> maintainers were quite responsive to the Erlang community requests,
> and the
> overall quality of their releases is close to excellence.
The quality wouldn't have any reason to drop as the people holding the
"keys to the kingdom" now would still have it. But on the other hand
the people having the desire to contribute patches or documentation
would have an easier time doing it. Likewise, experiments based on
erlang (loosely or tightly) would be easier for those not part of the
core team.
Linus summarized the general advantages of DVCS [git,hg,bzr|...] and
some of the method in the YouTube video
http://youtube.com/watch?v=4XpnKHJAok8
I think the general advantages could apply here.
Other communities (Scala and OCaml are currently dealing with this)
address this by setting up a repository of third party extensions that
play well together and hopefully are on a migration path to inclusion
in the baseline. This is a very different beast than opening up the
main repository.
I would completely support and agree with an ErlangX repository for
this purpose.
Having a full history, allowing people out of Ericsson to create
On 17 Mar 2008, at 16:39 , Massimo Cesaro wrote:
patches (that could then be forwarded to the core team for inclusion)
or to create temporary or permanent forks to test out features (that
would probably be much rarer and harder).
The first suggestion is one that is extremely well supported by DVCS:
any user can clone the repository (which is the equivalent of checking
it out in e.g. Subversion), then hack around (patch bugs, try to find
things to cleanup, or even just create documentation patches which are
already invaluable), save everything in his local repository and when
needed "patchbomb" an erlang dev list (basically send revision as
inline patches) so that they can be reviewed and considered for
inclusion.
The patchbomb step could be replaced by "attach patches to bugs" if a
bug tracker is opened (the former is the way Mercurial development
works, the latter is the way Django's works -- in the Python world)
Conceptual integrity would be preserved because only the people who
already have commit rights now would still have it, and these people
would therefore have veto power over patches and suggestions, being
able to enforce design decisions or style issues (and possibly ignore
patches altogether if they don't have the time).
DVCS would just make contributions easier by allowing everybody to
hack around locally (with history) and have an history of the Erlang
development for documentation as well as bug-finding (bisection
search) purposes.
The quality wouldn't have any reason to drop as the people holding the
>
> The language and its libraries are not static, they are maintained by
> Ericsson in a win-win situation for the end user. Until now, the
> Erlang
> maintainers were quite responsive to the Erlang community requests,
> and the
> overall quality of their releases is close to excellence.
"keys to the kingdom" now would still have it. But on the other hand
the people having the desire to contribute patches or documentation
would have an easier time doing it. Likewise, experiments based on
erlang (loosely or tightly) would be easier for those not part of the
core team.
i'll answer this question instead; "why should ericsson let the world
at large see the repos and the bug tracker?"
because it would improve the quality of the erlang based Ericsson
products.
mats
On 3/18/08, Massimo Cesaro <massimo...@gmail.com> wrote:
> My point is: we (Erlang professional users) should avoid the generation of
> different Erlang flavors, based on experiments or otherwise.
Why? Python has a handful of flavors based on different experiments
(Jython, Stackless, PyPy), and the community has not collapsed. If
someone really were to implement Jerlang, there's no reason you'd be
forced to use it.
Looking at the HiPE website, they mention it was merged into OTP in
2001, but their publications page lists documents from as far back as
1996. I read into this that HiPE was just such an experimental flavor
for the first 5 years of its life.
Why? Python has a handful of flavors based on different experiments
(Jython, Stackless, PyPy), and the community has not collapsed. If
someone really were to implement Jerlang, there's no reason you'd be
forced to use it.
Looking at the HiPE website, they mention it was merged into OTP in
Are you aware of that many production installations today of OTP are
locally patched versions?
So companies are having the problem of managing their patches and
porting them to new otp releases right now.
As a compromise, could otp start to publish each accepted change-set
between releases? More contained change-sets should make it easier to
fast forward patches and identify what changes in otp clash with your
local patches.
Or an even better compromise: publish them to an open git^W DSCM
repository directly instead of having the community do so.
The point of projects like Stackless and PyPy are not to get the
community to move to a new platform. They're domains for
experimenting with new ideas without complicating the baseline
implementation. If those experiments prove successful, they're likely
to gain wider attention and use (e.g., a lot of Stackless features
have been integrated into PyPy).
> I wouldn't call HiPE a flavor. HiPE is a native compiler derived from Erlang
> specifications which eventually was merged in the language distribution
> itself.
>From the HiPE manual at
http://www.it.uu.se/research/group/hipe/documents/hipe_manual.pdf:
"The HiPE system is intended as an experimental platform for research
in language implementation and compiler techniques as applied to
Erlang. One goal with this system is that [it] should be a complete
Erlang implementation."
This is exactly what Jython/Stackless/PyPy/IronPython/etc. are.
On Tue, Mar 18, 2008 at 4:38 AM, Mats Cronqvist
<mats.cr...@kreditor.se> wrote:
>
> Jim Miller wrote:
> > This is a fine exposition on how the tool can facilitate this process
> > but fails to address the question of "What is the economic (not money)
> > argument for making Erlang open source?"
> i don't understand this question.
> firstly, erlang is already open source.
Poor choice of words on my part. What I meant to say was "What is the
economic argument for opening the central source repository for access
to others outside of Ericcson?"
> secondly, what does "economic (no money)" mean?
Simply the weighing the costs (time, effort) against the return
(improved software quality, a better product). It can involve money
but doesn't necessarily.
> thirdly, argument for whom? Ericsson?
Of course, its their product
>
> i'll answer this question instead; "why should ericsson let the world
> at large see the repos and the bug tracker?"
> because it would improve the quality of the erlang based Ericsson
> products.
>
Open source does not translate into "better quality" software. One
can simply look at freshmeat or any of the other open source projects
out there to see that for every one or two quality open source
projects there are literally thousands that are horrible. As stated
by a few others on this discussion most open source projects are coded
by people who "pray and spray" and who do not go through the rigors of
true testing for a production environment. I know that not everyone
falls into this category, nor am I categorizing people on this list as
such.
Lets look at the cost of what it would take Ericsson to open up the
repository as is being discussed.
- Employee time spent moving things out of clearcase into another
repository while maintaining source history. Yes, one can argue that
simply taking a snapshot of the current state and starting a new
repository might be acceptable but I doubt that would be acceptable to
them. This takes a serious amount of time from an employee who would
otherwise be working on things that directly provide revenue for a
company (primary purpose of companies is to make money for the
shareholders). This is a one to two week investment just to get the
ball rolling.
- Now, for every patch that is submitted through the process, an
employee has to review the submission, verify whether it is acceptable
or not, approve or reject it, write a note to the submitter if its
been rejected (or check-in comment), and roll it into the baseline if
it is. This is for EVERY submission regardless of whether or not it
is part of the business goals of the company. Based on my experience
with having been the make mom for a bunch of good size projects, this
can take a huge amount of time. Very quickly it gets to the point
where that person is doing nothing but dealing with patches.
Interviews with Linus Torvalds indicate that the bulk of his time on
the kernel isn't spent doing coding, its dealing with submissions.
So the overall cost to start something like this is basically
allocating one whole person to it. In the companies I typically work
with (Washington DC area software shops) this is roughly $200K
What are the benefits:
- Quicker inclusion of new capabilities.
True, but are these capabilities that help Ericsson? Does a new
wrapper around a vector graphics library (libcairo) help a company
that writes the embedded software for network hardware? There are
some fantastic libraries out there but what percentage of them are
suitable for Ericsson's goals?
- Better code quality
Maybe. Most code I've seen written on the net does not do nearly
the job required for testing that I require for production quality
code my team releases. If you're Ericsson, and you're achieving 7 or
9-9s of system uptime, a single bug can introduce serious problems.
Do you trust the most software developers out there to be rigorous
enough to do this? Its tedious, hard, unsexy work and most people
just don't do it.
So, does Ericcson get more than $200K a year benefit out of opening
their repository? I'd have a hard time seeing it. A good read is
"The Cathedral and the Bazaar" where Eric Raymond goes through the
thought processes behind what makes for good, successful open source
software. In almost all of the cases, because the original author is
not making money off of the software, the cost/benefit analysis works
out in favor of opening it up. In this case, because Ericsson is
making money from the way they do it, it doesn't make sense.
Additionally because they are paid to do it, there is the motivation
to do the tedious, hard, unsexy work that makes the language and
platform as stable and robust as it is.
And by comparison, the opening of Java seems to have been a success,
but what was behind it? If one looks at the political and corporate
history around Java, they resisted opening for years and finally only
did so under pressure from the surging of the .NET platform. If Sun
wanted Java to survive, they needed the open source community to fully
embrace Java, which they weren't going to do without Sun opening it
up. Sun's business model and strategy for Java required acceptance of
the language which wasn't happening and they were losing ground to
.NET. (Go back in the business news archives and look at what was
going on between Sun and Microsoft, their business strategies, and how
Sun has been losing money, its very interesting reading). Sun was
quickly finding that they were not making money off of Java or its
products using the previous business model so they changed models. It
would be real interesting to see what type of money Sun had to spend
on their Java team before and after the change.
Surely not nearly as many as there are locally patched FreeBSD copies
out there. ;-) (although I wish there were...)
BR,
Ulf W
We have on our agenda to open up step by step in a way that we
think is advantageous for our own job and for the Erlang community.
We want to keep the quality on the same or higher level.
We don't want there to be many dialects of Erlang just because we
don't think that
is good for the community.
But it is a matter of prioritizing this over new features and it is
quite hard to motivate that this has higher priority than implementing
new features or improving performance.
See also comments below.
On 3/17/08, Torbjorn Tornkvist <to...@tornkvist.org> wrote:
> Hi,
>
> I just got back from QCon 2008 in London.
> Lots of interesting presentations, and among them
> one by Joe Armstrong about Erlang.
>
> One talk was about the Eclipse team and how they had
> gone from a closed development style to a style where
> all their internal processes was completely open to public(*).
>
> It is now 10 years (I think...) since Erlang became Open Source.
> The last year, the interest in Erlang has exploded and I think
> it is time to take the next step; open up the development of Erlang.
We have plans to make Erlang available in a SVN repository.
The plan is that we continue development in Clearcase , since we see
absolutely no advantage in changing that. We will synchronize to and
from the SVN repository
automatically at regular intervals (probably daily)
The repository will be available for anyone with read access..
We will be very restrictive with letting in persons as "committers"
>
> Now, I know that Erlang is buried in Clearcase somewhere in the
> dungeons of Ericsson and that it probably will be a major undertaking
> to change the "corporate policy". I think however that the matter
> is worth discussing.
Clearcase is not a problem.
>
> So, suggestions:
>
> - Put Erlang into a modern (distributed) VCS like Mercurial or Git.
I think we should use a widely used VCS , SVN is widely used and probably good
enough for this.
> - Make it possible to browse the full repository.
The whole point with this is that the source should be browsable.
A completely different thing is letting others than us commit to the repository.
This will be very restrictive in order to keep the stability of the product.
It is worth noting that so far 99% of the patches contributed to
erlang-bugs or erlang-patches need to be corrected or rewritten by us
before they make it into a release. And this has nothing with lack of
VCS to do. The reasons are more of the type:
- solution is not done the way we like to have it
- does not work on all platforms
- buggy
- unacceptable dependencies
- unacceptable incompatibility
> - Open up the bug-tracker to become fully public.
This is a different thing not tightly connected to the VCS issue above.
This would involve a lot of work for us and to what benefit?
It would require us to change bug-tracking system or to use 2 different ones
at the same time.
There is 2 sides of this.
1) The Open source user submitting a bug-report and tracking
bug-reports from other users.
2) Our reception of bug-reports, answering etc. From this point of view it is no
big difference for us in continuing as now with bugreports via
erlang-bugs and erlang-questions. There is nothing wrong with 1) but
for us it probably costs more than it tastes.
> - Publish the testserver result, e.g using CruiseControl.
Why?
All tests should be successful before we release anything.
> - Make the whole testsuite available for download and deployment.
We will probably release some of the test suites step by step.
> - Open up any other (as of today) internal dev.tools, wikis etc.
Why?
What do you think you are missing today?
>
> This way, it would be easy for the public to provide well tested
> patches that easily can be tracked. It would make it easier both
> for the OTP team to cherry-pick patches as well as make it easier
> to maintain non-accepted patches in ones own respositories.
See above about the need to rewrite 99% of the patches. Of course the
patches can be better if there are test-suites available, but I don't
expect that most contributors will be able to test and build on 10 to
20 different OS+CPU combinations.
The number of committers must therefore be very restricted. And it
must be possible to restrict commit to specific branches and subtrees
of the code.
A comparision with Eclipse gives:
Eclipse is run as a foundation supported by it's members, I understand is as
this foundation have both people and machinery to run web-sites,
repositories etc.
We could have a foundation for Erlang as well but we are not there
yet. Currently
it is the Erlang group at Ericsson that has to finance this. The
business case for that is not obvious for everyone within Ericsson.
In Eclipse there are a lot of rules and checklists before you can
commit something. I think all these would be relevant for Erlang as
well.
See http://www.eclipse.org/projects/dev_process/index-quick.php
>
> Cheers, Tobbe
>
> (*) www.jazz.net
>
> _______________________________________________
> erlang-questions mailing list
> erlang-q...@erlang.org
> http://www.erlang.org/mailman/listinfo/erlang-questions
>
In summary , we are as said not negative to opening up more.
But there must be a business case for everything we do.
It is not always enough with the motivation, "What's good for Erlang
is good for Ericsson". Even if I mostly agree to tha myself.
/Kenneth Erlang/OTP team Ericsson
> If I remember well, I know of a user contributed patch solving a
> real problem; this was acknowledge by the Erlang team, but was
> rewritten because it contained bugs/constraints/whatever that would
> impact on existing code.
> They'll find problems because is their job to run regression testing
> among the other tests.
This seems like an argument for opening up the test suite to other
people.
On CPAN, for example, there's a system for automated distributed
testing of new modules. Many volunteers around the world, with many
diverse systems, automatically download, install, and test new
uploads. This has the potential for much more complete and diverse
testing than even a single large company can manage (and at very low
cost).
-kevin
Thanx Kenneth, a great reply!
Just to make thing clear: my intention has never been to
suggest that Erlang should open up for public commit rights.
See comment below:
This is excellent news!
However, it is really beneficial to be able to clone a repository.
This way anyone can publish their patches which then can be merged
by anyone with a maintained history.
> We will be very restrictive with letting in persons as "committers"
>> Now, I know that Erlang is buried in Clearcase somewhere in the
>> dungeons of Ericsson and that it probably will be a major undertaking
>> to change the "corporate policy". I think however that the matter
>> is worth discussing.
>
> Clearcase is not a problem.
>> So, suggestions:
>>
>> - Put Erlang into a modern (distributed) VCS like Mercurial or Git.
> I think we should use a widely used VCS , SVN is widely used and probably good
> enough for this.
>> - Make it possible to browse the full repository.
> The whole point with this is that the source should be browsable.
>
> A completely different thing is letting others than us commit to the repository.
> This will be very restrictive in order to keep the stability of the product.
>
> It is worth noting that so far 99% of the patches contributed to
> erlang-bugs or erlang-patches need to be corrected or rewritten by us
> before they make it into a release. And this has nothing with lack of
> VCS to do. The reasons are more of the type:
> - solution is not done the way we like to have it
> - does not work on all platforms
> - buggy
> - unacceptable dependencies
> - unacceptable incompatibility
This is just as it should.
>
>
>
>> - Open up the bug-tracker to become fully public.
> This is a different thing not tightly connected to the VCS issue above.
> This would involve a lot of work for us and to what benefit?
> It would require us to change bug-tracking system or to use 2 different ones
> at the same time.
> There is 2 sides of this.
> 1) The Open source user submitting a bug-report and tracking
> bug-reports from other users.
> 2) Our reception of bug-reports, answering etc. From this point of view it is no
> big difference for us in continuing as now with bugreports via
> erlang-bugs and erlang-questions. There is nothing wrong with 1) but
> for us it probably costs more than it tastes.
For me it would be fantastic if I could see, and search, among the
bug-reports. To be able to see the status, and any planned actions.
We have been maintaining our own version of Erlang/OTP ever since
it became Open Source. I know of at least 3 non-Ericsson companies
in Stockholm that do this. It has always been a hassle: to check if
our fixes has been included in the latest release and if not, to
port them into the new code base. This is also connected with the
next paragraph below.
>
>
>> - Publish the testserver result, e.g using CruiseControl.
> Why?
Because if it was possible to track all the commits you do,
and have them verified by the test suites (e.g eccessible via
CruiseControl). It would be much easier for us to introduce,
maintain and discard our own patches. Perhaps, we even could
come up with alternative solutions when testcases break...
> All tests should be successful before we release anything.
>> - Make the whole testsuite available for download and deployment.
> We will probably release some of the test suites step by step.
>
>> - Open up any other (as of today) internal dev.tools, wikis etc.
> Why?
> What do you think you are missing today?
Well, I think that if there is just one focal point
where all the development and supporting processes take
place the easier should it be for everyone. I mean how
much confidential info do you have hidden that touches
your Erlang development processes ? I can't think of
anything... (ok, I left Ericsson 10 years ago :-)
Not having to think about what can be exposed must
make life easer for you, or ?
--Tobbe
Sorry, but this is the silliest post I've read on this list in years.
-Rick
Ever wish you could take back hitting the send button on your mail client?
I do :-)
Please allow me to apologize--did not mean to come off like such a jerk.
Erlang is a fantastic language and has been for years. I have production
systems requiring 24x7 uptime which I've maintained (fixed bugs, added
features) without ever taking the system down, not even for a second.
I don't see the java community touching the robustness of erlang/OTP anytime
soon--maybe not in our life time. I've talked with my "java friends" about
the differences, and they just can't hold a candle to what you get with erlang
out of the box (let alone if you know how to code OTP). Ditto for the .NET
folks.
Though it might be nice to open the code up more (I'm not giving an opinion
here, btw), I think it's rather far fetched to claim that "it's all over" if
Ericsson doesn't do X and Y. Every few months someone joins the list to remind
us how we all need to find new jobs (because erlang is going to go away real
soon, etc, etc), but that theory doesn't seem to match reality.
-Rick
Torbjorn Tornkvist wrote:
<snip>
>>> I just got back from QCon 2008 in London.
>>> Lots of interesting presentations, and among them
>>> one by Joe Armstrong about Erlang.
>>>
>>> One talk was about the Eclipse team and how they had
>>> gone from a closed development style to a style where
>>> all their internal processes was completely open to public(*).
>>>
>>> It is now 10 years (I think...) since Erlang became Open Source.
>>> The last year, the interest in Erlang has exploded and I think
>>> it is time to take the next step; open up the development of Erlang.
>> We have plans to make Erlang available in a SVN repository.
>> The plan is that we continue development in Clearcase , since we see
>> absolutely no advantage in changing that. We will synchronize to and
>> from the SVN repository
>> automatically at regular intervals (probably daily)
>> The repository will be available for anyone with read access..
>
> This is excellent news!
>
> However, it is really beneficial to be able to clone a repository.
> This way anyone can publish their patches which then can be merged
> by anyone with a maintained history.
>
We can easily set up a continuous import on Launchpad.net from a
subversion repository into a bazaar branch. This will make it easy for
people to clone the repository using bazaar, and should not require any
additional work. People could then maintain their own branches in
bazaar, and easily generate patches, merge from the official trunk, etc.
without needing any rights to the subversion repo.
--
Elliot Murphy | https://launchpad.net/~statik/
While I am partial to GIT already, the following document should be
read for how development can be done using a DSCM, not only git:
http://erlangish.blogspot.com/2007/10/erlware-and-git-development-model.html
These are the values I see:
* OTP would still receive patches over erlang-patches and have the
last word on what gets committed.
* Anyone gets commit access to _a_ repository so their parallel
branches can be kept up to date with the master OTP branch.
+ Experimental branches can live side by side until they reach OTP-quality.
* DSCM encourages change-sets that map 1:1 with an added
feature/bugfix, so the change history in itself is very readable
and educational as to how future similar changes should be made. (No
daily Clear-case-syncing commits that just bring the public repo up to
date please!)
PS.
GIT is not very small or going unmaintained away anytime soon:
> While I am partial to GIT already, the following document should be
> read for how development can be done using a DSCM, not only git:
>
> http://erlangish.blogspot.com/2007/10/erlware-and-git-development-model.html
>
> These are the values I see:
> * OTP would still receive patches over erlang-patches and have the
> last word on what gets committed.
> * Anyone gets commit access to _a_ repository so their parallel
> branches can be kept up to date with the master OTP branch.
> + Experimental branches can live side by side until they reach OTP-quality.
> * DSCM encourages change-sets that map 1:1 with an added
> feature/bugfix, so the change history in itself is very readable
> and educational as to how future similar changes should be made. (No
> daily Clear-case-syncing commits that just bring the public repo up to
> date please!)
Not to mention the incredible usefull git-bisect to search for bugs.
Especially with a small script that automatically tests if a version is
good or bad.
Eg. a bug is reported -> make a unittest and let bisect search where the
error went in.
This is nearly undoable in svn without a really fast lan connection and
even then it's quite timecomsuming.
cu,
michael
--
It's already too late!
Very true, but the same can be said about closed source software. The
reality is that many (most?) open _and_ closed source software has
poor software engineering and testing behind it. One nice thing about
projects that have open development models, is that you can quickly
get a sense of the quality of the software by looking at the test
suite, development practices.
For companies considering using Erlang, this matters. Especially when
it comes to bug tracking and testing. In these areas, transparency
leads to trust. Trust of this form leads to more users.
> Lets look at the cost of what it would take Ericsson to open up the
> repository as is being discussed.
>
> - Employee time spent moving things out of clearcase into another
> repository while maintaining source history. Yes, one can argue that
> simply taking a snapshot of the current state and starting a new
> repository might be acceptable but I doubt that would be acceptable to
> them. This takes a serious amount of time from an employee who would
> otherwise be working on things that directly provide revenue for a
> company (primary purpose of companies is to make money for the
> shareholders). This is a one to two week investment just to get the
> ball rolling.
This is a time consuming thing, but it is a one time cost.
> - Now, for every patch that is submitted through the process, an
> employee has to review the submission, verify whether it is acceptable
> or not, approve or reject it, write a note to the submitter if its
> been rejected (or check-in comment), and roll it into the baseline if
> it is. This is for EVERY submission regardless of whether or not it
> is part of the business goals of the company. Based on my experience
> with having been the make mom for a bunch of good size projects, this
> can take a huge amount of time. Very quickly it gets to the point
> where that person is doing nothing but dealing with patches.
> Interviews with Linus Torvalds indicate that the bulk of his time on
> the kernel isn't spent doing coding, its dealing with submissions.
I'm sure that Linus is completely annoyed to have to deal with all
those contributors to Linux ;-).
But seriously, most software projects (open and closed source) I know
of would _love_ to have more top notch contributors - especially ones
that work for free. I am not talking about hackers that submit buggy,
half baked code, but truly good ones. It is true that as a
development team grows, you have to spend more time coming up with a
good process - for instance one that distributes code review and
testing over a larger set of people.
But, one of the big benefits of opening up development is the
possibility of attracting some really good developers. In my mind,
this is a good thing, not a bad thing.
> So the overall cost to start something like this is basically
> allocating one whole person to it. In the companies I typically work
> with (Washington DC area software shops) this is roughly $200K
Again, I would view it as a truly great thing if people were
submitting so many patches to Erlang that such a person had to be
hired.
> What are the benefits:
>
> - Quicker inclusion of new capabilities.
> True, but are these capabilities that help Ericsson? Does a new
> wrapper around a vector graphics library (libcairo) help a company
> that writes the embedded software for network hardware? There are
> some fantastic libraries out there but what percentage of them are
> suitable for Ericsson's goals?
True, but isn't this a straw man argument. I can also think of many
improvements that Ericsson would potentially be interested in:
- An implementation of the erlang network protocol that runs over high
speed interconnects like infiniband (maybe this already exists?)
- SMP performance optimizations
- Better tools for integrating Erlang with other languages
Opening up the dev process could bring people out of the woodwork to
work on interesting things like this.
> - Better code quality
> Maybe. Most code I've seen written on the net does not do nearly
> the job required for testing that I require for production quality
> code my team releases. If you're Ericsson, and you're achieving 7 or
> 9-9s of system uptime, a single bug can introduce serious problems.
> Do you trust the most software developers out there to be rigorous
> enough to do this? Its tedious, hard, unsexy work and most people
> just don't do it.
I fully agree with this. But, I think this is why an open development
process is important. When people make an important choice - like
what language to use for a project, it is important for them to be
able to judge the quality of the language implementation and the
overall health of the language. With an open source language like
Python, it is very easy to get a sense of both of these things.
People like that.
True, using SVN, git or mercurial won't help this. But, having a well
defined and open development model would help this. In fact, I would
argue that the reason you get such poor quality patches is that there
is not formal, open and well documented way of contributing to Erlang.
If Erlang moved to an open dev model, the Erlang team could simply
document the requirements for all contributed code - things like
coding conventions, documentation, testing, dependencies. This would
encourage people to submit better quality code.
I completely agree with this. I don't think anyone (I hope) is
suggesting that Erlang development be turned into a democratic free
for all. You absolutely need extremely good, devoted people (like the
Erlang team) steering the project and making firm decisions about what
gets in and what does not. But, that doesn't mean the development
process can not be made more open and transparent and still maintain
the full integrity of the project and Ericsson's commercial interests.
> I'm afraid that's it's not so straightforward as it sound.
I do think it is straightforward - if you don't tell people what you
expect (in terms of the quality of code contributions and patches) you
can't expect much from them. Simply laying out the expectations would
go a long way toward improving the quality of code contributions that
the Erlang teams gets from the outside.
What else would people propose to encourage higher quality public
contributions to Erlang?
Cheers,
Brian
These conclusions here sound quite alien to me.
Open source is a meritocracy. It is free for all to participate and
prove their merit.
It is the product that matters.
When R12-B0 didn't prove good enough, then many deployments stayed
with/at/on[1] their R11 releases waiting for a better release.
In the list, I often see promises that detected issues/bugs will be
fixed in the next OTP release.
Often there is an early fix attached as a patch to the mail. Not a bad
thing. Not at all. Very good indeed.
But the patch is nothing else than an informal branch of the OTP
source tree. With a suitable DSCM it can be handled within source
control management.
The branch can be given a name. Additional work can be done on it,
version controlled. It can be merged with some third-party branch
that makes Erlang work on a new processor architecture equipped with
flux-capacitors. Or macs. Or just local patches / other public
pre-release fixes.
As to submitted patches, overworked OTP members can comment with
suggestions to what is required for them to swallow a patch whole for
OTP inclusion if they dont have time to work it right now.
Version controlled work can progress outside of the OTP team, but the
work can be committed back. Using the same tools the whole time.
> > I'm afraid that's it's not so straightforward as it sound.
>
> I do think it is straightforward - if you don't tell people what you
> expect (in terms of the quality of code contributions and patches) you
> can't expect much from them. Simply laying out the expectations would
> go a long way toward improving the quality of code contributions that
> the Erlang teams gets from the outside.
To learn from other projects:
What documentation does linux have about expected code quality besides
the hint to read and burn the gnu code conventions?
Isn't it about publishing your patches to the mailinglist and rework
them until you stop getting witty remarks of how your brain is more
like a "." than an "O" ? (Or stop being handed lollipops :)
Would it be good if good documentation existed so someone new to the
otp source tree and conventions could write a great patch. Yes indeed.
Has this ever happened for any sufficiently interesting project?
My two öre: The best documentation for what makes a patch good is a
history of previous good patches and their evolution from unacceptable
to acceptable. At least it works to do it this way.
[1]
I'm drunk and english prepositions have very vague rules.
This is much more important to fix than any issues people find that
erlang have with punctuation!
> As to submitted patches, overworked OTP members can comment with
> suggestions to what is required for them to swallow a patch whole for
> OTP inclusion if they dont have time to work it right now.
> Version controlled work can progress outside of the OTP team, but the
> work can be committed back. Using the same tools the whole time.
I completely agree with this: the core Erlang/OTP team should *not*
have to rewrite "99%" of the patches (Kenneth's number). They should
not, in fact, have to rewrite any patch (unless the rewrite or
corrections are trivial, or the patch submitter doesn't have the
necessary resources e.g. to test the patch on all erlang-supported
archs): it wastes their time that could be better spent, and it
prevents contributors from learning what is good Erlang/OTP style/patch.
In the problems Kenneth listed, 4 ought to be handled by the patch
submitter through feedback by the core team until acceptance of the
patch:
* "Solution is not done the way we like to have it", a better/more
acceptable approach should be suggested, and the submitter (or someone
else) could then go on and rewrite the patch according to this. Would
also teach contributors and would-be contributors the preferred coding
style/approaches.
* "buggy", if the patch doesn't work then Erlang/OTP core has no
reason to accept it
* "unacceptable dependencies", should be listed and the submitter
could go back to the drawing board, remove them and resubmit the
patch, which would also send hints to contributors and would-be
contributors about acceptable and unacceptable dependencies
* "unacceptable incompatibility", should likewise be listed/stated and
the submitter could then go work on debugging his patch.
That's usually the way it's done on e.g. the Mercurial list, and DVCS
have a very good support for that kind of workflows.
> I completely agree with this: the core Erlang/OTP team should *not*
> have to rewrite "99%" of the patches (Kenneth's number). They should
> not, in fact, have to rewrite any patch (unless the rewrite or
> corrections are trivial, or the patch submitter doesn't have the
> necessary resources e.g. to test the patch on all erlang-supported
> archs):
Take a look at a patch that someone actually submitted, for example:
http://www.erlang.org/pipermail/erlang-questions/2008-January/032345.html
it's a pretty typical user-contributed patch, i.e. it's a small
change, only about five lines of code, it fixes a definite problem,
and, most importantly, it's mostly crap. The guy who submitted the
patch
1. couldn't be bothered testing it properly
and 2. messed with some internals which he most likely doesn't
understand.
and 3. made some arbitrary and undocumented design decisions, e.g.
about how wide the hash should be
and 4. couldn't be bothered coming up with a solution which works
with HIPE and with old versions
i.e. he did the first 5% of the job and then wasn't prepared to put in
the hard work to actually finish. Feedback from the OTP group (your
suggestion) isn't going to fix that.
Matt
(the patch is mine, i.e. I am too lazy to do the job properly. Sorry.)
> i.e. he did the first 5% of the job and then wasn't prepared to put in
> the hard work to actually finish. Feedback from the OTP group (your
> suggestion) isn't going to fix that.
>
Maybe it wouldn't, maybe it would, maybe it wouldn't in that case but
would in others.
> (the patch is mine, i.e. I am too lazy to do the job properly. Sorry.)
I can't judge that you're too lazy to do the job properly as I'm too
lazy to do it at all.
PS: I fully know that tools and processes can't fix everything, or
more precisely can't fix anything, but they can help change things.
What I see here is that an automated test-case that exposes this
problem should be written.
And that's the kind of feedback you could have received.
Something that leaves the test failures as a thorn until the cause is
fixed (or the test removed :). If OTP has survived with this hole so
far, it can probably survive many more months.
Now as for your needs:
If your code depend on the behavior that your patch introduces, then
you probably want to make sure it is included each time you compile
OTP?
Wouldn't you want to forward port it as new OTP makes new releases you
intend to use? Back port it to that old system still running R9?
Wouldn't you want to publish this patch so others with the same issue
can look at your attempt, or perhaps inviting HiPE to implement
something equivalent?
Basically I'm arguing for the use of a DSCM. I don't get the feeling
that you argue against that. Are we actually disagreeing about
something? :)
Perhaps the most important issue I see here isn't if GIT or Mercurial
is chosen over Subversion, as both can already import from subversion.
The issue is if the public repository will have a commit log like:
OTP ClearCase changes for 2008-03-20 11:00
OTP ClearCase changes for 2008-03-20 14:00
OTP ClearCase changes for 2008-03-14 16:00
OTP ClearCase changes for 2008-03-13 09:00
OTP ClearCase changes for 2008-03-11 14:00
OTP ClearCase changes for 2008-03-11 11:00
Or if it will be a more lively one-commit-is-one-fixed-feature and
multiple living branches for pre-release fixes. Something more like
http://lwn.net/Articles/140350/
Or a less busy log: