Include (part of) contrib in 1.1 release?

[Stuart Sierra]

Clojure-contrib is currently closest thing to a "standard library."
It's the first non-core code people use, and the most commonly used.
It has never had a formal release. Some libs are very stable and
widely used; others are not.

Should we consider packaging clojure-contrib with the Clojure 1.1
release?

If yes, should we include only the stable/reliable libs? For now,
let's not worry about *which* libs are stable, just whether or not it
is desirable to separate them.

-SS

[Chas Emerick]

I think folding in some subset of c-c would be a great idea. I
definitely don't think the whole thing should be included, though:
there are a lot of bits that are simply too niche or (as you point
towards) not fully baked. I'm sure everyone has their own list, but
some consensus should be reached as to what libs really merit a
promotion.

Further, I would make the (potentially controversial?) suggestion that
if some sizable number of c-c libs are pulled into clojure proper,
that perhaps c-c has run its course? Between lein and the excellent
maven support in clojure-maven-plugin, it seems like each lib could
very easily make its own way, which would make various things a lot
easier: producing slimmer app-level builds, minimizing maintenance
headaches given absent contrib authors, etc.

- Chas

> --
>
> You received this message because you are subscribed to the Google
> Groups "Clojure Dev" group.
> To post to this group, send email to cloju...@googlegroups.com.
> To unsubscribe from this group, send email to clojure-dev...@googlegroups.com
> .
> For more options, visit this group at http://groups.google.com/group/clojure-dev?hl=en
> .
>
>

[Sean Devlin]

+1000000000000000000000000000000000000000000 (I know I don't have that
many points :))

* Contrib *is* the standard library. Not in name, but in usage.
* Code that uses contrib is usually considered "better" that code that
repeat functionality.
* Most "Clojure in a box" solutions include contrib

Here's why freezing contib will be awesome:
* Expected behavior for production
* Frozen Docs are more reliable. Currently, unless you are neck deep
in Clojure, you don't know where the docs for test & company are
* Freezing Contrib will give a time period to fix bugs, add (sometimes
non-existent) test coverage and improve docs.

The only catch is that there will a lot of debate deciding which libs
are stable. Some of the ones "on the edge" will have very involved
discussions.

Hmmm... then there will be the issue of changing API signature. Is it
acceptable for functions in contrib 1.2 to have a different signature
than contrib 1.1? Would something like Python's 2->3 be possible?
Could any Python Hackers qualified to comment on 2->3 add something?

Overall, though, freezing a version of contrib w/ core is a great
idea. It will take some time & effort.

Sean

[Phil Hagelberg]

Chas Emerick <ceme...@snowtide.com> writes:

> I think folding in some subset of c-c would be a great idea. I
> definitely don't think the whole thing should be included, though:
> there are a lot of bits that are simply too niche or (as you point
> towards) not fully baked. I'm sure everyone has their own list, but
> some consensus should be reached as to what libs really merit a
> promotion.

Strongly agree. It seems like 99% of nontrivial Clojure codebases use
contrib, especially for IO functions.

However, the danger right now is that moving libraries into Clojure
means they're subject to a much slower development pace. An example of
this is Clojure ticket #193[1], in which Stuart Sierra posted a patch to
clojure.test (a library he created) back in September which hasn't been
applied. Please don't interpret this as complaining about Rich being
slow; I know he is making great progress on the deftypes/protocol work
as well as coordinating a 1.1 release, and I'm really glad he's focusing
on that right now!

I just think it would be unfortunate if this were to happen to other
libraries. One solution in this case would be to add Stuart as a
committer to the Clojure repository with the understanding that he
should only touch clojure.test, and even then possibly only after
discussing the changes on this mailing list. Sure there's no way to
enforce this with permissions in git, but we've already built up a level
of trust that would hopefully cover this.

I'm unsure how this would look in the context of other libraries, but I
think having only two committers on the clojure repo is not going to
last forever; perhaps it's time to think about what that's going to look
like going forward.

-Phil

[1] - https://www.assembla.com/spaces/clojure/tickets/193

[Paul Stadig]

It seems to me that what Stuart was asking is if there should be a clojure distribution that has two jars clojure-1.1.jar and clojure-contrib-1.1.jar. This does not necessarily imply that any of the code in clojure-contrib would have to be folded into clojure core, just that there should be an official 1.1 jar for clojure-contrib. Correct me if I'm wrong, Stuart.

I think it would be very useful to make a 1.1 release that includes both clojure core and contrib, and even if it included some libs in contrib that are rough around the edges or incomplete. Not sure it is necessary to pull out only "stable" contrib libs for a 1.1 jar. Just a 1.1 freeze might be enough.

I don't want to detract from the other discussions, but I thought it would be useful to clarify what Stuart is saying.

Paul

[Chas Emerick]

It's definitely true that anything that goes into core will always be
subject to less churn (a good thing in many ways -- and if it seems
likely that a lib really needs rapid attention, then perhaps it
shouldn't be in core). The issue with multiple committers is the
problem of too many cooks in the kitchen.

An alternative approach would be for no further libs to be merged into
the clojure project, but for its build process to make it trivial for
Rich (or anyone else building core) to pull in whatever version of
dependent libs he wants. That would essentially only require the use
of either ivy or git submodules + a tweak to core's ant script, or
maven. Then each lib carries on as its maintainer(s) wish, and Rich
folds in libs blessed by whatever lieutenant he has on various posts.

- Chas

[Rich Hickey]

On Thu, Dec 3, 2009 at 2:04 PM, Chas Emerick <ceme...@snowtide.com> wrote:
> I think folding in some subset of c-c would be a great idea.  I
> definitely don't think the whole thing should be included, though:
> there are a lot of bits that are simply too niche or (as you point
> towards) not fully baked.  I'm sure everyone has their own list, but
> some consensus should be reached as to what libs really merit a
> promotion.
>
> Further, I would make the (potentially controversial?) suggestion that
> if some sizable number of c-c libs are pulled into clojure proper,
> that perhaps c-c has run its course?  Between lein and the excellent
> maven support in clojure-maven-plugin, it seems like each lib could
> very easily make its own way, which would make various things a lot
> easier: producing slimmer app-level builds, minimizing maintenance
> headaches given absent contrib authors, etc.
>

Since core is essentially feature-frozen already for 1.1, let's set
aside this separate discussion on folding in contrib libs, and stay
focused on the path to 1.1.

Thanks,

Rich

[Sean Devlin]

Chas:
I think contrib itself is a good thing, "as a staging ground for the language".  We still need it.  I had a recent experience with duck streams that my help illustrate the point.

I was writing some network code, and needs to transport a string over a Socket.  I knew I could use duck-streams to handle the stream stuff, so all I had to do was get the appropriate input/output  Well, it turns out that this is extremely simple with a Socket.  The code looks like this

(reader (.getOutputStream a-socket)) ;now do stuff with reader

As you may know reader & writer are multimethods, dispatched on class.  I simply added a case to reader & writer, and let recursion take care of the rest.  Now duck-streams can work with Sockets.  I sent Stuart S. and email, seeing if he was interested.  He took a look at my patch, liked it, and applied it the same day.  See Assembla ticket #46.

Time from idea to inclusion in contrib: 4 hours.

Why did this work?
* Contrib is centralized.  I knew of duck-streams because it ships w/ Clojure
* Contrib is open to review.  I was able to add a method that wasn't originally there, and improve the library for everyone.  Open source at its finest (not me, contrib).
* Contrib is loose.  The only parties involved were myself in Stuart.  Much less coordination involved.
* Luck.  The 4 hours number occured because Stuart & I both were hacking Clojure at the same time.

Lein is really good, but I don't think you get all of these factors with individual libraries.

Just my $.02

Sean

[Sean Devlin]

Rich,
Does this mean that there is a potential for a document mismatch, like with test?  If so, what type of mitigations will be in place?  Will it be easy to navigate to the master autodocs that Tom F. created from clojure.org?

Sean

[Chas Emerick]

Rich said his piece on this, so this now a more academic topic.

As far as I can see, contrib's only advantages are that (a) only those
with a lodged SCA can contribute, so migrating anything from contrib
to core has no legal implications, and (b) its libs are easily
discoverable because of its privileged status.

(a) is an issue that is above my pay grade. However, I don't think
the discoverability issue is particularly significant or
insurmountable given the toolchain and community building around lein,
maven, github, clojars, etc.

So, I'd agree about contrib's usefulness as a staging ground for core,
but I'm not sure that that's the case because of technical, community,
or collaboration reasons.

- Chas

[Stuart Sierra]

On Dec 3, 3:21 pm, Rich Hickey <richhic...@gmail.com> wrote:
> Since core is essentially feature-frozen already for 1.1, let's set
> aside this separate discussion on folding in contrib libs, and stay
> focused on the path to 1.1.

Agreed.
-SS

[Sean Devlin]

Chas,
Well, just to continue our academic discussion about contrib...

I found something really cool today that would be awesome in contrib:

http://muckandbrass.com/web/display/~cemerick/2009/12/04/String+Interpolation+in+Clojure

Okay, a cheap shot :). In all seriousness, this is awesome. Kick ass
work. I'm going to play with it for a bit and see how it flows.

I've had a chance to think about your contib points, and I think
you're on the right track. The staging ground is useful because it
provides a chance for code to get peer review, before there is any
discussion to put it in core.

The ability to make code discoverable is something we should tackle as
a community. Something else to think about :)

Seriously, awesome work with <<

Sean