clojure.lib
Stuart Halloway
Rich has asked me to gather some information from the community in
preparation for packaging and releasing a subset of contrib. The goal
here is to create a library (for now call it "clojure.lib") that
contains things that most people need, and make that library's release
process more rigorous than contrib's has been to date.
I plan to create a survey later this week--once I know from you all
what questions to put on it! Here is a partial list of things I have
been thinking about:
What do people want out of an official lib package? (There are
probably many answers here, hopefully with significant overlap.)
Which packages in contrib should be part of clojure.lib?
Of the clojure.lib packages, should the java-specific ones be in a
separate clojure.lib.java? (I'm looking at you, sql).
Who will be responsible for clojure.lib? (Where contrib is a "scratch
your itch" library, clojure.lib should have a unified vision, but also
distributed responsibility.)
Logistics: what should clojure.lib look like in terms of git, assembla,
etc.?
What coding and testing standards should clojure.lib have?
And finally, what important question did I omit? :-)
Thanks,
Stu
James Reeves
> Which packages in contrib should be part of clojure.lib?
I personally use the following packages almost all the time:
- clojure.contrib.def
- clojure.contrib.duck-streams
- clojure.contrib.str-utils
- clojure.contrib.java-utils
And these packages often:
- clojure.contrib.error-kit
- clojure.contrib.except
- clojure.contrib.zip-filter.xml
- clojure.contrib.fcase
I'm not sure if my usage is representative, but those packages would
be my choice for a "clojure.lib" library.
- James
Sean Devlin
c.c.seq-utils
c.c.duck-streams
Sean
> --
> You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
> To post to this group, send email to cloju...@googlegroups.com.
> To unsubscribe from this group, send email to clojure-dev...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/clojure-dev?hl=en.
>
Stuart Sierra
> Rich has asked me to gather some information from the community in
> preparation for packaging and releasing a subset of contrib.
My first thought: if clojure.lib is supposed to be the start of a
standard library, then I would encourage packaging it with the Clojure
distribution.
-SS
Richard Newman
> standard library, then I would encourage packaging it with the Clojure
> distribution.
That certainly avoids cross-versioning issues.
Dimitry Gashinsky
<stuart....@gmail.com> wrote:
> Hi all,
>
> Rich has asked me to gather some information from the community in
> preparation for packaging and releasing a subset of contrib. The goal
> here is to create a library (for now call it "clojure.lib") that
> contains things that most people need, and make that library's release
> process more rigorous than contrib's has been to date.
>
> I plan to create a survey later this week--once I know from you all
> what questions to put on it! Here is a partial list of things I have
> been thinking about:
>
> What do people want out of an official lib package? (There are
> probably many answers here, hopefully with significant overlap.)
>
> Which packages in contrib should be part of clojure.lib?
I am using these for projects at work right now:
command-line
duck-streams
except
pprint
prxml
repl-utils
shell-out
sql
trace
zip-filter.xml
They are very useful and quite stable.
>
> Of the clojure.lib packages, should the java-specific ones be in a
> separate clojure.lib.java? (I'm looking at you, sql).
I do not see sql more java specific, then duck-streams or other packages.
Regards,
DiG
Mark Engelberg
math
combinatorics
duck-streams
shell-out
java-utils
seq-utils
set
The problem with just surveying library usage is that for many of these libraries, often there's only one or two functions that I really believe would be better off in the "core".
Chas Emerick
I agree with this wholeheartedly: if something is to be promoted, then
the appropriate target would be the Clojure project itself. I can't
imagine adding another project and set of artifacts would simplify
things for users, nevermind the maintainers involved.
If that ends up being the case, then I presume the set of libs / fns
up for promotion would be pretty constrained, and likely not tied to
host APIs that don't have direct corollaries elsewhere (e.g. duck-
streams strikes me as an embodiment of a generally-applicable
approach, sql and jmx and shell_out, not so much).
- Chas
Meikel Brandmeyer
def (in particular defvar, name-with-attribues)
error-kit, condition (Is there a "standard" way of sorts of error
handling?)
pprint
Sincerely
Meikel
Konrad Hinsen
> Rich has asked me to gather some information from the community in
> preparation for packaging and releasing a subset of contrib. The goal
> here is to create a library (for now call it "clojure.lib") that
> contains things that most people need, and make that library's release
> process more rigorous than contrib's has been to date.
That's an excellent idea.
> I plan to create a survey later this week--once I know from you all
> what questions to put on it!
I can think of two sets of questions: one directed at Clojure users,
and one at the contributors of clojure.contrib. For Clojure users,
there is:
> What do people want out of an official lib package? (There are
> probably many answers here, hopefully with significant overlap.)
>
> Which packages in contrib should be part of clojure.lib?
I'd reframe that question as "Which contrib packages do you use most
in your code?". Otherwise people will list what they consider cool
even if it doesn't get used that much. It might also be nice to know
exactly what functions people really use. Perhaps some packages should
only be included partially in clojure.lib.
> Of the clojure.lib packages, should the java-specific ones be in a
> separate clojure.lib.java? (I'm looking at you, sql).
Anything else but sql in that category? I don't really know what "Java-
specific" means exactly.
The second set of questions concerns the contributors and we might as
well discuss it here (assuming pretty much everyone is around):
> Who will be responsible for clojure.lib? (Where contrib is a "scratch
> your itch" library, clojure.lib should have a unified vision, but also
> distributed responsibility.)
I'd start with a simple structure, given the current size of the
community. A set of volunteers to do the work, and two or three
"gurus" who take decisions when necessary (such as what to add etc.).
Rich would obviously one of them and I'd just let him nominate the
others. Dictatorship has worked well so far for Clojure.
> Logistics: what should clojure.lib look like in terms of git,
> assembla,
> etc.?
Same setup as for Clojure and current contrib. It seems to work well
and we are used to it. The main difference from contrib would be that
clojure.lib should be included in the official release jars of Clojure.
> What coding and testing standards should clojure.lib have?
I'd say as close as possible to clojure.core. The problem is just that
this is hard to describe in words.
Konrad.
Stuart Halloway
> then the appropriate target would be the Clojure project itself. I
> can't imagine adding another project and set of artifacts would
> simplify things for users, nevermind the maintainers involved.
Good point, will discuss with Rich today.
> If that ends up being the case, then I presume the set of libs / fns
> up for promotion would be pretty constrained, and likely not tied to
> host APIs that don't have direct corollaries elsewhere (e.g. duck-
> streams strikes me as an embodiment of a generally-applicable
> approach, sql and jmx and shell_out, not so much).
I personally want a package of the Java-specific stuff, particularly
io, sql, and shell-out, but not so badly that I would let
clojure.lib.java become an obstacle to shipping clojure.lib.
Stu
Stuart Halloway
>
> I'd reframe that question as "Which contrib packages do you use most
> in your code?". Otherwise people will list what they consider cool
> even if it doesn't get used that much. It might also be nice to know
> exactly what functions people really use. Perhaps some packages
> should only be included partially in clojure.lib.
Good point. Will do.
>> Of the clojure.lib packages, should the java-specific ones be in a
>> separate clojure.lib.java? (I'm looking at you, sql).
>
> Anything else but sql in that category? I don't really know what
> "Java-specific" means exactly.
I think most people use io (split across duck-streams and java-utils),
shell-out, and sql. io and shell-out might be made generic enough to
have parallel implementations on other clojure platforms though.
Going forward, I think libraries that help with tooling will be
important, but I may be the sole user of jmx today. :-)
Constantine Vetoshev
condition
java-utils
seq-utils
duck-streams
str-utils (can probably switch to str-utils2)
pprint
json.read and json.write
I think it would be fantastic to promote java-utils, seq-utils, and
pprint into clojure.core. A clojure.string included with Clojure
itself would be really helpful, too.
Phil Hagelberg
> The problem with just surveying library usage is that for many of
> these libraries, often there's only one or two functions that I really
> believe would be better off in the "core".
Indeed; there are several libs that I've only ever used one or two
functions from.
A while ago I brought up the idea of a clojure.io library. Perhaps
clojure.lib would be a better place for this? I'm all in favour of
canonizing duck-streams, but I'd like to move a handful of functions
from java-utils into it first (delete-file(-recursively), file).
I also think clojure.contrib.logging should be spun out into its own
third-party lib.
Here's the usage of different contrib libraries for our project at work:
duck-streams: 27
* copy: 11
* file-str: 7
* reader: 5
* writer: 5
* spit: 2
* to-byte-array: 3
* read-lines: 1
* append-writer: 1
java-utils: 23
* file: 13
* delete-file-recursively: 8
* as-str: 3
* delete-file: 2
logging: 20
def: 2 (defvar-, defn-memo)
pprint: 2
shell-out: 2
str-utils: 6
* str-join: 4
* re-sub: 3
seq-utils: 4
* indexed: 2
* partition-by: 1
* rand-elt:1
The vast majority of these are included in my clojure.io proposal.
-Phil
Sean Devlin
Where did the proposal end up again?
Sean
Phil Hagelberg
> Where did the proposal end up again?
Uh... nowhere really. Someone mentioned that Rich had some ideas up his
sleeve for a more functional approach to I/O, though it's my
recollection that it had been scrapped since he couldn't find a clean,
stateless way to handle it.
So if the clojure.io proposal (which was mostly just duck-streams-plus)
ends up being clojure.lib.io, there's room in the future for Rich to
include a more idiomatic I/O solution in Clojure proper as clojure.io
while still letting folks Get Stuff Done using clojure.lib. Strikes me
as the best of both worlds.
-Phil
Richard Newman
Thumbs down on including the JSON code in its current state (with no
offense implied).
I use org.danlarkin.json, which does indenting, keywordizes by
default, and works with any Reader or string. The contrib JSON lib
doesn't indent, uses a var to control keywordizing, and only works
with Strings or PushbackReaders.
Stuart Halloway
I would love to see clojure.io move forward with an objective of being
in clojure.lib. Would you be willing to start a branch so we can play
with it?
Stu
Stuart Halloway
Good points. I don't think Dan is a Clojure contributor yet. Would you
mind reaching out to him to see if he would be willing to sign the CA
so we can draw on his json library to produce clojure.lib.json?
Stu
Richard Newman
> you mind reaching out to him to see if he would be willing to sign
> the CA so we can draw on his json library to produce clojure.lib.json?
Done!
Laurent PETIT
Phil Hagelberg
> I would love to see clojure.io move forward with an objective of being
> in clojure.lib. Would you be willing to start a branch so we can play
> with it?
Sure; I'll see if I can do that soon and will report back. But if this
is the first commit to break ground on clojure.lib don't wait for me; I
might take a couple days to get to it. I can (re-)base it off an
existing clojure.lib branch just as easily.
-Phil
Perry Trolard
in contrib: Phil's clojure-http-client. If clojure.lib is meant to be
a standard library, not just a place specifically for mature contrib
libs, I'd vote to include it. (http://github.com/technomancy/clojure-
http-client)
Perry
Phil Hagelberg
Thanks for your vote.
Unfortunately if clojure.lib is part of Clojure itself, it sounds like
Regular Folks won't have commit access to it. While I am fairly happy
with the state of clojure-http-client, I am not ready to give up commit
access on it, especially after the same thing happened to clojure.test.
We had several clojure.test bugs stay open for months on end that had
simple patches ready to be applied, in some cases from the original
author of the library himself.
I would like to think that using automated dependencies is easy enough
now that we should try to keep clojure.lib down to the things that are
likely to be used by a very large portion of Clojure projects, perhaps
half or so.
-Phil
Matt Revelle
>
> Thanks for your vote.
>
> Unfortunately if clojure.lib is part of Clojure itself, it sounds like
> Regular Folks won't have commit access to it. While I am fairly happy
> with the state of clojure-http-client, I am not ready to give up commit
> access on it, especially after the same thing happened to clojure.test.
> We had several clojure.test bugs stay open for months on end that had
> simple patches ready to be applied, in some cases from the original
> author of the library himself.
>
> I would like to think that using automated dependencies is easy enough
> now that we should try to keep clojure.lib down to the things that are
> likely to be used by a very large portion of Clojure projects, perhaps
> half or so.
>
> -Phil
>
With the additional code that will come with clojure.lib, maybe more committers should be added or the process changed?
Dan Larkin
In fact I am a contributor. clojure-json received significant contributions from Darrell Bishop who isn't yet a contributor, but I've just emailed him to ask him if he wouldn't mind sending in his CA.
Feel free to include clojure-json in clojure.lib either wholesale or in parts. I haven't looked at it in a while, so I'm sure it could use some cleaning up.
One nit to pick, though, is the name 'clojure.lib'. Why not just have clojure.json? What's the deal with the .lib? Why is there clojure.xml and clojure.stacktrace, but a (proposed) clojure.lib.io and clojure.lib.json?
Dan
Mark Engelberg
Unfortunately if clojure.lib is part of Clojure itself, it sounds like
Regular Folks won't have commit access to it. While I am fairly happy
with the state of clojure-http-client, I am not ready to give up commit
access on it, especially after the same thing happened to clojure.test.
We had several clojure.test bugs stay open for months on end that had
simple patches ready to be applied, in some cases from the original
author of the library himself.
That's a good point, that libraries moving into the clojure core should be fairly stable since changes will be harder to accomplish. And I don't just mean stable in the sense of well-tested, but stable in the sense that the community seems satisfied with the names and signatures of the various functions, because more people will be depending on these things to not change.
For example, I haven't needed to make any changes to clojure.contrib.math or clojure.contrib.combinatorics for nearly a year. Although no one on this thread (other than I) mentioned using these libraries in their current projects, I get enough feedback on these libraries from time to time that I believe them to be used by others, and quite stable. I'm not really able to judge objectively whether these have any place in the core, but my point is that they are mature enough that they are ready to be in the core if so desired.
So maybe a last-minute overhaul of str-utils and a best-of compilation of io functions, while great ideas for the long-term, are exactly the kind of thing that *shouldn't* be moved to the core, until they've been thoroughly vetted for some number of months.
Stuart Halloway
continue to work in contrib and then move stuff that we agree* is ready.
Stu
*to be defined :-)
Stuart Halloway
Thanks! The name "clojure.lib" is a placeholder so we can talk about
it. If I understand correctly, everyone who has chimed in (so far)
wants "it" to be part of the clojure install (as opposed to a
subproject of contrib or a separate top-level project). If that is our
final answer than I agree with you that the names should be brought in
line with the existing clojure names.
Stu
Chas Emerick
On Jan 26, 7:15 pm, Phil Hagelberg <p...@hagelb.org> wrote:
On the issue of limited bandwidth / attention to bugfixes in included
libs, I'd throw out there that git submodules would seem to be a very
reasonable way to delegate maintenance of "clojure.lib" libraries,
while making it absolutely trivial for Rich & Co. to integrate the
results of that maintenance.
Specifically, making clojure.io or clojure.http-client (or whatever)
separate "projects", pulled into clojure core as submodules and
additional source roots, would perhaps simplify a lot of the process
that crops up w.r.t. maintenance issues -- and provide a clean
mechanism for "lieutenants" to be designated for libraries that are
important, but perhaps not forefront in the minds of the core
committers.
Cheers,
- Chas
Phil Hagelberg
>> Sure; I'll see if I can do that soon and will report back. But if
>> this is the first commit to break ground on clojure.lib don't wait
>> for me; I might take a couple days to get to it. I can (re-)base it
>> off an existing clojure.lib branch just as easily.
>
> I wasn't planning on starting a clojure.lib namespace (yet) -- just
> continue to work in contrib and then move stuff that we agree* is
> ready.
In that case I'm not sure what you mean. If you just want to take a look
at clojure.io in the context of Clojure proper, I've already got a
branch for that:
http://github.com/technomancy/clojure/tree/io
-Phil
Stuart Halloway
That's fine too! My point is that the repos and namespace we ship a
useful library under should be a late bound decision. Build it in
whatever place is easiest for you, and once folks are happy with it we
can decide where it should officially live.
Stu
Timothy Pratley
> The goal here is to create a library (for now call it "clojure.lib") that
> contains things that most people need, and make that library's release
> process more rigorous than contrib's has been to date.
SS has added three rigors to contrib:
1) matched version numbers
2) dependency based
3) packaged release of core+contrib
Which leads to using the packaged releases as a good default choice,
and easy to manage as a dependency. This will take care of all my
needs in terms of reliably having access to 'useful stuff'. So while
I've often asked for parts of contrib to be in core in the past, I am
less motivated to request them now :)
That said, there are plenty of excellent candidates for promotion, and
the possibility of new lib functionality is exciting! The key benefits
I can see of a vetted lib over contrib are:
1) Quality - provides a filter, contrib is very large to
explore/understand/evaluate
2) Stability - can rely on interface remaining fixed
3) Completeness - I consider IO to be the major consideration here
So my questions are:
* Should there be a breaking changes policy (sliding scale
core->lib->contrib? case-by-case? core&lib strict, contrib free?)...
actually I think this is just common-sense so probably no need to be
too formal about it.
* Can IO be locked down? (I am confused by the discussion thread on
clojure.io, it seems there are lots of differing ideas but that is
just my impression - Phil's code looks the business to me).
Regards,
Tim.
Phil Hagelberg
>> In that case I'm not sure what you mean. If you just want to take a
>> look at clojure.io in the context of Clojure proper, I've already got
>> a branch for that:
>>
>> http://github.com/technomancy/clojure/tree/io
>
> That's fine too! My point is that the repos and namespace we ship a
> useful library under should be a late bound decision. Build it in
> whatever place is easiest for you, and once folks are happy with it we
> can decide where it should officially live.
If the point is just to get more people to use it, then distributing it
as a separate library that folks can pull in as a dependency from
clojars is the best way to do that:
That's much easier than asking people to switch to my custom build of
Clojure.
In this case though, everyone already uses duck-streams anyway, so I'm
really not expecting there to be anything in this library that people
don't know about.
-Phil
Sean Devlin
Sean
Stuart Sierra
>The contrib JSON lib doesn't indent
That's a feature; JSON should be compact. If you want readability,
use YAML.
> uses a var to control keywordizing,
That's closer to the spec: JSON keys are strings. What if you get a
key like ":" or "/"?
> and only works with Strings or PushbackReaders.
Easily fixed.
And it's slightly faster. :)
-SS
Stuart Sierra
wrote:
> > and only works with Strings or PushbackReaders.
>
> Easily fixed.
Fixed.
-SS
Richard Newman
Please don't think I'm attacking you or your code.
I'm making no judgment about the code quality of either library: I
just want to make sure that things considered for 'promotion' into a
standard library get 'designed' with lots of input, resulting in a
better and more stable suite. I think that's important, given the
decreased flexibility that c.lib will have, even compared to contrib.
In this case, there are two commonly used JSON libraries, so it makes
a lot of sense to compare their features.
>> The contrib JSON lib doesn't indent
>
> That's a feature; JSON should be compact. If you want readability,
> use YAML.
I don't mean mandatory indenting. Having the option to print output
with indenting is a massive boon to debugging, and even to
programmatic interchange where hyper-efficiency isn't a concern.
In most cases I would rather sacrifice a few dozen bytes and a few
milliseconds for the ability to sanely use a pager or text editor on
JSON output.
If you mean YAML's block-indenting: that's not parseable by JSON
parsers, so it doesn't do the trick. (Is there a Clojure YAML library
that does indenting?)
>> uses a var to control keywordizing,
>
> That's closer to the spec: JSON keys are strings. What if you get a
> key like ":" or "/"?
It might be closer to the spec, but it's not as useful. Maps with
keyword keys are idiomatic Clojure, and map nicely to c.c.sql, MQL
queries, etc. etc.
I'm sure you know that, given that you added the option!
Yes, there are edge cases. ":" and "/" actually aren't (or at least
aren't enforced by Clojure):
user=> (keyword ":")
::
user=> (keyword "/")
:/
but I see no problem with raising an exception if a key can't be
keywordified.
For the record, I don't mind having this behavior be parameterized. It
would be nice if it were an optional argument to the JSON parser: vars
are less convenient when laziness is involved.
>> and only works with Strings or PushbackReaders.
>
> Easily fixed.
Great, thanks.
> And it's slightly faster. :)
Sorry, which one is faster? (I'm all for speed, though I'd rather have
safety, consistency, a stable memory profile, yadda yadda.)
Thanks,
-R
Stuart Sierra
some interesting code to integrate it with the pretty printer...
C.c.json is slightly faster.
-ss
Richard Newman
Fantastic; I look forward to trying this out soon. Thanks for putting
time into all this.