contrib 1.0.0 releases?
Sean Corfield
approval, I wondered what criteria are going to be applied for such
1.0.0 releases?
I ask because I have a couple of open issues on java.jdbc that I plan
to address and then bump to 0.1.0 - but I'd like to get a sense of
what the path to 1.0.0 might look like.
Specifically for java.jdbc, which is pretty stable and in production
use with a number of different databases, what would Clojure/core want
to see changed / added before a 1.0.0 release could be considered?
--
Sean A Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/
World Singles, LLC. -- http://worldsingles.com/
Railo Technologies, Inc. -- http://www.getrailo.com/
"Perfection is the enemy of the good."
-- Gustave Flaubert, French realist novelist (1821-1880)
Stuart Halloway
> approval, I wondered what criteria are going to be applied for such
> 1.0.0 releases?
>
> I ask because I have a couple of open issues on java.jdbc that I plan
> to address and then bump to 0.1.0 - but I'd like to get a sense of
> what the path to 1.0.0 might look like.
>
> Specifically for java.jdbc, which is pretty stable and in production
> use with a number of different databases, what would Clojure/core want
> to see changed / added before a 1.0.0 release could be considered?
Hi Sean,
It's exciting that somebody has brought this question to a head so quickly. Guess we (as in *all* of us) ought to think about guidelines. Here is a quick go, in no particular order:
1. Stability and production use, as you say.
2. Some amount of time in #1 to get community feedback. (How long?)
3. Conformance with Clojure idioms.
4. Careful review to avoid unnecessary dependencies.
5. Simplicity / Power / Focus.
6. Avoid AOT requirement if at all possible.
7. Motivated maintainer.
I am not a big user of java.jdbc, but reading through the source code this morning I have the following questions/comments:
1. What is the "internal" namespace buying us? I have been down this road a few times and never ended up liking it.
2. Exposing with-connection but not get-connection is not simple. Or do we intend for ordinary users to use the internal namespace?
3. The half-earmuffs are not idiomatic. What do they mean?
4. Is there an API that lets you pass the connection directly, instead of relying on the bound one? If not, that doesn't seem simple either.
Stu
Meikel Brandmeyer
Am Mittwoch, 12. Oktober 2011 14:19:17 UTC+2 schrieb Stuart Halloway:
3. The half-earmuffs are not idiomatic. What do they mean?
I use these quite often if a macro expands basically into a suitable call to a driver function. The function version then gets an earmuff on the RHS. I took this from scsh which uses this convention a lotl. clojure.core uses this also (eg. with-bindings*) as well as a trailing -fn (eg. with-redefs-fn) in other situations. I never got any negative feedback about the earmuff while advocating it as a convention. So I'm a little surprised it is not considered idiomatic.
Sincerely
Meikel
Luc Prefontaine
1. Stability and production use:
Well it's a dev tool but it's already used in our code base in some test programs, it's in our repo here.
Most of the code comes from the original contrib.trace tool written by Stuart #2 (or is it #6 :)
2. Some amount of time in #1 to get community feedback
Waiting for comments, however I have no idea who's using it so how much time is reasonable ?
May I point out that I already have white hairs :)
3. Conformance with Clojure idioms;
Waiting for comments...
4. Careful review to avoid unnecessary dependencies
Done (except for clojure.pprint), if I missed some, I need a vision overhaul (which may be the case
since I cannot read small print warnings in some web pages :)))
5. Simplicity / Power / Focus;
Aside from the trace-form addition, the code has remain basically unchanged. Waiting for comments.
6. Avoid AOT requirement if at all possible.
None so far.
7. Motivated maintainer.
I intend to die at my keyboard, I'm 50 so that leaves us hopefully a good 20 years of productive work :)
Comments ?
--
Luc P.
================
The rabid Muppet
Phil Hagelberg
Indeed; this is a very common idiom, though I've never heard it called
"half-earmuff". Perhaps Stuart was referring to something else?
-Phil
Sean Corfield
<stuart....@gmail.com> wrote:
> It's exciting that somebody has brought this question to a head so quickly. Guess we (as in *all* of us) ought to think about guidelines. Here is a quick go, in no particular order:
I'm accumulating this conversation here:
http://dev.clojure.org/display/design/Moving+Projects+Into+Contrib
(is there any reason why Confluence is so slow and seems so unstable?)
> 1. Stability and production use, as you say.
> 2. Some amount of time in #1 to get community feedback. (How long?)
A function of both time and number of users, I'd say? ClojureSphere
can give us some idea of open source usage (but not production use
behind the corporate firewall):
http://clojuresphere.herokuapp.com/data.json
http://clojuresphere.herokuapp.com/java.jdbc
http://clojuresphere.herokuapp.com/tools.logging
> 3. Conformance with Clojure idioms.
Some of us will need more assistance here than others! Which is partly
why I raised this question sooner rather than later.
> 4. Careful review to avoid unnecessary dependencies.
Agreed.
> 5. Simplicity / Power / Focus.
Related to #3, agreed.
> 6. Avoid AOT requirement if at all possible.
Agreed.
> 7. Motivated maintainer.
Yup. I added the following to the where did contrib go page because
I've received several direct emails from people asking about this
particular aspect of the migration:
"If a clojure.contrib namespace is listed here but has no migration
details, that means no one has volunteered to maintain that namespace.
Part of the rationale for modular contrib is that there should be an
active maintainer going forward so that the library can be kept
compatible with new versions of Clojure."
> 1. What is the "internal" namespace buying us? I have been down this road a few times and never ended up liking it.
Good question. I inherited that structure from Stephen and just stuck
with it. I've seen quite a few projects have some sort of "public API"
namespace that then uses/requires some internal / implementation
namespace. I don't know whether it's a _good_ idiom but it seems
fairly common. At World Singles, we've settled into splitting
namespaces (only) when they have a lot of "internal" helper functions
which would otherwise "clutter" the main API namespace.
Actually just looking over our codebase I think it's more a case that
we've split namespaces in two distinct situations:
1. where we have "private" helper functions and want to reduce clutter
in the main API namespace
2. where we have "specialized" functions related to a primary API namespace
#1 is less common than I first thought. #2 itself tends to cover two things:
1. exposing a (non-Clojure) public API in the main namespace which
delegates to a (Clojure idiomatic) internal API in the specialized
namespace
2. exposing rarely used / business logic specific APIs (in the
specialized namespace) alongside a generic API in the main namespace
So that leads me to consider the java.jdbc "internal" namespace as not
buying much - it could be folded into the main namespace with
everything made private (and probably several simplifications could
then be applied)...
> 2. Exposing with-connection but not get-connection is not simple. Or do we intend for ordinary users to use the internal namespace?
Stephen could probably provide more insight there. My experience with
using java.jdbc at World Singles is the internal namespace is not
intended for ordinary users (so making its contents fully private and
merging it into the main namespace would be better).
> 3. The half-earmuffs are not idiomatic. What do they mean?
I've seen this quite a bit in other projects where a foo macro (or
function) delegates to a foo* function for the implementation. Like
Meikel, I'm a little surprised this isn't considered idiomatic.
> 4. Is there an API that lets you pass the connection directly, instead of relying on the bound one? If not, that doesn't seem simple either.
No, there's a lot of more radical work I'd like to do on the API to
provide more composability. At World Singles, we've wrapped java.jdbc
in a layer that provides a much more streamlined, almost
CRUD-data-mapper-like API that's a lot less SQL-y. I figured 0.1.0
would be a "complete" and "stable" evolution of the original c.c.sql
and then what comes between 0.1.0 and 1.0.0 can be an additional set
of more idiomatic APIs?
I wasn't sure how much backward compatibility mattered to the larger
dev team? If c.j.jdbc is meant to be a replacement for c.c.sql - how
"drop-in" compatible does it need to be over time? Isn't stability of
API one of Rich's key tenets?
First item on the Library Coding Standards:
"Get the name and signature right. Rich strongly respects Java's
commitment to not break existing code. In practice, that means we can
tweak the implementation forever, but once we publish a name and
signature we need to stick with it. (In practice I think this means
that we want many people to review the name and sig, even if they
don't review the implementation details.)"
Sean Corfield
Looking over the Library Coding Standards page, I noticed this:
"Use the bang! only for things not safe in an STM transaction."
One of the questions I've had about java.jdbc is "Should the update
operations have ! on them?" and when I look at several other
libraries, they do indeed have ! on functions that perform mutation.
In CongoMongo, for example, it has ! on functions that alter var root
as well as on functions that mutate Java objects (such as set! on the
MongoOptions object) as well as on functions that mutate the data
store (create-collection! destroy! update!). Some of those actually
would be safe if executed repeatedly, some would not.
What is really considered idiomatic here?
Should java.jdbc use ! on operations that mutate the outside world?
(and how does that sit with backward compatibility given that its
current API does not?)
Chas Emerick
On Oct 12, 2011, at 7:59 PM, Sean Corfield wrote:
> On Wed, Oct 12, 2011 at 5:19 AM, Stuart Halloway
> <stuart....@gmail.com> wrote:
>> 3. Conformance with Clojure idioms.
>
> Looking over the Library Coding Standards page, I noticed this:
>
> "Use the bang! only for things not safe in an STM transaction."
>
> One of the questions I've had about java.jdbc is "Should the update
> operations have ! on them?" and when I look at several other
> libraries, they do indeed have ! on functions that perform mutation.
> In CongoMongo, for example, it has ! on functions that alter var root
> as well as on functions that mutate Java objects (such as set! on the
> MongoOptions object) as well as on functions that mutate the data
> store (create-collection! destroy! update!). Some of those actually
> would be safe if executed repeatedly, some would not.
>
> What is really considered idiomatic here?
>
> Should java.jdbc use ! on operations that mutate the outside world?
> (and how does that sit with backward compatibility given that its
> current API does not?)
If you're using a database library, and you invoke a `put` or `update` operation, I think it's safely obvious that that's side-effecting.
I use ! to disambiguate — to highlight a particularly volatile function so that typing/seeing that bang inspires that extra moment or two of consideration.
There's swap! and alter-meta!, but there's also alter-var-root and spit, so I think the best one can say is that each case is a judgement call.
IMO, by all means, tack ! onto fns that aren't STM-safe (and use io! as well), but I wouldn't limit it to that.
Cheers,
- Chas
Alex Miller
Have you considered (or maybe it's already done) wrapping db calls in
java.jdbc in io! for safety? I'm on the fence about how much to use
io! personally.
> --
> You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
> To post to this group, send email to cloju...@googlegroups.com.
> To unsubscribe from this group, send email to clojure-dev...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/clojure-dev?hl=en.
>
>
Sean Corfield
> Have you considered (or maybe it's already done) wrapping db calls in
> java.jdbc in io! for safety? I'm on the fence about how much to use
> io! personally.
I haven't considered it in depth but it does seem like a good idea for
the future. Maybe 0.2.0 or 0.3.0 :)
Alexander Taggart
Sean Corfield
working on addressing a number of point raised by Stuart and wanted to
get feedback on the direction I'm going...
>
> 1. Stability and production use, as you say.
> 2. Some amount of time in #1 to get community feedback. (How long?)
> 3. Conformance with Clojure idioms.
> 4. Careful review to avoid unnecessary dependencies.
> 5. Simplicity / Power / Focus.
> 6. Avoid AOT requirement if at all possible.
> 7. Motivated maintainer.
raised some good points related to 3, 4 and 5...
> 1. What is the "internal" namespace buying us? I have been down this road a few times and never ended up liking it.
weird aliases and a removed a number of delegation functions.
> 2. Exposing with-connection but not get-connection is not simple. Or do we intend for ordinary users to use the internal namespace?
in 0.2.0) except for ClojureQL (which accesses all sorts of things in
the internal namespace, including re-binding the *db* var!). However,
ClojureQL hasn't been updated in five months and could continue to use
0.1.4, until it stops abusing a lot of the internals of java.jdbc. And
only one user has reported this as an issue.
Regardless, I'm moving to an API that allows the db-spec to be passed
into every function. See below.
> 3. The half-earmuffs are not idiomatic. What do they mean?
delegates to a function - I see a lot of cases where the function is
named to match the macro but with * appended. If it's not idiomatic, I
guess we should all stop doing it :)
Regardless, I'm actually moving away from this style (merging the
internal namespace in helped get rid of some of this anyway).
> 4. Is there an API that lets you pass the connection directly, instead of relying on the bound one? If not, that doesn't seem simple either.
https://github.com/seancorfield/jsql
The likely new API is outlined near the bottom of the readme. It's
meant to be simple and more functional than the current API. In
particular, it doesn't rely on dynamic variables and rebinding.
The DSL outlined in the first half of the readme is mostly a
convenience for common SQL operations that supports naming conventions
for entities (tables & columns etc) and identifiers (when building
result sets). Currently that's handled via dynamic vars and rebinding.
My plan is to get community feedback on the direction of the new API
and initially build it on top of clojure.java.jdbc (so it'll be built
on battle-tested code).
Then I'll add the code either directly to clojure.java.jdbc or perhaps
a sub-namespace (thoughts?), with enough machinery that the old API
functions can be rewritten to use the new API under the hood. It'll
probably end up being a fairly complete rewrite of the old code...
Once it's stabilized and feedback from the community is favorable,
I'll make it the primary API and deprecate the old, less idiomatic
API.
Input from everyone is welcome!
(and, just as a note, I won't be taking pull requests on that repo to
ensure I don't have any issues merging that code into
clojure.java.jdbc!)
Sean A Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/
World Singles, LLC. -- http://worldsingles.com/
Shantanu Kumar
>
> Agreed. Here's where I''m going with my current thinking:
>
> https://github.com/seancorfield/jsql
>
> The likely new API is outlined near the bottom of the readme. It's
> meant to be simple and more functional than the current API. In
> particular, it doesn't rely on dynamic variables and rebinding.
>
> The DSL outlined in the first half of the readme is mostly a
> convenience for common SQL operations that supports naming conventions
> for entities (tables & columns etc) and identifiers (when building
> result sets). Currently that's handled via dynamic vars and rebinding.
dynamic vars; in that respect your suggested API looks good to me.
Regarding the DSL/convenience API – I did few similar experiments in
the past and eventually realized:
(a) The convenience API seems to add syntactic clutter over plain SQL,
especially after the initial enthusiasm of using the API fades away.
It is much more convenient to have something like:
(query "SELECT * FROM EMP WHERE age >= :min-age and pay <= :max-
pay"
{:min-age 27 :max-pay 80000})
(b) These days several NoSQL databases are providing JDBC drivers that
do not implement even the SQL-92 standard. A SQL convenience DSL may
not be useful for all JDBC connections.
(c) Database specific DSL/convenience APIs may belong in external
libraries, but not into CJJ.
Shantanu
Chas Emerick
>> 4. Is there an API that lets you pass the connection directly, instead of relying on the bound one? If not, that doesn't seem simple either.
>
> Agreed. Here's where I''m going with my current thinking:
>
> https://github.com/seancorfield/jsql
>
> The likely new API is outlined near the bottom of the readme. It's
> meant to be simple and more functional than the current API. In
> particular, it doesn't rely on dynamic variables and rebinding.
>
> The DSL outlined in the first half of the readme is mostly a
> convenience for common SQL operations that supports naming conventions
> for entities (tables & columns etc) and identifiers (when building
> result sets). Currently that's handled via dynamic vars and rebinding.
>
> My plan is to get community feedback on the direction of the new API
> and initially build it on top of clojure.java.jdbc (so it'll be built
> on battle-tested code).
>
> Then I'll add the code either directly to clojure.java.jdbc or perhaps
> a sub-namespace (thoughts?), with enough machinery that the old API
> functions can be rewritten to use the new API under the hood. It'll
> probably end up being a fairly complete rewrite of the old code...
>
> Once it's stabilized and feedback from the community is favorable,
> I'll make it the primary API and deprecate the old, less idiomatic
> API.
>
> Input from everyone is welcome!
(1) Big +1 to eliminating the use of dynamic vars at the bottom. The approach being taken in jsql looks good, and reminds me of the changes I made in clutch to get away from dynamic vars as well. The other side of that transition is very, very comfortable. :-)
(2) I found the maps-only contract of with-query-results (due to resultset-seq) to be very awkward; I really only want vectors coming out of resultsets. I can imagine that being an atypical use case and maybe bad form in general (since it tightly couples the specifics of query column ordering and the code that works with the results), but I wanted vectors enough to redefine resultset-seq for my application so it only emits vectors. Itself bad form, but it works well enough. I don't know if this point is of general interest enough to warrant consideration for later revs of java.jdbc, but I thought I'd share.
Cheers,
- Chas
Sean Corfield
> (1) Big +1 to eliminating the use of dynamic vars at the bottom. The approach being taken in jsql looks good, and reminds me of the changes I made in clutch to get away from dynamic vars as well. The other side of that transition is very, very comfortable. :-)
> (2) I found the maps-only contract of with-query-results (due to resultset-seq) to be very awkward; I really only want vectors coming out of resultsets. I can imagine that being an atypical use case and maybe bad form in general (since it tightly couples the specifics of query column ordering and the code that works with the results), but I wanted vectors enough to redefine resultset-seq for my application so it only emits vectors. Itself bad form, but it works well enough. I don't know if this point is of general interest enough to warrant consideration for later revs of java.jdbc, but I thought I'd share.
positional information so heavily. Can you elaborate on why this was
useful and how you ensured the ordering is what you expect and how you
maintain it?
al...@puredanger.com
For fixed queries I can imagine maps being much more usable of course.
Kevin Downey
<al...@puredanger.com> wrote:
> We do translation of arbitrary queries and deal generically with the results so vectors would actually be better for us too. Vectors would reduce memory usage and might have better performance, although as always would need to test to know for sure.
And what is good, Phaedrus,
And what is not good—
Need we ask anyone to tell us these things?
Sean Corfield
back as something like this:
{:columns [:col1 :col2 :colC :colD]
:rows [[1 2 3 4]
[1 3 5 7]
[9 8 7 6]]}
Chas? Alex? Others?
Currently, large result sets can be fetched lazily and processed
inside a with-connection / with-query-results, so you are asking for a
fully realized vector of vectors, for processing outside that, yes?
If this format is useful to enough people, it's worth adding an option
for c.j.jdbc to support this.
David McNeil
{:columns [:col1 :col2 :colC :colD]
:rows [[1 2 3 4]
[1 3 5 7]
[9 8 7 6]]}
For our use here at Revelytix, I think it would be more useful to have rows be a lazy sequence of vectors.
-David
Sean Corfield
> For our use here at Revelytix, I think it would be more useful to have rows
> be a lazy sequence of vectors.
realized, so either the query function returns a fully realized result
set (of some format) or you pass in a function to operate on each row
or on the whole result set.
But, yes, if this format is supported, a sequence of vectors would be
more reasonable.