Clojure User Survey, preparation for 1.1

[the.stua...@gmail.com]

If you have trouble viewing or submitting this form, you can fill it out online:
http://spreadsheets.google.com/viewform?formkey=dFJSd1p4YXh0d0VxV0xjdk42MTU5RkE6MA

Clojure User Survey

This user survey will help move Clojure toward an official 1.1 release.

How do you get Clojure? *

• Download release
• Github
• Maven or Ivy

What distribution of Clojure do you use primarily? *

• 1.0 release
• I picked one development snapshot and stuck with it
• I use the latest development snapshot
• I follow Github "master" branch
• I follow Github "new" branch

What about chunked sequences? *

• I don't know what they are
• The help me
• They hurt me
• I don't notice any difference

Powered by Google Docs Report Abuse - Terms of Service - Additional Terms

[Sean Devlin]

A couple questions about 1.1. If Rich or somebody "in the know" could
answer, that would be great

* Is there a "ballpark" estimate for a release date?
* Will there be a 1.1 version of contrib frozen?
* Will there be an announced API freeze before 1.1 release?
* Will there be a period for docs & test coverage to improve/"get up
to speed" with the 1.1 release?

Thanks!
Sean

On Nov 23, 4:55 pm, the.stuart.sie...@gmail.com wrote:
> If you have trouble viewing or submitting this form, you can fill it out  

> online:http://spreadsheets.google.com/viewform?formkey=dFJSd1p4YXh0d0VxV0xjd...

[Kevin Downey]

http://clojure-log.n01se.net/date/2009-11-23.html#16:16

irc discussion of 1.1 that led to stuart's creation and sending of survey

> --
>
> You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
> To post to this group, send email to cloju...@googlegroups.com.
> To unsubscribe from this group, send email to clojure-dev...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/clojure-dev?hl=.
>
>
>



--
And what is good, Phaedrus,
And what is not good—
Need we ask anyone to tell us these things?

[Meikel Brandmeyer]

Hi,

On Nov 23, 11:13 pm, Sean Devlin <francoisdev...@gmail.com> wrote:

> A couple questions about 1.1.  If Rich or somebody "in the know" could
> answer, that would be great
>
> * Is there a "ballpark" estimate for a release date?
> * Will there be a 1.1 version of contrib frozen?
> * Will there be an announced API freeze before 1.1 release?
> * Will there be a period for docs & test coverage to improve/"get up
> to speed" with the 1.1 release?

I'd also like to throw in something, I'm fairly unhappy about.

The work on defprotocol/deftype/chunked seqs/streams etc. is all cool
and important. But sometimes some really trivial patches, which could
save the day for someone right now, go mouldy in the bug tracker. Eg.
fixing clojure.zip/branch?. Rich, please get some Lieutnants filtering
the patches for you and introduce a bug day once a week/every two
weeks/a month (depending on the actual need) to iron out such issues.

A short break on the feature work now and then might also bring a bit
fresh air in your thoughts.

Just my 0,02€.

Sincerely
Meikel

[Kevin Downey]

Rich has said the 1.1 release will not include protocols and
datatypes, check the irc logs for more info

> --
>
> You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
> To post to this group, send email to cloju...@googlegroups.com.
> To unsubscribe from this group, send email to clojure-dev...@googlegroups.com.

> For more options, visit this group at http://groups.google.com/group/clojure-dev?hl=en.

[Meikel Brandmeyer]

Hi,

On Nov 24, 7:13 am, Kevin Downey <redc...@gmail.com> wrote:

> Rich has said the 1.1 release will not include protocols and
> datatypes, check the irc logs for more info

Yes. I know that. I didn't say he should include defprotocol/deftypes
in the next release. I said, that he should not forget about now,
while thinking about tomorrow.

The bug tracker is just a ever growing list of issues. Some of them
are closed after while others still date back from google code... I
understand, that not each and every patch should go in immediately.
Care has to be taken and eg. for extensions like the recently added
bound-fn discussion is required.

But bug fixes waiting for over half a year to be included can be a
serious pain. You either have to rewrite your code to work around the
bug (bleh), or you have to patch the bug in your private clojure
version (again bleh). This is even worse if the bugfix is rather
trivial.

Even a cleanup closing old/now irrelevant/rejected issues or giving
some feedback ("make it work for this use case", "what happens here?")
would be an improvement. I feel honored if I can contribute to
Clojure. I will happily work on my patches, if they don't fulfill the
requirements. I even updated a patch several times to stay in sync
with master. But doing the work to see the patch then rot in the
tracker is - hmm.. - not nice. A "won't accept a patch in that form"
or "this is not an issue" would be better.

Sincerely
Meikel

Example: I asked Rich in IRC in March[1], whether I should open a
ticket, because AOT compiled namespaces loose their metadata. Then Tom
[2] opened a ticket in June. Now Phil[3] again trips over this issue.
It seems to be very low priority, still people hit this issue
regularly. Even a simple "AOT namespaces won't support metadata" would
be an answer.

[1]: http://clojure-log.n01se.net/date/2009-03-06.html
[2]: http://www.assembla.com/spaces/clojure/tickets/130-Namespace-metadata-lost-in-AOT-compile
[3]: http://groups.google.com/group/clojure/browse_thread/thread/b32e872d0233ed39/7f87b913532e828c

[Konrad Hinsen]

On 23.11.2009, at 23:17, Kevin Downey wrote:

> http://clojure-log.n01se.net/date/2009-11-23.html#16:16

Skimming through that IRC log, I noticed a discussion about "direct
linking" of function calls. Isn't this just the same as using a let
around function definition? For example transforming

(defn foo [x] (reduce + (map count x)))

to

(let [map map count count reduce reduce + +]
(defn foo [x] (reduce + (map count x))))

should make foo close over the versions of map, count, reduce, and +
that were available when it was defined. Of course it works only when
unqualified symbols are used for referring to the functions, but
that's the case in 99% of Clojure code.

If my observation is correct, then all it takes to facilitate direct
linking is a macro that applies the above let trick on all of
clojure.core plus a user-supplied list of functions. That macro would
be proposed as part of clojure.core as an optimization aid. The big
advantage: no change to the language, nor even the compiler, and
complete transparency to client code. I'd much prefer this to a
compile-time option that changes the behaviour of var lookup in a
subtle way and doesn't show up explicitly in the source code.

Konrad.

[Sean Devlin]

I also agree w/ Miekel. Tickets could use some faster response
times. What would be a good way to solve this issue?

Sean

> [2]:http://www.assembla.com/spaces/clojure/tickets/130-Namespace-metadata...
> [3]:http://groups.google.com/group/clojure/browse_thread/thread/b32e872d0...

[Rich Hickey]

On Nov 24, 4:03 am, Meikel Brandmeyer <m...@kotka.de> wrote:

> [2]:http://www.assembla.com/spaces/clojure/tickets/130-Namespace-metadata...
> [3]:http://groups.google.com/group/clojure/browse_thread/thread/b32e872d0...

Where's the patch?

Rich

[Meikel Brandmeyer]

Hi,

Am 25.11.2009 um 22:28 schrieb Rich Hickey:

> Where's the patch?

I'm sorry. I provide patches where I can. AOT compilation is beyond
that.

Sincerely
Meikel

[Rich Hickey]

I didn't mean to imply that you write it, but you seemed to be saying
that a fix was sitting around for 6 months waiting to be included.

Rich

[Rich Hickey]

So a client would have to change from defn to defn-direct or defn-fast
or something? It's possible. One thing the current implementation (in
new branch) is doing is detecting the difference between protocol vars
and non, since the former should never be directly linked, but a macro
could be made to do the same. The open questions with direct linking
have to do with how and when you specify you want it, and the
granularity. In some sense, a macro is just saying you always want the
granularity to be per function and the specification the use of the
'direct' macro. Answering those questions will determine if a the
macro solution is easy and flexible enough, vs saying, say, :require
some-ns :linkage :direct or something. Using a macro doesn't change
some of the gotchas, i.e. rebinding a var expecting it to apply to all
nested code will not do so for direct linked code, no matter which
mechanism is used to direct link.

I'm not sure having a bifurcation of defns is something I'd want for
this. Right now my impression is, when it comes to invoked vars
(direct linking is not for *vars*), people are paying a tax for a
degree of dynamism that goes unused 99% of the time, so perhaps the
default should be the other way.

Direct linking (to clojure*) is implemented and up already in the new
branch compiler. I encourage people to play with it and share their
experiences. I'd welcome also an attempt at a macro version so we can
compare ease of use, configuration and performance.

Rich

[Konrad Hinsen]

On 26.11.2009, at 04:51, Rich Hickey wrote:

> So a client would have to change from defn to defn-direct or defn-fast
> or something?

Yes. Or wrap a macro around a whole bunch of functions definitions,
e.g.:

(with-direct-linking
(defn foo1 [] ...)
(defn foo2 [] ...)
... )

where with-direct-linking just expands into a big let. I'd probably
prefer this, as it's only one macro that works with all variants of
defn and also with functions created otherwise, such as a top-level
letfn. Whoever wants defn-direct can easily define that in terms of
with-direct-linking.

> I'm not sure having a bifurcation of defns is something I'd want for
> this. Right now my impression is, when it comes to invoked vars
> (direct linking is not for *vars*), people are paying a tax for a
> degree of dynamism that goes unused 99% of the time, so perhaps the
> default should be the other way.

True, but my concern is conceptual simplicity. Right now there is a
simple model for how vars work, and their usage for naming functions
is not special in any way. Special-casing invoked vars makes the rules
a lot more complicated.

> Direct linking (to clojure*) is implemented and up already in the new
> branch compiler. I encourage people to play with it and share their
> experiences. I'd welcome also an attempt at a macro version so we can
> compare ease of use, configuration and performance.

I'll try a macro version if I find some time.

Konrad.

[Meikel Brandmeyer]

Hi,

On Nov 26, 2:14 am, Rich Hickey <richhic...@gmail.com> wrote:

> I didn't mean to imply that you write it, but you seemed to be saying
> that a fix was sitting around for 6 months waiting to be included.

Ok. That was a misunderstanding. This is an issue sitting around for 8
months (5 of them in the tracker). There is no point in tracking
issues if they are not addressed. Anyway, Tom now did some
investigations and it seems to be a tricky problem.

Sincerely
Meikel

[Konrad Hinsen]

On 26.11.2009, at 04:51, Rich Hickey wrote:

> Direct linking (to clojure*) is implemented and up already in the new
> branch compiler. I encourage people to play with it and share their
> experiences. I'd welcome also an attempt at a macro version so we can
> compare ease of use, configuration and performance.

Have a look at clojure.contrib.macros/with-direct-linking. Some usage
examples:

(with-direct-linking
(defn foo [x] (map count x))
(defn bar [x] (zero? x)))

This basic variant creates a let form with all the public symbols from
clojure.core whose current values are functions but not macros. Other
symbols and/or other namespaces can be given as an optional first
argument that must be a vector of symbols:

(with-direct-linking [clojure.core/map clojure.core/zero?]
(defn foo [x] (map count x))
(defn bar [x] (zero? x)))

This does direct linking for map and zero? but not for count.

(with-direct-linking [*ns*]
(defn foo [x] (map count x))
(defn bar [x] (zero? x)))

This does direct linking for all function-valued vars that already
exist in the current namespace. Watch out when using this
interactively, as the effect of redefining a var can be surprising.

(with-direct-linking [clojure.core clojure.zip clojure.contrib.seq-
utils]
(defn foo [x] (map count x))
(defn bar [x] (zero? x)))


A few observations from first experiments with this macro:

+ It acts on all usages of functions, not only invocations.

- One probably frequent usage pattern is a bit cumbersome: direct
linking for functions defined previously in the same namespace. This
requires wrapping each individual function in with-direct-linking
[*ns*] ...) to make sure that the previous definition is taken into
account.

+ It works even with Clojure 1.0.


Konrad.

[Laurent PETIT]

Hello Konrad,

One question regarding this macro and interaction with other ones :
does this work with called macros inside the user code, where the
macros expand to syntax quoted forms ... ?

2009/11/26 Konrad Hinsen <konrad...@fastmail.net>:

> --
>
> You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
> To post to this group, send email to cloju...@googlegroups.com.
> To unsubscribe from this group, send email to clojure-dev...@googlegroups.com.

> For more options, visit this group at http://groups.google.com/group/clojure-dev?hl=en.
>
>
>

[Konrad Hinsen]

On 26.11.2009, at 16:57, Laurent PETIT wrote:

> One question regarding this macro and interaction with other ones :
> does this work with called macros inside the user code, where the
> macros expand to syntax quoted forms ... ?

Good point... It works in the sense of yielding correct results, but
it doesn't optimize as much as one would hope for. Function calls
written using qualified symbols are not optimized, and as you noticed
those result mostly from the expansion of macros with syntax quote.
That is indeed a problem.

Konrad.

[Allen Rohner]

I've been using this macro for several months:

(defmacro no-var-lookups
"Replaces v with a local variable of the same name. Used to avoid
the overhead of var lookups in performance-intensive code. vs is a
sequence of symbols that should not be looked up."
[vs & body]
(let [letbody (clojure.contrib.seq-utils/flatten (map (fn [x] [x x])
vs))]
`(let [~@letbody]
~@body)))

My application follows the typical 80/20 rule, where 80% of the
runtime cost comes from a few functions, all in the same file. This
default goes the other way from you guys have been talking about, but
I can say I've been using this and have been fairly happy with it. I
typically use it like:

(no-var-lookups [foo bar]
(defn baz [x]
(map foo (bar x))))

My experience matches Rich's in that 99% of the time, I don't use
binding. no-var-lookups would be annoying to use everywhere, but I'm
fine with it only in performance critical code.

One other thing worth considering in this discussion: binding is
extremely useful in testing, and I'd really prefer a solution that
allows flexibility in what can be rebound at test-time.

Allen

[Laurent PETIT]

But it may not totally work, either, since when the code will be
evaluated, you'll freeze the value of the vars in the ref for part of
the user code, but still let code from syntax quoted macro calls do
var lookup. So there's the possibility that the value for the "same"
symbol is not the same at runtime ... :-(

Would it be posssible to catch the macroexpansion of the code, and
transform it by changing all totally qualified symbols with their non
qualified counterpart ? If so, what about aliases, then ?

2009/11/26 Konrad Hinsen <konrad...@fastmail.net>:

[Konrad Hinsen]

On 26 Nov 2009, at 22:52, Laurent PETIT wrote:

> But it may not totally work, either, since when the code will be
> evaluated, you'll freeze the value of the vars in the ref for part of
> the user code, but still let code from syntax quoted macro calls do
> var lookup. So there's the possibility that the value for the "same"
> symbol is not the same at runtime ... :-(

What we are discussing here is an optimization for the frequent case
of functions bound to vars whose value never changes. If you expect a
function to change, you shouldn't use such methods.

> Would it be posssible to catch the macroexpansion of the code, and
> transform it by changing all totally qualified symbols with their non
> qualified counterpart ? If so, what about aliases, then ?

Yes: do a full macroexpansion (clojure.contrib.macro-utils/mexpand-
all) and then use a code walker to replace qualified symbols with
something else. That something else shouldn't be their unqualified
counterparts, because there may well be several qualified symbols that
map to the same unqualified one. One solution is to use gensym to make
new unqualified symbols.

What I dislike about this is again its complexity, as for Rich's
special-casing of invocation. Normal macro expansion is something
every Clojure programmer is familiar with. Anything else must be
learnt in addition.

A practical and yet unanswered question is how frequent such
situations are. How many time-critical operations result from the
expansion of syntax-quoted forms? I don't know.

Konrad.

[Rich Hickey]

While I take your point about special casing of invocation, I'm
unconvinced about the preferability of such a macro. Already there are
cases in which it won't do a complete job. Handling of things defined
in the same file, macroexpansions etc. Also, should people try to
apply it around anonymous functions they will be surprised to see no
benefit as the dereference in the binding will happen every time.

And the intrusiveness on code is very high, either wrapping groups of
defns (it's very troublesome to have giant top-level gathering
expressions), or over and over again around every function you want to
be fast. It is a big problem if fast == cumbersome.

I don't think the complexity story is significantly different,
especially as such a make-it-fast macro will see extensive use, and
will frequently be a recommended technique for perf-boosting.
Explaining what the macro does will cover the same topics as
explaining the optimization. Also, if the macro is going to have the
same ability to be turned on or off, it will require similar knobs and
switches. And a big issue is that it bakes in the granularity of
specification - no way to say that this application calls that library
via direct linking. An optimization can support multiple overlapping
granularity specifications, from a single invocation to library-wide.

Rich

[Konrad Hinsen]

On 28 Nov 2009, at 16:55, Rich Hickey wrote:

> And the intrusiveness on code is very high, either wrapping groups of
> defns (it's very troublesome to have giant top-level gathering
> expressions), or over and over again around every function you want to
> be fast. It is a big problem if fast == cumbersome.

I agree it's a solution only for optimizing speed-critical sections.
When used everywhere, the source code quickly becomes a mess.

I was wondering if it might be a better idea to attack the problem at
the other end, by introducing the notion of constants. A constant
would be a var with a special marker (e.g. on the metadata). Any
attempt to change its value or to create a thread-local binding would
cause na exception. The compiler could then safely replace constants
by their literal value. Function definitions (defn) would be default
create constants, with a separate macro for the rarer case of function
definitions intended for rebinding.

Advantages:
- conceptually simple
- safe (it is impossible to modify a var that the compiler has assumed
to be constant)
- covers all uses of functions, not just invocations
- covers constants that are not functions (e.g. (defconstant pi Math/
PI))

Disadvantages:
- redefining functions during development becomes more cumbersome.

The idea has been floating around already, though not in the context
of optimization:

http://groups.google.com/group/clojure/browse_thread/thread/1227a7a7c7c2b352/fb50f2533464fc12?lnk=gst&q=%22immutable+defs%22#fb50f2533464fc12

http://groups.google.com/group/clojure/msg/d8faa72ab82b7ab6

Konrad.

[Rich Hickey]

I think constants (and more flavors of 'vars' in general) are
interesting, but they don't really fit this problem. That is because
linkage is properly a decision of the caller, not the callee. While
some functions know they are designed to be dynamically rebound (and
in the current implementation can be marked as such with {:dynamic
true} metadata), it is quite another thing to globally decide a
function can never be rebound. There's no reason one set of code
consuming a library can't link to it directly while another doesn't
(in the same app), leaving open the possibility for the latter to do
tracing, mocking etc.

Also, constants trip up on vars that are declared, or otherwise
redefined as part of a bootstrap. In the current implementation the
value of the var is captured at the point of instantiation of the
class of the callee (I've also played with point-of-first-call of
callee), and thus the var in question might have seen a prior mutation
or two. Compatibility with that is desirable.

Rich

[Rich Hickey]

That said, I take your point - if there was a different type of var
(constant isn't quite it) that was used by defn, then that kind of var
could have its own set of rules/expectations regarding linkage (and
not necessarily fixed ones), rather than special-casing current vars
and complicating their semantics.

Keeping in mind the 'anything fixed by the callee can't be undone by
the caller', what capabilities would these have - would they be
rebindable at the root, rebindable dynamically, just like current vars
but link-once by default? And what would they be called?

Rich

[Andy Kish]

One thing that needs to be addressed for the 1.1 release is the
missing docs for stuff that's been moved from contrib into clojure,
particularly clojure.test. Sean has brought this to the attention of
the mailing list before.

I can’t find any documentation on how to use them anymore. They’re not
on clojure.org and they’re not in the clojure contrib docs. Here’s the
list:
clojure.contrib.test-is -> clojure.test
clojure.contrib.stacktrace -> clojure.stacktrace
clojure.contrib.template -> clojure.template
clojure.contrib.walk -> clojure.walk

Andy K.

[Sean Devlin]

I agree, this doc issue should be fixed before release. Here's Tom's
autodoc:

http://tomfaulhaber.github.com/clojure/

It's a workaround. Template & Walk seem to have disappeared entirely,
though. Did they get renamed?

Sean

[Tom Faulhaber]

Doc is available here: http://tomfaulhaber.github.com/clojure/

This will get rolled into an "official" place RSN, but it's there for now. This is not a work-around, but rather a work in progress. It will end up as part of the official clojure stuff just like the contrib autodoc has.

template and walk missing was an oversight in my config file. It's fixed now.

Tom

[Konrad Hinsen]

On 1 Dec 2009, at 13:33, Rich Hickey wrote:

> That said, I take your point - if there was a different type of var
> (constant isn't quite it) that was used by defn, then that kind of var
> could have its own set of rules/expectations regarding linkage (and
> not necessarily fixed ones), rather than special-casing current vars
> and complicating their semantics.
>
> Keeping in mind the 'anything fixed by the callee can't be undone by
> the caller', what capabilities would these have - would they be
> rebindable at the root, rebindable dynamically, just like current vars
> but link-once by default? And what would they be called?

I'd say it all depends on what kind of run-time manipulation you want
to allow. Most applications would work fine with never-changing vars
for referring to functions. The biggest need for modifying already-
defined functions seems to be in development tools: mocking,
profiling, tracing, debugging. Perhaps we should start by compiling a
list of candidate techniques - can anyone come up with more? In
particular with techniques used inside production-quality
applications, rather than development tools?

For development tools, I guess that dynamic rebinding should be
sufficient, so current vars with link-once default would be fine.

Konrad.

[Jonathan Smith]

Maybe I am missing the point, but there are a lot of situations in
which I would want a dynamic rebinding of a function. A big example is
right in clojure proper with memoize.

Imagine if you will a user-application with a login-dialog function.

It is something like:
(defn login []
(make-password-screen)
(wait-until-logged-in-and-return-user-name))

Assuming that the user-management stuff stays in one thread, easy way
to do log-ins would be:
(.start
(new Thread
#(binding [login (memoize login)]
(ui-loop-or-whatever))))

Then you could call (login) whenever you reach a 'secure' piece of
code, and either you would have to re-log-in or you would have already
logged in. (It could be a version of memoize that times-out).

I can think of other situations where I would want to describe
functionalities of my application via binding functions rather than by
passing values.

Imagine writing a simulation program where you have a main program,
and then you want to plug in a few different algorithms to test.

I imagine it being a lot cleaner to have a single generically-named
function for each algorithm-type which gets manipulated, than it would
be to have to pass functions all over the place (or even a hash of
function that has to get stuff pulled out of it every time).

While it is true that 99% of use cases don't require it, that 1% is a
fairly powerful 1%.

If we take 'code as data' to heart, I think this would be a bad move,
as it seems that it would make certain functions more first-class than
other functions. It could lead to a lot of non-transparent behavior
when you rebind a function.

(And for what benefit? Optimization?)

[Richard Newman]

> In particular with techniques used inside production-quality
> applications, rather than development tools?

Don't forget "fixing". If a library function incorrectly dies when
given a keyword, say:

(binding [some-lib-function
(fn [x]
(some-lib-function
(if (keyword? x) (name x) x)))]

;; ... head off into the wilderness.
)

This kind of behavior modification can be used to implement "before"
and "after" methods, argument rewriting a la advice, cross-cutting
logging, and so on. Granted, not often in long-term "production"
fashion (because usually it's to work around a bug which will
eventually be fixed), but most mature production apps I've seen carry
around a bunch of baggage like this. I'm sure we've all seen comments
like ";; bug12345: work around broken libxxx when day is Wednesday"!

Relegating a technique to the "only used for development" toolbox is
like saying nobody will ever need a telnet REPL on a production
server. Anyone who has had to fix a bug affecting a customer or a
production service knows you can never have too many tools :)

[[ Sidenote: things like this are a great reason to have a load
script, or the call into your main loop or whatever, be interpreted
code loaded at runtime from the disk. The ability to add fixes, hacks,
and redefinitions without rebuilding, repackaging, and redeploying a
jar can sometimes save your bacon. ]]