last-var-wins: a simple way to ease upgrade pain

[Stuart Halloway]

For a long time, people have been suggesting that various libraries be
promoted from contrib into clojure core. Last week, we started making
some of the necessary code changes.

==== THE PROBLEM ====
While the changes to Clojure itself are only additive and non-
breaking, they can nevertheless cause breakages in your code:

(1) If you defined your own function 'foo', and Clojure gets a new
core function named 'foo', your definition would cause an error.
(2) Third party libs can create situation #1 behind your back. In
fact, this situation is almost guaranteed to occur when promoting
functions from contrib! If clojure.contrib.seq/group-by goes into
core, then people who were using the contrib version could get an error.

I (and other library authors) spent some time and effort to adapt to
these changes, but it is a PITA. It has already caused cascading
breakage in circumspec, labrepl, leiningen, swank, and incanter (to
name a few).

One solution would be consensus around, and rigorous correct use of,
an elegant dependency management tool. Since that ain't likely, here's
another possibility:

==== A SIMPLER SOLUTION: WARN, DON'T ERR ====
So Rich has given us something else: as of yesterday's commit
ab9a567faecc8cfde4625654fe9bb92988d7494d, attempting to define or
refer a var that is already referred from another namespace causes a
*warning*, not an *error*. So, for the example of 'group-by' being
moved into core, you might see the following warning message when
using a library that isn't updated yet:

WARNING: group-by already refers to: #'clojure.core/group-by in
namespace: incanter.core, being replaced by: #'incanter.core/group-by

That's a lot better than being dead with an error.

==== GETTING RID OF THOSE WARNINGS ====
In this case, the warning is innocuous: Incanter uses the (now
deprecated) contrib group-by, and doesn't get the new one added to
clojure.core. Library authors can then either

(1) Update their code to use the new things core gives them. If they
defined the function themselves, just remove the definition. If they
referred in a third-party implementation, simply removed the use/
require/refer.
(2) Continue to use their own implementation of the name. This
requires *adding* an exclude, e.g. (:refer-clojure :exclude [group-by])


==== PROMOTION AND DEPRECATION ====

With "warn on name collision" in place, the sane way to promote from
one library to another is to copy the vars into their new home, but to
also leave the old ones in place (for a while). The old vars can be
marked with {:deprecated "some.version.no"}, and old clients can
continue to use them.

So, this morning (commit 95dddbbdd748b0cc6d9c8486b8388836e6418848), I
put the promoted seq fns back into contrib as well. The result of this
should be that old code will work (with warnings telling you which
libs need updates) instead of stopping dead (with an error).


==== THIS AIN"T SET IN STONE! ====
We are testing this approach out. If it causes unforeseen foulups, we
can always go back to the old way.

But I for one think it is awesome. It is more consistent with
Clojure's dynamic approach in general. Not only does it make upgrades
less painful, it also makes it easier to try things out at the REPL.
No more "you can't def that cause somebody else did already." Now life
at the REPL is "Do what I told you, period."

I hope you will like this idea and encourage Rich to make it permanent.

Stu

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en

[Douglas Philips]

On 2010 May 5, at 10:32 AM, Stuart Halloway wrote:

> ==== A SIMPLER SOLUTION: WARN, DON'T ERR ====

> That's a lot better than being dead with an error.

Is there a way to turn those warnings back into errors for those
really paranoid of us?

> ==== PROMOTION AND DEPRECATION ====
> With "warn on name collision" in place, the sane way to promote from
> one library to another is to copy the vars into their new home, but
> to also leave the old ones in place (for a while). The old vars can
> be marked with {:deprecated "some.version.no"}, and old clients can
> continue to use them.

Is there some way to control where the warnings "go"? If they happen
in a server whose stdout/stderr aren't being read by a human (or which
has been directed to /dev/null)...

I think it is a great idea in general. IIRC Common Lisp addressed this
with shadowing-import...

-D

[Meikel Brandmeyer]

Hi,

+1 for this change! I often start putting functions in a namespace,
trying around in the Repl and then refactoring stuff out into other
namespaces. But the original functions are still in place. Evaluating
the first namespace often blows up with the "already there" error. So
I have to manually unmap all offending definitions. *ieck* Or take
some other more radical measures like restarting my nailgun server.
This change is really appreciated.

Sincerely
Meikel

[Chas Emerick]

We've been able to treat ns declarations *as* declarations for the
most part, which is nice. IMO, last-var-wins pulls the veil away even
more on the fact that namespaces are probably the most pervasively-
stateful part of clojure.

I remember various proposals floating around a long time ago to load
the entire file of each ns and do some name resolution before defining
any vars so as to avoid the need for declare. Maybe that could be
resurfaced, and local declarations always given priority over referred
namespaces? I've no idea how difficult that would be, and may be way
out of scope for a 1.2 release, but it seems like that would kill a
variety of birds with one stone, and maybe slightly simplify the
mental model that one needs to have in place to understand namespaces.

- Chas

[Sean Devlin]

+1 after 1.2

[Meikel Brandmeyer]

Hi,

On Wed, May 05, 2010 at 08:24:39AM -0700, Chas Emerick wrote:

> variety of birds with one stone, and maybe slightly simplify the
> mental model that one needs to have in place to understand namespaces.

The model is already quite easy, no? Everything is just a sequence
of statements read by Clojure. A ns form does not create a namespace
but declares one. It is only created, if it doesn't exist already.
And from then on this namespace is modified by further statements.

The question with the new style is: to which filter does f refer to?

(ns some.name.space)

(defn f [pred x] (filter pred x)) ; core filter?

(defn filter [pred x] ...)

Sincerely
Meikel

[Laurent PETIT]

Hi,

2010/5/5 Meikel Brandmeyer <m...@kotka.de>:

> Hi,
>
> On Wed, May 05, 2010 at 08:24:39AM -0700, Chas Emerick wrote:
>
>> variety of birds with one stone, and maybe slightly simplify the
>> mental model that one needs to have in place to understand namespaces.
>
> The model is already quite easy, no? Everything is just a sequence
> of statements read by Clojure. A ns form does not create a namespace
> but declares one. It is only created, if it doesn't exist already.
> And from then on this namespace is modified by further statements.
>
> The question with the new style is: to which filter does f refer to?
>
> (ns some.name.space)
>
> (defn f [pred x] (filter pred x)) ; core filter?
>
> (defn filter [pred x] ...)

I bet on core filter, since compilation will hard link to the fn (for
clojure.core) or resolve the symbol into a var, which at compile time
will still be the var pointing to core.

/me crosses fingers and hopes he finally "got things" :-)

[Meikel Brandmeyer]

Hi Laurent,

On Wed, May 05, 2010 at 10:12:52PM +0200, Laurent PETIT wrote:

> > The question with the new style is: to which filter does f refer to?
> >
> > (ns some.name.space)
> >
> > (defn f [pred x] (filter pred x)) ; core filter?
> >
> > (defn filter [pred x] ...)
>
> I bet on core filter, since compilation will hard link to the fn (for
> clojure.core) or resolve the symbol into a var, which at compile time
> will still be the var pointing to core.
>
> /me crosses fingers and hopes he finally "got things" :-)

What happens when re-evaluating (parts) of the namespace? Eg. during
development in the Repl?

There are pitfalls one has to be aware of. I still like the warning
instead of the error.

Sincerely
Meikel

[Laurent PETIT]

2010/5/5 Meikel Brandmeyer <m...@kotka.de>:

> Hi Laurent,
>
> On Wed, May 05, 2010 at 10:12:52PM +0200, Laurent PETIT wrote:
>
>> > The question with the new style is: to which filter does f refer to?
>> >
>> > (ns some.name.space)
>> >
>> > (defn f [pred x] (filter pred x)) ; core filter?
>> >
>> > (defn filter [pred x] ...)
>>
>> I bet on core filter, since compilation will hard link to the fn (for
>> clojure.core) or resolve the symbol into a var, which at compile time
>> will still be the var pointing to core.
>>
>> /me crosses fingers and hopes he finally "got things" :-)
>
> What happens when re-evaluating (parts) of the namespace? Eg. during
> development in the Repl?

Let's see in practice:

lpetit@lpetit-laptop:~/projects$ mkdir -p tmp
lpetit@lpetit-laptop:~/projects$ echo "(ns foo) (def f \"f in foo \")"
> tmp/foo.clj
lpetit@lpetit-laptop:~/projects$ cat tmp/foo.clj
(ns foo) (def f "f in foo ")
lpetit@lpetit-laptop:~/projects$ java -cp clojure/clojure.jar:tmp/ clojure.main
Clojure 1.2.0-master-SNAPSHOT
user=> (ns bar (:use foo))
nil
bar=> (defn before [] (str "before redefining: " f))
#'bar/before
bar=> (before)
"before redefining: f in foo "
bar=> (def f "f in bar")
WARNING: f already refers to: #'foo/f in namespace: bar, being
replaced by: #'bar/f
#'bar/f
bar=> (defn after [] (str "after redefining: " f))
#'bar/after
bar=> (after)
"after redefining: f in bar"
bar=> (before)
"before redefining: f in foo "
bar=> (defn before [] (str "before redefining: " f))
#'bar/before
bar=> (before)
"before redefining: f in bar"
bar=>

So I was right ..., and simulating reloading 'before uses the new
context when in REPL. So those warnings are to be taken very seriously
into account.
But it's not worse than writing a new version of a macro and
forgetting to recompile all the code depending directly or indirectly
on the macro ...

> There are pitfalls one has to be aware of. I still like the warning
> instead of the error.

I also am very interested in this new behaviour. Time will tell, though.

[Meikel Brandmeyer]

Hi,

On Wed, May 05, 2010 at 11:08:26PM +0200, Laurent PETIT wrote:

> But it's not worse than writing a new version of a macro and
> forgetting to recompile all the code depending directly or indirectly
> on the macro ...

Or redefining a defmulti loosing the methods. This was supposed to be
addressed by a recent change, however I haven't tested that, yet.

As I said: one always has to be aware of the pitfalls.

Sincerely
Meikel

[gary ng]

I have a question related to this.

(ns foo (:use clojure.core))

(defn + [x y] x)

'+' is already referred because of the use and is an error right now. But this is a legitimate use of the symbol as 'foo' can be matrix and matrix addition is different from standard number addition. Or is there a better for these kind of primitive symbols overload in clojure or symbol overload in general  ? 

:exclude may not be what I want here as I may still call for the basic + on numbers. In Haskell for example, this is not an issue because of its type class support so I can have multiple '+' doing different things.

[Meikel Brandmeyer]

Hi,

On Wed, May 05, 2010 at 02:56:09PM -0700, gary ng wrote:

> I have a question related to this.
>
> (ns foo (:use clojure.core))
>
> (defn + [x y] x)
>
> '+' is already referred because of the use and is an error right now. But
> this is a legitimate use of the symbol as 'foo' can be matrix and matrix
> addition is different from standard number addition. Or is there a better
> for these kind of primitive symbols overload in clojure or symbol overload
> in general ?
>
> :exclude may not be what I want here as I may still call for the basic + on
> numbers. In Haskell for example, this is not an issue because of its type
> class support so I can have multiple '+' doing different things.

Right now, this can be handled as:

(ns foo
(:refer-clojure :as core :exclude (+)))

(defn +
[matrix1 matrix2]
... (core/+ num1 num2) ...)


Sincerely
Meikel

[Chas Emerick]

On May 5, 3:47 pm, Meikel Brandmeyer <m...@kotka.de> wrote:
> Hi,
>
> On Wed, May 05, 2010 at 08:24:39AM -0700, Chas Emerick wrote:
> > variety of birds with one stone, and maybe slightly simplify the
> > mental model that one needs to have in place to understand namespaces.
>
> The model is already quite easy, no? Everything is just a sequence
> of statements read by Clojure. A ns form does not create a namespace
> but declares one. It is only created, if it doesn't exist already.
> And from then on this namespace is modified by further statements.

I don't know -- I've been so far down the rabbit hole for so long that
I sometimes find it hard to think about where things stand relative to
the ideal.

I do wish that ns declarations were an all-or-nothing proposition.
I've worked myself into some very unusual REPL states when a namespace
loads only partially due to an error (usually an incredibly useless
one, but that's just an impl issue). I can imagine that having a ref
per namespace would make this a straightforward simplification, and
would probably shake out of a CinC impl.

- Chas

[gary ng]

On Wed, May 5, 2010 at 3:18 PM, Meikel Brandmeyer <m...@kotka.de> wrote:

Hi,

On Wed, May 05, 2010 at 02:56:09PM -0700, gary ng wrote:
Right now, this can be handled as:

(ns foo
 (:refer-clojure :as core :exclude (+)))

(defn +
 [matrix1 matrix2]
 ... (core/+ num1 num2) ...)

Thanks. though it is still a bit limited in the sense that in each ns, there can only be one implementation of a symbol,  though I think that is the characteristic of all symbol based dynamic language. 

I am writing a toy DSL where I would like it to be able to do something like this:

(+ today 3) => a date which is 3 days from today

(+ 1 2) => 3

such that the following is possible

(+ today (- 14 remind-prior)) 

=> 2 weeks from today - a fixed remind before days 

=> a date when I am going to be reminded for an event 2 weeks from today

[Meikel Brandmeyer]

Hi,

On 6 Mai, 03:38, gary ng <garyng2...@gmail.com> wrote:

> I am writing a toy DSL where I would like it to be able to do something like
> this:
>
> (+ today 3) => a date which is 3 days from today
> (+ 1 2) => 3
>
> such that the following is possible
>
> (+ today (- 14 remind-prior))
> => 2 weeks from today - a fixed remind before days
> => a date when I am going to be reminded for an event 2 weeks from today

I'm deeply suspicious of such a behaviour. Why would + on a
date mean adding days? Why not hours? minutes? seconds?
months? years? I would always prefer plus-days over such a
behaviour, because I wouldn't have to think everytime, what +
adds.

But that is only a matter of taste, I fancy.

Sincerely
Meikel

[Richard Newman]

>> (+ today (- 14 remind-prior))
>> => 2 weeks from today - a fixed remind before days
>> => a date when I am going to be reminded for an event 2 weeks from
>> today
>
> I'm deeply suspicious of such a behaviour. Why would + on a
> date mean adding days?

Frink (as one example) resolves this through unit tracking. Examples
from <http://futureboy.homeip.net/frinkdocs/#DateTimeHandling>:

#1969-08-19 16:54 Mountain# + 1 billion seconds
AD 2001-04-27 06:40:40.000 PM (Fri) Mountain Daylight Time

#2002-12-25# - now[] -> days
105.70975056709

[gary ng]

On Wed, May 5, 2010 at 10:34 PM, Meikel Brandmeyer <m...@kotka.de> wrote:

I'm deeply suspicious of such a behaviour. Why would + on a
date mean adding days? Why not hours? minutes? seconds?
months? years? I would always prefer plus-days over such a
behaviour, because I wouldn't have to think everytime, what +
adds.

But that is only a matter of taste, I fancy.

Convention, mostly. Say in the security trading settlement world, they use terms like T+3 to mean transaction date + 3 days. Which is why I said, toy DSL. It is used in an implicit context. Everyone in that business knows what T+3 means. Just like we know what CC means when looking at email.

[Meikel Brandmeyer]

Hi,

On 6 Mai, 08:22, gary ng <garyng2...@gmail.com> wrote:

> Convention, mostly. Say in the security trading settlement world, they use
> terms like T+3 to mean transaction date + 3 days. Which is why I said, toy
> DSL. It is used in an implicit context. Everyone in that business knows what
> T+3 means. Just like we know what CC means when looking at email.

Well, then you might want to look into multimethods (or Protocols if
you
follow the edge).

(ns my.name.space

(:refer-clojure :as core :exclude (+)))

(defmulti + (fn [x & _] (type x)))

(defmethod + Integer
[& args]
(apply core/+ args))

(defmethod + org.joda.time.DateTime
[date & days]
(reduce #(.plusDays %1 %2) date days))

Sincerely
Meikel

[Michael Wood]

On 6 May 2010 03:38, gary ng <garyn...@gmail.com> wrote:
>
> On Wed, May 5, 2010 at 3:18 PM, Meikel Brandmeyer <m...@kotka.de> wrote:
>>
>> On Wed, May 05, 2010 at 02:56:09PM -0700, gary ng wrote:
>> Right now, this can be handled as:
>> (ns foo
>>  (:refer-clojure :as core :exclude (+)))
>>
>> (defn +
>>  [matrix1 matrix2]
>>  ... (core/+ num1 num2) ...)
>
> Thanks. though it is still a bit limited in the sense that in each ns, there
> can only be one implementation of a symbol,  though I think that is the
> characteristic of all symbol based dynamic language.
> I am writing a toy DSL where I would like it to be able to do something like
> this:
> (+ today 3) => a date which is 3 days from today
> (+ 1 2) => 3
> such that the following is possible
> (+ today (- 14 remind-prior))
> => 2 weeks from today - a fixed remind before days
> => a date when I am going to be reminded for an event 2 weeks from today

Perhaps this is something like what you're looking for?

http://richhickey.github.com/clojure-contrib/generic.arithmetic-api.html

--
Michael Wood <esio...@gmail.com>

[Mike Meyer]

On Wed, 5 May 2010 23:22:06 -0700
gary ng <garyn...@gmail.com> wrote:

> On Wed, May 5, 2010 at 10:34 PM, Meikel Brandmeyer <m...@kotka.de> wrote:
> > I'm deeply suspicious of such a behaviour. Why would + on a
> > date mean adding days? Why not hours? minutes? seconds?
> > months? years? I would always prefer plus-days over such a
> > behaviour, because I wouldn't have to think everytime, what +
> > adds.
> > But that is only a matter of taste, I fancy.
> Convention, mostly. Say in the security trading settlement world, they use
> terms like T+3 to mean transaction date + 3 days. Which is why I said, toy
> DSL. It is used in an implicit context. Everyone in that business knows what
> T+3 means. Just like we know what CC means when looking at email.

But that just boils down to a matter of taste, with your taste
determined by the industry that dominates your background. So wiring
it into a DSL is reasonable, and it would be nice if the system didn't
get in the way of doing so.

<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/consulting.html
Independent Network/Unix/Perforce consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org