shadowing imports with definitions

[Matthew Flatt]

I've pushed a change to `module` so that a definition can shadow any
import, not just initial imports from the `#lang` line.

This change is intended to reduce compatibility problems that are
created by adding an export to an existing library. The change doesn't
avoid or solve the problem in general, because conflicting imports
still trigger an error. So, it's still better to avoid adding exports
where there's a good chance of conflicts (e.g., because the added name
is short or follows a common pattern). This change might provide a
useful amount of flexibility when adding an export really seems better.

I'm sure we've discussed this change before, and I have been opposed.
Unlike the initial import, where all bindings within a module could be
considered nested under it, other `require`s and definitions all bind
in the same scope within a module. Sam has convinced me that more
liberal shadowing is worth a try, anyway.

For now, order matters: definitions can only shadow bindings from
preceding `require`s. Lifting that constraint, so that a definition can
effectively suppress a later import, may or may not be a good idea; in
any case, it's tricky with the old macro expander, and we can revisit
this ordering question later (if the initial experiment works out).
It's possible for an imported macro to be used in an expansion before a
shadowing definition is discovered, as in internal definition contexts.
An identifier still cannot be imported multiple times (with different
bindings) or defined multiple times.

Naturally, I advise against intentionally writing code that depends on
shadowing imports, not only because it makes code less clear, but
because we should revert the change if the experiment doesn't turn out
well.

[Alexis King]

Would it perhaps be possible to support compilation “warnings” in addition to flat-out errors? I think the logic behind this change is good—simply adding exports being a potentially breaking change is problematic—but being completely silent about it is also possibly confusing. Having a concept of warnings in addition to halting errors would at least warn developers when building modules that something is going on.

Alexis

> --
> You received this message because you are subscribed to the Google Groups "Racket Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-dev+...@googlegroups.com.
> To post to this group, send email to racke...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-dev/20150623133639.202626501C7%40mail-svr1.cs.utah.edu.
> For more options, visit https://groups.google.com/d/optout.

[Matthew Flatt]

Adding a warning sounds useful to me. I've pushed that change.

> https://groups.google.com/d/msgid/racket-dev/F85A7CA9-2E61-4365-A4D6-A357301CAE9
> B%40gmail.com.

[Greg Hendershott]

If I understand correctly, the way to silence the warning would be to
change the require to use only-in or except-in?

So in a sense, the new behavior is a sort of implicit transformation
of (require x) => (require (except-in x (all-defined-here))) [where
"(all-defined-here)" is something I just made up. :)] And the way to
silence the warning is to replace the "all-defined-here" by the
specific unwanted new identifier. Or, change the require to use
only-in. Which reminds me:

In discussions about the package system, and major version changes,
and what it means to be backward compatible, I recall we've observed
that new versions of a package shouldn't provide any new things.
Strictly speaking. But in practice, systems _that_ strict, like COM
interfaces, end up with a lot of ISomeInterface2 gunk. So probably
that's too strict. The alternative is to handle it on the import end,
which means you should always use "only-in" if you want to be sure a
package update won't break you by providing something new. That's
probably the least-worst of the two.

TL;DR Always use only-in and brush thrice daily. But if you're not
100% diligent, at least you'll get some protection plus a warning from
this latest change.

Is that about right?

[Greg Hendershott]

p.s. I guess a third way is to always use prefix-in, which IIUC is the
convention in Clojure. (I feel having prefixes on everything is
distracting, but, it's another possible way.)

[Alexis King]

It continues to be my opinion that the package system needs versioning with dependency version resolution. The NPM system wouldn’t work, since it can maintain multiple different versions in a project at the same time, but that’s probably a bad idea, anyway. The Bower, Clojure, and Ruby package versioning systems would be compatible with Racket’s concept of a package, and they work well. The current package management model is, in my opinion, incredibly dangerous.

Still, that would be a major project, so that aside, I agree that the backwards-compatibility story in Racket is subpar but can still be improved without a complete overhaul of the system (though perhaps not solved completely). As you mention, Clojure is basically prefix-in by default, but it does have a way to require identifiers without prefixing. I don’t think this is an intrinsically bad system, but it’s incompatible with the standard library’s naming conventions (i.e. racket/vector identifiers would need to be unprefixed so you could import them with (prefix-in vector- racket/vector)).

Just some musings of mine—this is a problem I’ve spent a sizable amount of time thinking about, although I haven’t done much to try and solve it.