Namespaced maps

[Alex Miller]

Just wanted to bring to your attention a new piece of syntax: namespace maps (http://dev.clojure.org/jira/browse/CLJ-1910). We are trying to make working with namespaced keys/symbols in maps easier. Here's the text from the ticket in case there are questions/comments.

A common usage of namespaced keywords and symbols is in providing attribute disambiguation in map contexts:

{:person/first "Han" :person/last "Solo" :person/ship {:ship/name "Millenium Falcon" :ship/model "YT-1300f light freighter"}}

The namespaces provide value (disambiguation) but have the downside of being repetitive and verbose.

Namespaced maps are a reader (and printer) feature to specify a namespace context for a map.

• Namespaced maps combine a default namespace with a map and yield a map.
• Namespaced maps are reader macros starting with #: or #::, followed by a normal map definition.
  • #:sym indicates that sym is the default namespace for the map to follow.
  • #:: indicates that the default namespace auto-resolves to the current namespace.
  • #::sym indicates that sym should be auto-resolved using the current namespace's aliases OR any fully-qualified loaded namespace.
    • These rules match the rules for auto-resolved keywords.
• A namespaced map is read with the following differences from normal maps:
  • A keyword or symbol key without a namespace is read with the default namespace as its namespace.
  • Keys that are not symbols or keywords are not affected.
  • Keys that specify an explicit namespace are not affected EXCEPT the special namespace _, which is read with NO namespace. This allows the specification of bare keys in a namespaced map.
  • Values are not affected.
  • Nested map keys are not affected.
• The edn reader supports #: but not #:: with the same rules as above.
• Maps will be printed in namespaced map form only when:
  • All map keys are keywords or symbols
  • All map keys are namespaced
  • All map keys have the same namespace

Examples:

;; same as above - notice you can nest #: maps and this is a case where the printer roundtrips
user=> #:person{:first "Han" :last "Solo" :ship #:ship{:name "Millenium Falcon" :model "YT-1300f light freighter"}}
#:person{:first "Han" :last "Solo" :ship #:ship{:name "Millenium Falcon" :model "YT-1300f light freighter"}}

;; effects on keywords with ns, without ns, with _ ns, and non-kw
user=> #:foo{:kw 1, :n/kw 2, :_/bare 3, 0 4}
{:foo/kw 1, :n/kw 2, :bare 3, 0 4}

;; auto-resolved namespaces (will use user as the ns)
user=> #::{:kw 1, :n/kw 2, :_/bare 3, 0 4}
:user/kw 1, :n/kw 2, :bare 3, 0 4}

;; auto-resolve alias s to clojure.string
user=> (require '[clojure.string :as s])
nil
user=> #::s{:kw 1, :n/kw 2, :_/bare 3, 0 4}
{:clojure.string/kw 1, :n/kw 2, :bare 3, 0 4}

;; to show symbol changes, we'll quote the whole thing to avoid evaluation
user=> '#::{a 1, n/b 2, _/c 3}
{user/a 1, n/b 2, c 3}

;; edn reader also supports (only) the #: syntax
user=> (clojure.edn/read-string "#:person{:first \"Han\" :last \"Solo\" :ship #:ship{:name \"Millenium Falcon\" :model \"YT-1300f light freighter\"}}")
#:person{:first "Han", :last "Solo", :ship #:ship{:name "Millenium Falcon", :model "YT-1300f light freighter"}}

Open questions:

1. Current patch does not allow whitespace between ns and map definition. Should we allow whitespace there for formatting?
2. Should autoresolution support fully-qualified loaded namespaces? (like auto-resolved keywords)
3. Should there be a printer flag to suppress printing namespaced maps?
4. Patch changes print-method for maps but not pprint. Should it?

[Steve Miner]

> A common usage of namespaced keywords and symbols is in providing attribute disambiguation in map contexts:
>
> {:person/first "Han" :person/last "Solo" :person/ship {:ship/name "Millenium Falcon" :ship/model "YT-1300f light freighter”}}
>
> The namespaces provide value (disambiguation) but have the downside of being repetitive and verbose.

I must be missing something. The stated downside is not a problem for my code. It effortlessly handles the namespaced keywords. :-)

Could you say more about the issue? Is it a problem of human readability?

Steve Miner

[Alex Miller]

On Thu, Apr 7, 2016 at 4:13 PM, Steve Miner <steve...@gmail.com> wrote:

Could you say more about the issue?  Is it a problem of human readability?

Yes, generally people find maps full of namespaced keywords to be harder to read (and write).

[Nicola Mometto]

Following up the conversation started in #clojure-dev, here's some more thoughts on this after sleeping on it.

- I really don't like the idea of special-casing `_` here, users are already confused about idioms like `[.. & _]` thinking that `_` is some special token/variable. Making it an actual special token in some occasion wouldn't help.
- I also don't like how we're making the single `:` auto-qualify keywords when used within the context of a qualifying-map. Auto-qualifying keywords has always been the job of the double `::`, changing this would introduce (IMO) needless cognitive overhead.
- The current impl treats `#:foo{'bar 1}` and `'#:foo{bar 1}` differently. I can see why is that, but the difference might be highly unintuitive to some.
- The current proposal makes it feel like quote is auto-qualifying symbols (*), when that has always been the job of syntax-quote. (*) I know that's not correct, but that's how it's perceived.

Here's an alternative syntax proposal that handles all those issues:

- No #::, only #:foo or #::foo
- No auto-resolution of symbols when the namespaced-map is quoted, only when syntax-quoted
- No special-casing of `_`
- No auto-resolution of single-colon keywords

Here's how the examples in that ticket would look:

#:person{::first "Han", ::last "Solo", ::ship #:ship{::name "Millenium Falcon", ::model "YT-1300f light freighter"}}

;=> {:person/first "Han" :person/last "Solo" :person/ship {:ship/name "Millenium Falcon" :ship/model "YT-1300f light freighter"}}

#:foo{::kw 1, :n/kw 2, :bare 3, 0 4}
;=> {:foo/kw 1, :n/kw 2, :bare 3, 0 4}

{::kw 1, :n/kw 2, bare 3, 0 4}
;=> {:user/kw 1, :n/kw 2, :bare 3, 0 4}

Note in the previous example how we don't need `#::` at all -- `::` already does that job for us

(require '[clojure.string :as s])

#::s{::kw 1, :n/kw 2, bare 3, 0 4}
;=> {:clojure.string/kw 1, :n/kw 2, :bare 3, 0 4}

`{a 1, n/b 2, ~'c 3}
;=> {user/a 1, n/b 2, c 3}

Again, no need for `#::` here, we can just rely on the existing auto-qualifying behaviour of `.

`#:foo{a 1, n/b 2}
;=> {foo/a 1, n/b 2}

I think this would be more consistent with the existing behaviour -- it's basically just making `#:foo` or `#::foo` mean: in the top-level keys of the following map expression, resolve keywords/symbols as if *ns* was bound to `foo`, rather than introducing new resolution rules and special tokens.

I realize that this proposal wouldn't work with EDNReader as-is, given its lack of support for `::` and "`". I don't have a solution to that other than "let's just bite the bullet and implement them there too", but maybe that's not acceptable.

> --
> You received this message because you are subscribed to the Google Groups "Clojure Dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to clojure-dev...@googlegroups.com.
> To post to this group, send email to cloju...@googlegroups.com.
> Visit this group at https://groups.google.com/group/clojure-dev.
> For more options, visit https://groups.google.com/d/optout.

[Rich Hickey]

Yes, there is a cognitive load when one encounters many long, identical prefixes in pulling out the pertinent parts. And, there is the ‘typing’ problem - more keystrokes is just a disincentive for some people.

I think this engenders far less use of fully-qualified keywords than there should be. I’m hoping to make that more straightforward and common. This will involve things like this namespaced-map support, as well as corresponding enhancements to destructuring to come.

With this in place, I think we will be able to hang more value props on namespaced keywords. Stay tuned… :)

Rich

[Alex Miller]

I replied on the ticket to to this proposal.

[Mike Drogalis]

To kick in a real world example, we use namespaced keywords everywhere in Onyx. It's clearly worth it, so we continue to do it - but man it can be a pain: https://github.com/onyx-platform/onyx/blob/0.9.x/src/onyx/peer/function.clj#L20-L21

[Ambrose Bonnaire-Sergeant]

Thanks Mike, that's a compelling example.

Ambrose

[Gary Fredericks]

This is a bit prior to the proposal, but I would appreciate an explanation of the benefits of namespace keywords in different scenarios. The canonical usage in my mind is for adding keys to maps whose contents you don't fully control (like metadata on vars, ring middleware that adds keys to a request map, etc.). I'm less clear on the benefits for data that the user has more direct control over: e.g., when Mike says "it's clearly worth it", I have no idea what sort of benefits he has in mind.

Most of the other times I've thought about namespaced keywords it involves a context where I have several different sets of keys from different sources that I need to represent together, but I can do this just as well with nested maps. The tradeoffs between those two approaches aren't obvious to me.

Gary Fredericks
(803)-295-0195
frederi...@gmail.com
gfredericks.com

[Mike Drogalis]

On Fri, Apr 8, 2016 at 8:20 AM, Gary Fredericks <frederi...@gmail.com> wrote:

This is a bit prior to the proposal, but I would appreciate an explanation of the benefits of namespace keywords in different scenarios. The canonical usage in my mind is for adding keys to maps whose contents you don't fully control (like metadata on vars, ring middleware that adds keys to a request map, etc.). I'm less clear on the benefits for data that the user has more direct control over: e.g., when Mike says "it's clearly worth it", I have no idea what sort of benefits he has in mind.

The example that I posted falls under the category of a map that users can add keys to, similar to how Ring operates. Sorry for the lack of explanation there. 

[Gary Fredericks]

The only idea I've had of a benefit beyond the convenience of "now I can put all these keys in the same map" is that it allows data to take more meaning with it when it escapes your system, similar to the way edn's tagged literals can be processed by generic tools.

Gary Fredericks
(803)-295-0195
frederi...@gmail.com
gfredericks.com

[Sean Corfield]

On 4/8/16, 6:39 AM, "Rich Hickey" <cloju...@googlegroups.com on behalf of richh...@gmail.com> wrote:
>I think this engenders far less use of fully-qualified keywords than there should be. I’m hoping to make that more straightforward and common. This will involve things like this namespaced-map support, as well as corresponding enhancements to destructuring to come.

I agree. I used namespaced keywords in a handful of places but I would use them a lot more if they easier to use (less typing, less clutter), because I like them _in principle_.

I think the proposal overall is great but I agree that assigning special meaning to _ in this context seems like the wrong thing to do. Since that seems present only to support literals that are a mix of namespaced keywords and bare keywords – can you (Rich, or Alex?) explain the use case that makes it worthwhile to have these bare keywords in the same map?

Sean Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"Perfection is the enemy of the good."
-- Gustave Flaubert, French realist novelist (1821-1880)

[Alex Miller]

On Friday, April 8, 2016 at 11:07:23 AM UTC-5, Sean Corfield wrote:

 

I think the proposal overall is great but I agree that assigning special meaning to _ in this context seems like the wrong thing to do. Since that seems present only to support literals that are a mix of namespaced keywords and bare keywords – can you (Rich, or Alex?) explain the use case that makes it worthwhile to have these bare keywords in the same map?

I do not think we have a strong use case for needing this, although it's something you could imagine wanting. As I mentioned on the ticket, consider this a placeholder feature, up for removal or replacement before release. 

[Michał Marczyk]

On 8 April 2016 at 16:43, Mike Drogalis <madru...@gmail.com> wrote:
>
> To kick in a real world example, we use namespaced keywords everywhere in Onyx. It's clearly worth it, so we continue to do it - but man it can be a pain: https://github.com/onyx-platform/onyx/blob/0.9.x/src/onyx/peer/function.clj#L20-L21

Note that if onyx.core is an actual Clojure namespace, you could save
some typing on the lines you highlighted:

{:keys [onyx.core/results onyx.core/messenger onyx.core/state
onyx.core/replica onyx.core/peer-replica-view
onyx.core/serialized-task] :as event}

by requiring [onyx.core :as c] (for example) and then saying (note the
double colon)

::c/results ::c/messenger ::c/state …

Minimal example:

(require '[clojure.string :as s])

(let [{:keys [::s/foo]} {:clojure.string/foo 1}] foo)
;= 1

Cheers,
Michał

[James Reeves]

On 8 April 2016 at 15:43, Mike Drogalis <madru...@gmail.com> wrote:

To kick in a real world example, we use namespaced keywords everywhere in Onyx. It's clearly worth it, so we continue to do it - but man it can be a pain: https://github.com/onyx-platform/onyx/blob/0.9.x/src/onyx/peer/function.clj#L20-L21

My reading of the proposed syntax indicates that it wouldn't solve this particular problem, or have I misunderstood?

If you have a destructuring map like:

{:keys [onyx.core/results onyx.core/messenger onyx.core/state

        onyx.core/replica onyx.core/peer-replica-view

        onyx.core/serialized-task] :as event}

Then the keys are held in a vector. My understanding of the new syntax is that it only affects the keys of maps, and doesn't affect nested data structures. You therefore couldn't write:

#:onyx.core{:keys [results messenger state

                   replica peer-replica-view

                   serialized-task] :as event}

Or even:

#:onyx.core{results :results messenger :messenger

            state :state replica :replica

            peer-replica-view :peer-replica-view

            serialized-task :serialized-task :as event}

I wonder if it might be an idea to expand this syntax to vectors as well? That way destructuring keys could be made concise:

{:keys #:onyx.core[results messenger state

                   replica peer-replica-view

                   serialized-task] :as event}

Apologies if this idea has already been suggested and I just missed it.

- James

[James Reeves]

On 8 April 2016 at 19:10, Michał Marczyk <michal....@gmail.com> wrote:

On 8 April 2016 at 16:43, Mike Drogalis <madru...@gmail.com> wrote:

Note that if onyx.core is an actual Clojure namespace, you could save
some typing on the lines you highlighted:

{:keys [onyx.core/results onyx.core/messenger onyx.core/state
            onyx.core/replica onyx.core/peer-replica-view
onyx.core/serialized-task] :as event}

by requiring [onyx.core :as c] (for example) and then saying (note the
double colon)

::c/results ::c/messenger ::c/state …

This is deviating off topic a little, but as Mike points out, you can only shorten keyword namespaces that relate to actual code namespaces.

So:

    (alias 'fb 'foo.bar)

Would fail if 'foo.bar was not an existing "real" namespace.

However, with Datomic and other systems that use namespaced keywords heavily, it's often the case that keyword namespaces don't relate to code namespaces. While we're talking about ideas to make using namespaced keywords easier, perhaps we could enhance the behaviour of clojure.core/alias to support aliasing arbitrary namespaces, and not just those defined by in-ns?

- James

[Alex Miller]

Destructuring is a separate area and one that Rich and I are working on right now. 

Your last example on vectors is an interesting idea.

--

[Francis Avila]

I second this re some ability to create a keyword alias for non-code namespaces. I would use double-colon (::a/b) much, much more if I had this ability and the need for this proposal becomes much less compelling. (This proposal is still a better story for edn).

I am concerned that this proposed syntax gives its keywords no visual difference from an unnamespaced keyword. It seems like it would be easy to miss the map prefix, or forget to do the mental arithmetic to prepend the namespace. The advantage of double-colon has always been that it makes clear you need to do some indirection.

What if we do this instead (although I'm not sure how the reader could implement it cleanly):

#:person{::first "John" ::last "Doe" :id 10}
;=> {:person/first "John" :person/last "Doe" :id 10}

Pros

1. The double colon without namespace part is currently illegal. Now it is legal only inside these namespaced maps.
   
2. Requiring the :: prefix for replacement gives a clear visual indicator that namespace magic is occurring and your mind needs to do some dereferencing to determine the final namespace.
   
3. It also eliminates the need for the magic _ namespace for when you do not want automatic namespace substitution.

Cons

1. it can only work with keywords, not symbols. Not sure if this is desirable or not, or whether the namespace-map proposal is supposed to work on symbol keys too.
   
2. I'm not sure how to extend this pattern to destructuring,e.g. (let [{:keys #:person[:first extra]} {:person/first "John" :extra 10] [first extra]) looks wrong.

Strawman proposal, multiple namespace aliases:

#:[{_ person d db} {::first "first" ::d/id 10}]
;=>  {:person/first :db/id 10}

[Rich Hickey]

Please, I do not want bikeshedding on :_/foo. It is a placeholder. The feature it holds a place for is supporting full heterogeneity when a namespace prefix is present - there will need to be a way to get unqualified keys.

[James Reeves]

On 8 April 2016 at 20:38, Francis Avila <fav...@breezeehr.com> wrote:

Pros

1. The double colon without namespace part is currently illegal. Now it is legal only inside these namespaced maps.

In edn it's illegal, but in Clojure ::foo is equivalent to :<current-ns>/foo.

Your syntax would still work, but :: would have to mean something different inside a namespaced map.

- James

[Rich Hickey]

Yes, but really it allows the data to take its meaning anywhere, even within the same app/library. Does unqualified :age mean the same thing everywhere in a code base? Maybe, but maybe there’s some :age 42 and some :age :paleolithic. Until you use namespaced keys all of your meaning is context-dependent.

Rich

[Gary Fredericks]

Thanks, that's helpful.

Gary Fredericks
(803)-295-0195
frederi...@gmail.com
gfredericks.com

[Francis Avila]

That's right, I forgot about that feature. That weakens my proposal considerably because it increases confusion. Another reader tag may be necessary to distinguish.

[Sean Corfield]

On 4/8/16, 12:40 PM, "Rich Hickey" <cloju...@googlegroups.com on behalf of richh...@gmail.com> wrote:
>Please, I do not want bikeshedding on :_/foo. It is a placeholder.

Let me restate my question to you / Alex then without the distraction of _ -- I don’t want to get sidetracked by that…

>The feature it holds a place for is supporting full heterogeneity when a namespace prefix is present - there will need to be a way to get unqualified keys.

Can you (Rich, or Alex?) explain the use case that makes it worthwhile to have these bare keywords in the same map?

You say “there will need to be a way to get unqualified keys” but I think I need to be convinced by a compelling example of _why_ you would need to have both the namespaced keywords and also bare keywords in the same literal?

Is that likely to be common enough to _require_ syntax for it (whatever that syntax might be) as opposed to assoc’ing or merge’ing in the bare keywords?

(merge #:user{:name “Sean” :age 53}
{:bare “stuff”})

Thanks!

Sean Corfield -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

[Colin Fleming]

Mike, is there a reason you're not using the ::alias/keyword form suggested by Michał?

I must admit, I'm not seeing a huge win for this new proposal since I use the aliased form a lot and it's been sufficiently concise for my uses, at least - my vote would be that the overloaded extra syntax is not worth it.

Perhaps it's just a matter of publicising the aliased form more? In my experience, many devs aren't aware of it. Perhaps adding support for aliasing non-code nses as proposed in this thread would help too.

[Rich Hickey]

It’s not a necessity, it’s a matter of completeness/regularity. merge may be fine in code, but it doesn’t work in edn files.

Rich

[Michał Marczyk]

This new syntax is similar in a way to syntax-quote. Have you

considered making it more outwardly similar and providing a similar

unquote mechanism?

E.g.

#`{…} form             ;; would be called "syntax-qualify form"

#`{nil :foo nil bar} … ;; qualify keywords with :foo/…, symbols with bar/…

#`{:q :quux} …         ;; expand :q/… to :quux/…

Etc.

#`:foo        ;; abbr. for #`{nil :foo}

#`:foo #`bar  ;; merge maps, equivalent to #`{nil :foo nil bar}

And finally

#~form

could mean "ignore the current syntax-qualify level for form".

Then

#`:foo {:a 1 :asdf/b 2 #~:c 3}

would be read in as

{:foo/a 1 :asdf/b 2 :c 3},

and

#`{nil :foo nil bar :q :quux} {:a 1 b 2 :q/c 3}

would be read in as

{:foo/a 1 bar/b 2 :quux/c 3}

Cheers,

Michał

[Michał Marczyk]

On 11 April 2016 at 21:36, Michał Marczyk <michal....@gmail.com> wrote:

This new syntax is similar in a way to syntax-quote. Have you

considered making it more outwardly similar and providing a similar

unquote mechanism?

Well, I should have said "superficially similar". Also the look & feel I'm suggesting is more of a mix between the syntax-quote l&f and the reader meta l&f.

There's no shortage of details to be ironed out: for example one might feel that #`:foo {…} feels like it should perhaps namespace-qualify all keywords in {…}, whereas the behaviour useful here is to namespace-qualify keys in {…}. In contrast, it might be useful for #`:foo […] to namespace-qualify keywords occurring immediately within […] but not nested etc.

Cheers,

Michał

[Sean Corfield]

On 4/11/16, 12:12 PM, "Rich Hickey" <cloju...@googlegroups.com on behalf of richh...@gmail.com> wrote:
>It’s not a necessity, it’s a matter of completeness/regularity. merge may be fine in code, but it doesn’t work in edn files.

That’s a convincing use case. Thanks Rich!

Sean