puzzled by RuntimeException

[Greg D]

The sequence in the transcript below shows runtime exceptions when a numeric keyword is followed by a list starting with a symbol or character.

Would anyone help me with a reason for the failing cases?

user=> (clojure-version)
"1.6.0"
user=> '(:42 a)
(:42 a)
user=> '(:42 "a")
(:42 "a")
user=> '(:42 \a)
(:42 \a)
user=> '(:42 nil)
(:42 nil)
user=> '(:42 true)
(:42 true)
user=> '(:42 :43)
(:42 :43)
user=> '(:42 (a))

RuntimeException EOF while reading, starting at line 1  clojure.lang.Util.runtimeException (Util.java:221)

RuntimeException Unmatched delimiter: )  clojure.lang.Util.runtimeException (Util.java:221)
user=> '(:42 ("a"))
(:42 ("a"))
user=> '(:42 (\a))

RuntimeException EOF while reading, starting at line 1  clojure.lang.Util.runtimeException (Util.java:221)
RuntimeException Unmatched delimiter: )  clojure.lang.Util.runtimeException (Util.java:221)

user=> '(:42 (nil))

RuntimeException EOF while reading, starting at line 1  clojure.lang.Util.runtimeException (Util.java:221)

user=> RuntimeException Unmatched delimiter: )  clojure.lang.Util.runtimeException (Util.java:221)


user=> '(:42 (true))

RuntimeException EOF while reading, starting at line 1  clojure.lang.Util.runtimeException (Util.java:221)

user=> RuntimeException Unmatched delimiter: )  clojure.lang.Util.runtimeException (Util.java:221)


user=> '(:42 (:43))
(:42 (:43))
user=>

[Stephen Gilardi]

The sequence in the transcript below shows runtime exceptions when a numeric keyword is followed by a list starting with a symbol or character.

Would anyone help me with a reason for the failing cases?

user=> (clojure-version)
"1.6.0"
user=> '(:42 a)
(:42 a)
user=> '(:42 "a")
(:42 "a")
user=> '(:42 \a)
(:42 \a)
user=> '(:42 nil)
(:42 nil)
user=> '(:42 true)
(:42 true)
user=> '(:42 :43)
(:42 :43)
user=> '(:42 (a))

RuntimeException EOF while reading, starting at line 1  clojure.lang.Util.runtimeException (Util.java:221)

RuntimeException Unmatched delimiter: )  clojure.lang.Util.runtimeException (Util.java:221)
user=> '(:42 ("a"))
(:42 ("a"))
user=> '(:42 (\a))

I tried to duplicate this result with the clojure.main repl, but was unable to do so. All entries in your post read in without raising an exception.

I used this command line to launch the repl:

% java -cp ~/.m2/repository/org/clojure/clojure/1.6.0/clojure-1.6.0.jar clojure.main

—Steve

[Greg D]

Steve,

Thanks.  I did a quick check, and it seems that I don't get exceptions when I start the repl as you do.

I normally start mine with 'lein repl'.  You've given me a good lead to investigate.

Greg

[Greg D]

I believe this is a problem in REPL-y, which is used when using 'lein repl'.

I used Cider to start a nREPL server, then used 'leing repl :connect' to get the REPL-y interface.

The problem was evident in the latter, but not the former.  I opened an issue for REPL-y.

Thanks again, Steve

[Stephen Gilardi]

You’re welcome!

Following on your evidence, I tried running REPL-y from a checkout and found that the current head of master (0.3.1-SNAPSHOT) doesn’t throw the exception, but the current release (0.3.0) does.

I used git bisect to find the commit that changed the behavior and found this:

https://github.com/trptcolin/reply/issues/132 (Reader error on number-like keywords in maps)

The fix is here: https://github.com/cgrand/sjacket/pull/16 (Number-like keywords)

So, to avoid the exception you saw when using number-like keywords with lein repl, REPL-y needs a release and leiningen needs to pick it up.

… and we got a view of a slice of the Clojure lib ecosystem… three layers deep. Neat.

Cheers,

—Steve

[Alex Miller]

Just FYI, some background on keywords that start with a number...

The reader docs (http://clojure.org/reader) states that symbols (and keywords follow symbols in these rules) must "begin with a non-numeric character". The current Clojure reader accidentally accepts these due to a bug in the LispReader regex that reads symbols. We applied a fix for this bug early in Clojure 1.6 (http://dev.clojure.org/jira/browse/CLJ-1252). We then discovered that this broke code in a number of libraries and applications (often in test data, but also in some real code). Due to this issue, we rolled back the change. 

There is still an open ticket to do something to resolve this (likely by changing what we say is allowed) - this may have impacts on EDN as well. http://dev.clojure.org/jira/browse/CLJ-1286

Alex

[Stephen Gilardi]

> Just FYI, some background on keywords that start with a number...
>
> The reader docs (http://clojure.org/reader) states that symbols (and keywords follow symbols in these rules) must "begin with a non-numeric character".

Thanks, Alex. The REPL-y and sjacket issues cited the Clojure issues in Jira and I looked through them.

There seems to be an assumption in the discussion I’ve seen that the current description of keywords at http://clojure.org/reader means that the characters *after* the leading colon in a keyword have to follow the rules for a symbol.

I think another reasonable way to interpret the description of keywords is that the spelling of a keyword *including the leading colon* has to follow the rules for a symbol — with the obvious exception that a symbol’s spelling *cannot* begin with a colon. Since a colon is always non-numeric, keywords *always* pass the “begins with a non-numeric character” test.

When I looked in the past at what it takes to parse a token in Clojure, I concluded that the syntax of numbers vs. non-numbers (including the syntax for numbers with a radix component) was chosen such that a token represents a number if and only if it has a leading numeric character. I’ve always interpreted the “no leading numeric character” rule for the spelling of symbols (and keywords, including the leading colon) as in support of that simple distinction. More generally, the reader can classify a token by its leading character, or leading 2 characters in the case of a leading #.

I don’t see a similarly compelling reason to disallow numeric characters immediately after the leading colon in a keyword.

—Steve

[Alex Miller]

On Wednesday, April 23, 2014 11:24:44 AM UTC-5, squeegee wrote:

> Just FYI, some background on keywords that start with a number...
>
> The reader docs (http://clojure.org/reader) states that symbols (and keywords follow symbols in these rules) must "begin with a non-numeric character".


Thanks, Alex. The REPL-y and sjacket issues cited the Clojure issues in Jira and I looked through them.

There seems to be an assumption in the discussion I’ve seen that the current description of keywords at http://clojure.org/reader means that the characters *after* the leading colon in a keyword have to follow the rules for a symbol.

This was not an assumption - Rich told me to interpret the page in this way.

 

I think another reasonable way to interpret the description of keywords is that the spelling of a keyword *including the leading colon* has to follow the rules for a symbol

— with the obvious exception that a symbol’s spelling *cannot* begin with a colon. Since a colon is always non-numeric, keywords *always* pass the “begins with a non-numeric character” test.

When I looked in the past at what it takes to parse a token in Clojure, I concluded that the syntax of numbers vs. non-numbers (including the syntax for numbers with a radix component) was chosen such that a token represents a number if and only if it has a leading numeric character. I’ve always interpreted the “no leading numeric character” rule for the spelling of symbols (and keywords, including the leading colon) as in support of that simple distinction. More generally, the reader can classify a token by its leading character, or leading 2 characters in the case of a leading #.

I think you are reasoning from implementation, which may change. The http://clojure.org/readers page should be the normative reference. (as much as anything in Clojure is normative :) 

 

I don’t see a similarly compelling reason to disallow numeric characters immediately after the leading colon in a keyword.

There is not any particularly compelling reason which is why we deemed it of lesser importance than breaking people's existing programs and reverted the change.

 

—Steve

[Greg D]

Thanks Alex and Steve,

I've based a ton of work on keywords where the second character is numeric.

The http://clojure.org/readers page should be the normative reference.

The work is based on reliance on the definitions in the readers page. I believe it is unambiguous in demanding a colon as the first character of a keyword, and allowing the second character to be numeric.

I am frightened that opinions differ; especially Rich's, obviously.

[Alex Miller]

On Friday, April 25, 2014 9:48:10 AM UTC-5, Greg D wrote:

Thanks Alex and Steve,

I've based a ton of work on keywords where the second character is numeric.

The http://clojure.org/readers page should be the normative reference.

The work is based on reliance on the definitions in the readers page. I believe it is unambiguous in demanding a colon as the first character of a keyword, and allowing the second character to be numeric.

Well I'll contend it is at least ambiguous because I read it differently. :) I think everyone can agree that keywords begin with a colon. My understanding is that symbols and keywords share naming rules as a result of "Keywords are like symbols". I think the interpretation depends on whether you see the ":" as being part of the name; I would say not (that's how keywords are represented in print form but the : is not stored or returned as part of the namespace or name). Symbols and keywords share the same symbolPat regex in LispReader with I believe the same intent for allowed names.

 

I am frightened that opinions differ; especially Rich's, obviously.

I think at this point it is unlikely that "keywords starting with numbers" are going to become invalid (and I would like to formalize that in the reader page). Even if the reader could not read them, keywords (intentionally) can be created that cannot be read by the reader via the "keyword" function and this is a widely used feature. 

[Ben Wolfson]

On Fri, Apr 25, 2014 at 11:19 AM, Alex Miller <al...@puredanger.com> wrote:

On Friday, April 25, 2014 9:48:10 AM UTC-5, Greg D wrote:

Thanks Alex and Steve,

I've based a ton of work on keywords where the second character is numeric.

The http://clojure.org/readers page should be the normative reference.

The work is based on reliance on the definitions in the readers page. I believe it is unambiguous in demanding a colon as the first character of a keyword, and allowing the second character to be numeric.

Well I'll contend it is at least ambiguous because I read it differently. :) I think everyone can agree that keywords begin with a colon. My understanding is that symbols and keywords share naming rules as a result of "Keywords are like symbols". I think the interpretation depends on whether you see the ":" as being part of the name;

Since the very next thing after "Keywords are like symbols" is "except: They can and must begin with a colon, e.g. :fred." and the rule for symbols is that they "begin with a non-numeric character", it seems pretty forgivable to assume that the colon is part of the keyword (the fact that it isn't stored is immaterial; if you know it's a keyword you also know it starts with a colon) and that it can therefore contain numeric characters immediately after the colon. "Begin" should mean "begin" in both bullet points, right?

--
Ben Wolfson
"Human kind has used its intelligence to vary the flavour of drinks, which may be sweet, aromatic, fermented or spirit-based. ... Family and social life also offer numerous other occasions to consume drinks for pleasure." [Larousse, "Drink" entry]

[Alex Miller]

On Friday, April 25, 2014 1:32:32 PM UTC-5, Ben wrote:

On Fri, Apr 25, 2014 at 11:19 AM, Alex Miller <al...@puredanger.com> wrote:

On Friday, April 25, 2014 9:48:10 AM UTC-5, Greg D wrote:

Thanks Alex and Steve,

I've based a ton of work on keywords where the second character is numeric.

The http://clojure.org/readers page should be the normative reference.

The work is based on reliance on the definitions in the readers page. I believe it is unambiguous in demanding a colon as the first character of a keyword, and allowing the second character to be numeric.

Well I'll contend it is at least ambiguous because I read it differently. :) I think everyone can agree that keywords begin with a colon. My understanding is that symbols and keywords share naming rules as a result of "Keywords are like symbols". I think the interpretation depends on whether you see the ":" as being part of the name;

Since the very next thing after "Keywords are like symbols" is "except: They can and must begin with a colon, e.g. :fred." and the rule for symbols is that they "begin with a non-numeric character", it seems pretty forgivable to assume that the colon is part of the keyword (the fact that it isn't stored is immaterial; if you know it's a keyword you also know it starts with a colon) and that it can therefore contain numeric characters immediately after the colon. "Begin" should mean "begin" in both bullet points, right? 

It is confusing! :)  I apply "begin" to the name, not the token. What about auto-resolved keywords that begin with :: ? I think it is more consistent to apply the same rule to the name part of : and :: keywords (and ideally symbols). And there is no reason to define a "first character" rule if the first character must be ":". 

But all of this is immaterial - we currently allow keywords starting with numbers and seem to have decided this is ok. I would like to get Rich to approve a change to the page and do so.