clojure.edn won't accept clojure.java.io/reader? How to work around this and why isn't this documented anywhere?

[Fluid Dynamics]

=> (with-open [in (io/reader (io/resource "foo"))] (edn/read in))
ClassCastException java.io.BufferedReader cannot be cast to java.io.PushbackReader  clojure.edn/read (edn.clj:35)

Er, what? Aren't these things supposed to just plug into each other and work OOTB? Do I need to muck about with interop to use edn with any input source other than strings, then?

[Andy Fingerhut]

In regards to your question "Why isn't this documented anywhere?", it is documented somewhere -- in the documentation string for clojure.edn/read, the very function you were attempting to use:

user=> (doc clojure.edn/read)
-------------------------
clojure.edn/read
([] [stream] [opts stream])
  Reads the next object from stream, which must be an instance of
  java.io.PushbackReader or some derivee.  stream defaults to the
  current value of *in*.

etc.

As far as why it requires a PushbackReader, I didn't design the API.  Yes, some things in Clojure require using Java interop, and in many (but not all) cases, file I/O requires it.

Andy

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Fluid Dynamics]

On Monday, December 8, 2014 2:26:42 AM UTC-5, Andy Fingerhut wrote:

In regards to your question "Why isn't this documented anywhere?", it is documented somewhere -- in the documentation string for clojure.edn/read, the very function you were attempting to use:

user=> (doc clojure.edn/read)
-------------------------
clojure.edn/read
([] [stream] [opts stream])
  Reads the next object from stream, which must be an instance of
  java.io.PushbackReader or some derivee.  stream defaults to the
  current value of *in*.

What's *not* documented is that io/reader doesn't output something that edn/read can use directly, nor is there documented an officially recommended workaround for this.

AFAICT just wrapping the reader output in "(java.io.PushbackReader. ...)" works.

Still, this is awkward, verbose, and prevents the (nontrivial) use of edn in a platform-neutral way by referring only to Clojure functions without direct interop. Well, except for the even more awkward workaround of slurp and read-string, with the accompanying need to hold the entire file in memory *twice* for a short time.

As far as why it requires a PushbackReader, I didn't design the API.  Yes, some things in Clojure require using Java interop, and in many (but not all) cases, file I/O requires it.

Perhaps io/reader should output a PushbackReader, if only for convenience's sake.

Also, how does this work on ClojureCLR or ClojureScript? Neither of those platforms has a java.io.PushbackReader, and I'm not even sure what the equivalent of the clojure.java.io namespace is for them, unless the "java" in that name is misleading.

[Ashton Kemerling]

Nothing in the Java.io namespace was made by the clojure team, so it's not their fault that reader and pushbackreader aren't cross compatible. I'm assuming that they need something from pushbackreader for performance reasons, but that's just a guess. 

Clojurescript and ClojureCLR must have different solutions because they're just different platforms. Abstracting the whole high performance IO api from each of these platforms is silly and probably unwise.  

--Ashton

Sent from my iPhone

[László Török]

2014-12-08 8:12 GMT+00:00 Fluid Dynamics <a209...@trbvm.com>:

On Monday, December 8, 2014 2:26:42 AM UTC-5, Andy Fingerhut wrote:

In regards to your question "Why isn't this documented anywhere?", it is documented somewhere -- in the documentation string for clojure.edn/read, the very function you were attempting to use:

user=> (doc clojure.edn/read)
-------------------------
clojure.edn/read
([] [stream] [opts stream])
  Reads the next object from stream, which must be an instance of
  java.io.PushbackReader or some derivee.  stream defaults to the
  current value of *in*.

What's *not* documented is that io/reader doesn't output something that edn/read can use directly, nor is there documented an officially recommended workaround for this.

FWIW I don't think that io/reader(v1.2) was intended to output something that edn/read can use directly, as it was added well before clojure.edn (v1.5).

AFAICT just wrapping the reader output in "(java.io.PushbackReader. ...)" works.

There you go.

 

Still, this is awkward, verbose, and prevents the (nontrivial) use of edn in a platform-neutral way by referring only to Clojure functions without direct interop. Well, except for the even more awkward workaround of slurp and read-string, with the accompanying need to hold the entire file in memory *twice* for a short time. 

As far as why it requires a PushbackReader, I didn't design the API.  Yes, some things in Clojure require using Java interop, and in many (but not all) cases, file I/O requires it.

Perhaps io/reader should output a PushbackReader, if only for convenience's sake.

io/reader is not meant to be used solely as an input to edn/read.

 

Also, how does this work on ClojureCLR or ClojureScript? Neither of those platforms has a java.io.PushbackReader, and I'm not even sure what the equivalent of the clojure.java.io namespace is for them, unless the "java" in that name is misleading.

Exactly! There is no clojure.java.io as they are on a different host. Different host implies likely different I/O capabilities. 

BTW cljs has cljs.reader/read-string

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

László Török

[Fluid Dynamics]

On Monday, December 8, 2014 9:32:28 AM UTC-5, Las wrote:

Still, this is awkward, verbose, and prevents the (nontrivial) use of edn in a platform-neutral way by referring only to Clojure functions without direct interop. Well, except for the even more awkward workaround of slurp and read-string, with the accompanying need to hold the entire file in memory *twice* for a short time. 

As far as why it requires a PushbackReader, I didn't design the API.  Yes, some things in Clojure require using Java interop, and in many (but not all) cases, file I/O requires it.

Perhaps io/reader should output a PushbackReader, if only for convenience's sake.

io/reader is not meant to be used solely as an input to edn/read.

AFAICT, PushbackReader is substitutable anywhere a reader is expected, but apparently a plain unwrapped BufferedReader is not.

[Michał Marczyk]

On 8 December 2014 at 21:17, Fluid Dynamics <a209...@trbvm.com> wrote:
> On Monday, December 8, 2014 9:32:28 AM UTC-5, Las wrote:

>> […]

>> io/reader is not meant to be used solely as an input to edn/read.
>
>
> AFAICT, PushbackReader is substitutable anywhere a reader is expected, but
> apparently a plain unwrapped BufferedReader is not.

user=> (line-seq (java.io.PushbackReader. (io/reader (io/file ".bashrc"))))
ClassCastException java.io.PushbackReader cannot be cast to
java.io.BufferedReader clojure.core/line-seq (core.clj:2955)

It works with the plain unwrapped BufferedReader that io/reader returns.

Unfortunately PushbackReader and BufferedReader are both classes
rather than interfaces and they both have methods that the other class
does not. So, there isn't a single good choice for what a "reader"
function in Clojure on the JVM should return.

Cheers,
Michał

[Fluid Dynamics]

On Monday, December 8, 2014 4:01:28 PM UTC-5, Michał Marczyk wrote:

On 8 December 2014 at 21:17, Fluid Dynamics <a209...@trbvm.com> wrote:
> On Monday, December 8, 2014 9:32:28 AM UTC-5, Las wrote:
>> […]
>> io/reader is not meant to be used solely as an input to edn/read.
>
>
> AFAICT, PushbackReader is substitutable anywhere a reader is expected, but
> apparently a plain unwrapped BufferedReader is not.

user=> (line-seq (java.io.PushbackReader. (io/reader (io/file ".bashrc"))))
ClassCastException java.io.PushbackReader cannot be cast to
java.io.BufferedReader  clojure.core/line-seq (core.clj:2955)

It works with the plain unwrapped BufferedReader that io/reader returns.

Unfortunately PushbackReader and BufferedReader are both classes
rather than interfaces and they both have methods that the other class
does not. So, there isn't a single good choice for what a "reader"
function in Clojure on the JVM should return.

 
Jeez. Who made this mess? And what happens on other VM targets? What does line-seq expect in CLJS or CLR?

[Andy Fingerhut]

To Fluid Dynamics:

I attempted to send this privately to the email address a209...@trbvm.com, but got a permanent failure to deliver.  Hence the open letter.

I understand that some things do not work as you wish them to, and I understand that one can be frustrated or shocked by the state of libraries that you find confusing or broken, but there are other ways to express this that can be less inflammatory.

For example, instead of "Jeez. Who made this mess?" one might say "That's an unfortunate state of affairs.  Do you know how it got this way, or whether there are existing Clojure libraries that smooth some of this over for the developer?"

Thanks,

Andy Fingerhut

--

[Stuart Sierra]

As the original author of the function that eventually became clojure.java.io/reader, it was one of those unfortunate decisions that seemed like a good idea at the time and cannot be changed without breaking backwards compatibility.

Long before EDN existed, I wrote clojure.contrib.io https://github.com/clojure/clojure-contrib/blob/1.2.x/src/main/clojure/clojure/contrib/io.clj

This included a function `reader` that returned a java.io.BufferedReader: http://docs.oracle.com/javase/7/docs/api/java/io/BufferedReader.html

BufferedReader was a convenient type to return because it supports .readLine, which is used by clojure.core/line-seq: https://github.com/clojure/clojure/blob/1.1.x/src/clj/clojure/core.clj#L1954-L1960

However, clojure.core/read and later clojure.edn/read were written in terms of java.io.PushbackReader, because they need the ability to look ahead one character in the stream while parsing it:
http://docs.oracle.com/javase/7/docs/api/java/io/PushbackReader.html

java.io.PushbackReader and java.io.BufferedReader are both subclasses of java.io.Reader, but there is no class in the JDK which combines the features of both PushbackReader and BufferedReader. Java does not permit multiple inheritance of concrete classes.

In Java it is common to have several layers of Reader sub-classes wrapped around each other, so that's what we do.

We cannot change clojure.java.io/reader to return a different type without breaking a lot of existing code that expects it to return a BufferedReader.

This was reported as an issue in 2009 and discussed on the mailing list:
https://groups.google.com/forum/#!topic/clojure/_tuypjr2M_A
http://dev.clojure.org/jira/browse/CLJ-82

It turns out there are some subtle issues which can cause incorrect behavior were clojure.core/read to blindly wrap a PushbackReader around its argument:
https://groups.google.com/d/msg/clojure/_tuypjr2M_A/W1EcEbMUg_cJ

In conclusion, this is a minor nuisance, well-known to Clojure developers, for which no good solution has been identified.

–S

[Fluid Dynamics]

On Monday, December 8, 2014 7:32:29 PM UTC-5, Stuart Sierra wrote:

As the original author of the function that eventually became clojure.java.io/reader, it was one of those unfortunate decisions that seemed like a good idea at the time and cannot be changed without breaking backwards compatibility.

Long before EDN existed, I wrote clojure.contrib.io https://github.com/clojure/clojure-contrib/blob/1.2.x/src/main/clojure/clojure/contrib/io.clj

This included a function `reader` that returned a java.io.BufferedReader: http://docs.oracle.com/javase/7/docs/api/java/io/BufferedReader.html

BufferedReader was a convenient type to return because it supports .readLine, which is used by clojure.core/line-seq: https://github.com/clojure/clojure/blob/1.1.x/src/clj/clojure/core.clj#L1954-L1960

However, clojure.core/read and later clojure.edn/read were written in terms of java.io.PushbackReader, because they need the ability to look ahead one character in the stream while parsing it:
http://docs.oracle.com/javase/7/docs/api/java/io/PushbackReader.html

Seems like the simplest fix at the library level would be to make core/read and edn/read accept both classes and wrap BufferedReaders in PushbackReaders themselves, then.
 

It turns out there are some subtle issues which can cause incorrect behavior were clojure.core/read to blindly wrap a PushbackReader around its argument:
https://groups.google.com/d/msg/clojure/_tuypjr2M_A/W1EcEbMUg_cJ

That sounds like magic. The user wrapping a PushbackReader around a BufferedReader doesn't cause problems, but the library function doing so does? Why would where the wrapping takes place make a difference? Is the *only* problem the rare case of reading more than one object from the same stream? A docstring warning to wrap manually *in those cases* would suffice, then, no?

[Matching Socks]

How many programmers does it take to change a light bulb?!

http://dev.clojure.org/jira/browse/CLJ-1611

[Luc Préfontaine]

Dunno the answer but I know how many buddhist monks are needed, exactly three:

a) the first one readies itself for the bulb swap! by repeating a mantra

b) the second meditates to make the first monk levitate toward the fixture 

c) the third one immolates itself to provide light for the entire duration of the operation

Sorry for all the buddhist monks that may be offended by the above :)

I like black humor very much and this is probably the only joke that I can write

on this list that will not qualify me hopefully for eternal damnation... Euh moderation...

Luc P.

Sent from my iPad

On Dec 8, 2014, at 20:11, Matching Socks <phill...@gmail.com> wrote:

How many programmers does it take to change a light bulb?!

http://dev.clojure.org/jira/browse/CLJ-1611

[Dylan Butman]

+10 for bringing this thread around. 

[dmiller]

Also, how does this work on ClojureCLR or ClojureScript? Neither of those platforms has a java.io.PushbackReader, and I'm not even sure what the equivalent of the clojure.java.io namespace is for them, unless the "java" in that name is misleading.

ClojureCLR implements PushbackInputStream, PushbackTextReader and LineNumberingTextReader just so it can implement the Lisp reader, the EDN reader, and a few other things following the ClojureJVM code.

https://github.com/clojure/clojure-clr/tree/master/Clojure/Clojure/Readers

-David 

[Ryan Schmitt]

It turns out there are some subtle issues which can cause incorrect behavior were clojure.core/read to blindly wrap a PushbackReader around its argument:
https://groups.google.com/d/msg/clojure/_tuypjr2M_A/W1EcEbMUg_cJ

That sounds like magic. The user wrapping a PushbackReader around a BufferedReader doesn't cause problems, but the library function doing so does? Why would where the wrapping takes place make a difference? Is the *only* problem the rare case of reading more than one object from the same stream? A docstring warning to wrap manually *in those cases* would suffice, then, no?

It's not a question of where the wrapping takes place, it's a question of who holds a reference to the wrapped Reader and what their expectations are. The reason that clojure.core/read cannot wrap its argument in a PushbackReader is that it doesn't know who else holds a reference to that argument. Let's say I have a BufferedReader that initially contains the following two Edn elements:

"string1""string2"

Let's also say that I have a version of clojure.edn/read that accepts a BufferedReader and (unsoundly) wraps it in a PushbackReader. After the first read call, the PushbackReader clojure.core/read created will contain the following:

"string2"

But the BufferedReader might only contain the following:

string2"

Since the PushbackReader created by clojure.core/read will be garbage collected after the function returns, it will be impossible to parse any more data correctly out of this reader. And this won't even break all of the time--it will only occasionally break, in the cases where a non-whitespace element gets pushed back into the PushbackReader and then garbage collected.

As for the motivation behind using a PushbackReader at all: http://en.wikipedia.org/wiki/Parsing#Lookahead

[Ryan Schmitt]

Can anyone come up with an example of a case where the call to clojure.edn/read leaves an unread character in the PushbackReader? Looking briefly at the EdnReader source, I only see cases where the pushback buffer is used internally, in order to save a character that is later consumed during the same call to read. If this is true, and if this behavior could be guaranteed for all time and for all possible versions of Edn, then it would in fact be possible--although not necessarily worth it--for clojure.edn/read to accept a BufferedReader, or even an arbitrary Reader, as input.

[Fluid Dynamics]

If that's the only issue, then there's not really a problem as long as read accepts a PushbackReader and doesn't further wrap one if it is given one. Then the user in an instance like the above can do their own wrapping *when they must*, but can also eschew it when it won't be a problem because they intend to discard the whole stream after the one call to read.