Strange behavior with clojure.java.io/copy

[Juha Syrjälä]

Hi,

I found a strange behavior in implementation of clojure.java.io/copy function

https://github.com/clojure/clojure/blob/ee1b606ad066ac8df2efd4a6b8d0d365c206f5bf/src/clj/clojure/java/io.clj#L391

(defn copy

  "Copies input to output.  Returns nil or throws IOException.

  Input may be an InputStream, Reader, File, byte[], char[], or String.

  Output may be an OutputStream, Writer, or File.

  Options are key/value pairs and may be one of

    :buffer-size  buffer size to use, default is 1024.

    :encoding     encoding to use if converting between

                  byte and char streams.   

  Does not close any streams except those it opens itself 

  (on a File)."

  {:added "1.2"}

  [input output & opts]

  (do-copy input output (when opts (apply hash-map opts))))

Actual copying is implemented here when copying from an InputStream to an OutputStream.

https://github.com/clojure/clojure/blob/ee1b606ad066ac8df2efd4a6b8d0d365c206f5bf/src/clj/clojure/java/io.clj#L306

(defmethod do-copy [InputStream OutputStream] [^InputStream input ^OutputStream output opts]

  (let [buffer (make-array Byte/TYPE (buffer-size opts))]

    (loop []

      (let [size (.read input buffer)]  ;;; XXX point 1

        (when (pos? size)               ;;; XXX point 2

          (do (.write output buffer 0 size)

              (recur)))))))

Here .read function at point 1 is https://docs.oracle.com/javase/7/docs/api/java/io/InputStream.html#read(byte[])

The javadoc states following

Reads some number of bytes from the input stream and stores them into the buffer array b. The number of bytes actually read is returned as an integer. This method blocks until input data is available, end of file is detected, or an exception is thrown.

If the length of b is zero, then no bytes are read and 0 is returned; otherwise, there is an attempt to read at least one byte. If no byte is available because the stream is at the end of the file, the value -1 is returned; otherwise, at least one byte is read and stored into b.

--- 

Meaning that return value -1 means end of stream, and return value 0 doesn't mean end of stream. However condition at point 2 in code above, stops recursion when .read returns value that is smaller than 1.

Now consider a case where .read returns this sequence of values in consecutive calls:

1024, 0, 1024, 201, -1

clojure.java/io copies only the first 1024 bytes, when the whole stream has 2249 bytes. Is this the intended behavior? Should the condition at point 1 be "(not (neg? size))" ?

[Juha Syrjälä]

I posted this also to ask.clojure.org forum at https://ask.clojure.org/index.php/8450/strange-behavior-with-clojure-java-io-copy

[alpeware]

The read method returns the number of bytes consumed from the input stream and stored in the provided buffer.

The only case the read method would return 0 is when a buffer of length 0 is provided since there would be no place to store any bytes consumed from the input stream.

In most cases, a fixed length buffer is allocated and the input stream is consumed in chunks by reading it into the buffer, do something with the contents of the buffer and then iterate until the whole stream has been consumed.

I hope this helps.

[Dirk Wetzel]

As alpeware already said, .read will not return [1024, 0, 1024, 201, -1] because it will not return a zero unless buffer-size is zero.

This bit of the Java-Doc: If the length of b is zero, then no bytes are read and 0 is returned;

So if the check is changed to (<= 0 size), passing a buffer-size of 0 (for whatever silly reason) would result in an infinite loop.

While not my decision to make, I personally don't think that this would be desirable behaviour.

[Andy Fingerhut]

That bit of Java-Doc says nothing about the behavior when providing a buffer b with a non-0 length.

If it said: "If the length of b is non-zero, then the return value will never be 0", then I might agree with you.  Also, if you knew for some reason that the implementation guaranteed this.  (I have not looked at the implementation to know either way.)

Andy

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+u...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/clojure/9deb8024-ce4b-4ef4-88d0-bbd817152ab6%40googlegroups.com.

[Andy Fingerhut]

Ah, I should have read further in the Java doc here, though, before posting.  In particular, the last sentence of this paragraph: "If the length of b is zero, then no bytes are read and 0 is returned; otherwise, there is an attempt to read at least one byte. If no byte is available because the stream is at the end of the file, the value -1 is returned; otherwise, at least one byte is read and stored into b."

https://docs.oracle.com/javase/8/docs/api/java/io/InputStream.html#read-byte:A-

If implementations actually follow that documentation, then it appears that a non-0 buffer size should never result in a return value of 0.

Andy

[Dirk Wetzel]

You mentioned an interesting point there:

If implementations actually follow that documentation, [...]

I know that at least some libraries (f.e. "org.apache.commons.io") only stop reading when -1 is returned and will continue reading if 0 is returned.

That behaviour is definitely more friendly towards implementations that don't follow the java-doc exactly.

@ Alex Miller:

To cover both cases, maybe take the buffer-size into account when making the decision about whether or not to continue reading?

Something along those lines:

(or (pos? size)

    (and (= 0 size)

         (pos? buffer-size)))

Am Montag, 26. August 2019 07:01:01 UTC+2 schrieb Andy Fingerhut:

Ah, I should have read further in the Java doc here, though, before posting.  In particular, the last sentence of this paragraph: "If the length of b is zero, then no bytes are read and 0 is returned; otherwise, there is an attempt to read at least one byte. If no byte is available because the stream is at the end of the file, the value -1 is returned; otherwise, at least one byte is read and stored into b."

https://docs.oracle.com/javase/8/docs/api/java/io/InputStream.html#read-byte:A-

If implementations actually follow that documentation, then it appears that a non-0 buffer size should never result in a return value of 0.

Andy

On Sun, Aug 25, 2019 at 9:54 PM Andy Fingerhut <andy.f...@gmail.com> wrote:

That bit of Java-Doc says nothing about the behavior when providing a buffer b with a non-0 length.

If it said: "If the length of b is non-zero, then the return value will never be 0", then I might agree with you.  Also, if you knew for some reason that the implementation guaranteed this.  (I have not looked at the implementation to know either way.)

Andy

On Sun, Aug 25, 2019 at 9:21 PM 'Dirk Wetzel' via Clojure <clo...@googlegroups.com> wrote:

As alpeware already said, .read will not return [1024, 0, 1024, 201, -1] because it will not return a zero unless buffer-size is zero.

This bit of the Java-Doc: If the length of b is zero, then no bytes are read and 0 is returned;

So if the check is changed to (<= 0 size), passing a buffer-size of 0 (for whatever silly reason) would result in an infinite loop.

While not my decision to make, I personally don't think that this would be desirable behaviour.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clo...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clo...@googlegroups.com.

[Matching Socks]

A bit awkward to clutter a standard library with too much speculative forgiveness of other software's potential failure to follow a spec.  Where would it end?

Anyway, the best improvement here would help with all InputStream consumers, not just io/copy.  

If you suffer from a faulty InputStream that returns zero when it simply does not feel like blocking, couldn't you either fix it or wrap it in a "TolerateZeroBytesReadByRetryingInputStream" that blocks until it gets something other than zero?