How to reopen an scalax.io.Input

[Ben]

Hi,
I would like to find out how to reopen and Input. I have code similar
to the following:

val in:Input = Resource.fromURL("file://someFile")
val lessThan5 = in.bytesAsInts.takeWhile(_ < 5)

However when the code complete the underlying Input stream is closed.
How do I reopen the Input.
Thanks,
Ben

[Jesse Eichar]

It will open and close the stream each time you access the data. In fact in the example you gave, the stream should not be opened at all (assuming your are not in the REPL) because LongTraversable is a non-strict (lazy) collection.  So the url should not be opened until you execute foreach or some other operation that actually accesses the data.

Jesse

[Ben]

Let me be more specific. I am trying to read from the stdout of a
process. I have a while loop that runs until the process has exited.
Inside the loop
I try to read 10 bytes from stdout and print them. The first time
through the loop all works as expected. The second time through the
loop I get the following
error message:

[error] (run-main) java.io.IOException: Stream closed
java.io.IOException: Stream closed
at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:
162)
at java.io.BufferedInputStream.read(BufferedInputStream.java:325)
at java.nio.channels.Channels
$ReadableByteChannelImpl.read(Channels.java:385)

My Code:
while (process.hasExited() != true){
println(stdout.bytes.take(10).asInput.slurpString(Codec.UTF8))

[Jesse Eichar]

Maybe a few more details are in order.  There are a few usage patterns for the Input object you created.  If you look at the signature it is a InputStreamResource.  Which means you can use the input API and get the bytes or ints from the input stream and perform collection operations on then (in a non-strict manner).  Currently foldLeft and limitFold are the most flexible methods for processing the file but often take, drop etc... are sufficient.  

Consider:

val first5 = in.take(5)

val second5 = in.drop(5).take(5)

As the variable names imply, first5 is the first 5 elements of the input and second5 is the second.  The issue is that when they are processed, the input will be opened twice.  So a limitFold would be a better way to get the data.  

Another usage pattern is the scala-arm usage.  using acquireAndGet will provide access to the underlying inputStream, problem with that is that you have to use the input API and buffer arrays etc... In my opinion, yuck.

I am not a big fan of either of these patterns so I am working on an API that looks something like:

for {

  processor <- in.processor

  first5 <- processor.take(5)

  second5 y- processor.take(5)

} {

 // do something with the data

}

In this example the input is opened only once.  This will be out in a bit over a month I would say and I am working on docs for this API.  

Jesse

[Ben]

Thanks for the quick and clear answers. I will look forward to your
new additions to the API.

> On Sun, Jan 22, 2012 at 2:12 PM, Jesse Eichar <jesse.eic...@gmail.com>wrote:
>
>
>
>
>
>
>
> > It will open and close the stream each time you access the data. In fact
> > in the example you gave, the stream should not be opened at all (assuming
> > your are not in the REPL) because LongTraversable is a non-strict (lazy)
> > collection.  So the url should not be opened until you execute foreach or
> > some other operation that actually accesses the data.
>
> > Jesse
>

[Jesse Eichar]

Ah.  And that is why testing and outside point of view are required.  In my effort make sure no resources are ever looked open I forgot that there are cases where streams actually have to be left open.  So for now all my Input objects automatically close the streams.

I will add a method to resource, maybe something like Resource.persistentStream which will create input/output objects that don't close the streams after use.

For now...  I guess hacks are in order to keep it open, for example decorate the stream and make the close method do nothing...  

Not so good for scripts.  But this issue will be fixed in the next snapshot.

Jesse

[Jesse Eichar]

follow https://github.com/jesseeichar/scala-io/issues/48 for notification of the fix.

Jesse

[Josh Suereth]

Looks like iteratee I/o a bit.  I'll have to check out the impl

[Jesse Eichar]

On Jan 22, 2012, at 19:24, Josh Suereth <joshua....@gmail.com> wrote:

Looks like iteratee I/o a bit.  I'll have to check out the impl

Shhh don't tell :P. actually the design is more from some IO monad articles I read through.  While I appreciate the iteratee design, I just never fell in love.

Jesse

[Josh Suereth]

I haven't seen it express well in scala, although I'm trying some crazy ideas.

[Ben]

Just a thought.
I think the simple non-resource managed case should be the default.
The library user could then
decorate it with the resources managed trait at runtime if required.

[Jesse Eichar]

A good thought.  Although I prefer to make safe behaviour the most easily discoverable but allow unsafe behaviour be possible for those in the know.  I first considered adding Resource factory options to create unmanaged method but decided that just made the API huge for what I hope is not the standard usecase.  I have instead added an unmanaged method to Resource and have some new  JavaConverters directly to unmanaged resources.

[Ben]

In scala it is very easy to create a loan pattern that closes the open
streams as required. As such, I would focus on the thin interface.
Get it right, then build the fat interface with arm on top.

[Jesse Eichar]

fair point.  

[Josh Suereth]

If you want the low-level stuff, just use the java APIs.   I'm a big fan of the ARM stuff in scala I/O.  

The loaner pattern is 'easy' but we see it used *everywhere* when doing I/O.  Backing it in as the default for the *high level* API is genius IMHO.  It covers 90% of my I/O use cases.  When I need nitty gritty low-level, I usually aim for Netty.

In my opinion, a low-level Netty API with a high level API like what Jesse has covers my I/O situations.  The ARM library allows some advanced loaner-pattern like activities, but the ability to compose *behavior* objects in I/O is pretty amazing.

I'm personally a huge fan of Jesse's idea to create processors that when evaluated will run over a program.   This kind of programming is far more natural for I/O than any other.  In fact, I'd even say that decomposing accessing resources from the processing of data in that resource is a huge win.   This is why the Haskell/scalaz community is all about Iteratees.  It's the right level of abstraction for some really composable modular I/O.  (Note:  Netty is basically an OO/mutable implementation of the iteratee pattern).

So, if you want the raw low-level stuff, java is probably your best I/O bet.  I'm looking for something easy to use that doesn't put a lot of work (like creating a loaner pattern in every project) on me.  I can just do   for(line <- Path("foo").lines) doProcessing(line), I think we're winning.   If I start having to do crazy things like:

withResource(Path("foo").source) { source =>

  for { line <- source.lines } doProcessing(line)

}

we've lost.  Also note that managed resources compose, so I can do this:

def diff(file1: Path, file2: Path) = for {

  (line1, line2) <-  file1.lines zip file2.lines

  if line1 != line2

} yield diffLine(line1,line2)

Which is *far* cleaner than the loaner pattern version:

def diff(file1: Path, file2: Path) = withResource(file1.source) { s1 =>

  withResource(file2.source) { s2 =>

     for {

      (line1, line2) <-  source.lines zip source.lines

      if line1 != line2

    } yield diffLine(line1,line2)

  }

}

While I agree that the underlying raw access API needs to be there, I think the *default* needs to be the managed-resource API.  It's far more powerful and less verbose.  The mental shift into it isn't very hard.  The only downside to it is it's "not like java's API", which is a good thing.

- Josh

- Josh

[Jesse Eichar]

Thanks for the rebuttal Josh. One of the dangers I recognize is that I am the library author and therefore completely biased so I try to take all points of view into account since I have to be skeptical of my reactions.

...  That is what a psychology degree can do to a person, even a dusty unused one.

Jesse

[David Jung]

Sorry to dig up something so old, but I'm about to go to the unmanaged resource route (using .asUnmanagedInput) for my project and I was wondering how to close the resource once I was finished using it?

I just tried a .close() call, but one doesn't seem to exist.

[Jesse Eichar]

Well the idea of unmanaged means you aren't supposed to close it, otherwise you would manage it.  

But I assume you have a special case so I guess you should be able to call the open method to get the underlying resource and then close that.

Jesse

--
 
---
You received this message because you are subscribed to the Google Groups "Scala Incubator" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-incubat...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.