Pipeline interoperability with IOStream and IOBuffer

[William Wong]

Hello, 

I'm trying to continue the discussion of 

https://github.com/JuliaLang/julia/issues/15479

julia> run(pipeline(IOBuffer("a xyz b"), `grep xyz`))
ERROR: MethodError: `uvtype` has no method matching uvtype(::Base.AbstractIOBuffer{Array{UInt8,1}})
 in _jl_spawn at process.jl:253
 in anonymous at process.jl:415
 in setup_stdio at process.jl:403
 in spawn at process.jl:414
 in spawn at process.jl:293
 in run at process.jl:530

I feel like we should be able to use PipeBuffer/IOBuffer with pipeline.  Do the Julia devs believe we should never expect to be able to pipeline using IOBuffer/PipeBuffer?

It seems like many other people are expecting to be able to do this too:

https://github.com/JuliaLang/julia/issues/14437

https://github.com/JuliaLang/julia/issues/3823#issuecomment-157714083

A point brought up in the the issues say that one is a file abstraction and one is a stream abstraction.  I now have a bit better understanding of what that means but I couldn't find any official documentation on the differences between a stream and a file.

This is especially confusing if you consider 

filestream = open("somefile", "w")  - is a file as a stream abstraction but

buffer = IOBuffer() - is a stream (as the docs currently say) as a file abstraction.

The former works with pipeline, but not the latter.

In either case, it seems that I need an IOStream to pipe data into a pipeline command. How can I turn data into a stream that is not a file?

https://groups.google.com/forum/#!msg/julia-users/R-F3F97leh4/o4zKINZbbvUJ asks a similar question but readall produces a string, I still need to stream it into a pipeline command.

Thank you,

Will

[Steven G. Johnson]

On Thursday, July 21, 2016 at 6:09:12 PM UTC-4, William Wong wrote:

I'm trying to continue the discussion of 

https://github.com/JuliaLang/julia/issues/15479

The github issue is the appropriate place for discussion; it's not helpful to split the discussion into two separate forums. 

(If you want to see progress on an issue, the most effective way is to submit a pull request attempting to fix the issue.)

[Tom Breloff]

To be fair, he was told, in that issue, that julia-users was a more appropriate place to ask this type of question. 

[William Wong]

To add to that, this discussion had actually initially started out on Julia-Users

https://groups.google.com/forum/#!searchin/julia-users/pipeline$20iostream/julia-users/-66VWszB-mM/b9wno137BgAJ

I was also hoping that someone could help me with this question here:

In either case, it seems that I need an IOStream to pipe data into a pipeline command. How can I turn data into a stream that is not a file?

https://groups.google.com/forum/#!msg/julia-users/R-F3F97leh4/o4zKINZbbvUJ asks a similar question but readall produces a string, I still need to stream it into a pipeline command.

[Steven G. Johnson]

On Friday, July 22, 2016 at 8:55:44 AM UTC-4, Tom Breloff wrote:

To be fair, he was told, in that issue, that julia-users was a more appropriate place to ask this type of question. 

No, he was told to ask on julia-users about usage questions — in this case, if he can't figure out how to write from an IOBuffer to an IOStream or to a process.  (No changes to Base are required to do this.)

In contrast, if you want to continue to press for changes to the Julia standard library, the correct forum is the Github issue.

[William Wong]

Hi Steven,

Thank you very much for taking the time to try to try to answer my question. 

Perhaps if it is not clear enough, yes I have a usage question that I am asking here.  I'll try to rephrase it. Could you kindly answer this?

I have data (could be a string) that is stored in IOBuffer. I want to send it into a process, i.e. via a pipeline.  Is there a way to turn a string or any byte array into a stream? An example of what I want is in the code snipped in OP.

For example, java has something like this http://stackoverflow.com/questions/247161/how-do-i-turn-a-string-into-a-stream-in-java

I understand (as stated in the github issue) IOBuffer is not directly supported in pipelines (although the docs clearly say otherwise) and if you feel like that should be a discussion on the issue, then I will continue that particular discussion on the github issue. Just wanted to note that the devs there already dismissed this feature request as not necessary and labelled it a "documentation issue".

Thank you,

Will