Capture the output of Julia's console

[Laszlo Hars]

I tried to get answers in various threads to my question below, but so far nobody could help. Let me try in a separate thread… I use the latest Julia 0.3.0 nightly under Windows.

The problem: I have computer generated Julia code. The simplest case is when a block of Julia instructions is copied to the clipboard, which can be simply pasted to the Julia console, and run.

I'd like to capture the Julia output seen in the console, and either log it in a file, or better yet, return it to the caller application via the Windows clipboard. There are two problems

-          If the pasted Julia instructions have syntax errors, the Julia error messages cannot be caught with some surrounding Julia instructions, because the execution of the pasted block of code is aborted

-          Normal console output does not seem to always appear in STDOUT. I'd like to get the output of code, like

for i = 1:3 println((i,i^2)) end # results appear in STDOUT

or

a=1;b=2;a+b # nothing is seen in STDOUT

The Julia nightly versions a couple months ago allowed simple redirections of STDOUT and STDERR to user pipes, from where the data could be read. Everything used to work fine, but not anymore.

Thus my question is: can we redirect everything written to the Julia console to a pipe or to file; or otherwise capture all the output text?

[Stéphane Laurent]

I really don't know how it works, but this is what Yihui Xie's R package runr does for R.

[Laszlo Hars]

Thanks, Stéphane, for the link. That package uses Julia as a TCP server, and passes the Julia code as a string. The result is returned in an IOBuffer. It is certainly an alternative to pasting the clipboard to the console, but does not solve my problems: print() and multi-line input is not handled properly, even though error messages can be captured by "try parse(...) catch err" constructs (not in the code).

Maybe the solution is to pre-process the Julia code to be evaluated, to handle print() and multi-line input in several parse() calls? Assuming that the output of the Julia console is impossible to capture in a Julia variable.

[Steven G. Johnson]

On Tuesday, June 24, 2014 11:29:37 PM UTC-4, Laszlo Hars wrote: 

The Julia nightly versions a couple months ago allowed simple redirections of STDOUT and STDERR to user pipes, from where the data could be read. Everything used to work fine, but not anymore.

The redirect_stdout() and redirect_stderr() functions to redirect stdio to pipes still works. (That's how standard i/o is captured in IJulia.) 

[Laszlo Hars]

I know they work by themselves. I just don't know, how I can use them to capture all the console output in a variable, or in a file. Could you post some functional code?

[Steven G. Johnson]

On Wednesday, June 25, 2014 4:07:48 PM UTC-4, Laszlo Hars wrote:

I know they work by themselves. I just don't know, how I can use them to capture all the console output in a variable, or in a file. Could you post some functional code?

julia> rdstdout, wrstdout = redirect_stdout()

(Pipe(open, 0 bytes waiting),Pipe(open, 0 bytes waiting))

julia> print("hello")

julia> s = readavailable(rdstdout)

"hello"

julia> clipboard(s) 

[Laszlo Hars]

Thanks, Steve, but it just does not work for my problem. In my first post I wrote that this simple redirection does not work with regular console output, only with print(). Try it yourself:

~~~

rdstdout, wrstdout = redirect_stdout()

1+2

s = readavailable(rdstdout)

clipboard(s)

~~~

- the clipboard does not change.

Even if it did, the problem remains with error messages. Try to replace "1+2" with "1)" in the above code and see how you can get the resulting error message to the clipboard.

[Laszlo Hars]

...anyone? I'd appreciate ideas, even if they are untested.

[Jameson Nash]

just briefly, the problem is you are attempting to do two different things:

1) capture stdio data. I posted code for this in response to one of your earlier requests

2) implement a REPL (read-eval-print-loop). you have implemented the read-eval parts. the basic repl uses Base.showerror to display errors and display("text/plain", ans) to print the last result returned from each eval

alternatively, you could create a pipe to julia, and reuse the basic repl, but it really depends on your desired result

[Laszlo Hars]

Thanks, Jameson, for your reply.

1) Unfortunately, the code you posted earlier does not work for my problem (as I described in the first post here), and I could not fix it. This is why I asked for help.

2) Implementing a full REPL (working on multi-line input; gracefully handling error messages in the middle, and evaluate the rest; allow print(); work with nested strings...) is a lot of work. This is why I wanted to use Julia's REPL. I did try a pipe, like the code in the R interface to Julia, but I could not get the messages about syntax errors.

My problem is still open.

[Steven G. Johnson]

It might help if you gave a clearer description of your higher-level goals.   What are you ultimately trying to accomplish?

[Laszlo Hars]

The immediate goal is to capture everything the Julia console outputs for a block of input (pasted from the Windows clipboard) and log it in a file, or put it in the clipboard. The calling program then gets this data as IJulia does, and processes it. (I can do it, with clearing the Julia screen, pasting code there, marking everything, copy to clipboard... but it is ugly and slow.)

One application of it is my AutoHotkey macro for MS Word. It has three hotkeys. #1: takes the text from the caret backwards to a prescribed string (double space), and passes this to Julia, replacing the selection with the result. I type "There are  7*24*60*60" now I press the #1 hotkey, which replaces  7*24*60*60 with 604800 computed by Julia, and I go on with typing, "seconds in a week." The final sentence then reads: "There are 604800 seconds in a week." The #2 hotkey keeps the selection and copies the result after it, with an "=" in between, while the #3 hotkey takes the whole current paragraph (with Shift-Enter separated lines) and evaluates that in Julia. E.g.

for i = 1:9

println((i,i^3))

end

This gives me a small table of the cubes direct in MS Word. All these used to work until a couple months ago, but when the new Julia console was introduced, it broke. I used a separate thread to read everything from STDERR and to copy it to a Julia variable, returned by another function call. These stopped working, and my nice AutoHotkey macros are useless.

There are two problems: 1) the results of direct expressions (like "1+2") do not appear in STDOUT in the Julia console, while print() calls produce no results, which can be assigned to a variable, 2) I cannot capture syntax error messages with simply surrounding the computer generated Julia code with extra instructions. This happens very often with misplaced parentheses and similar errors, and in MS Word I'd like to get the error message. 

[Peter Simon]

I would like to second this request.  I've implemented a "diary" function similar to Matlab's, but it fails to capture REPL output produced by e.g., assignment of a value to a variable, when the source line is not terminated by a semicolon.

[Steven G. Johnson]

On Saturday, June 28, 2014 4:40:21 PM UTC-4, Laszlo Hars wrote:

One application of it is my AutoHotkey macro for MS Word. It has three hotkeys. #1: takes the text from the caret backwards to a prescribed string (double space), and passes this to Julia, replacing the selection with the result.

Rather than using the REPL, wouldn't it be far easier just to write a tiny Julia script that reads strings, executes them, and prints the result (or puts it on the clipboard or whatever)?   e.g.


      while true
       println("RESULT: ", eval(parse(readline())))
       end

You could easily modify this with a try...catch block to catch exceptions and do whatever you want with it, etcetera.      It seems like this would be easier and much more flexible than hacking the REPL.

[Laszlo Hars]

Yes, it is a possibility, as I wrote above: "Maybe the solution is to pre-process the Julia code to be evaluated, to handle print() and multi-line input in several parse() calls." It is not trivial. The print calls don't produce results, so I have to redefine print() and println(). We need to break up the code to be pasted to Julia as single-line commands, because parse does not handle multi-line code, etc. I was hoping that someone has already some Julia code that captures the console output, or can tell the trick. It does not seem to be the case.

[Stéphane Laurent]

These days I have experienced Yihui's runr package and this is indeed not very satisfactory yet. 

Do you know whether we can straightforwardly get in Julia, the console output as a ready-to-print character string, for example here:

julia> [1,2,3]

3-element Array{Int64,1}:

 1

 2

 3

I'd like to get:

 

"3-element Array{Int64,1}:\n 1\n 2 \n 3"

[Michael Hatherly]

This might not be the best way to do it, but the following has worked for me in the past.

The output to the REPL is produced using display() (I think), so you can use the following to get the same kind of output as a string:

julia> buf = IOBuffer();
julia> td = TextDisplay(buf);
julia> display(td, [1,2,3]);
julia> str = takebuf_string(buf)

"3-element Array{Int64,1}:\n 1\n 2\n 3"

If some has a simpler version I'd be grateful to see it as well.

-- Mike

[Steven G. Johnson]

On Saturday, June 28, 2014 10:29:35 PM UTC-4, Laszlo Hars wrote:

Yes, it is a possibility, as I wrote above: "Maybe the solution is to pre-process the Julia code to be evaluated, to handle print() and multi-line input in several parse() calls." It is not trivial.

You can capture print output by redirecting stdio as I mentioned above (just saving the original STDOUT in a variable so that you can still print your final output to it).  Multiline input can be handled by Base.parse_input_line(STDIN), which is what the REPL uses to read multiline input and works the same way (it detects if the line is the beginning of a valid but incomplete expression and reads another line if so). 

For example, try:

const orig_STDOUT = STDOUT
rdstdout, wrstdout = redirect_stdout()

while true
    try
        result = eval(Base.parse_input_line(STDIN))
        println("RESULT = ", result)
    catch e
        print("EXCEPTION = ")
        Base.showerror(STDOUT, e)
    end
    s = readavailable(rdstdout)
    clipboard(s)
    print(orig_STDOUT, s)
end

[Steven G. Johnson]

On Sunday, June 29, 2014 7:24:35 AM UTC-4, Michael Hatherly wrote:

This might not be the best way to do it, but the following has worked for me in the past.

The output to the REPL is produced using display() (I think), so you can use the following to get the same kind of output as a string:

The display is just using writemime(io, "text/plain", x).     So, you can just use

            stringmime("text/plain", x)

to get a string representation identical to the REPL output.

But it depends on what you want: there are a lot of print/string variants that print things in slightly different ways.  For example, you can use repr(x) to get unabbreviated output -- the difference will arise for things like large matrices: if you want the abbreviated output as in the REPL, use stringmime, whereas if you want the full output you can use repr.    For scalars etc. the output will be identical.   Or you can use print(x) or string(x), which will differ e.g. for strings (when you print "foo", do you want to print the quotation marks or not?).

[Laszlo Hars]

Thanks Mike, this would allow fixing the problem, that display results don't appear in STDOUT. The help says:

   Display "x" using the topmost applicable display in the display

   stack, typically using the richest supported multimedia output for

   "x", with plain-text "STDOUT" output as a fallback.

Redefining display() with hard coded STDOUT as the first parameter could be one way. Another is temporarily modify the "display stack", to put STDOUT on the top.

--- Can someone tell us how to do these correctly? (Base.display(x) = println(x) seems to work reasonable well.)

Now "only" the problem of catching syntax error messages remain

[Steven G. Johnson]

On Sunday, June 29, 2014 12:04:00 PM UTC-4, Laszlo Hars wrote:

Thanks Mike, this would allow fixing the problem, that display results don't appear in STDOUT. The help says:

   Display "x" using the topmost applicable display in the display

   stack, typically using the richest supported multimedia output for

   "x", with plain-text "STDOUT" output as a fallback.

Redefining display() with hard coded STDOUT as the first parameter could be one way. Another is temporarily modify the "display stack", to put STDOUT on the top.

--- Can someone tell us how to do these correctly

Just call

pushdisplay(TextDisplay(STDOUT))

(Do this after the redirect_stdout in my example code.)

(Though I think it is should probably be considered a bug that this is needed when running "julia foo.jl", i.e. in non-interactive mode.)

[Steven G. Johnson]

On Sunday, June 29, 2014 12:04:00 PM UTC-4, Laszlo Hars wrote:

Now "only" the problem of catching syntax error messages remain

My example code already catches syntax errors.   I don't see the difficulty here.

[Michael Hatherly]

Thanks for the thorough explanation Steven!

[Michael Hatherly]

No problem, though Steven's explanation and example are the right way to go of course. In fact, together with the discussion in this thread and re-reading the section on IO in the docs Julia's IO system is making a lot more sense now. Thanks everyone!

[Laszlo Hars]

Thanks, Steve. I did not know about "Base.parse_input_line". It can very well be the solution together with capturing STDOUT with redefining display(). My only concern is using an undocumented function. It may change or get removed.

[Tomas Lycken]

> My only concern is using an undocumented function. It may change or get removed.

To be fair, you're using a pre-release language. Even though there are probably some things that are very unlikely to change, there's no promise of backward-compatibility yet, as far as I've understood things. And if that specific piece of functionality would change or be removed, it'll probably be replaced by something that does the same thing, but better.

// T

[Steven G. Johnson]

On Sunday, June 29, 2014 12:34:33 PM UTC-4, Laszlo Hars wrote:

Thanks, Steve. I did not know about "Base.parse_input_line". It can very well be the solution together with capturing STDOUT with redefining display(). My only concern is using an undocumented function. It may change or get removed.

That's true, but hacking the REPL is far more likely to break in future releases.

Undocumented internal functions in Julia have a tendency to become documented external functions when they prove useful outside of Base.   There is a reasonable case to be made for exporting and documenting parse_input_line (although it might get renamed in the process).  Maybe open an issue?

(And you shouldn't redefine display; instead push something onto the display stack as I explained above.   For one thing, redefining display(x) is not enough to work for code that explicitly calls display("text/plain", x).)

[Laszlo Hars]

I spent several hours struggling with display() not using STDOUT. I found that
pushdisplay(TextDisplay(STDOUT))
does not make display() to use STDOUT. I think it has something to do with the Latex symbol support, because my old functions stopped working around the time it was introduced.

[Laszlo Hars]

This is what I have in my juliarc.jl file
~~~
pushdisplay(TextDisplay(STDOUT)) # display() does not write to STDOUT: evalu("1+2") vs. evalu("print(1+2)")
const STDOUT_orig = STDOUT

function evalu(s::String)        # paste to console: evalu("""<Julia commands> """)\n
   try
      rd, = redirect_stdout()
      eval(Base.parse_input_line(s))
      print("!")
      s = chop(readavailable(rd))# remove extra ! preventing hang when nothing is written to stdout
   catch s
   finally
      redirect_stdout(STDOUT_orig)
   end
   clipboard(s)                  # return result/error-message
end
~~~
Either I overlooked something obvious, or the normal Julia output to the console is not available in STDOUT. A couple of days ago redifining display() did work, but with the latest nightly (and on another PC) I have no luck any more. Has something changed in REPL?

[Steven G. Johnson]

On Tuesday, July 1, 2014 1:20:25 PM UTC-4, Laszlo Hars wrote:

This is what I have in my juliarc.jl file
~~~
pushdisplay(TextDisplay(STDOUT))

I think the problem is that .juliarc.jl is run before the REPL is initialized, whereas you want to push the display afterwards.  i.e. your TextDisplay is no longer on top of the display stack when the REPL runs.

Why are you using juliarc.jl for this?   It seems more sensible to put the script you want into a file myrepl.jl, and then just run julia myrepl.jl ... 

[Keno Fischer]

Actually, I fixed this so that any display pushed in .juliarc.jl will be on top of the display stack.

[Laszlo Hars]

>(Steve) juliarc.jl is run before the REPL is initialized
I thought about that possibility, and tried also manually typing in

pushdisplay(TextDisplay(STDOUT))

when the REPL is up and running. There is no difference: the Julia console output just does not appear on my STDOUT

>(Keno) I fixed this so that any display pushed in .juliarc.jl will be on top of the display stack
Does it mean the the problem is in somewhere else, or it will work with the next nightly?

[Laszlo Hars]

> Does it mean that the problem is in somewhere else, or it will work with the next nightly?
Nope, with the current nightly prerelease Julia version console display still does not appear in STDOUT. Redefinition of Base.display() to print() does not help, either, so the problem is still open. Any more ideas?

[Steven G. Johnson]

It works for me on MacOS.  e.g. putting the code below into a file capture.jl, then running
      julia capture.jl
I can then type e.g. display("foo") and "foo" gets included in the text copied to the clipboard.

#####################################################

const orig_STDOUT = STDOUT
rdstdout, wrstdout = redirect_stdout()

pushdisplay(TextDisplay(STDOUT))

[Laszlo Hars]

Thanks, Steve. Good to know that something works under MacOS. Unfortunately, I get nothing to the Windows clipboard with your 2-engine method, which was the problem that started this thread. (Also, my application does not need an infinite Julia loop: I can just surround the code to be executed with output redirection Julia commands - if ever they would work.)

[Jameson Nash]

pasting Steve's code into the windows terminal in 0.3 works for me also.

although note that readavailable is also perfectly free to return only one character at time (or any amount, from 1 byte up to and including the amount of data that has been written to it -- it might not even need to return a complete UTF8 characters out of the stream). the only way to be certain you have captured all of the data that has been written to STDOUT is to back out your changes (remove it from the display list, `redirect_stdout(orig_STDOUT)`, and `close(wrstdout)`) then call `readall(rdstdout)`

[Laszlo Hars]

Oh, I see. It is not a 2-engine solution. It was not clear to me from Steve's post.

Pasting the code to the Julia console does work, but we need editing of the results. Here are some simple test cases:

1+2
RESULT = 3

print(1+2)
3RESULT = nothing

for i = 1:3
println((i,i^3))
end
(1,1)
(2,8)
(3,27)
RESULT = nothing

The clipboard in the end is

(1,1)(2,8)(3,27)RESULT = nothing

[Laszlo Hars]

Steve's code works even without "pushdisplay(TextDisplay(STDOUT))", which tells us that the problem of not seeing regular Julia console output on STDOUT may be because we don't read it properly. We may have to wait, close the write pipe, redirect the output back to the original... or something else, like printing something. Maybe, the  "print(orig_STDOUT, s)" command does the trick?  This should be easy to Julia experts, but drives newbies crazy.

[Laszlo Hars]

Looking into Steve's code we see it cheats. Our problem has been that the Julia console output does not appear on STDOUT, only the results of print() do. Steve's code evaluates the input expression and then prints it. What we see in STDOUT is not the Julia console output, but results of "println("RESULT = ", result)". We can have this effect without an infinite loop, what I ended up using during the last couple of days:

const STDOUT_orig = STDOUT

function evalh(s::String) # paste to console: evalh("""<Julia commands> """)\n

try

rd,wr = redirect_stdout()

s = eval(Base.parse_input_line(s))

close(wr) # needed for readall()

s = readall(rd) * (s == nothing ? "" : string(s))

[Steven G. Johnson]

On Thursday, July 3, 2014 9:20:55 PM UTC-4, Laszlo Hars wrote:

Looking into Steve's code we see it cheats. Our problem has been that the Julia console output does not appear on STDOUT, only the results of print() do. Steve's code evaluates the input expression and then prints it.

The point I have been trying to make is that you probably shouldn't be using the REPL ("console") at all for this sort of thing, which is not what the REPL was designed for; you should be writing your own read/print script.  Outside of the REPL, you have no choice but to use print (or similar) if you want to see a result.

[Laszlo Hars]

> shouldn't be using the REPL ("console") at all for this sort of thing

Fair point, even though using REPL has advantages: we can always go to the Julia console to see what has been executed, look at the history, use help, try out variations of the code directly there... Nevertheless, the thread was started saying that evaluation of computer generated code was just one of the potential applications of capturing the console output. Logging Julia output to a file was another, and there are several more.

Here we are after 40 posts in this thread and we still don't know, how to capture the Julia REPL output.

[Steven G. Johnson]

Nevertheless, the thread was started saying that evaluation of computer generated code was just one of the potential applications of capturing the console output. Logging Julia output to a file was another, and there are several more.

Here we are after 40 posts in this thread and we still don't know, how to capture the Julia REPL output.

Short answer: declare your own subtype MyTerminal <: Base.Terminals.TextTerminal <: IO, implementing the abstract TextTerminal interface in base/Terminals.jl but with a write(t::MyTerminal, x) method that also writes x to a file or whatever it is that you want to do with it, then set Base.active_repl.t to an instance of MyTerminal.

For example, I put together a quick hack at https://gist.github.com/stevengj/88502943fe0478933492 that allows you to log the output of the REPL to a file.  Just do
     Base.active_repl.t = LoggingTerminal(Base.active_repl.t, open("foo.out", "w"))
and the REPL output will be logged to "foo.out" (though you may need to flush(Base.active_repl.t) to see it).

However, the real answer is that hacking the REPL like this exposes a lot of undocumented guts, and is ugly, and is probably not the way to do what you want.

* If you want to send commands from an external program to Julia and then send the results back, the right thing is probably to implement your own read-eval-print loop like in the example code I showed earlier.
        -- If you want to do this while still being able to execute Julia commands interactively in the REPL, you can run something like my example code in a background task.

* If you want logging of REPL output, and you don't just want to copy-and-paste from your terminal into a text editor to save the output, then probably you want a more full-featured environment like the IJulia notebook.

--SGJ

[Laszlo Hars]

Thanks, Steve. I'd never imagined that the solution is that complicated. Now I have something to digest for a couple of weeks :)

[Terence Copestake]

I'm not convinced I've understood this thread 100%, but the issue caught my interest so I went away and had a play. I've put together a gist of a .juliarc.jl file that ended up being a retake on things already mentioned here, so it's possible that it's still not the solution you're looking for.

The gist is up at https://gist.github.com/tcopestake/3b3c0891d4ca29df9f09

(Note that I've been testing this in a REPL opened with the --color=no option)

Typing commands in the REPL, I can see output on the screen and it's also logged in the julia-stdout.txt file.

Is this not what you're looking for? Perhaps if you can point out what my code doesn't do that you need it to do, I can have another look into it.

Stay safe.

[Stefan Karpinski]

The idea of reading and writing to a REPL by reading and writing unframed inputs and output over a UNIX-style stream pair is fundamentally broken. The biggest issue is that the REPL interaction is fundamentally message-oriented while streams are, by definition, not. There's also the issue that you need much more than "just a send a chunk of code and get a result back" because sometimes there's no result to send back – it's an error instead, and there are many kinds of errors, including ones where the input was incomplete or irrecoverably incorrect. This is inherent to the problem of REPL style interaction. You can either handle the problem in an incomplete way and live with the brokenness of it, or you can do it right and live with the fact that it's complex. This is emphatically not some unnecessary complexity that Julia has foisted on you – if you want something that acts like a REPL, you have to implement the front end of a REPL.

[Laszlo Hars]

>(Terry) Typing commands in the REPL, I can see output on the screen and it's also logged in the julia-stdout.txt file.

Thanks!. Your code does log normal output. We have to extend it a little to log also error messages, but warnings are logged correctly.

>(Stefan) reading and writing to a REPL by reading and writing unframed inputs and output over a UNIX-style stream pair is fundamentally broken.

I don't see, why it has to be fundamentally broken. If the REPL was written with print() calls, instead of display(), we would not have a problem. Or, if display() sent data to STDOUT like print(). We can live with a few unsupported features, like triple-quoted strings, but handling no output or catching error messages are easy. Even if I just enter "1)", which is unrepairable. The code I write and use myself does not have to be perfect, just good enough. The simple solution I posted last works for me 99.9% of the time, which I cannot claim for most commercial software.

[Stefan Karpinski]

If you're just sending unframed data across a pipe, when does the receiving side evaluate the code?

[Laszlo Hars]

> when does the receiving side evaluate the code?

There are different solutions. E.g. I can send its length information with the data, and call parse-evaluate when that many bytes arrived. (I coded this up, and it is working, too). However, the function I posted for evaluating computer generated code does not use input streams, but properly terminated strings are pasted to the console, that is, it uses messages for the input. The return path is also safe: whenever the evaluation is done, the next Julia instructions are executed, which save the output. I did not find problems with this approach, except with the aforementioned triple-quoted strings, which is not a big deal. Multi-line input and output, error messages, etc. all work reliably.

The problem I see is the unpredictable behavior of the REPL. With an older Julia version I get some output twice. Can I rely on the current behavior of the REPL (when the display() results don't appear on STDOUT), or in another nightly version they would appear again?

Some kind of documentation on the REPL I/O would really help us.

[Laszlo Hars]

Terry's idea works also for evaluating generated Julia code, too, by just pasting the commands to the REPL window. We only need to mark the end of the computation with printing a special character (e.g. of ASCII code 250) and then the code below copies the result to the clipboard:

# for tests: put these to ".juliarc.jl"
# last pasted command: print('·') copies data to clipboard, resets data collection

STDOUT0 = STDOUT
 
rd,wr = redirect_stdout()
redirect_stderr(STDOUT) # warnings OK, error messages fail!
pushdisplay(TextDisplay(wr))
 
@async begin
   c = ""
   while true
      d = readavailable(rd) # blocks loop until there is data in rd
      c *= string(d) # collects pieces of result
      write(STDOUT0, d) # sends output to console (via original STDOUT)
      flush(STDOUT0)
      if search(c,'·') > 0
         clipboard(c)
         c = ""
      end
   end
end

Only the problem with error messages remain. The recent nightly builds of Julia don't seem to write the error messages to STDERR, and so redirecting it does not help.

[Laszlo Hars]

I hoped that at some point Julia error messages would again appear in STDERR. No luck. Even the Release candidate 0.3.0-rc1 fails to write, e.g. DomainError to STDERR (provoked by 1^-1 entered in the Windows REPL). Maybe the documentation should mention that STDERR is for warnings, only.

[Kevin Squire]

Interesting.  Can you (or did you) file an issue?

Cheers,

   Kevin

[Laszlo Hars]

I don't know, how to file an issue. Where can I get instructions? (The STDERR behavior may be intentional. I kept asking about it in this group, for months, but nobody was interested.)

[Jameson Nash]

I think the change in the STDERR behavior was intentional, once we got rid of the limitations of using libreadline. Now we can run multiple simultaneous independent REPL loops, thanks to Keno's work. Redirecting STDERR and having that propagate to the stderr stream used by the REPL was a limitation in 0.2 which no longer exists in 0.3. I wouldn't expect it to be reintroduced.

Steve Johnson has provided solutions several times. Such as https://gist.github.com/stevengj/88502943fe0478933492, if you want to hook into the existing REPL, or

      while true
       println("RESULT: ", eval(parse(readline())))
       end

if you want to roll your own.

Implementing a good REPL requires interacting with the user, so it is inherently a hard task and thus may needs a fairly large amount of code to configure and manage the program's state to stay synchronize with the user's expectations.

PS: https://github.com/JuliaLang/julia/issues/7633

[Kevin Squire]

Per Jameson's response and my own testing, it seems like the STDERR issues have been resolved (perhaps after rc1?):

$ julia -e 'throw(DomainError())' 1> stdout.txt 2> stderr.txt

$ cat stdout.txt 

$ cat stderr.txt 

ERROR: DomainError

 in process_options at ./client.jl:213

 in _start at ./client.jl:354

For future reference, you can search for current issues and file new ones at https://github.com/JuliaLang/julia.  Click on the Issues tab on the right.  Search is located at the top.  

Cheers,

   Kevin

[Kevin Squire]

Or not.  See https://github.com/JuliaLang/julia/issues/7633.

[Laszlo Hars]

> the STDERR behavior was intentional

1) It just does not seem to be logical: warning messages still get written to STDERR. Why? And what is the use of STDERR, in general? STDERR is in the documentation, without telling anything about it.

2) The REPL behavior ought to be documented, otherwise new users would naively think that error messages are sent to STDERR, as regular console (REPL) output is available in STDOUT, and user input is seen in STDIN. So what is so special about error messages sent to STDERR? (Btw. error messages are not sent to STDOUT, either, they are just displayed in the console window.)

> Steve Johnson has provided solutions several times

Unfortunately, his solutions either don't work or they are so complicated, that I cannot include them into my programs.

> println("RESULT: ", eval(parse(readline())))

As we discussed it several times, this does not work with multi-line input. Sure, I can write my own REPL, but this is not the issue here. We should not have to do it, since there is one included in the Julia distribution. It just does not seem to have sufficient documentation.

[Steven G. Johnson]

As I understand it, the REPL makes a private copy of the output-stream descriptors early on, which is why you can't redirect them.    This is actually quite useful behavior, because otherwise it would be impossible to debug code using redirect_stdout etc. in the REPL (because the REPL would stop working as soon as you redirect).   (The REPL also needs to be able to talk to the terminal at a lower level than just stdout in order to support things like line editing.)  Also note that the REPL is a program written in Julia, not a part of the Julia language or standard library, so you wouldn't necessarily expect to see documentation of the REPL's low-level terminal-communication internals.

On Wednesday, July 16, 2014 8:48:11 PM UTC-4, Laszlo Hars wrote:

I don't know, how to file an issue. Where can I get instructions? (The STDERR behavior may be intentional. I kept asking about it in this group, for months, but nobody was interested.)

The difficulty here is that you are trying to hack the REPL for a task to which it is poorly suited.  Multiple alternative suggestions have been offered for better ways to communicate between Julia and an external process, but you've rejected them.   Basically, you should set up a channel to receive messages, execute the code yourself, catch errors yourself, and then you can do whatever you want with the output, rather than trying to intercept it from the REPL.   We didn't write all of your code for you, but we gave enough examples that you should be able to implement what you need.

Note that this is independent from whether you want the REPL to be running too, in order to be able to use Julia interactively with the same data.   Just run your receive-execute-respond loop in another thread (asynchronous task), and you can have the REPL running at the same time.

Another example is IJulia, which receives messages with Julia code from another process (the IPython front-end) and then sends back the results, including errors.   IJulia never invokes the REPL at all.

[Steven G. Johnson]

On Wednesday, July 16, 2014 11:17:32 PM UTC-4, Laszlo Hars wrote:

> the STDERR behavior was intentional

1) It just does not seem to be logical: warning messages still get written to STDERR. Why? And what is the use of STDERR, in general? STDERR is in the documentation, without telling anything about it.

You are confusing two output channels:

1) A Julia program may have side effects, e.g. it may call println(...), warn(....), etc.   These are written to some stream, by default STDOUT for print and STDERR for warn, and in general you should use STDOUT and STDERR just like stdout and stderr in C (and they use the same file descriptors as libc's stdout and stderr, internally, which is useful for Unix pipes etc.).   This is a property of the Julia language and standard library.

2) When you execute a code block in the REPL, the REPL also prints its result value (if any), or any exception that was thrown.  This is not a part of the code or a part of the Julia language or standard library, it is a behavior of the REPL, which is an interactive program that happens to be bundled with Julia.   There is no reason to use the same output channel here as the ones used for side effects of Julia code, and in fact it is sometimes useful to have it be a different channel (e.g. if redirect_stdout has redirected Julia side effects, but you still want to run the REPL).  And the REPL can be easily replaced by a different interactive environment, e.g. IJulia or Julia Studio, and these environments may do different things with the results/exceptions produced by expressions that are evaluated.

 

2) The REPL behavior ought to be documented

The low-level mechanisms by which the REPL communicates with the terminal are not part of the Julia language or standard library.

or they are so complicated, that I cannot include them into my programs.

> println("RESULT: ", eval(parse(readline())))
As we discussed it several times, this does not work with multi-line input. Sure, I can write my own REPL, but this is not the issue here. We should not have to do it, since there is one included in the Julia distribution. It just does not seem to have sufficient documentation.

We explained to you several times how to capture multi-line input.  You can either use parse_input_line like the REPL or (better) you can devise your own communication protocol (like the one used between IJulia/IPython), that communicates code blocks to be evaluated in an unambiguous way.

[Laszlo Hars]

> you wouldn't necessarily expect to see documentation of the REPL's low-level terminal-communication internals

I understand. However, as we learn the language we use the REPL. It is the first thing we see of Julia. A clear distinction in the documentation would be really helpful telling what of the Julia features works in the REPL and what does not.

> trying to hack the REPL for a task to which it is poorly suited

This "hacks" have been working for me. I could achieve everything I wanted with 8-10 lines of Julia code.

> Multiple alternative suggestions have been offered for better ways to communicate between Julia and an external process, but you've rejected them

I did not ask help "to communicate between Julia and an external process". I posted several code snippets, which do accomplish that task, with 8-10 lines of code. I asked help to capture the output of the REPL. I could not imagine that it was that hard. I believe it should not be.

It is an entirely different conversation to discuss if I need to capture the output of the REPL, and if I could achieve the same or better results with something else. I would be glad to discuss that, but the point here was simply how I can capture the output of Julia's console. As I understand now, the only way is to define my own terminal and replace the REPL terminal with it.

[Laszlo Hars]

> We explained to you several times how to capture multi-line input.

You missed my points again. This was not the issue in this thread. Still, if you look at the code snippets I posted here, you can see that I knew how to handle multi-line input. I did not need your "friendly" reminder or explanations.