How do you do interactive development in Julia?

[Andrei Zh]

I'm trying to find my way developing Julia code interactively. In other languages (e.g. Python, R, Octave, etc.) when working on some piece of code I open a file with it and a console. Each time I change something in the file, I send that part to console and thus get new state. When I change a lot of things, I just send the whole file, replacing all definitions. It's quite convenient, and I'm pretty sure many of you are familiar with this practise.

In Julia, however, I faced several challenges.

1. When developing modules (and I really like modular systems) I have to either run "using MyMod" after each change, or use qualified names (e.g. "MyMod.somefunc()"), which is really annoying. In Python, for example, all definitions sent to console go right to a global namespace, which is pretty convenient within single module. And for several modules there's IPython's "%autoreload 2".

2. If I abandon modules, on other hand, I can't redefine constants. So if, for example, I define type Point in global namespace, then change it and want to load new definition, I just get "invalid redefinition of constant Point" error.

So I want to know how YOU cope with these issues. Are there any best practises for interactive development? Are there any workarounds for cases I mentioned?

Thanks you,
Andrei

[Johan Sigfrids]

There is a Autoreload.jl package modeled after IPython's autoreload extension.

[J Luis]

I use a trick that makes use of the fast Julia's start time. So I create a small "do_it.jl" file with only, let's say

using MyModule
exec_this_fun()

than on the shell (Windows cmd in my case), I just do

julia do_it.jl

This has the advantage of working when the "reload("MyModule") doesn't work, and I have found many cases where it doesn't (for instances when writing ccal wrappers to some C library)

[Sebastian Nowozin]

Hi,

I use a similar method; however, what would really help is an option to "drop to the REPL" after code execution, in order to inspect the state of variables.  Say, to call "julia --no-exit test1.jl" and after the last line in test1.jl the REPL would activated.

Similarly, in Matlab, I often write the initial prototype code by inserting a large number of "keyboard" statements; when the keyboard statement is reached MATLAB stops and goes to the REPL.  Then I inspect a few variables and issue a "dbcont" statement to continue to the next instance of the keyboard statement.  I know Julia has a debugger, and it may support this use, but I have not tried whether this is possible to use it for a similar workflow.  After such a prototype run I typically extract the verified code into a test case without any keyboard statements.

Best,
Sebastian

[Sebastian Nowozin]

Hah, I just checked the help message of Julia once again and indeed the "-L" option seems to accomplish exactly what I want.
Sorry for the noise ;-)

Sebastian

[Andrei Zh]

Thanks for your answers. I used strategy with something like "do_it" script when working in Octave, but I found that it doesn't provide full REPL experience. In REPL I can experiment with different approaches, save state of long-running computations, see results and decide where to go further without replaying all operations. With "do_it" script all these things are just harder and tale longer (though I still replay everything from time to time to clear possibly dirty state).

Autoreload.lj looks good, but it seems like problem lies deeper in Julia itself. Today I found that if I write "using MyMod" and then change any definition (say, function "myfun()") from it, there's no way to update this definition in Main. Consider following case study. I have following in file "MyMod.jl":

-------------------------
module MyMod
export myfun

function myfun()
    println("version 1")
end

end
------------------------

Then in REPL I do:

 julia> using MyMod

 julia> myfun()
 version 1


Looks ok. Now let's change printed expression to "version 2" and send new module to REPL:

julia> include("/home/slipslop/work/MiLK/MyMod.jl")                                                                                                                   
 Warning: replacing module MyMod

Ok, this is what I expected. MyMod is completely replaced with a new one. And what about "myfun"?

 julia> myfun()
 version 1

Nope, it's not reloaded by itself (though qualified name "MyMod.myfun()" works fine). Let's re-import names from MyMod:

 julia> using MyMod
 Warning: using MyMod.myfun in module Main conflicts with an existing identifier.

 julia> myfun()
 version 1

So Julia sees conflicting name and does NOT replace definition. "reload()", "areload()", "arequire()", etc. just reload module, but they don't help to update existing names in the Main. The only way to see updated version is to use qualified name:

 julia> MyMod.myfun()
 version 2

Is there any way to update definitions that where imported into Main with "using" statement?

[Tim Holy]

On Sunday, July 06, 2014 02:52:45 PM Andrei Zh wrote:
> Is there any way to update definitions that where imported into Main with
> "using" statement?

Not that I know of. https://github.com/JuliaLang/julia/pull/7487 may be of
interest to you.

--Tim

[Andrei Zh]

Thanks Tim, this is another interesting capability, and I'm definitely looking forward to try it out in Julia REPL. It's a bit unusual though, so I'm still looking for smoother way to work interactively.

I checked one of popular packages (namely, Distributions.jl) and I like their approach. They put all the code for related functionality into somefile.jl without any bothering about modules, play around with it (redefine without any problem) and then, when they are done, just include this file into SomeModule. This is not what I've got used to in Python, but it still works quite well.

Anyway, further ideas for better interactive development are still welcome.

[Sebastian Nowozin]

Hi Andrei, Tim,

another related issue I sometimes have with interactive development in
Matlab is to load/save workspaces entirely.

For example, in a typical experiment I will have a short driver script
"run_exp1.m" which contains as its last line a statement like "save
result_exp1.mat", saving everything, including options defining the
experiment and the results.
(Then typically for me a second script like "plot_exp1.m" is used to
load this file and create some plots.)

I have looked around a little but there seems to be no way to save all
variables listed in whos() in Julia. There is the MAT package and I
have used it before to save arrays, but what about general Julia data
structures?

Are there plans for universal serialization/deserialization of data
structures in a portable and future-proof manner, such that it would
be suitable for storing experimental results?

Thanks,
Sebastian

[Tim Holy]

https://github.com/timholy/HDF5.jl

--Tim

[Rob J. Goedman]

Andrei,

Not sure if my way is better, but it seems to work ok for me. It is also biased towards OSX, although I know a similar approach has been used on other platforms (for R).

I have Julia's REPL window side by side to TextMate (or similar editor). I extended Julia's plugin for Textmate with 1 menu item, Command-R, in addition to Command-E. Command-E sends the current selection to Julia's REPL. This command has always been there I believe.

Command-R sends the entire file to the REPL by inserting an include("..."), mainly because Command-E creates too much clutter in the history.

Usually I'm editing a file and then re-submit it to the REPL by typing Command-R in the editor. Occasionally though this is not sufficient and one or more modules have to be reloaded as you point out.

I kill the REPL (Ctrl-D), now back in the shell, UP-ARROW will bring up something like 'clear; julia', RETURN starts a new version of the REPL, UP-ARROW again recalls the last command executed in the REPL, RETURN to execute.

Thus it takes a minimum of 5 key-strokes, sometimes a few more because I tried out some stuff in the REPL. In those cases CTRL-r is quite handy.

Works in my workflow, YMMV.  I think key is that restarting the REPL and loading the needed modules is fairly quick. While developing packages I have found that the restart time is usually easy to control. 

The other gotcha is that I usually have multiple terminals open, and need to ensure the REPL is the last used terminal. Again, in my case I've gotten used to it. Or on OSX I could probably find the REPL terminal in AppleScript and activate it. But I haven't been bitten enough to put in that effort.

Regards,

Rob J. Goedman

goe...@icloud.com

[Andrei Zh]

Hi Rob,

Thanks for your answer. I use very similar approach with Emacs and ESS mode[1]: I've bound command "ess-eval-line-and-step" to "C-d" (CONTROL / COMMAND + D) to be able to send single line and "ess-load-file" to "C-c C-c" (double CONTROL / COMMAND + C) to reload the entire file. I don't like idea of reloading REPL, though. Not because it takes 5 commands (I can actually automate most of these keystrokes), but because this way you lose your current state. This may be really painful, especially when your computations took half an hour or so (which is often the case in image processing, recommendation engines, etc.)

I think, though, that your ideas combined with clear() function ([2]) from Tim's comment should work pretty well. 

[1]: https://github.com/emacs-ess/ESS/wiki/Julia
[2]: https://github.com/JuliaLang/julia/pull/7487

[Rob J. Goedman]

Hi Andrei,

I'll have a look at your pointers before further extending the TextMate bundle, particularly [2] looks promising.

Thanks, and regards,

Rob J. Goedman

goe...@icloud.com