Julia Utopia: Share your tips and tricks to efficient coding in Julia

[David Parks]

I'm a few weeks into Julia and excited and motivated to learn and be as efficient as possible. I'm sure I'm not alone. I know my way around now, but am I as efficient as I can be? 

What haven't I tried? What haven't I seen? What haven't I asked?

For those of you who have been around longer, could you share your advice on efficient day-to-day development style?

For example:

• What IDE do you use? Are you using Atom? A combination of Atom and the REPL? Something else?
• How do you debug complex code efficiently? How do you debug other peoples code efficiently?
• Do you have a favorite way of visualizing your work?
• Are there must have tools? packages? utilities?
• Any simple day-to-day efficiency/advice you could share with others who didn't yet know to ask.

[Eric Forgy]

I'm also interested in this question. Having been lurking around for more than a year, I've seen this kind of question come up a few times and don't remember the answers, but remember seeing answers and they were helpful. I think it would be cool/helpful if Julia Computing had an "About Us" page with bios and answers to similar questions to the above for each person.

[Jacob Quinn]

I'll take a stab here. For context, I've been coding in Julia since 2012, contributed to Base and some data processing packages.

* I currently use Atom; it's come a long way since it started, both the IDE itself and Julia support. It seems to have way more momentum in both respects as well (vs. say, sublime). I don't use the inline evaluation very much; not sure why, but I just haven't ever been a huge fan of that workflow. I typically just copy paste into the terminal (iTerm2 on mac), and enjoy using the *very* rich REPL features Julia provides (tab completion, history search, symbols, etc.).

* For debugging code, I typically rely on good ole `println` or `@show`, plus building things up incrementally (validating things as you build up). I recently gave Gallium.jl a spin; for context, I've never coded in a language with a "real debugger", so I'm not sure I'm the ideal candidate here, but I found it somewhat difficult to navigate the right granularity of stepping through code (should I go to the next line? next call? step into it?). I'm 100% positive this is due to me needing to spend more time with it and better learn "debugging" practices; note it's currently being developed on master, so 0.5/nightlies.

* I don't do a lot of visualizing of my work, so no great advice/opinions here.

. 

* For day-to-day, here's a smattering of things I find myself reaching for all the time:

  * modules: I rarely get very far developing new code without throwing it in a module; this allows to develop much easier interactively and iteratively because you can just `reload("MyModule")` and you're ensured all the types/methods get redefined. 

  * @which: One of the handiest macros in Base to help you navigate Julia's powerful dispatch system; use to ensure that the right methods are being called. It also comes in very handy for just taking a quick peek at somebody else's function and where it was defined if I want to take a closer look.

  * @time and @profile: also very handy macros when you get into performance tuning. Check out the Profiling section in the manual to learn some of the ins and outs, but it becomes extremely helpful to get a feel for where time is being spent in your code

  * @code_lowered, @code_typed, @code_llvm, @code_warntype: (there's also @code_native if you're into the real nitty gritty). These are your best friends! For greater context, search YouTube for Jeff Bezanson's JuliaCon talk where he talks through the various levels of "compilation" in Julia; it goes a looooong way to understanding exactly what's going with your code. @code_warntype is particularly helpful for spotting type instabilities which can kill performance. It's also helpful to get a feel for the actual machinery of your code

Anyway, hopefully that sparks a few ideas.

-Jacob

[Tamas Papp]

I use Julia with Emacs+ESS+julia-mode. I find it very efficient,
especially as I use Emacs for everything else too (LaTex, other
programming languages, e-mail, git interface).

I debug using print statements. I find @show very valuable, also see
https://github.com/timholy/DebuggingUtilities.jl . I am looking forward
to using Gallium.jl though, but at the moment I am in the middle of a
project with a deadline so I stick to the current stable version and
don't play around with 0.5.

I find @edit really useful, for looking up code for matching
methods. Also methodswith.

I find Plots.jl and Gadfly.jl very convenient for quick plots (and of
course also for production-quality plots). I have been using the latter,
and experimenting with the former now.

Favorite packages outside the core include Parameters.jl. JuliaStats
packages are really well-designed and high quality too. But of course
which packages you find useful would depend on your application domain.

Coming from Common Lisp and R, I had to wrap my head a different
workflow. In interactive languages I used previously, I kept
manipulating the image interactively, refining functions, building up
the good old "big ball of mud", then I would clean up, all without ever
restarting the process (especially in CL).

Workflow in Julia is quite different, especially because of the infamous
issue #265. I put things in modules very early, and keep reloading. I
have been experimenting with an interactive workflow using
eval(module,...) via ESS, but until #265 is fixed I put that on the
backburner.

So because I call workspace() so often, at the moment I break up things
into modules and small "scripts" which load them, load some data and
generate other data or plots, each a single well-defined step (eg
datacleaning, MCMC, simulations). I found JLD invaluable for this. To a
certain extent, this reminds me of a C/Fortan workflow, but in a nice
way :D It is certainly more disciplined then doing everything
interactively and then (occasionally) not being able to reproduce the
results.

Having used Julia for a while I am much more relaxed about optimizing code
now. I try not to do silly things (like row-major access for a
column-major structure), but in general I postpone profiling and
optimizing to much later, and sometimes don't do it at all. I find this
efficient because I spend much more time writing and rewriting code than
running it, and Julia is very efficient out of the box, so extracting
another factor of 2-5 with a lot of work is not a major concern unless I
use the code often.

On Thu, May 12 2016, David Parks wrote:

> I'm a few weeks into Julia and excited and motivated to learn and be as
> efficient as possible. I'm sure I'm not alone. I know my way around now,
> but am I as efficient as I can be?
>
> What haven't I tried? What haven't I seen? What haven't I asked?
>
> For those of you who have been around longer, could you share your advice
> on efficient day-to-day development style?
>
> For example:
>

> - What IDE do you use? Are you using Atom? A combination of Atom and the
> REPL? Something else?
> - How do you debug complex code efficiently? How do you debug other
> peoples code efficiently?
> - Do you have a favorite way of visualizing your work?
> - Are there must have tools? packages? utilities?
> - Any simple day-to-day efficiency/advice you could share with others

[Sisyphuss]

I use jupyter notebook in an aggressive way. I consider working in Atom when it releases its Linux portable version (at least without installing).

I use `print` and `@show`. But I find the format of `@show` for matrix is not very friendly. And I agree with Tamas. I also program with a very incremental way to avoid bug for large project.

I use `winston` to visualize my simulation result, because it's simple and already satisfies my needs. Tom's `Plots.jl` is also a good choice.

No for me.

I agree with Tamas, one great difficulty of using Julia is to refuse the temptation of over optimizing the code.

[Penn Taylor]

I don't have much to add to what others have said about workflow. Lots of good advice so far, especially about putting things into modules early on and putting effort into getting short, tightly-focused functions.

Here's the one piece of advice I wish I had been given early on: Multiple dispatch is the heart and soul of Julia. The language has lots of other neat features, but multiple dispatch --and the associated type system-- is the key to effectively modeling problems in Julia. It's the one thing you should concentrate on really understanding, and the one thing you should really pay attention to when looking at well-designed code others have written. If you're coming from an OOP background, it may take some time to come to terms with how multiple dispatch allows, encourages, and sometimes demands that you structure your model.

[baillot maxime]

Me I'm using 'Atom' now best IDE for me with a small problem with the workspace, it can be triqui if you want to run codes with include files in it.

 I don't like the way of jupyter but it's a good tool if you want to do a lectures.

also I'm still looking for a Matlab type IDE (damn matlab! I'm too used to it...)

About plotting i'm using 'PyPlot' but I don't like the idea of using an external program (by external I want to say no coded in julia) 'Gadfly' is good for simple 2D print. I will try `Plots.jl` look like a good package.

I hope it will help you and all the other suggestion of the other people was very good (I just learn some stuff :) )

[Liye zhang]

(1) I use Debug.jl to debug the code, and it works.

https://github.com/toivoh/Debug.jl  

(2) I use JuliaDT for code editor, which is similar to PyDev. The current version is 0.2. The future version would be a very good julia IDE. I currently use it for my daily work.

https://github.com/JuliaComputing/JuliaDT

(3) The packages depend on your research field.  For me, I mainly using packages such as JUMP, LightGraphs.jl, DataFrams and Gadfly.

On Friday, May 13, 2016 at 1:01:04 AM UTC+8, David Parks wrote:

[Cedric St-Jean]

Jupyter has completely changed my workflow in Python and Julia. I've got my notebooks numbered 01_Planning_Invasion, 02_Moving_the_Troops, etc. I write most of my code inside the notebook interactively, then if it's good enough, it gets promoted to a .jl file. It takes about a week to complete a notebook. Once it's done, I make a test out of the final calculation, and move on to the next problem.

Emacs+ESS is OK, but it doesn't integrate well with the notebook workflow.

Autoreload.jl is really good once you know how to use it, though unfortunately there's a number of gotchas to get there.

For debugging code, I typically rely on good ole `println` or `@show`, plus building things up incrementally (validating things as you build up). I recently gave Gallium.jl a spin; for context, I've never coded in a language with a "real debugger", so I'm not sure I'm the ideal candidate here, but I found it somewhat difficult to navigate the right granularity of stepping through code (should I go to the next line? next call? step into it?). I'm 100% positive this is due to me needing to spend more time with it and better learn "debugging" practices; note it's currently being developed on master, so 0.5/nightlies.

I don't find stepping to be very useful either, but the ability to go up and down the stack to inspect variables and objects is significantly better than using print statements. Is that in Gallium?

[Chris Rackauckas]

I work on numerical/scientific code, so my experience may be different than more traditional programming uses. 

Atom is great. I tried the original JunoLT, and it put me off from Julia for awhile, but I feel at home in Atom. Nice tip: there's a package for hidpi if you have a 4K screen. It will automatically resize your text to match the difference. I have a 4K widescreen monitor turned vertical and can swap my code over there to show code paradise, but then if I swap it to another screen (to have a big pdf open) it changes instantly. 

Honestly, the debugging could be better. The errors are really good so you usually know what line it is and what kind of problem it is, but I would like a debugger (and haven't tried to build Gallium yet). But println debugging works well.

Plots.jl is great. I was using a mixture of PyPlot and Gadfly, but am moving my package over to Plots.jl because it lets you use like every package. That's actually the key to Julia. Julia with its named functions and multiple dispatch has the ability to wrap things really well. Thus there are very powerful packages which wrap things. Plots wraps a bunch of backends so you can use the same plot command with a bunch of different backends. JuMP lets you define a nonlinear problem and try out a bunch of different solvers. Etc. Things that would normally be multiple different scripts just to test things out, is now the difference of 1 line. 

Must have packages? DifferentialEquations.jl. I kid... but not really. I made something like this in MATLAB a few years ago, but it was really slow due to what MATLAB does with anonymous functions, and many other reasons. I built this in Julia quite quickly and it works well. The flexibility of Julia gives the same situation like I described in the last paragraph where it's really simple to define new problems/algorithms and so developing new numerical methods is much easier.

But the true good packages are the amazing performance-based macros. ParallelAccelerator.jl is killer: you just add @acc in front of a loop and get a really fast basically fully-optimized C++ program. Devectorize.jl can devectorize vector-functions to make readable but fast code. JuMP and all of the optimization packages it can use is amazing if you do that kind of work as well. CudaRT.jl is really easy to use and gives a quick interface to CUDA code. 

Two pieces of advice. The first is to get on Gitter. People are really helpful on Gitter, don't be afraid to ask. It's been really helpful for me. The second, is the machinefile. I don't know if you use HPCs, but if you use an HPC using multiple nodes for a computation in Julia is as simple as this: add the MPI stuff to your job script, and pass the machinefile to Julia. That's it. Now your @paralllel loop runs on 192 cores. I wrote how to do this in more detail on my blog. Coming from doing C+MPI coding to do this stuff, I will never go back. Now code that takes MATLAB amount of time to develop can do almost what the C+MPI code could do (which takes sooo much longer to make).

[Scott T]

I've been writing papers that have scientific results powered by Julia, and I want every part of the paper to be reproducible if necessary. So I'm finding myself writing a lot of Jupyter notebooks to tinker around with things, make or read data files, make and save plots (as svg, png and pdf), and then writing a simple .jl file that uses NBInclude.jl to execute any relevant notebooks. Then I'll write a short Makefile which calls that jl file to generate the figures as part of my make process. The combination of jupyter and NBinclude is proving to be really nice. Anything reusable will get made into a module. 

[Scott Jones]

I'll also take a stab at this.  For context, I've been programming in Julia full time for the last 13 months, as well as mentoring a couple of my colleagues using Julia, and have contributed to various parts of Julia (string conversions, unit tests, documentation, among other things).

* I use Emacs (mostly because I haven't had time to move my customizations / key bindings I've had for years over to Atom, which the rest of my colleagues are using for Julia development)

* For debugging, I tend to try things out first in v0.5 (even though many things don't work the same as v0.4.5), because of the much better backtrace information, and also because I've been trying out Gallium.jl, but it still doesn't work tracing through my code, although I'm confident that the remaining issues are soon dealt fixed. It still seems rather primitive though compared to the debugging environment I used to have for C/C++ with MS Visual Studio, or using XCode.  Because of that, I end up decorating a lot of my code with print statements, using a macro that can either be compiled completely away, or based in bits I set on a per module basis, that prints out a string. I found the @printf macros and the $ string interpolation rather cumbersome, so I made my own string utility package https://github.com/ScottPJones/StringUtils.jl, that combines C and Python style formatting with Swift style string interpolation (building off of Dahua Lin's, Tom Breloff's (& others') work in https://github.com/JuliaLang/Formatting.jl) Being able to set defaults for how different types are formatted (thanks, Tom!) saves a LOT of time typing when adding a bunch of debugging output statements

* The code I write is mostly low level, performance critical stuff (sparse data support, serialization/deserialization, string handling, database access), so I use `@timev`, `@profile` and https://github.com/johnmyleswhite/Benchmarks.jl, as well as all of the other macros, i.e. `@which`, `@time`, `@profile`, and `@code_*` that Jacob already mentioned, to make sure I avoid things like type instability, excessive allocations, unnecessary conversions, etc.

* Make sure you write unit tests and use code coverage tools.

* Document everything!  We always put things in modules, and add a docstring to describe each module at the top of each file. We even make sure to document "private" functions (i.e. not exported), it makes life much easier when you can simply type `?<name>` or `?<module>.<name>` to get some helpful info while in the REPL.

* Some very useful packages: JSON.jl (reading and writing configuration data), CSV.jl (loading lots of tabular data - again, thanks Jacob!), SQLite.jl (quick storage, easy to access from different languages), ODBC.jl (for more complex database access needs, esp. useful to me now that decimal floating point values and different Unicode encodings are now handled correctly), DecFP.jl (ability to deal with decimal floating point arithmetic, which can be important when dealing with certain types of financial data, not stuff quants look at, but simple stuff like doing correct calculations with currency amounts, such as getting the sales tax amount correct, simply try `round(.70*.05,2)` to see what I mean)

* I've also started using Tom Breloff's https://github.com/tbreloff/Plots.jl (first used it to help my son with his science fair project, realized it's useful to me to visualize some of my benchmarking results, instead of eyeballing screens and screens of text output).

* I also found https://github.com/weijianzhang/MatrixDepot.jl useful, to get a number of different types of matrices to help test & benchmark some sparse matrix construction code I had implemented (because of performance issues I'd run into with sparse matrices in Base)

Finally, there's a theme to many of my previous points, i.e. build on top of what a lot of other smart people have created.

A great resource to help find those is Svaksha's http://svaksha.github.io/Julia.jl/

Hope that helps!

Scott

P.S. If you are concerned about the efficiency of the code you write, in addition to how efficiently you are developing your code, I'd recommend getting Avik Sengupta's new book: https://www.packtpub.com/application-development/julia-high-performance

[Henri Girard]

I am new to Julia before I used sage, my first aim is to do with julia what I do with sage, and it's not easy specially with plotting, I choose pyplot because it's matplotlib and sage uses it too.
The best way to get along with languages is to use git github, very quickly we can access most tutorials and projects, I am looking for RLC circuit but I don't find much except very specialize projects I can't understand. But in two months I have gathered a lot of examples, I will make a notebook or few, to collect all these wonders.
I like julia because it's wonderfull symbol capacity \pi tab for example \hbar tab, that's really what I am looking for, for many years now. I started learning maths/physics about 20 years ago, I had no notion at all of it. But now I master quiet well the domain !
The only other soft I know, using directly symbols is mathsudio, but doesn't work well on recent linux, it's a win stuff. Free but not open source... Which is very limiting sometimes.

[Lyndon White]

My tip: Use your `.juliarc.jl` to add your current work to your path.

So much less annoying than constantly put `push!(LOAD_PATH, directory)` at the top of various files.

For example I have:

function add_to_path(directory)
        if isdir(directory)
                push!(LOAD_PATH, directory)
        end
end

add_to_path("/mnt_volume/phd/prototypes/SenseSplittingWord2Vec/src/")
add_to_path("/mnt/phd/prototypes/SenseSplittingWord2Vec/src/")

So if the folder with my current project can be found, it is added to the path.
If not then, not.

----------------------------

 managing your cluster with `-L` rather than with `--machine-file`.

Which I am less sure is a good idea.
See this StackOverflow question