Calling Julia Statements from Python.

[ah...@linuxism.com]

Hello folks,

I'm building a mathematical homework system in Django(web framework in python), and my mentor wants me to link Julia to be used within it.

Unfortunately, there is no python package that can be used to call
Julia statements from python. The only way to do it is to build my own python package.

I will write one, however, I don't know anything about Julia development files.

Where do I start in order to be able to build a python package that will be used to evaluate Julia statements from it?

--
Ahmed

[Viral Shah]

See jlapi.c in julia/src and also see the recent mailing thread on calling julia from Ruby and R.

-viral

[Stefan Karpinski]

Also, we do already have this ability, it's buried in https://github.com/JuliaLang/julia-ipython. We'll split this into a bunch of more usable pieces next week. Jeff, Steve Johnson and I just spent a week hacking on this stuff with the IPython team and managed to get a pure-Julia IPython kernel talking to the IPython notebook front end with authentication and tab completion. Calling Julia from Python had actually been possible for some time.

[Fernando Perez]

On Thu, Jul 25, 2013 at 9:41 PM, Stefan Karpinski <ste...@karpinski.org> wrote:
> Also, we do already have this ability, it's buried in
> https://github.com/JuliaLang/julia-ipython. We'll split this into a bunch of
> more usable pieces next week. Jeff, Steve Johnson and I just spent a week

It's worth noting that the code above *already works fine* for simply
calling Julia from Python, with zero IPython in the picture:

>>> import julia

>>> j = julia.Julia()

>>> j.run("[1:10]")
array([ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10])

>>> j.run("[i+2 for i=1:5]")
array([3, 4, 5, 6, 7])


So if all the OP needs is to call Julia from Python, a simple 'python
setup.py install' should do it.

Cheers,

f

[John Hall]

I'm a novice at python (more experienced in Matlab) and just starting in Julia (because numpy is not running as fast as Matlab was). I'm running 64bit Windows 7 and I have python installed using anaconda and Julia installed using the Julia 0.2 binary (both work). I've tried to figure this out, but it's still not working for me. Here's what I did:

1) Went to C:\Program Files\Julia\julia-dca8d76c37\Git (julia-dca8d76c37 is the name the binary gave it) and opened up git-cmd.bat
2) cd C:\Program FIles\Julia
3) enter: git clone https://github.com/JuliaLang/julia-ipython
4) After that (and julia-ipython shows up in the right place), open python.exe (also tried using an iPython notebook, running the same code there)
5) enter:
import os
os.chdir("C:\Program Files\Julia\julia-ipython\python")
setup.py install #error on this line, also tried python setup.py install

I get an error on the setup.py line. I'm mostly used the libraries that come with anaconda (numpy, scipy, etc) so I haven't had to do anything that requires setup.py. After this didn't work I installed the latest ipython using the directions from http://ipython.org/install.html for anaconda (so I think before I wasn't using v 1.0, but now I am). Still doesn't work.

Any tips?

[John Hall]

I realized that I made the stupid mistake of trying to run python.setup.py install in python instead of from a cmd or powershell session. So now I can run the
import Julia
command, but when I'm testing the next line of the example
j = julia.Julia()
I get the error: "JuliaMagicError: error starting up the Julia process"
running the commands.getstatusoutput line that leads to this error. I get status=1 and JULIA_HOME=

"'{' is not recognized as an internal or external command,\noperable program or batch file."

Not sure what that means.

[David Smith]

I get this same error.  Was this ever fixed?

[John Hall]

I was never able to get it to work. It seems like they're trying to give iJulia that functionality as well, but I haven't been able to get iJulia to work yet. I'd be interested if you have any progress.

[John Lynch]

I might have a go too then.

I've been battering my head against femtolisp so a bit of Python would be a nice break.

[Stefan Karpinski]

It would be great to turn the Python-Julia bindings buried in IJulia into a legitimate Python package. I'm not sure what's necessary for that, but I suspect that it would get used quite a bit.

[John Lynch]

OK.  I followed the process on Linux Debian Jessie with Python 2.7.5 and it worked.  I will try it in 3.3 next.  Then if that is still working I'll try it on Windows XP in virtualbox.

[John Lynch]

Python 3 provides more of a challenge so I'll have to look at this:

>>> import julia
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "./julia/__init__.py", line 3, in <module>
    from core import Julia
ImportError: No module named 'core'

John, David,

Were you using Python 2 or Python 3?

[John Hall]

Python 2 (specifically I was using 64bit Anaconda on Windows 7, I think the Python version was like 2.7, but I don't think it's that anymore).

[David Smith]

Python 2.7.5 |Anaconda 1.8.0 (x86_64)| (default, Oct 24 2013, 07:02:20) 

[GCC 4.0.1 (Apple Inc. build 5493)] on darwin

[John Lynch]

Thanks.  The Python was probably 2.7.2 or 2.7.3.

I will try this with enthought free from earlier this year (2.7.3).  First I did need to upgrade IPython to 1.1.  I checked that ipython notebook worked.

Then I called julia.bat (this is julia 0.2) and added ijulia with Pkg.add("IJulia") which put it into C:\Documents and Settings\Administrator\.julia

cd C:\Documents and Settings\Administrator\.julia\IJulia\python
python setup.py install
python
import julia
j=julia.Julia()

at which point I get what I suspect is the identical error.  I suspect that Macs with Python 2.7 work (can anyone confirm?) and that Python 3 will be an issue for everyone.  I'll have a look at windows first because it probably has the largest number of people affected.

[John Hall]

There was an error when it was installing. When I run IPython --profile julia it just brings up IPython and then when I tried some code that only would have worked in Julia I get an error.

On Tue, Nov 19, 2013 at 6:52 PM, John Lynch <mjohn...@gmail.com> wrote:

David, John,

Does your IJulia actually work.  When I run IPython --profile julia its not functioning even though it shows "profile" julia.

As you can see the simple test to determine if this is python or julia fails:


.

[John Lynch]

Actually the last one isn't a valid test.

ipython --profile julia doesn't give you ipython in linux either.

The windows equivalent that should work is:

ipython notebook --profile julia

And it doesn't for me.  Does it for you when you test it with:

m=50 ; 5m

[John Lynch]

When you say "There was an error when it was installing."

exactly what do you mean?

[David Smith]

ipython notebook --profile julia works for me.

[John Lynch]

Thanks David.

Can you confirm that it will accept the line:

mm=50 ; 50m

to ensure that its not Python (which would throw an error at 50m instead of 50*m)?

I have created a fresh XP ve and I'm going to install the latest Anaconda 2.7 so that I see it the same way you guys do.


Separately I have identified part of the problem with Python 3.  Its compiling to the wrong names/places so I need to review that.

2.7.5
byte-compiling /usr/local/lib/python2.7/dist-packages/julia/magic.py to magic.pyc
byte-compiling /usr/local/lib/python2.7/dist-packages/julia/core.py to core.pyc
byte-compiling /usr/local/lib/python2.7/dist-packages/julia/__init__.py to __init__.pyc

3.3.3
byte-compiling /usr/local/lib/python3.3/dist-packages/julia/magic.py to magic.cpython-33.pyc
byte-compiling /usr/local/lib/python3.3/dist-packages/julia/core.py to core.cpython-33.pyc
byte-compiling /usr/local/lib/python3.3/dist-packages/julia/__init__.py to __init__.cpython-33.pyc

[David Smith]

That line works for me (apart from the typo).  I also ran versioninfo() inside the notebook environment.  This was the return:

Julia Version 0.2.0+41
Commit ca2f144* (2013-11-19 04:43 UTC)
Platform Info:
  System: Darwin (x86_64-apple-darwin11.4.2)
  WORD_SIZE: 64
  BLAS: libopenblas (USE64BITINT DYNAMIC_ARCH NO_AFFINITY)
  LAPACK: libopenblas
  LIBM: libopenlibm

[John Hall]

I had tested a code similar to what is mentioned above when running ipython notebook --profile julia. That code doesn't work either.

@John Lynch, when I ran Pkg.status(), IJulia listed under Required packages as 0.0.0. I removed the IJulia package and added it again. The original output is below. It says to fixup Nettle, which I did, then Nettle told me to install RPMmd, then it told me to run fixup on that. But fixup didn't work on RPMmd.

INFO: Installing BinDeps v0.2.9
INFO: Installing IJulia v0.0.0
INFO: Installing JSON v0.2.3
INFO: Installing Nettle v0.0.0
INFO: Installing REPLCompletions v0.0.0
INFO: Installing URIParser v0.0.0
INFO: Installing ZMQ v0.1.6
INFO: Running build script for package IJulia
Found IPython version 1.1.0 ... ok.
Creating julia profile in IPython...
(Existing KernelManager.kernel_cmd setting in ipython_config.py is untouched.)
(Existing IPythonWidget.execute_on_complete_input setting in ipython_qtconsole_c
onfig.py is untouched.)
(Existing NotebookApp.port setting in ipython_notebook_config.py is untouched.)
(Existing Julia IPython png logo file untouched.)
(Existing Julia IPython svg logo file untouched.)
(Existing tooltip.js file untouched.)
(Existing custom.js file untouched.)
INFO: Running build script for package Nettle
WARNING: An exception occured while building binary dependencies.
You may have to take manual steps to complete the installation, see the error me
ssage below.
To reattempt the installation, run Pkg.fixup("Nettle").

 in build at pkg.jl:259WARNING:
backtraces on your platform are often misleading or partially incorrect

ERROR: syntax: extra token "uri" after end of expression
 in include at boot.jl:238
 in include_from_node1 at loading.jl:96
 in include at boot.jl:238
 in include_from_node1 at loading.jl:96
 in reload_path at loading.jl:121
at C:\Users\John\AppData\Roaming\julia\packages\URIParser\src\parser.jl:297
at C:\Users\John\AppData\Roaming\julia\packages\URIParser\src\URIParser.jl:10
at C:\Users\John\AppData\Roaming\julia\packages\BinDeps\src\dependencies.jl:132
at C:\Users\John\AppData\Roaming\julia\packages\BinDeps\src\BinDeps.jl:474
at C:\Users\John\AppData\Roaming\julia\packages\Nettle\deps\build.jl:1

[John Lynch]

I can confirm that IJulia (ipython notebook --profile=julia) is working with the latest anaconda 2.7.5 on windows.  It fails with the julia.core.JuliaMagicError: error starting up the Julia process.

So no need to test that David.  I need to check an unpackaged python/ipython/notebook install as well though.

Clean install of Python 2.6, setuptools, pyreadline, ipython[all]
Follow procedure above for julia.

OK.  Exactly the same issue.

[Fernando Perez]

On Tue, Nov 19, 2013 at 12:47 PM, Stefan Karpinski <ste...@karpinski.org> wrote:

It would be great to turn the Python-Julia bindings buried in IJulia into a legitimate Python package. I'm not sure what's necessary for that, but I suspect that it would get used quite a bit.

It already is, it ships with a proper setup.py file that uses all the standard python machinery, just run

./setup.py install --whatever-options

and it installs like any other Python package.

We could pull it into a *separate* repo if you guys want, that's completely trivial to do.

Cheers

f

--
Fernando Perez (@fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail

[Stefan Karpinski]

What would it take to make it installable via pip and/or easy_install?

[Viral Shah]

On 20-Nov-2013, at 12:32 pm, Stefan Karpinski <ste...@karpinski.org> wrote:

> What would it take to make it installable via pip and/or easy_install?

+1 to make it installable via pip and easy_install.

-viral

[Steven G. Johnson]

It doesn't seem like pip packaging would be very hard.  The main prerequisite is a good Python programmer (with some experience interacting with low-level C libraries) who wants to call Julia from Python on a regular basis, and who is willing to be the primary maintainer of that code.   It needs not only packaging, but also a fair amount of code cleanup, testing, robustification, and syntax nice-ification on the Python side.

The basic problem now is that none of the "maintainers" really use it.  For my own part, I'm only currently interested in calling Python from Julia, not the other way around.

[John Lynch]

I waited a week to see if there were volunteers.  Has anyone stepped up to do this?

I'll put my hand up to start around Christmas if no one more suitable steps forward. 

My qualifications:
- writing code and learning languages is a hobby of mine but I'm committed to some C work for about a month.
- a great deal of C/C++ primarily in dll form
- user level knowledge of Python including Pandas, Cython, and Numba
- user level knowledge of Julia and great interest in how the language is built
- started as an engineer writing tens of thousands of lines of assembler including single board level bioses

But I've learned from Femtolisp and my attempt to understand the Julia parser that there are limits that stop me in my tracks!  Completely.  Thus I may need some advice from the original authors.

I think that this task is simpler, is worth doing, and is probably within my capabilities.  The existing code is much simpler and clearer than the Julia parser.  But if anyone else more capable wants the task please put your hand up.

[Pablo W.]

I am very interested to follow that project so I can certainly help a bit, at least by testing it.

There is one easy way to make a bilingual package that would be installable with pip and with Julia. (I got this idea from the in-development package https://github.com/EconForge/Smolyak.git)
It consists in moving the content of the python directory into the root directory. That way the package is compliant with the expected Julia layout and the expected python one since it has a setup.py file in the root. It can then be pip-installed directly from github pushed to the pip repository. The only real work would then be to merge the readme file and the documentation if any.

That would solve the immediate problem. Then there is the question whether the development of IJulia and the Julia wrapper have enough in common to justify this move.

Pablo

[Tobi]

I think that the development of a python-julia package should go hand in hand with making the julia C API stable/public.
Since the functionality seems to be already there, the most important thing seems to be writing documentation for the C API.
Further some more symbols have to be exposed in libjulia.

[David Smith]

I fully support this and can test and make small contributions, but I don't have the skills to spearhead it.

[Tobi]

I can try to write down some things although I have just scratched the surface of the julia API. This will likely be an iterative process as one might have to extend the C API when trying to write some real example code that uses the julia API.

This discussion should probably take place at the respective julia issue: https://github.com/JuliaLang/julia/issues/3111

[Jake Bolewski]

I've started experimenting with a better python->julia interface here: https://github.com/jakebolewski/pyjulia.  One immediate improvement was to use python's meta-import hooks to make importing julia functions feel more natural.

With these changes you can do stuff like:

from julia.FFTW import fft
from julia import ones
assert(fft(ones(100)[0]==100)

# or to import all functions exported from base
from julia.Base import *

# or import an installed package
from julia import Color

This works pretty well but it kind of constrains you to having one global julia environment, I haven't found a satisfactory way around that (and other similar libs.  such as oct2py and rpy2 seem to do the same thing).

As Steven said, python -> julia integration will be mostly making a nicer interface to returned PyCall objects.  PyCall does all of the heavy lifting, so you shouldn't have to work with julia's c-api very much.

Best,

Jake 

[Steven G. Johnson]

John, it would be great if you could work on this.  I'd be happy to advise as needed.

[Tobi]

I am working on some documentation for the C API here:
https://github.com/tknopp/julia/blob/docuCAPI/doc/manual/embedding.rst
It is still at a very early stage but maybe this can be a starting point.

[John Lynch]

Thanks Tobi,
It'll be a month before I can get onto this so I look forward to your continued progress.

[Steven G. Johnson]

Jake, that syntax looks great, and your pyjulia repo seems like a good starting point for future work.  Having a single global Julia environment doesn't seem like a problem to me.  (After all, you can only have a single global Python environment too.)

[Fernando Perez]

On Fri, Nov 29, 2013 at 5:08 AM, Steven G. Johnson <steve...@gmail.com> wrote:

Jake, that syntax looks great, and your pyjulia repo seems like a good starting point for future work.  Having a single global Julia environment doesn't seem like a problem to me.  (After all, you can only have a single global Python environment too.)

+1

One comment: from the looks of it, this repo started from a copy of the Python code in the old IJulia repo (which is totally OK). I'd really like to ask that you don't throw away the magic function support in the process:

https://github.com/JuliaLang/IJulia.jl/blob/master/python/julia/magic.py

The magics don't introduce an IPython dependency at all in the core, but they provide a very easy and convenient syntax for python users inside IPython (with both line and cell magics).  That code was already there in the repo you (Jake) started from, I don't see a good reason to drop it (I wrote it in good part because I actually *use* that code in practice).

Cheers,

f