Julia, Python and Cython

11,509 views
Skip to first unread message

Dag Sverre Seljebotn

unread,
Apr 21, 2012, 7:29:06 AM4/21/12
to juli...@googlegroups.com, Robert Bradshaw, mark florisson, Travis E. Oliphant, Fernando Perez, Peter Wang
Hi Julia-dev list,

I'm one of the core developers of Cython, a tool for translating Cython
code (a blend of Python and C) into C code using the CPython API,
loadable into the standard CPython interpreter. It's become somewhat of
a de facto standard in the scientific Python community when Python
itself isn't enough.

I recently learned about Julia -- and so far I love it! Congratulations!

To some of the Cython developers and users, Cython has been something of
a stop-gap solutions. It gets the job done, but it's not pretty. So
while I have invested lots of time in Cython and have a heavy emotional
investment as well, I still wish for Cython to be swept away -- or the
need for it to disappear.

What I hope can be done here is to avoid walled gardens. We already have
R and Python as somewhat separate open source gardens (although Python
folks will occasionally call into R) and it's very destructive. Open
source computation doesn't have more cycles than we need!

I believe Julia and Python is a match made in heaven:

Python has:

- an enourmous amount of libraries

- a pretty large userbase in scientific computing and engineering

- some enthusiastic developers/companies/foundations wishing to drive
the platform further for scientific computing

But, there's also

- a crappy interpreter speed-wise (no JIT), and the best bet for a
better one, PyPy, has its own problems

- a CPython development community who simply doesn't "get" scientific
computing (not their problem at all, but it can cause problems for us).

And Julia appears to have exactly a) nice language geared for
performance, b) the JIT technology that we lack, c) developers who "get it".

It's a common theme when we scientific Python users talk that we don't
really use Python for the *language*. We use it for the community and
the libraries. This is why I'm afraid to jump to Julia outright (and
won't do it) -- it's way too easy to underestimate the pain of building
the userbase and the thousands of libraries for every little task. Just
consider the amount of pain it is for the Python community to make the
transition from Python 2 to Python 3!

So I believe both Julia and "SciPy" (as a term for the scientific Python
community) would gain hugely be a close cooperation. And I don't think
this is an arbitrary match at all! -- I believe Python to be
best-of-breed scientific ecosystems (in competition with R), and it
looks like Julia may become the best-of-breed scientific language and
interpreter.

Here's an example roadmap. I'm not saying anything about what *you*
should do, it's just food for thought. Perhaps this is more about what
Python users or Cython developers should do. I just want to get the
conversation started. (Also there's a discussion going on on the
Cython-dev list *right now* that's sort of relevant, more below.)

1) Make sure Julia is easily callable from Python.

Scientific Python users are quite used to diving into other languages
(earlier C and C++, now more and more Cython) to get speedups. Julia
should be one of those choices, and it should be well enough done that
Julia gets promoted by core scientific Python teachers (instead of
Cython as is often the case today).

2) Make sure Python can be transparently called from Julia. Even ship
Python with Julia! You don't need to reimplement all the libraries under
the sun in Julia -- it may be faster, it may be purer, but using
existing libraries gets the job done now, and the important ones can
always be ported to Julia later for speed.

This is not a one-way transaction -- once there's a single non-trivial
but useful piece of code implemented in Julia only (and under a non-GPL
open source license), you can bet that scientific Python distributions
will start to bundle it.

Technology:

CPython has a nice C API. If Julia can both be called from and call into
C (fast!), an "interpreter-level" binding will not be a problem.

On the Cython list we're currently discussion a proposal to annotate
Python objects with C-level function signatures, so that when native
functions are "boxed" and passed throug Python, you can unbox the
function pointer instead of boxing and unboxing the arguments.

This could be used to make sure Julia routines are callable *without*
argument boxing and unboxing from Cython, and that scipy routines, such
as ODE solvers etc. (implemented in C or Fortran) can likewise call back
into callbacks written in Julia without argument boxing/unboxing.

Spec draft:

http://wiki.cython.org/enhancements/cep1000

Best point of entry for ongoing discussion:

http://article.gmane.org/gmane.comp.python.cython.devel/13557

Finally, for some light background reading, here's some comments of the
clashes scientific Python has had with PyPy, and the dangers to be avoided:

http://technicaldiscovery.blogspot.com/2011/10/thoughts-on-porting-numpy-to-pypy.html

http://blog.streamitive.com/2011/10/19/more-thoughts-on-arrays-in-pypy/

Any and all comments appreciated.

Dag Sverre Seljebotn

Aron Ahmadia

unread,
Apr 21, 2012, 8:11:52 AM4/21/12
to juli...@googlegroups.com, Robert Bradshaw, mark florisson, Travis E. Oliphant, Fernando Perez, Peter Wang, Jed Brown, Andy Terrel, Dag Sverre Seljebotn
Hi Julia and Python developers,

For those of you who don't know me, I'm a research scientist in supercomputing, and part of my day job is educating scientists about how to use high performance computing resources, from supercomputer to high performance computing library to the programming language they need.  I have spent some time in the last two years working on a project incorporating  a set of hyperbolic differential equation solvers with PETSc, a very large (300kLOC of C) tool for scientific computing, and a big part of the success of the project came from using Python, numpy, and petsc4py (powered by Cython).  Our project is new, but the underlying software is very mature.  It would have been significantly harder in today's Julia due to a lack of automated wrapper generation and access to PETSc through petsc4py.

I've been thinking about this a lot as well.  Dag, thanks for initiating this and providing several useful links.  I am going to shamelessly cc a few people who I think will be interested in watching this discussion (and are capable of muting this in GMail if they're annoyed by it).  

I think that Dag's first recommendation,  allowing Julia to be easily called from Python, could be very useful to some portion of my community.  If the LLVM performance is close enough to C (and perhaps more importantly, can be ported effectively to SIMD and GPU accelerators), then this provides a viable, high-level, hardware-context-portable alternative to C, C++, and Fortran for writing numerical kernels, with JIT for free.  In the context of supercomputing, this is predicated for the BlueGene users on IBM and the US Department of Energy showing a little more commitment to LLVM, as JIT support in LLVM for PPC64 is stale and broken (http://lists.cs.uiuc.edu/pipermail/llvmdev/2011-March/038935.html).  This would also require a deep Julia-LLVM-CUDA integration for emitting GPU kernels (completely reversing my advice in another julia thread).

Cheers,
Aron

Stefan Karpinski

unread,
Apr 21, 2012, 12:42:55 PM4/21/12
to juli...@googlegroups.com, Robert Bradshaw, mark florisson, Travis E. Oliphant, Fernando Perez, Peter Wang
I definitely agree that cooperation between different scientific computing tools is both good and necessary. We're already leveraging as many high-quality C and Fortran libraries as we can. Calling Julia from Python and vice versa is a much tougher nut to crack though, from a purely technical standpoint.

We want to be able to compile Julia code into C-callable shared libraries, but that's going to take a fair amount of work; this would make it as easy for Python to call Julia code as it is to call C code. I'm not sure how much boilerplate and magic incantation is required for calling shared libraries from Python, but from what I recall it's pretty minimal. Whatever incantations there are can certainly be easily assisted with from the Julia side when generating libraries.

I'm not sure what's necessary to call Python from Julia. I've never embedded a python interpreter in a C process, but it should be basically the same exact thing: just write ccall code that does whatever the embedding C program would do. Making sure data structures are compatible is a trickier propposition, but fortunately the lingua franca of basic C data types (i.e. the stuff that Fortran 77 can do) gets you pretty damned far — that's actually all our ccall interface even supports at this point. Passing native Python data structures to and from Julia is, I suspect, never going to work, but passing NumPy array memory around might very well, precisely because the layout is C/Fortran compatible.

A great starting point would be for someone to take a crack at embedding a Python interpreter in a Julia process using ccall (I might do this because it looks fun and easy :-). I think that calling Python from Julia is the more fruitful direction anyway because there's obviously far more Python code out there that could be useful to call from Julia than there is Julia code that might be useful to call from Python. It's also the easier direction to get working immediately.

Stefan Karpinski

unread,
Apr 21, 2012, 1:05:03 PM4/21/12
to juli...@googlegroups.com, Robert Bradshaw, mark florisson, Travis E. Oliphant, Fernando Perez, Peter Wang
On Sat, Apr 21, 2012 at 12:42 PM, Stefan Karpinski <ste...@karpinski.org> wrote:
 
A great starting point would be for someone to take a crack at embedding a Python interpreter in a Julia process using ccall (I might do this because it looks fun and easy :-). I think that calling Python from Julia is the more fruitful direction anyway because there's obviously far more Python code out there that could be useful to call from Julia than there is Julia code that might be useful to call from Python. It's also the easier direction to get working immediately.

Ok, well this was trivial:

julia> libpython = dlopen("libpython")
Ptr{Void} @0x00007fe7cbb4b080

julia> ccall(dlsym(libpython,:Py_Initialize), Void, ())

julia> ccall(dlsym(libpython,:PyRun_SimpleString), Int32,
             (Ptr{Uint8},), "print 'Hello from Python.'")
Hello from Python.
0

Dag Sverre Seljebotn

unread,
Apr 21, 2012, 2:27:40 PM4/21/12
to juli...@googlegroups.com, Robert Bradshaw, mark florisson, Travis E. Oliphant, Fernando Perez, Peter Wang
On 04/21/2012 06:42 PM, Stefan Karpinski wrote:
> I definitely agree that cooperation between different scientific
> computing tools is both good and necessary. We're already leveraging as
> many high-quality C and Fortran libraries as we can. Calling Julia from
> Python and vice versa is a much tougher nut to crack though, from a
> purely technical standpoint.

I also think it's about more than the tools -- the scientific Python
community is, I believe, what comes closest to prior art for what you
want to achieve. High-level scientific computing but conscious about
performance.

For instance, packaging issues is something SciPy people have probably
spent hundreds of hours of thinking about. Of course, Python packaging
is broken in a hundred ways -- but you should talk to people who knows
about that (e.g., David Cournapeau) just to be warned about how NOT to
do it and what you should avoid.

Perhaps some Julia devs could come and give a talk about Julia at a
SciPy conference this summer? (provided the committee find that on topic
enough...I'd hope so).

> We want to be able to compile Julia code into C-callable shared
> libraries, but that's going to take a fair amount of work; this would
> make it as easy for Python to call Julia code as it is to call C code.
> I'm not sure how much boilerplate and magic incantation is required for
> calling shared libraries from Python, but from what I recall it's pretty
> minimal. Whatever incantations there are can certainly be easily
> assisted with from the Julia side when generating libraries.

Hopefully it could be made totally transparent; at least that's my hope.
The set of language idioms/semantics that can sanely be mapped between
Julia and Python is much larger than what C provides.

The reason I stress this is because I believe your payoff would be
rather large -- it's *much* less work to create a Python bridge than
port over all the web frameworks, networking libraries, machine learning
libraries, and whatnot. Believe me -- a Python bridge is easy :-)

> I'm not sure what's necessary to call Python from Julia. I've never
> embedded a python interpreter in a C process, but it should be basically
> the same exact thing: just write ccall code that does whatever the
> embedding C program would do. Making sure data structures are compatible
> is a trickier propposition, but fortunately the lingua franca of basic C
> data types (i.e. the stuff that Fortran 77 can do) gets you pretty
> damned far — that's actually all our ccall interface even supports at
> this point. Passing native Python data structures to and from Julia is,
> I suspect, never going to work, but passing NumPy array memory around
> might very well, precisely because the layout is C/Fortran compatible.

Well, you can just use proxy objects, a "PyObject" type in Julia. The
proxy aspect would hopefully mostly disappear through macros/JITing, and
you're left with basically what Cython does. Using Python libraries
through Julia could easily be *faster* than through the Python
interpreter (unless your FFI to C is something silly).

The structures of Python objects are *very* easily accessible from C,
and there is no reason you can't make them so from Julia. In Cython,
we're able to turn almost any Python code into C that is compiled by
gcc. We're 100% in the Python environment, doing everything just as the
CPython interpreter would in every way, but it is C-compiled code.

But yes, having a native Python list and a native Julia list map to the
same thing, and be modifiable simultaneously in both languages without
copying, would not work.

You should definitely be able to do

pycall(pyobj, "methodname", 3, 4.4)

reather easily.

However, to make it 100% transparent and allow

methodname(pyobj, 3, 4.4)

you may perhaps need to add a feature or two to the Julia language
(though probably nothing specific to Python).

I regret that I don't have time to dive into this myself the coming
month. But I'm more than willing to answer any questions you may have
about the this and the CPython API, if any Julia developers want to play
with it.

Dag

>
> A great starting point would be for someone to take a crack at embedding
> a Python interpreter

> <http://docs.python.org/py3k/extending/embedding.html> in a Julia

> - some enthusiastic developers/companies/__foundations wishing to

> http://wiki.cython.org/__enhancements/cep1000


> <http://wiki.cython.org/enhancements/cep1000>
>
> Best point of entry for ongoing discussion:
>

> http://article.gmane.org/__gmane.comp.python.cython.__devel/13557


> <http://article.gmane.org/gmane.comp.python.cython.devel/13557>
>
> Finally, for some light background reading, here's some comments of
> the clashes scientific Python has had with PyPy, and the dangers to
> be avoided:
>

> http://technicaldiscovery.__blogspot.com/2011/10/thoughts-__on-porting-numpy-to-pypy.html
> <http://technicaldiscovery.blogspot.com/2011/10/thoughts-on-porting-numpy-to-pypy.html>
>
> http://blog.streamitive.com/__2011/10/19/more-thoughts-on-__arrays-in-pypy/

Dag Sverre Seljebotn

unread,
Apr 21, 2012, 2:41:24 PM4/21/12
to juli...@googlegroups.com

On the other hand, almost nothing in Python libraries relies on using
the exact right type, but just duck typing the right interface -- in 99%
of the cases, if you call a Python library function that expects a list,
it would be happy with a proxy for a Julia list instead.

But that requires proxying Julia objects Python-side -- does Julia have,
or will it in time get, an API for interacting with Julia
types/objects/functions from C? (I guess I should just look in the docs
and code...)

Dag

Jeff Bezanson

unread,
Apr 21, 2012, 3:38:18 PM4/21/12
to juli...@googlegroups.com, Robert Bradshaw, mark florisson, Travis E. Oliphant, Fernando Perez, Peter Wang
Dag, great to hear from you. It's a milestone for me when people in
the SciPy community start noticing what we're doing!

You have an excellent appraisal of the situation. What julia loves to
do is generate native code, so using it together with python for
top-level integration, plus other possible combinations, would be a
great match and I think I can see your vision there. We do want to be
able to generate C-callable shared objects, which of course would be
usable from python, but that is probably not enough.

We should probably do some deeper integration and implement things
like pycall()/PyObject. Many python libraries are in a "sweet spot"
where there is not much performance to gain, e.g. because they do lots
of I/O. I don't know when we will get around to this, but it's very
much worth thinking about. Would we be able to focus exclusively on
python 3 at this point, or should we worry about both?

I can't really think through all the details now, but this will be an
ongoing process.

Thanks for the very helpful message.

-Jeff

On Sat, Apr 21, 2012 at 7:29 AM, Dag Sverre Seljebotn

Fernando Perez

unread,
Apr 21, 2012, 4:00:56 PM4/21/12
to juli...@googlegroups.com, Robert Bradshaw, mark florisson, Travis E. Oliphant, Peter Wang
On Sat, Apr 21, 2012 at 11:27 AM, Dag Sverre Seljebotn
<d.s.se...@astro.uio.no> wrote:
> Perhaps some Julia devs could come and give a talk about Julia at a SciPy
> conference this summer? (provided the committee find that on topic
> enough...I'd hope so).

+1. I think it would be great to have this, I'd already floated that
same idea in idle chat with some colleagues a few days ago. If some
Julia devs want to do so, I'm sure we could talk to the program
committee to ensure that the talk is accepted, as I think it's an
extremely relevant and valuable topic.

Cheers,

f

Fernando Perez

unread,
Apr 21, 2012, 4:02:15 PM4/21/12
to juli...@googlegroups.com, Robert Bradshaw, mark florisson, Travis E. Oliphant, Peter Wang
On Sat, Apr 21, 2012 at 12:38 PM, Jeff Bezanson <jeff.b...@gmail.com> wrote:
> Would we be able to focus exclusively on
> python 3 at this point, or should we worry about both?

No, you must worry about both. With isolated exceptions, by and large
the 'scipy community' is still squarely using 2.x, and so is Sage
(another key player in the 'python-based scientific computing' space).
So if anything, focus on 2.x first, but not on 3.x.

Cheers,

f

Barry Smith

unread,
Apr 21, 2012, 4:06:45 PM4/21/12
to juli...@googlegroups.com, Robert Bradshaw, mark florisson, Travis E. Oliphant, Peter Wang

Prepare for the future, not the past. 3.x all the way.

Dag Sverre Seljebotn

unread,
Apr 21, 2012, 4:12:28 PM4/21/12
to juli...@googlegroups.com
On 04/21/2012 10:06 PM, Barry Smith wrote:
>
> Prepare for the future, not the past. 3.x all the way.

Remember that for success in the long term, one needs success in the
short term. Anybody in the SciPy community who would be interested in
adopting and helping out in implementation will be on 2.x for another
couple of years.

BUT, we don't need to have this discussion: Breakage is mainly in the
Python language. The C API, which is what is relevant for Julia+Python
integration, only changed superficially between Python 2 and Python 3.
If you focus on Python 2 first, getting Python 3 support is a couple of
days work.

The same Cython-generated C files compile with both Python 2 and 3 with
the help of a couple of #ifdefs.

Dag

Stefan Karpinski

unread,
Apr 21, 2012, 8:47:22 PM4/21/12
to juli...@googlegroups.com, Robert Bradshaw, mark florisson, Travis E. Oliphant, Peter Wang
I would love to do this. I suspect Jeff would be down too. Let's make it happen given that the SciPy conference moderators are down with it. Would be a good deadline for having some nice Julia/SciPy integration too.

Konrad Hinsen

unread,
Apr 22, 2012, 12:38:45 PM4/22/12
to juli...@googlegroups.com
--On 21 avril 2012 20:27:40 +0200 Dag Sverre Seljebotn
<d.s.se...@astro.uio.no> wrote:

> Hopefully it could be made totally transparent; at least that's my hope.
> The set of language idioms/semantics that can sanely be mapped between
> Julia and Python is much larger than what C provides.

Indeed. A "python object" proxy type in Julia could implement interfaces to
iterators, indexing, etc. Attribute access (and thus method call) requires
a bit more thought, because Julia doesn't have real equivalents. Access to
fields in Julia types is determined by static typing, not dynamic lookup.

Access from Python functions defined in Julia should be no problem at all.
Access to Julia-defined data types should be easy if Julia has reflection
mechanisms, which I haven't explored yet.

I agree that a Julia-Python bridge would be *very* nice to have!

Konrad.

Jeff Bezanson

unread,
Apr 22, 2012, 2:14:26 PM4/22/12
to juli...@googlegroups.com
> bit more thought, because Julia doesn't have real equivalents. Access to
> fields in Julia types is determined by static typing, not dynamic lookup.

Not quite true; we do dynamic field lookup in some cases as necessary.
But it is true that julia objects are not dictionaries.
pycall(obj, :method, ...) can be implemented, and if we allow
overloading dot then obj.method might even be made to work.

Dag Sverre Seljebotn

unread,
Apr 22, 2012, 5:18:33 PM4/22/12
to juli...@googlegroups.com

method(obj, ...) would be more idiomatic though?

But I don't know whether it is possible (or desirable) to implement
support for that -- guess it would require something like a
"findunknownfunction" function/hook that one could write a PyObject
method for?


Dag

Dag Sverre Seljebotn

unread,
Apr 22, 2012, 5:55:56 PM4/22/12
to juli...@googlegroups.com, Jeff Bezanson, Robert Bradshaw, mark florisson, Travis E. Oliphant, Fernando Perez, Peter Wang
On 04/21/2012 09:38 PM, Jeff Bezanson wrote:
> Dag, great to hear from you. It's a milestone for me when people in
> the SciPy community start noticing what we're doing!
>
> You have an excellent appraisal of the situation. What julia loves to
> do is generate native code, so using it together with python for
> top-level integration, plus other possible combinations, would be a
> great match and I think I can see your vision there. We do want to be
> able to generate C-callable shared objects, which of course would be
> usable from python, but that is probably not enough.
>
> We should probably do some deeper integration and implement things
> like pycall()/PyObject. Many python libraries are in a "sweet spot"
> where there is not much performance to gain, e.g. because they do lots
> of I/O. I don't know when we will get around to this, but it's very
> much worth thinking about. Would we be able to focus exclusively on
> python 3 at this point, or should we worry about both?
>
> I can't really think through all the details now, but this will be an
> ongoing process.
>
> Thanks for the very helpful message.

About the performance, you're of course right that that's often not
needed. PyPI has 20000 packages now (want to control a browser? Write
OpenOffice documents? Parse BiBTEX? Work with JSON or XML? Almost
anything is there.) For most such "utility nice-to-haves" performance
doesn't matter at all.

However, there's another class of more computational libraries where
performance matters: PyZMQ, mpi4py, petsc4py, PyTables, Pandas,
scikits-learn, SciPy, all the dozen wrappers around mathematics
libraries in SAGE, etc..

And, incidentally, these are mostly written in Cython (with SciPy only a
subset, but that may in time grow). Which gives you some opportunities:

- We want to define a stable ABI for Cython, so that through some
introspection you'd get the corresponding C function for a Python
callable that you can call directly (the CEP 1000 discussion I linked to
in my introductory post is a humble beginning on that). For something
like ZeroMQ that gives you one C wrapper function around the raw library.

- Write a Julia backend for Cython. This is not as insane as it sounds
-- 1 1/2 years ago Enthought funded a .NET/IronPython backend for Cython
and used it to port a subset of SciPy to .NET (I was part of that effort
myself). Basically, Cython would generate Julia code that calls the
CPython API or use the C FFI -- and in the case of wrappers, could allow
you to go straight from Julia to the wrapped library. (A robust Julia
backend would be harder than the C++/CLR, but not impossible, and
PyPy+Cython faces some of the same challenges -- I'll write more when/if
you become interested).

Of course, this is a lot of work, but we're talking several months, not
several years. I'm not saying this affects anything in the *present*, I
just want to paint you a picture of what may be possible.

You've got some major strategic decisions ahead of you with how you're
going to grow a library base and a user base for Julia, good luck with
that! Feel free to ask if there's anything (when this thread dies I may
forget to check this list that often, so CC me or the cython-dev list).

Dag

Konrad Hinsen

unread,
Apr 23, 2012, 4:58:31 AM4/23/12
to juli...@googlegroups.com
Dag Sverre Seljebotn writes:

> On 04/22/2012 08:14 PM, Jeff Bezanson wrote:
> >> bit more thought, because Julia doesn't have real equivalents. Access to
> >> fields in Julia types is determined by static typing, not dynamic lookup.
> >
> > Not quite true; we do dynamic field lookup in some cases as necessary.
> > But it is true that julia objects are not dictionaries.
> > pycall(obj, :method, ...) can be implemented, and if we allow
> > overloading dot then obj.method might even be made to work.
>
> method(obj, ...) would be more idiomatic though?

I don't see how one could implement this in Julia without explicitly defining
a Julia function for each Python method. This could be handled by some interfacing
tool of course.

But more importantly, I think this is not a good idea because it
raises false expectations. Julia functions do multiple dispatch based
on the types of all arguments, whereas Python dispatches only on the
object type. pycall(obj, :method, ...) makes this clearer.

Konrad.

Dag Sverre Seljebotn

unread,
Apr 23, 2012, 5:15:02 AM4/23/12
to juli...@googlegroups.com
On 04/23/2012 10:58 AM, Konrad Hinsen wrote:
> Dag Sverre Seljebotn writes:
>
> > On 04/22/2012 08:14 PM, Jeff Bezanson wrote:
> > >> bit more thought, because Julia doesn't have real equivalents. Access to
> > >> fields in Julia types is determined by static typing, not dynamic lookup.
> > >
> > > Not quite true; we do dynamic field lookup in some cases as necessary.
> > > But it is true that julia objects are not dictionaries.
> > > pycall(obj, :method, ...) can be implemented, and if we allow
> > > overloading dot then obj.method might even be made to work.
> >
> > method(obj, ...) would be more idiomatic though?
>
> I don't see how one could implement this in Julia without explicitly defining
> a Julia function for each Python method. This could be handled by some interfacing
> tool of course.

You seem to have an assumption that the Julia languages stays the way it
is and will never be extended. But the Julia language is whatever the
authors make it.

> But more importantly, I think this is not a good idea because it
> raises false expectations. Julia functions do multiple dispatch based
> on the types of all arguments, whereas Python dispatches only on the
> object type. pycall(obj, :method, ...) makes this clearer.

I don't have an opinion on whether it is desirable, I'm way too new to
Julia to say.

I don't agree with your specific concern though -- you could make it so
that you would be totally free to create additional methods in Julia
taking other arguments into account. It just happens that modules
imported from Python (as seen from Julia) only contains functions
overloads on the first argument, and don't specify a type for the rest.

The real issue I see is in mapping Python semantics to Julia is that
Python is so namespace-based. I don't think it's a problem to map
"animal.make_sound(x, y)" to Julia, but I do think it's a problem to map
"os.path.realpath", or "scipy.special.sph_jn" to Julia.

Neither of the following are natural in current Julia, you'd be
painfully aware that you are calling into Python in either case:

pycall(pygetattr(os, :path), :realpath, x)
pycall(os.path, :realpath, x)
os.path.realpath(x)

But I see that namespacing is not out of the question for Julia:

https://github.com/JuliaLang/julia/issues/57

Dag

Konrad Hinsen

unread,
Apr 23, 2012, 9:24:25 AM4/23/12
to juli...@googlegroups.com
Dag Sverre Seljebotn writes:

> > I don't see how one could implement this in Julia without explicitly defining
> > a Julia function for each Python method. This could be handled by some interfacing
> > tool of course.
>
> You seem to have an assumption that the Julia languages stays the way it
> is and will never be extended. But the Julia language is whatever the
> authors make it.

Of course. I assume for the moment that the Julia authors are unlikely
to introduce a language feature (as opposed to an implementation
feature) for the sole purpose of facilitating interaction with Python.

> I don't agree with your specific concern though -- you could make it so
> that you would be totally free to create additional methods in Julia
> taking other arguments into account. It just happens that modules
> imported from Python (as seen from Julia) only contains functions
> overloads on the first argument, and don't specify a type for the rest.

Right now, Julia doesn't have modules or namespaces, and thus no imports.
All function names are global.

Introducing namespaces alone wouldn't make much of a difference to my
argument. Even if functions can reside in namespaces, as long as code
from anywhere can add method implementations to functions anywhere
else, the Julia programmer's expectation would be that all functions
do multiple dispatch. A special case for Python functions would make
sense only if modules containing functions were separate entities
protected from each other. That's perhaps a good idea, but it's also
wild speculation at this time.

> The real issue I see is in mapping Python semantics to Julia is that
> Python is so namespace-based. I don't think it's a problem to map
> "animal.make_sound(x, y)" to Julia, but I do think it's a problem to map
> "os.path.realpath", or "scipy.special.sph_jn" to Julia.

Right. The last item of the "Zen of Python" has not yet made its way into
the Julia universe ;-)

> Neither of the following are natural in current Julia, you'd be
> painfully aware that you are calling into Python in either case:
>
> pycall(pygetattr(os, :path), :realpath, x)

This could be improved even without namespace support in Julia. Python
modules are just Python objects, so all we need is a nice way to write
Python attribute access. I can't think of anything nice that could be done
without any change at all to the current Julia language, but some
lightweight options are

pycall(:os.path.realpath, x) # requires allowing dot in symbol names

os.path.realpath(x) # requires . to be defined as a special field
# access operator that each type can implement
# arbitrarily, with current field selection
# as the default case.

Konrad.

Patrick O'Leary

unread,
Apr 23, 2012, 9:41:43 AM4/23/12
to juli...@googlegroups.com
On Monday, April 23, 2012 8:24:25 AM UTC-5, Konrad Hinsen wrote:
  os.path.realpath(x)           # requires . to be defined as a special field
                                # access operator that each type can implement
                                # arbitrarily, with current field selection
                                # as the default case.

Making getfield() a generic method is still an open question:
https://groups.google.com/d/topic/julia-dev/tNuNU-VQdFQ/discussion

Tim Holy

unread,
Apr 23, 2012, 10:11:15 AM4/23/12
to juli...@googlegroups.com
On Monday, April 23, 2012 03:24:25 PM Konrad Hinsen wrote:
> pycall(:os.path.realpath, x) # requires allowing dot in symbol names

symbol("os.path.realpath") seems to work. Also, this does something
interesting,
julia> :(os.path.realpath)
os.:path.:realpath

but I'm not sure if it's good or not...

--Tim

Dag Sverre Seljebotn

unread,
Apr 23, 2012, 10:34:56 AM4/23/12
to juli...@googlegroups.com
On 04/23/2012 03:24 PM, Konrad Hinsen wrote:
> Dag Sverre Seljebotn writes:
>
> > > I don't see how one could implement this in Julia without explicitly defining
> > > a Julia function for each Python method. This could be handled by some interfacing
> > > tool of course.
> >
> > You seem to have an assumption that the Julia languages stays the way it
> > is and will never be extended. But the Julia language is whatever the
> > authors make it.
>
> Of course. I assume for the moment that the Julia authors are unlikely
> to introduce a language feature (as opposed to an implementation
> feature) for the sole purpose of facilitating interaction with Python.

That's such a strange statement to me, it feels like we're on entirely
different planets! Interacting with other languages (not specifically
"Python", but X) is not "sole", it's the alpha and omega of succeeding
as a language these days.

Look, I didn't really come here to discuss the specifics of a
Julia<->Python bridge (though I find that interesting). What I wondered
about was the overall strategy of Julia. How will Julia be able to
thrive, unlike so many other LISP-inspired languages that came and went?
Will it:

a) focus on a narrow niche (a "glorified MATLAB") with few libraries

b) full steam ahead on implementing packages in Julia (problem being
that "the last 5%" is different for every user -- e.g., Python as 20000
packages now)

c) focus on quick, easy, transparent usage of libraries from Python
(and/or Ruby, Perl, R...) to quickly make up for being a new-comer on
the language smorgasbord

Anyway, I think Jeff and Stefan understood my point and have answered
this to my satisfaction.

Dag

Konrad Hinsen

unread,
Apr 23, 2012, 10:49:48 AM4/23/12
to juli...@googlegroups.com
Dag Sverre Seljebotn writes:

> > Of course. I assume for the moment that the Julia authors are unlikely
> > to introduce a language feature (as opposed to an implementation
> > feature) for the sole purpose of facilitating interaction with Python.
>
> That's such a strange statement to me, it feels like we're on entirely
> different planets! Interacting with other languages (not specifically
> "Python", but X) is not "sole", it's the alpha and omega of succeeding
> as a language these days.

I actually agree with that. My point in assuming (for the time being)
that no Python-specific changes will be made to Julia is in exploring
first how far one can get without such changes. I use "assume" as in
a mathematical proof ("let's assume we know A, then we can show B").

Konrad.

Dag Sverre Seljebotn

unread,
Apr 23, 2012, 10:56:03 AM4/23/12
to juli...@googlegroups.com

Konrad Hinsen <google...@khinsen.fastmail.net> wrote:


Yes, on rereading what you wrote I see that I read a lot more between the lines than on them. My fault, I apologise.

Dag


>Konrad.

--
Sent from my Android phone with K-9 Mail. Please excuse my brevity.

Stefan Karpinski

unread,
Apr 23, 2012, 12:47:11 PM4/23/12
to juli...@googlegroups.com
The fact that being able to overload the dot operator might help in transparently interoperating with Python is a valuable data point — one that hadn't come up before. It pushes a bit in the direction of allowing overloading of dot syntax, along with namespace access potentially being implemented that way, and allowing data frames to have their columns accessed via dot syntax.

Adding language features *just* to interop with Python seems myopic, but considering interop with Python when considering what features to have in a greater context (including potential interop with Ruby, Perl, etc.), makes sense. We've considered interop with static languages like C and Fortran extensively and even discussed interop with C++, but decided that the combination of no ABI standard and insane name mangling makes that a non-starter. Python is an excellent first point to consider for interop with other dynamic languages. Cython, because of it's focus on C interop, performance, and heavy use in scientific computing is an even better starting point.
c++ interop.jpg

Bill Hart

unread,
Apr 23, 2012, 1:43:00 PM4/23/12
to juli...@googlegroups.com
Is there nothing in the clang library which allows platform independent C++ name mangling?

Bill.

Bill Hart

unread,
Apr 23, 2012, 1:44:47 PM4/23/12
to juli...@googlegroups.com
Arghh, that was the dumbest question ever. Please ignore it.

Bill.

Stefan Karpinski

unread,
Apr 23, 2012, 3:35:30 PM4/23/12
to juli...@googlegroups.com
Just to complete the cycle, I'll post the fact that this thread is now on the second page of Hacker News: http://news.ycombinator.com/item?id=3876579.

Matthew Emmett

unread,
Apr 23, 2012, 8:21:31 PM4/23/12
to juli...@googlegroups.com, Robert Bradshaw, mark florisson, Travis E. Oliphant, Fernando Perez, Peter Wang


On Saturday, April 21, 2012 12:42:55 PM UTC-4, Stefan Karpinski wrote:
We want to be able to compile Julia code into C-callable shared libraries, but that's going to take a fair amount of work; this would make it as easy for Python to call Julia code as it is to call C code. I'm not sure how much boilerplate and magic incantation is required for calling shared libraries from Python, but from what I recall it's pretty minimal. Whatever incantations there are can certainly be easily assisted with from the Julia side when generating libraries.

The ctypes Python module (that is included in the base libraries) makes calling C callable routines routines from Python fairly painless.  Using ctypes you can load libraries dynamically and optionally specify prototypes for your C callable functions (this makes calling C even easier, eg, basic types like int are automatically converted to C types). Furthermore, NumPy's ndpointer makes it very easy to pass references to NumPy arrays.  Anyway, something to consider.

Matt

PS, great work on Julia!
 

Konrad Hinsen

unread,
Apr 24, 2012, 2:54:14 AM4/24/12
to juli...@googlegroups.com
Stefan Karpinski writes:

> Python is an excellent first point to consider for interop with
> other dynamic languages. Cython, because of it's focus on C
> interop, performance, and heavy use in scientific computing is an
> even better starting point.

Cython uses the "Python platform", so it's best to consider both in
parallel in my opinion.

Something that could turn out to be tricky is interfacing Julia arrays
and NumPy arrays. That's pretty much a "must have" feature for
scientific applications, so it's worth investing some effort
there. While the basic storage layout is the same on both sides, the
mapping of element types is likely to be platform dependent, and
NumPy's record arrays probably require a Julia type definition for
each array's elements. Nothing looks impossible, but it's going to be
some work.

Konrad.

Aron Ahmadia

unread,
Apr 24, 2012, 3:27:34 AM4/24/12
to juli...@googlegroups.com
Something that could turn out to be tricky is interfacing Julia arrays
and NumPy arrays. That's pretty much a "must have" feature for
scientific applications, so it's worth investing some effort
there. While the basic storage layout is the same on both sides, the
mapping of element types is likely to be platform dependent, and
NumPy's record arrays probably require a Julia type definition for
each array's elements. Nothing looks impossible, but it's going to be
some work.
 
+1 to this.  One of the big killer features for Cython is its great numpy interop (thanks Dag and GSoC!).  In fact, this feature is what most of our scientific Python software relies on in some way.  When deciding how to interface at the API level, it's a little dangerous to look purely at the data storage layout in numpy, as right now the two main layouts are row-major and column-major, but it's pretty well-known in high performance computing that the optimal layout is block-oriented, and implementations of this format are starting to come into existence.  If we want to be forward-looking, it's more important to define a good API for making copies that is orthogonal to the underlying layout of the data.  In the simple case, you support streaming values row-wise and column-wise, and this is where to start.  You should try to leave the door open, though, for more sophisticated data putters/getters that operate on panels/blocks.

A

Jeff Bezanson

unread,
Apr 24, 2012, 4:22:31 AM4/24/12
to juli...@googlegroups.com
That sounds like a drag. We'd have to reimplement numpy's interface,
in addition to everything else we need/want to do. I'd put my hopes on
using lowest-common-denominator fortran storage, and standard data
formats (like HDF5 maybe, but I don't know enough about it).

Most of all, we (SciPy and julia) should look at the big picture of
how data is accessed and used. There should be some kind of
cloud-oriented platform where data can be pushed and pulled from
multiple environments, and allow parallel computations to move to the
data. And a web interface involving some kind of standard object model
for visualization, where people could work on all this stuff together
possibly from multiple languages at the same time. Kind of like
OpenDoc for scientific computing, but hopefully not doomed to failure.

Konrad Hinsen

unread,
Apr 24, 2012, 4:44:25 AM4/24/12
to juli...@googlegroups.com
Jeff Bezanson writes:

> That sounds like a drag. We'd have to reimplement numpy's interface,
> in addition to everything else we need/want to do. I'd put my hopes on
> using lowest-common-denominator fortran storage, and standard data
> formats (like HDF5 maybe, but I don't know enough about it).

In-memory data and out-of-memory data are different issues for me.
For the former, you have to be able to work with the data as the
supplier (whoever that is) has prepared it, for performance
reasons. For the latter, portability among platforms and languages is
the prime issue (for me at least). I'd rather discuss them separately.

For in-memory arrays, we can certainly start with the most common simple
cases, which comes down to Fortran-style storage. This still requires
correct element type mapping, but that's true for any inter-language
interface, even with C and Fortran. Type mapping isn't really difficult,
it's just a pain to test on all platforms.

In the long run, I am almost sure there will be pressure to support
more complex arrays, such as NumPy's record arrays, which are used
much like R's data frames and are just too convenient to give up once
you are used to them. The "impedance mismatch" is that the element
type information is dynamic in Python but static in Julia.


> Most of all, we (SciPy and julia) should look at the big picture of
> how data is accessed and used. There should be some kind of
> cloud-oriented platform where data can be pushed and pulled from
> multiple environments, and allow parallel computations to move to the

That's certainly an important topic, but one I'd see addressed outside
any specific programming language. OPeNDAP (http://opendap.org/) looks
like a good starting point.

One problem is to support very different storage and access modes with
different priorities. Data stored in the cloud is used very
differently from data stored on an IBM BlueGene where fast coordinated
parallel access from thousands of nodes is the priority.

Konrad.

Dag Sverre Seljebotn

unread,
Apr 24, 2012, 5:00:41 AM4/24/12
to juli...@googlegroups.com
On 04/24/2012 10:44 AM, Konrad Hinsen wrote:
> Jeff Bezanson writes:
>
> > That sounds like a drag. We'd have to reimplement numpy's interface,
> > in addition to everything else we need/want to do. I'd put my hopes on
> > using lowest-common-denominator fortran storage, and standard data
> > formats (like HDF5 maybe, but I don't know enough about it).
>
> In-memory data and out-of-memory data are different issues for me.
> For the former, you have to be able to work with the data as the
> supplier (whoever that is) has prepared it, for performance
> reasons. For the latter, portability among platforms and languages is
> the prime issue (for me at least). I'd rather discuss them separately.

You have the case where the data is in memory, it just happens to be so
in different machines all over the world.

These days the vogue is all about pushing computations to the data, not
the data to the computations.

Dag

Jeff Bezanson

unread,
Apr 24, 2012, 5:08:38 AM4/24/12
to juli...@googlegroups.com
> In the long run, I am almost sure there will be pressure to support
> more complex arrays, such as NumPy's record arrays, which are used
> much like R's data frames and are just too convenient to give up once
> you are used to them. The "impedance mismatch" is that the element
> type information is dynamic in Python but static in Julia.
>

But the element types do not have to be mapped to separate julia
composite types. It's just a collection of arrays with some extra
metadata and fancier indexing behavior. And the element type of a
julia array can certainly be determined at run time.

Konrad Hinsen

unread,
Apr 24, 2012, 5:30:34 AM4/24/12
to juli...@googlegroups.com
Jeff Bezanson writes:

> But the element types do not have to be mapped to separate julia
> composite types. It's just a collection of arrays with some extra
> metadata and fancier indexing behavior. And the element type of a
> julia array can certainly be determined at run time.

The layout of NumPy record arrays is that of a normal array with each
element being something like a composite type (it's a bit more general
than that). In other words, the fields of each element are stored
contiguously in memory. This can of course be seen as separate arrays
for each field by adjusting the strides, but I suspect that kind of
view leads to inefficient access patterns.

Konrad.

Konrad Hinsen

unread,
Apr 24, 2012, 5:36:31 AM4/24/12
to juli...@googlegroups.com
Dag Sverre Seljebotn writes:

> You have the case where the data is in memory, it just happens to be so
> in different machines all over the world.
>
> These days the vogue is all about pushing computations to the data, not
> the data to the computations.

That's the vogue for cloud computing, but we also have the case of a
database that holds data to be downloaded for client-side
processing. That's the situation that OPeNDAP addresses.

Konrad.

Jeff Bezanson

unread,
Apr 24, 2012, 6:51:56 AM4/24/12
to juli...@googlegroups.com
There are still various ways to implement it. Patrick's strpack is a
great example of course. He chose to generate types to allow field
access with dot, but that isn't necessary. Something similar to his
code generators for pack and unpack could be used to access struct
elements within a byte array. Or one could dispense with the
metaprogramming and access fields with something more
interpreter-like. So there are many approaches, including generating
types at run time.

We are planning to add more flexible struct and array-of-struct layout
support to the core language, which will allow us to generate fast
code for those cases, but will not support all the same formats as
numpy record arrays, at least not for a while. So some user-level
implementation might have to be part of the picture.

I have seen cases where "fields" are stored as separate homogeneous
arrays, but of course what you want there depends on the access
pattern.

Is OPeNDAP something worth seriously integrating with?

Konrad Hinsen

unread,
Apr 24, 2012, 8:36:04 AM4/24/12
to juli...@googlegroups.com
Jeff Bezanson writes:

> Something similar to his code generators for pack and unpack could
> be used to access struct elements within a byte array. Or one could

That's what I was thinking off as well.

> We are planning to add more flexible struct and array-of-struct layout
> support to the core language, which will allow us to generate fast
> code for those cases, but will not support all the same formats as

That sounds good!

> Is OPeNDAP something worth seriously integrating with?

It's pretty easy to do (they provide a C library, and the interface is made
to resemble netCDF), and I don't think it requires any action from language
developers. Anyone who wants OPeNDAP can just write a Julia interface layer.

Apparently OPeNDAP is well accepted in fields of science, but not in mine,
so I can't say how important it is to support.

Konrad.

nbecker

unread,
May 2, 2012, 8:46:44 AM5/2/12
to juli...@googlegroups.com, Robert Bradshaw, mark florisson, Travis E. Oliphant, Fernando Perez, Peter Wang
I'd just like to add, that although it might be technically more difficult, I think the preferred use case is python calling julia.  The reason is, that all top-level programs want to be written in python.  All the nice libraries for ui, command line parsing, etc are in python.  As many have observed, python makes a very good language for gluing hybrid systems together.

Jeff Bezanson

unread,
May 2, 2012, 10:52:51 AM5/2/12
to juli...@googlegroups.com
I agree. At the very least we want to be able to compile julia
programs to shared objects that export a native C API, which will
allow them to be used in many contexts.

Steven G. Johnson

unread,
Feb 23, 2013, 2:05:22 PM2/23/13
to juli...@googlegroups.com, Robert Bradshaw, mark florisson, Travis E. Oliphant, Fernando Perez, Peter Wang, d.s.se...@astro.uio.no
I thought I should post a brief update to this thread about recent progress on the issues discussed here last year.

On Saturday, April 21, 2012 7:29:06 AM UTC-4, Dag Sverre Seljebotn wrote:

2) Make sure Python can be transparently called from Julia. Even ship
Python with Julia! You don't need to reimplement all the libraries under
the sun in Julia -- it may be faster, it may be purer, but using
existing libraries gets the job done now, and the important ones can
always be ported to Julia later for speed.

This is available now, via the PyCall package (https://github.com/stevengj/PyCall.jl).  You can now do, for example:

    using PyCall
@pyimport pylab x = linspace(0,2*pi,1000); y = sin(3*x + 4*cos(2*x)); pylab.plot(x, y, @pykw color="red" linewidth=2.0 linestyle="--") pylab.show()

All the data types here are translated automatically, bi-directionally, under the hood. Multidimensional arrays (via NumPy) and
dictionaries can be shared without making a copy. There are also lower-level interfaces, e.g. to improve performance in cases
where you know the return types of the Python functions, and a PyObject wrapper for data types that cannot yet be translated
directly to native Julia types.

Thanks to Julia's ability to natively call C functions and Python's well-documented C API, this could be implemented entirely
in Julia---the ability to mix direct C calls with a high-level language is quite powerful. (I'm quite certain that orders of magnitude
more effort would have been required to add similar functionality to Octave, Matlab, or Scilab, having hacked on all three. In
fact, I don't have to guess: the PIMS project implemented a Python interface for Scilab, and required thousands of lines of C++
code (http://www.ohloh.net/p/pims), and it's not clear that they have as much functionality as PyCall, which took a week and a
thousand lines of Julia. It's just a lot harder to bridge two high-level languages by writing code in a low-level language.)

--SGJ


Tim Holy

unread,
Feb 23, 2013, 2:43:29 PM2/23/13
to juli...@googlegroups.com
Remarkable work, Stephen. I really look forward to using it.

I had intended to try to keep many of my lab's machines following 0.1, but
this will by itself probably be sufficient incentive to break with that!

Best,
--Tim

Kevin Squire

unread,
Feb 23, 2013, 3:29:31 PM2/23/13
to juli...@googlegroups.com
I agree--this is very impressive!  I have a lot of Python code, some of which I want to transition to Julia, and this is going to make that job a lot easier.  Thank you!

Kevin

Steven G. Johnson

unread,
Feb 23, 2013, 9:00:35 PM2/23/13
to juli...@googlegroups.com
Steven G. Johnson wrote:
> This is available now, via the PyCall package
> (https://github.com/stevengj/PyCall.jl).

Now that #2380 is merged, you can use PyCall without patching Julia
(provided you are using an up-to-date git master) via Pkg.add("PyCall").

Dag Sverre Seljebotn

unread,
Feb 25, 2013, 3:38:42 AM2/25/13
to Steven G. Johnson, juli...@googlegroups.com, Robert Bradshaw, mark florisson
That much functionality in less than thousand lines is very impressive
indeed, great job!

Dag Sverre

Neal Becker

unread,
Feb 25, 2013, 11:53:47 AM2/25/13
to juli...@googlegroups.com
Sounds interesting, but wouldn't it be more fun to call julia code from python?

Cameron McBride

unread,
Feb 25, 2013, 12:11:19 PM2/25/13
to juli...@googlegroups.com
On Mon, Feb 25, 2013 at 11:53 AM, Neal Becker <ndbe...@gmail.com> wrote:
> Sounds interesting, but wouldn't it be more fun to call julia code from python?

No. Then you'd have to write python! :)

The genius of this functionality is that existing python libraries can
be used to extend julia (language capabilities and applications) until
more native libraries grow up. (Also, a binding like this might be
easier to port over changes in Julia syntax, potentially even being
more stable. right?)

Cameron

p.s. Aesthetically, I honestly found it somewhat sad that this got up
and running. Any code using this will suffer the same runtime issues
of python, which is a significant reason I'm not using python in the
first place. This might even stunt development of equivalent julia
libraries! But, as Steven points out, the pragmatic side of getting
functional code up running faster (development fast) probably far
outweighs those risks.

Fernando Perez

unread,
Feb 25, 2013, 6:44:10 PM2/25/13
to juli...@googlegroups.com
On Mon, Feb 25, 2013 at 11:53 AM, Neal Becker <ndbe...@gmail.com> wrote:
> Sounds interesting, but wouldn't it be more fun to call julia code from python?

Like this, you mean?

http://nbviewer.ipython.org/5034046/JuliaMagic.ipynb

:)

Jeff and I just whipped this up while skipping talks at the SIAM
CSE'13 conference. Best possible use of conference time.

BTW, I'm having a problem with executing multiline blocks, for some
reason it seems that jl_eval_string only runs the first line. Is that
true? Am I missing something?

Cheers,

f

John Myles White

unread,
Feb 25, 2013, 9:57:45 PM2/25/13
to juli...@googlegroups.com
As I said on Twitter, this is super awesome.

-- John

Rahul Dave

unread,
Feb 25, 2013, 10:19:03 PM2/25/13
to juli...@googlegroups.com
Super! Not a socket send to the julia webserver?

-- 
Rahul Dave
Sent with Sparrow

Fernando Perez

unread,
Feb 25, 2013, 10:23:11 PM2/25/13
to juli...@googlegroups.com
On Mon, Feb 25, 2013 at 10:19 PM, Rahul Dave
<rahuldave.m...@gmail.com> wrote:
> Super! Not a socket send to the julia webserver?

No, this is a pure ipython kernel with a magic that initializes the
Julia interpreter by linking directly to libjulia.so:

https://gist.github.com/fperez/5034046#file-juliamagic-py

It took a bit of hacking from both Jeff and I to make it work, but the
principle is simple enough that we had it pretty much all sorted out
in an escalator ride. This isn't even necessarily the best way to do
it in the long term, it's really just a quick experiment to see what
we can do. I fully expect one of you experts to write a proper
solution soon, I just wanted to see what's possible and wanted to hack
instead of writing my talk for tomorrow :)

On the plus side, I have one more demo for the talk!

Cheers,

f

Stefan Karpinski

unread,
Feb 25, 2013, 10:28:17 PM2/25/13
to Julia Dev
Nice! There's always a certain cachet to offhandedly saying "oh, this is just something we hacked up yesterday" when you're doing the demo too :-)

Rahul Dave

unread,
Feb 25, 2013, 10:29:04 PM2/25/13
to juli...@googlegroups.com
Very sweet! Wonder if this is the right approach, actually, as proper bi-directional var conversion can be done..thanks for the gist!
Rahul

-- 
Rahul Dave
Sent with Sparrow

Fernando Perez

unread,
Feb 25, 2013, 10:42:14 PM2/25/13
to juli...@googlegroups.com
On Mon, Feb 25, 2013 at 10:28 PM, Stefan Karpinski <ste...@karpinski.org> wrote:
> Nice! There's always a certain cachet to offhandedly saying "oh, this is
> just something we hacked up yesterday" when you're doing the demo too :-)

Which I fully intend to do :)

Fernando Perez

unread,
Feb 25, 2013, 10:43:42 PM2/25/13
to juli...@googlegroups.com
On Mon, Feb 25, 2013 at 10:29 PM, Rahul Dave
<rahuldave.m...@gmail.com> wrote:
> Very sweet! Wonder if this is the right approach, actually, as proper
> bi-directional var conversion can be done..thanks for the gist!

If you have a chance to stop by, I'll be at MIT on Friday and we can
look at this code further with Jeff to think of a more robust path
forward. My talk will be at 4, but I'll have some time earlier than
that.

Cheers,

f

Steven G. Johnson

unread,
Feb 26, 2013, 8:14:17 PM2/26/13
to juli...@googlegroups.com
Neal Becker wrote:
> Sounds interesting, but wouldn't it be more fun to call julia code from python?

That's certainly another possibility, although even there I think the
Python<-->Julia conversion/wrapping routines in PyCall will eventually
be helpful.

There are at least three audiences here:

1) People who want to write new programs (and/or work interactively) in
a high-level language without being forced to switch to a low-level
language for inner loops or algorithms that are awkward to vectorize.
Julia is potentially attractive to this audience, but in order to
leverage existing software you need an easy way to call C (available now
via ccall), Python (available now via PyCall), etcetera, the more the
better.

2) Library authors who want their work to be as widely accessible as
possible, and hence need to export a C interface (from which one can
easily wrap Python, Octave, R, Julia...). These people need the ability
to statically compile Julia code with an "extern C" interface, something
I believe Jeff is planning.

3) Python users who are happy writing in Python and switching to another
language for performance-critical code, but would like that language to
be a high-level language like Julia rather than C. (2) would suffice
here, but would still require C glue (partially automated with SWIG).
But it would be nicer to have a "Foo = jlimport(Foo.jl)" that can
directly import Julia modules and do transparent type conversions.
Before this can be implemented, more of the internal C API of Julia
needs to settle down and be documented. (An implementation of this
could call PyCall on the Julia side to do the data conversions.)

I chose to implement (1) for a variety of reasons, but the most basic
reason is that (2) and (3) currently require more work on the Julia
internals whereas (1) could be implemented entirely as an external
module in pure Julia (modulo a few small bugfixes and feature requests).

But I'm a big fan of inter-language interoperability in general.
Numerical computation is one of the few areas of software engineering
where people regularly write new software using code that dates back to
the 1960s, in part because writing and debugging robust numerical
routines can be so tricky (even if you have the requisite specialized
knowledge), and that creates a huge incentive to retain existing libraries.

--SGJ

Viral Shah

unread,
Feb 26, 2013, 9:40:46 PM2/26/13
to juli...@googlegroups.com
This is really cool!

Konrad Hinsen

unread,
Feb 27, 2013, 2:58:36 AM2/27/13
to juli...@googlegroups.com
--On 26 février 2013 20:14:17 -0500 "Steven G. Johnson" <ste...@alum.mit.edu> wrote:

> There are at least three audiences here:

Let me add one that doesn't exist yet but will in a few years:

4) People whose problem is best solved by combining libraries written in
Julia and libraries written in Python.

> But I'm a big fan of inter-language interoperability in general.

That's exactly the point. People choose a programming language for various
reasons including personal taste and the environment they evolve in. I
don't expect scientists to ever "agree" on using a single language. They
did back in the Fortran era mostly because there was little choice. I
expect to see more rather than less diversity in scientific programming
languages in the future, so interoperability is an important point.

Actually, I hope that one day programming language designers and
implementers will think about interoperability right from the start. What
we should have, ideally, is a common low-level layer that any language can
build on to facilitate interoperability. That layer would take care of
memory handling, including garbage collection, process/thread scheduling,
etc. Something like the JVM but optimized for scientific applications.

Konrad.

Dag Sverre Seljebotn

unread,
Feb 27, 2013, 4:40:25 AM2/27/13
to juli...@googlegroups.com
On 02/27/2013 08:58 AM, Konrad Hinsen wrote:
Slightly related, I'm thinking more about common method dispatches and
safe calling of callbacks. COM or CORBA without the baggage.

One of my "projects I want to complete and already put some of work in"
is a specification that would make Cython and Numba and CPython
extensions share a standard for method dispatches and calling callbacks
in an "unboxed" manner, e.g., so that you could pass "numpy.sin" to a
Cython function and automatically unbox the underlying C function
pointer, rather than boxing all the arguments to it. (Think Go-style
interfaces).

This is needed to have a migration path from Cython etc. to smarter
LLVM-based tools (or perhaps even migration from Cython to Julia).

Julia could potentially tack on to that, and that could be the start of
such a low-level standard.

I'll announce it when I have something working and you can see if it is
valuable (best case is this autumn).

(My basic idea is: By relying on perfect hashing, it's possible to have
something that's NOT fixed at compiled-time, like C++ vtables, but still
only has a 3-4 cycles of overhead in a branch-predicted hot loop,
because if the entry exists in the hash table you *always* find it on
the first attempt. Dumb tools like Cython and CPython extensions written
in C can use that directly; smarter tools like Julia and Numba of course
can try to cache the pointer instead.)

Dag Sverre

Konrad Hinsen

unread,
Feb 27, 2013, 1:04:40 PM2/27/13
to juli...@googlegroups.com
--On 27 février 2013 10:40:25 +0100 Dag Sverre Seljebotn
<d.s.se...@astro.uio.no> wrote:

> One of my "projects I want to complete and already put some of work in"
> is a specification that would make Cython and Numba and CPython
> extensions share a standard for method dispatches and calling callbacks
> in an "unboxed" manner, e.g., so that you could pass "numpy.sin" to a
...

That sounds interesting!

> Julia could potentially tack on to that, and that could be the start of
> such a low-level standard.

As long as you stay within the Python universe, all code (Python, Cython,
Numba) uses the same memory manager. If you want to add Julia to the club,
you have to worry about data ownership and garbage collection. That doesn't
look like a trivial extension.

Konrad.

Steven G. Johnson

unread,
Feb 27, 2013, 1:15:59 PM2/27/13
to juli...@googlegroups.com
On Wednesday, February 27, 2013 2:58:36 AM UTC-5, Konrad Hinsen wrote:
--On 26 février 2013 20:14:17 -0500 "Steven G. Johnson" <ste...@alum.mit.edu> wrote:
> There are at least three audiences here:

Let me add one that doesn't exist yet but will in a few years:

4) People whose problem is best solved by combining libraries written in
Julia and libraries written in Python.

This doesn't seem like a separate case to me.  If these people are writing in Julia or Python, then they fall into cases (1) or (3), respectively.   If they are writing in some other language, then they probably want a C API (which they can either call directly or wrap with e.g. SWIG), which falls into case (2).
 

Dag Sverre Seljebotn

unread,
Feb 27, 2013, 1:18:32 PM2/27/13
to juli...@googlegroups.com
Yes, I was mainly thinking about arrays and native-type scalars here.
Really, for those low-level numerical algorithms that stay around for
decades, that usually gets you very far, and the higher-level API tends
to be written in the "client" language anyway to be Pythonic
(/"Julianic"? "Julish?")

Dag Sverre

Stefan Karpinski

unread,
Feb 27, 2013, 1:26:23 PM2/27/13
to Julia Dev
"Julian" is the adjective we seem to have settled on.

Patrick O'Leary

unread,
Feb 27, 2013, 1:27:31 PM2/27/13
to juli...@googlegroups.com
On Wednesday, February 27, 2013 12:18:32 PM UTC-6, Dag Sverre Seljebotn wrote:
...Pythonic
(/"Julianic"? "Julish?")

"Julian" is the agreed-upon term.

Neal Becker

unread,
Feb 27, 2013, 2:33:35 PM2/27/13
to juli...@googlegroups.com
I'd just like to explain my perspective.

I am very happy programming in Python, but need both performance numeric tools,
and my own specialized code (my field is communications and signal processing).

Now I'm using python/numpy, and have a library of tools in c++ that have been
packaged to use boost::python and ndarray to interface with numpy.

https://github.com/ndarray/ndarray

It's worked reasonably well, but:

1. Could use more performance
2. ndarray is now the 3rd in my series of re-coding my c++ libraries for the
c++/python interface of the month. It now doesn't seem to have much momentum.
It has a lot of good features, but lacks in other areas (and I'm not up to
adding major improvements, especially if I'm the only one ending up using it).

I've played with cython a bit, but I don't find it very appealing.

Perhaps julia could be used to implement some python-callable code? It's nice
that it has some good batteries included - some good performing libraries (some
look even better than what I'm currently using). Of course, julia would need to
accept/work with numpy arrays for this ecosystem to work.


Konrad Hinsen

unread,
Feb 27, 2013, 2:56:30 PM2/27/13
to juli...@googlegroups.com
It's a bit different in that (1) those people might have no inherent
preference for writing in Julia or in Python and (2) they will need
seemless interoperability in both directions in order to combine
non-trivial libraries written each for their respective language.

Dag Sverre Seljebotn writes:

> > As long as you stay within the Python universe, all code (Python,
> > Cython, Numba) uses the same memory manager. If you want to add Julia to
> > the club, you have to worry about data ownership and garbage collection.
> > That doesn't look like a trivial extension.
>
> Yes, I was mainly thinking about arrays and native-type scalars here.

Arrays already need memory management.

Konrad.

Steven G. Johnson

unread,
Feb 27, 2013, 9:10:26 PM2/27/13
to juli...@googlegroups.com
Neal Becker wrote:
> Perhaps julia could be used to implement some python-callable code? It's nice
> that it has some good batteries included - some good performing libraries (some
> look even better than what I'm currently using). Of course, julia would need to
> accept/work with numpy arrays for this ecosystem to work.

NumPy arrays are not a problem, Julia can wrap them natively without
copies (and my PyCall package does this, although it will be a lot nicer
once #2345 is fixed).

Steven G. Johnson

unread,
Mar 9, 2013, 8:02:21 PM3/9/13
to juli...@googlegroups.com

On Wednesday, February 27, 2013 2:33:35 PM UTC-5, nbecker wrote:
Perhaps julia could be used to implement some python-callable code?  It's nice
that it has some good batteries included - some good performing libraries (some
look even better than what I'm currently using).  

This is now possible.  The PyCall package can now convert Julia functions into callable Python objects.  Not only does this allow Julia to call Python functions expecting callback arguments, but since IPython uses PyCall in its experimental Julia interface (https://github.com/JuliaLang/julia/issues/1330 and https://github.com/fperez/juliamagic), you can now call Julia functions from Python in IPython:

IPython 1.0.dev -- An enhanced Interactive Python.

In [1]: %load_ext juliamagic
Finding Julia install directory...
Initializing Julia...
Initializing Julia PyCall module...

In [2]: %julia fib(n) = n < 2 ? n : fib(n-1) + fib(n-2)
Out[2]: <PyCall.jlwrap fib>

In [3]: Out[2](20)
Out[3]: 6765

(And, of course, you can copy the Julia magic from IPython to do the same thing in arbitrary Python code.)

--SGJ

Fernando Perez

unread,
Mar 10, 2013, 5:39:33 PM3/10/13
to juli...@googlegroups.com
Mmmh, I'm trying to test this out for a few talks I'm giving next week
where I'd like to demo it, but for the last few days I can't get Julia
to build at all. I'm kind of puzzled, since nobody else is reporting
errors, but I'm kind of stuck...

I had a build with a lot of system libraries enabled that was working
last week in Boston and then after an update started giving errors.

I've now backed off to a completely fresh checkout and even that isn't
building for me anymore:

http://pastebin.com/RT6B1wGV

I'm on ubuntu 12.10. Does this error ring a bell for anyone? It must
be something on my side, but I'm kind of stumped...

Cheers,

f

Jeff Bezanson

unread,
Mar 10, 2013, 5:44:38 PM3/10/13
to juli...@googlegroups.com
Try make -C deps/libuv clean

Fernando Perez

unread,
Mar 10, 2013, 11:58:52 PM3/10/13
to juli...@googlegroups.com
On Sun, Mar 10, 2013 at 2:44 PM, Jeff Bezanson <jeff.b...@gmail.com> wrote:
> Try make -C deps/libuv clean

Thanks! Oddly, I'd ran an aggressive 'git clean -fdx' which should
have nuked all files unknown to git. I don't understand why that
didn't work and this manual git clean did.

But in any case, it worked. Back in business now, thanks :)

f

Fernando Perez

unread,
Mar 11, 2013, 12:15:33 AM3/11/13
to juli...@googlegroups.com
On Sun, Mar 10, 2013 at 8:58 PM, Fernando Perez <fpere...@gmail.com> wrote:

> didn't work and this manual git clean did.

s/manual git/manual make/

Jeff Bezanson

unread,
Mar 11, 2013, 12:42:37 AM3/11/13
to juli...@googlegroups.com
Submodules are strange. They never seem to do quite what you expect
yet I'm not sure what I'd do to fix them.

Fernando Perez

unread,
Mar 11, 2013, 12:51:26 AM3/11/13
to juli...@googlegroups.com
On Sun, Mar 10, 2013 at 9:42 PM, Jeff Bezanson <jeff.b...@gmail.com> wrote:
> Submodules are strange. They never seem to do quite what you expect
> yet I'm not sure what I'd do to fix them.

Aha! I didn't realize that the build process did submodule checkouts.
Good to know, thanks. That explains it.

Cheers,

f

Viral Shah

unread,
Mar 11, 2013, 3:51:11 AM3/11/13
to juli...@googlegroups.com
A few things in deps/ are submodules - libuv, Rmath, openlibm, and nginx. Only libuv updates tend to break the build, given that it is deeply integrated with julia. make -C deps clean-uv is a good first thing to try when you have build breakage, and the usual make clean / make cleanall does not fix things.

-viral

mark florisson

unread,
Mar 11, 2013, 5:34:41 AM3/11/13
to juli...@googlegroups.com
Have you considered git subtrees? That way people that check out your
repositories immediately get the dependencies as well (which for large
dependencies might not be what you want). But it's a lot better if you
occasionally also develop the dependencies, since submodules make it
very easy to lose your work.

Patrick O'Leary

unread,
Mar 11, 2013, 9:08:11 AM3/11/13
to juli...@googlegroups.com
I'm not sure that the libuv thing is actually a submodule problem, seeing as we are getting the new source tree. It's that make doesn't notice that that source has changed.

Fernando Perez

unread,
Mar 13, 2013, 9:31:36 PM3/13/13
to juli...@googlegroups.com
Hey guys,

sorry to keep pestering, but I still can't get this thing to build:

(master)dreamweaver[julia]> make
SuiteSparse_wrapper.c: In function ‘jl_cholmod_sparse_copy_out’:
SuiteSparse_wrapper.c:135:25: error: ‘SuiteSparse_long’ undeclared
(first use in this function)
SuiteSparse_wrapper.c:135:25: note: each undeclared identifier is
reported only once for each function it appears in
SuiteSparse_wrapper.c:144:27: error: ‘spt’ undeclared (first use in
this function)
SuiteSparse_wrapper.c:144:52: error: expected expression before ‘)’ token
make[2]: *** [/home/fperez/tmp/src/julia/usr/lib/libsuitesparse_wrapper.so]
Error 1
make[1]: *** [julia-release] Error 2
make: *** [release] Error 2


I keep trying every combination of make clean/cleanall and git
cleaning and I can't get a build to finish.

I have tried both using all system libs and using none (the various
SYSTEM_X flags all to 0 or all to 1). Neither option has helped.

Any tips for the dummies amongst us?

Thanks!

f

Viral Shah

unread,
Mar 13, 2013, 10:46:57 PM3/13/13
to juli...@googlegroups.com
Can you delete the SuiteSparse-4.0.2* files and directories in deps/ and try again? SuiteSparse_long is defined in a SuiteSparse header, and it would seem that the headers are not created correctly, which is causing the wrapper to fail.

-viral

Viral Shah

unread,
Mar 13, 2013, 10:52:46 PM3/13/13
to juli...@googlegroups.com
Are you building with USE_SYSTEM_SUITESPARSE=1 by any chance?

-viral



On 14-Mar-2013, at 7:01 AM, Fernando Perez wrote:

Fernando Perez

unread,
Mar 14, 2013, 12:46:31 AM3/14/13
to juli...@googlegroups.com
Hey,

On Wed, Mar 13, 2013 at 7:52 PM, Viral Shah <vi...@mayin.org> wrote:
> Are you building with USE_SYSTEM_SUITESPARSE=1 by any chance?

Yup... Paul just told me to set that to 0, trying again...

f

Viral Shah

unread,
Mar 14, 2013, 12:52:21 AM3/14/13
to juli...@googlegroups.com
If you are on debian / ubuntu, just install the suitesparse-dev package, and it should go through.

If you are setting that to 1, you need to have /usr/include/suitesparse, or provide a location where the definition of SuiteSparse_long can be found.

-viral

Fernando Perez

unread,
Mar 14, 2013, 1:02:14 AM3/14/13
to juli...@googlegroups.com
On Wed, Mar 13, 2013 at 9:52 PM, Viral Shah <vi...@mayin.org> wrote:
> If you are on debian / ubuntu, just install the suitesparse-dev package, and it should go through.
>
> If you are setting that to 1, you need to have /usr/include/suitesparse, or provide a location where the definition of SuiteSparse_long can be found.

Yup, I'm on Ubuntu 12.10, and I already have libsuitesparse-dev:

dreamweaver[~]> apt-cache show libsuitesparse-dev
Package: libsuitesparse-dev
Priority: optional
Section: libdevel
Installed-Size: 5116
Maintainer: Ubuntu Developers <ubuntu-dev...@lists.ubuntu.com>
Original-Maintainer: Maintainer: Debian Science Maintainers
<debian-scienc...@lists.alioth.debian.org>
Architecture: amd64
Source: suitesparse
Version: 1:3.4.0-2ubuntu3
[etc...]

With that flag to 1 it won't build here...

I've re-cloned into a separate repo now to get around any brokenness
from submodules...

Trying again...

Viral Shah

unread,
Mar 14, 2013, 1:16:27 AM3/14/13
to juli...@googlegroups.com
The simplest thing you can do is clone the repo afresh, and just type make. It will take an hour or so, but you should get a build without any trouble whatsoever.

-viral

Fernando Perez

unread,
Mar 14, 2013, 1:20:32 AM3/14/13
to juli...@googlegroups.com
OK, got it to build with Paul's suggestion of setting the
libsuitesparse system choice to 0, but all others to 1.

Whew... Thanks. Back to the Julia/python bridge itself.. I really
want this working cleanly before I head over to Boston next week
(well, in reality, before my pydata keynote next Wednesday :)

Fernando Perez

unread,
Mar 14, 2013, 1:21:56 AM3/14/13
to juli...@googlegroups.com
we're in business...

Viral Shah

unread,
Mar 14, 2013, 2:09:32 AM3/14/13
to juli...@googlegroups.com
Sweet! I am really looking forward to getting to a point where I can run ipython+julia myself on my mac. So far, I have been too chicken to try it out, but I am hoping that at some point you will send instructions to the list and put them on the ipython webpage, and I can take the plunge.

In general, the julia build is set up for typing `make` and preparing a build that will give full functionality and good performance out of the box. It takes some time the first time though. As soon as you deviate from there with make options, you may need to start looking under the hood.

-viral

Fernando Perez

unread,
Mar 14, 2013, 2:58:14 PM3/14/13
to juli...@googlegroups.com
Hey,

I figured you'd like this:

http://nbviewer.ipython.org/urls/raw.github.com/fperez/juliamagic/master/JuliaIPython.ipynb

Note that the plots are made *inside* the Julia calls, but they come
back natively to IPython.

Cheers,

f

Rahul Dave

unread,
Mar 14, 2013, 4:09:15 PM3/14/13
to juli...@googlegroups.com
This is very sweet! And mindbending in the sense that you are using plt in julia to plot matplotlib. Very very sweet.

This would seem to be a way more seamless and fasterarchitecture then what I was thinking in terms of following  https://github.com/arokem/python-matlab-bridge which does matlab bindings over a socket to the matlab java webserver

-- 
Rahul Dave
Sent with Sparrow

Viral Shah

unread,
Mar 14, 2013, 4:40:27 PM3/14/13
to juli...@googlegroups.com
This is insanely cool. Using matplotlib gives a great amount of graphics functionality right away.

-viral

Fernando Perez

unread,
Mar 14, 2013, 4:44:32 PM3/14/13
to juli...@googlegroups.com
On Thu, Mar 14, 2013 at 1:09 PM, Rahul Dave
<rahuldave.m...@gmail.com> wrote:
> This is very sweet! And mindbending in the sense that you are using plt in
> julia to plot matplotlib. Very very sweet.

Yup, it's a bit acrobatic :)

I just added another example at the end showing how to manipulate a
figure created in julia back on the python side. Since a lot of the
matplotlib examples out there are in pure python, this should help.

> This would seem to be a way more seamless and fasterarchitecture then what I
> was thinking in terms of following
> https://github.com/arokem/python-matlab-bridge which does matlab bindings
> over a socket to the matlab java webserver

Oh certainly! Ariel did a great job there to work within the
limitations of matlab, but with PyCall we'll have a far, far more
elegant, clean and powerful setup. Don't worry :)

best,

f

Stefan Karpinski

unread,
Mar 14, 2013, 4:59:35 PM3/14/13
to Julia Dev
This is really remarkable.

Steven G. Johnson

unread,
Mar 14, 2013, 10:58:23 PM3/14/13
to juli...@googlegroups.com
On Thursday, March 14, 2013 4:09:15 PM UTC-4, rahuldave wrote:
This is very sweet! And mindbending in the sense that you are using plt in julia to plot matplotlib. Very very sweet.

For extra mind-bendingness, here is an example of a mutually recursive Fibonacci function that calls back and forth between Julia and Python:

In [1]: %load_ext juliamagic
Finding Julia install directory...
Initializing Julia...
Initializing Julia PyCall module...

In [2]: %julia fib(n, fib2) = n < 2 ? n : fib2(n-1, fib) + fib2(n-2, fib)
Out[2]: <PyCall.jlwrap fib>

In [3]: fib = _

In [4]: def fib2(n, fib):
   ...:     if n < 2:
   ...:         return n
   ...:     return fib(n-1, fib2) + fib(n-2, fib2)
   ...:

In [5]: fib2(20, fib)
Out[5]: 6765


 --SGJ

Jeff Bezanson

unread,
Mar 15, 2013, 12:08:54 AM3/15/13
to juli...@googlegroups.com
Witchcraft, I say! Nothing should be this interoperable; this is
*software* darnit!

Fernando Perez

unread,
Mar 18, 2013, 11:47:20 PM3/18/13
to juli...@googlegroups.com
On Thu, Mar 14, 2013 at 7:58 PM, Steven G. Johnson
<steve...@gmail.com> wrote:
> On Thursday, March 14, 2013 4:09:15 PM UTC-4, rahuldave wrote:
>>
>> This is very sweet! And mindbending in the sense that you are using plt in
>> julia to plot matplotlib. Very very sweet.
>
>
> For extra mind-bendingness, here is an example of a mutually recursive
> Fibonacci function that calls back and forth between Julia and Python:

Yup! Beautiful :)

Very, very much looking forward to Friday!

f

Fernando Perez

unread,
Mar 19, 2013, 7:34:27 PM3/19/13
to juli...@googlegroups.com
Mmh, trouble in paradise: this example was crashing for me so I just
updated my Julia/pycall builds to master, and now the whole thing
crashes if I run it with a remote IPython kernel. It's OK in 'plain
ipython', but if I open

ipython console

which lets me get a remote kernel but still see stdout easily, this happens:

In [1]: %load_ext juliamagic
Finding Julia install directory...
jpath: /home/fperez/tmp/src/julia/usr/lib/libjulia-release.so
Initializing Julia...
Initializing Julia PyCall module...

In [2]: %julia 1+2
fatal: error thrown and no exception handler available.
kernel died, restart ([y]/n)? y


###

Any ideas? There must be something odd with stdout/stderr where being
connected to a remote port just blows up.

If you guys can figure it out I'll be glad to pull, otherwise I'll
demo my static notebook tomorrow at my PyData keynote and we'll look
into it Friday.

cheers,

f

Steven G. Johnson

unread,
Mar 21, 2013, 9:43:58 PM3/21/13
to juli...@googlegroups.com
Just got the "plain ipython" stuff working on my Mac, but am getting the same crash with the remote notebook usage that you are seeing.  (I had never tried the remote notebook before.)   Any idea which version of Julia/PyCall was working so that we can do a git bisect?

Steven G. Johnson

unread,
Mar 21, 2013, 9:58:34 PM3/21/13
to juli...@googlegroups.com
On Thursday, March 21, 2013 9:43:58 PM UTC-4, Steven G. Johnson wrote:
Just got the "plain ipython" stuff working on my Mac, but am getting the same crash with the remote notebook usage that you are seeing.  (I had never tried the remote notebook before.)   Any idea which version of Julia/PyCall was working so that we can do a git bisect?

Okay, found a working version and managed to do a git bisect.  The culprit seems to be this patch to PyCall:

    https://github.com/stevengj/PyCall.jl/commit/b38e11a3ef90001ca306420a021c09c40083464c

Not sure yet what about this patch is causing a problem with remote notebooks (but not with plain ipython), but for now a workaround should be to check out an older PyCall version (cd ~/.julia/PyCall and git checkout 5af63444cb6db6fdd78b58634b25f974e89963aa)

Steven

Steven G. Johnson

unread,
Mar 21, 2013, 10:22:12 PM3/21/13
to juli...@googlegroups.com
Just pushed a patch to PyCall to work around this issue:

      https://github.com/stevengj/PyCall.jl/commit/b6165b5e09d3cbb1b74cd34f46cd9fd39a0dd7dc

the culprit seems to be switching to the builtin "type" function rather than types.TypeType for compatibility with Python 3 (where types.TypeType has disappeared).

Apparently, my "pybuiltin" function to look up builtins (in __main__.__builtins__) is broken in IPython remote mode; any idea why?  Fortunately, the pybuiltin function is not used for much, so this isn't a big deal at the moment.
Reply all
Reply to author
Forward
0 new messages