iPython developments
Christopher Barker
I've forwarded this from the IPython-dev list -- they are doing some
really cool stuff, and this looks like something EVERY Python IDE should
use for running python code from the IDE. Thought you all might want to
know.
-Chris
-------- Original Message --------
Subject: [IPython-dev] Qt/Curses interfaces future: results of the
weekend mini-sprint (or having fun with 0mq)
Date: Tue, 23 Mar 2010 14:01:37 -0700
From: Fernando Perez <fpere...@gmail.com>
To: IPython Development list <ipyth...@scipy.org>
Hi all,
I realize that we have a significant task ahead in sorting some thorny
issues out to get 0.11 ready for release, especially with respect to
GUI handling. But the big refactor of 0.11 starts to give us a clean
codebase on which new/better interfaces using ipython can be built,
and for that several people want to contribute right away, so it's
best that we build a path forward for those contributions, even if
some of us still have to finish the 0.11 struggle.
Wendell is looking into curses and Gerardo and Omar in Colombia are
looking at Qt, and it would be fantastic to be able to get these
developments moving forward soon. So Brian and I got together over
the weekend and did a design/coding sprint that turned out to be
surprisingly fun and productive. This is the summary of those
results. I hope this thread can serve as a design brief we'll later
formalize in the docs and which can be used to plan for a possible
GSOC submission, for example (which the Qt guys have in mind).
The basic issue we need to solve is the ability to have out-of-process
interfaces that are efficient, simple to develop, and that support
fully asynchronous operation. In today's ipython, you type code into
a program that is the same tasked with executing the code, so that if
your code crashes, it takes the interface down with it. So we need to
have a two-process system where the user-facing client and the kernel
that executes code live in separate processes (we'll retain a minimal
in-process interface for embedding, no worries, but the bulk of the
real-world use should be in two processes).
We want the user-facing client (be it readline-, curses- or qt-based)
to remain responsive when the kernel is executing code, and to survive
a full kernel crash. So client/kernel need to communicate, and the
communication should hopefully be possible *even when the kernel is
busy*, at least to the extent that low-level messaging should continue
to function even if the kernel is busy with Python code.
Up until now our engines use Twisted, and the above requirements can
simply not be met with Twisted (not to mention Twisted's complexity
and the concern we have with it not being ported soon to py3). We
recently stumbled on the 0mq messaging library:
and Brian was able to quickly build a set of Python bindings for it
(see link at the 0mq site, I'm writing this offline) using Cython.
They are fast, we have full control over them, and since Cython is
python-3 compliant, it means we can get a py3 version anytime we need.
0mq is a really amazing library: I'd only read about it recently and
only used it for the first time this weekend (I started installing it
at Brian's two days ago), and I was blown away by it. It does all the
messaging in C++ system threads that are 100% Python-threads safe, so
the library is capable of queuing messages until the Python layer is
available to handle them. The api is dead-simple, it's blazingly
fast, and we were able to get in two intense days a very real
prototype that solves a number of problems that we were never able to
make a dent into with Twisted. Furthermore, with Twisted it was only
really Brian and Min who ever wrote major amounts of code for
Ipython: twisted is really hard to grasp and has layers upon layers of
abstraction, making it a very difficult library to pick up without a
major commitment. 0mq is exactly the opposite: Brian explained the
basic concepts to me in a few minutes (I haven't read a single doc
yet!), we did some queuing tests interactively (by just making objects
at an ipython prompt) and we then started writing a real prototype
that now works. We are very much considering abandoning twisted as we
move forward and using 0mq for everything, including the distributed
computing support (while keeping the user-facing apis unchanged).
So what's in this example? For now, you'll need to install 0mq and
pyzmq from git; for 0mq, clone the repo at:
git://github.com/sustrik/zeromq2.git
then run
./autogen.sh
./configure --prefix=your_favorite_installation_prefix
make
make install
This should give you a fully working 0mq. Then for the python
bindings, clone Brian's repo and get the kernel branch:
git clone git://github.com/ellisonbg/pyzmq.git
cd pyzmq
git co -b kernel origin/kernel
then
cp setup.cfg.template setup.cfg
and edit setup.cfg to indicate where you put your libraries. This is
basically the prefix above with /lib and /include appended.
Then you can do the usual
python setup.py install --prefix=your_favorite_installation_prefix
The prototype we wrote lives in examples/kernel. To play with it,
open 3 terminals:
- T1: run kernel.py, just leave it there.
- T2 and T2: run frontend.py
Both T2 and T3 are simple interactive prompts that run python code.
You can quit them and restart them, type in both of them and they both
manage the same namespace from the kernel in T1. Each time you hit
return, they synchronize with each other's input/output, so you can
see what each client is sending to the kernel. In a terminal this is
done in-line and only when you hit return, but a curses/qt client with
a real event loop can actually fire events when data arrives and
display output from other clients as it is produced.
You can 'background' inputs by putting ';' as the last character, and
you can keep typing interactively while the kernel continues to
process. If you type something that's taking a long time, Ctrl-C will
break out of the wait but will leave the code running in the
background (like Ctrl-Z in unix).
This is NOT meant to be production code, it has no ipython
dependencies at all, no tab-completion yet, etc. It's meant to:
- let us understand the basic design questions,
- settle on the messaging api between clients and kernel,
- establish what common api all clients can use: a base to be shared
by readline/curses/qt clients, on top of which the frontend-specific
code will go.
So I'd encourage those of you who are interested in this problem to
have a look and let us know how it goes. For now the code lives in
pyzmq because it makes for a great zmq example, but we're almost ready
to start from it putting real ipython machinery. For thinking about
this design though, it's a lot easier to work with a tiny prototype
that fits in 3 files than to deal with all of ipython's complexity.
Cheers,
f
_______________________________________________
IPython-dev mailing list
IPyth...@scipy.org
http://mail.scipy.org/mailman/listinfo/ipython-dev
--
Christopher Barker, Ph.D.
Oceanographer
Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception
Edward K. Ream
<Chris....@noaa.gov> wrote:
> Hi folks,
>
> I've forwarded this from the IPython-dev list -- they are doing some really
> cool stuff, and this looks like something EVERY Python IDE should use for
> running python code from the IDE. Thought you all might want to know.
Edward
------------------------------------------------------------------------------
Edward K. Ream email: edre...@gmail.com
Leo: http://webpages.charter.net/edreamleo/front.html
------------------------------------------------------------------------------
Tal Einat
<Chris....@noaa.gov> wrote:
> Hi folks,
>
> I've forwarded this from the IPython-dev list -- they are doing some really
> cool stuff, and this looks like something EVERY Python IDE should use for
> running python code from the IDE. Thought you all might want to know.
>
> -Chris
Thanks for this, Chris!
FWIW, IDLE uses a subprocess for its shell, communicating with it via
a pure-Python RPC mechanism written especially for it. I couldn't
agree more that this is something any UI which embeds a Python
interpreter should do.
However, 0mq sounds like overkill for just running commands in a
separate process. Such use doesn't really require the super-efficient
message-passing which 0mq is meant to provide, and using 0mq for this
adds a dependency which isn't pure-Python.
IDLE's RPC mechanism is old and quirky, so I wouldn't necessarily
recommend using it for other projects, but there are other pure-Python
alternatives such as RPyC and XML-RPC which fit the bill.
Just my two cents...
- Tal