IbPy and Gnumeric

[cocoteuf]

Hello
I am new with IbPy and Python. I am trying to use IbPy within Gnumeric
via the python customize function. I can run my own python function
but was not succesfull to create a function for calling for instance
the RT bid price. Any experience with Gnumeric + IbPy?

Tks

[glenn]

There are a number of problems. First, IbPy needs to have a python
thread available to listen to IB. The python extensions are only
"hot" when in a callback (when gnumeric invokes a python function)...
this includes threads as python threads aren't true threads... so,
when gnumeric isn't "in python" the entire python infrastructure is
suspended including all python threads.

There really isn't any workaround. One needs to build infrastructure,
typically another process which does IB stuff which can be accessed
from gnumeric's python.

Which immediately gets you into the second problem: gnumeric has no
mechanism to handle events of any kind other than user interaction (no
timer... nothin) other than an undocumented mechanism under finance
which will listen on a named pipe. The stream on the pipe is name-
value pairs separated by a colon. Therefore, you can't even
automatically poll the python infrastructure (which would be pretty
ugly design anyways)

I built something which tried all this, even building a generic timer
using it (I used twisted for the server side BTW).

There were a couplea emails with the gnumeric developer of this file
listener gizmo who claimed it was really a hack for proof-of-principle
and we discussed how event handling might be more generally applied to
gnumeric. To date, no joy, but I'm sure the gnumeric team would love
someone to pick this up.

I'd suggest looking into openoffice given you're, basically, over the
wire anyway. Its nowhere near as clean as gnumeric but you're pretty
much stuck unless you wanna get down and dirty with the lowest levels
of gnumeric. I don't think its particularly nasty but I'd guess that
getting this kind of feature introduced would be "non-trivial"

-glenn

>
> Tks

[arbakra]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Thank you Glenn
Do you know a way to have openoffice linked to IB with IbPy?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkmAO1gACgkQYurTgRzE9cmpKgCgphZxIBBb1JE/uvTHXCh/OHsm
y5kAoL0C1RpuVqJ12E566weJMGvekzBc
=UAs9
-----END PGP SIGNATURE-----

[glenn]

On Jan 28, 3:02 am, arbakra <arbakra....@gmail.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Thank you Glenn
> Do you know a way to have openoffice linked to IB with IbPy?

I once did a bit of looking. There are python interfaces of various
maturities although I think I remember more about interfaces in Java
and possibly CORBA (not something with which one dabbles in lightly,
particularly with Python)

However, you really need to consider what you wanna do. If you want
live number blinky stuff in the spreadsheet, gnumeric with a slight
hack to invoke python to call (using the obscure pipe read hack) your
IbPy process periodically is probably easier than playing with
OpenOffice. Basically, you make a python function depend on values
updated from the pipe hack thing... your python call gets invoked, you
do a synchronous request to your IbPy process and return the values.

However, if you're willing to hit refresh occasionally, even this
isn't necessary. You just design something to invoke the same IbPy
process when you tickle gnumeric...

When I considered this problem in some depth, I concluded that if you
wanted to build a live table, its better to just hack a 2D display
thing in PyQt (Troy was insistent that Qt was the way to go... and he
was very much right... all the more so given the recent LGPL licensing
of Qt) and have all the knobs you want. For fancier analytic type
stuff, live updates are probably overrated anyways so refresh is
fine... (not to mention that gnumeric bogs down very quickly... its
not really designed to be a live update gizmo)

Therefore, you build python remote access to an IbPy process which can
be accessed both from your gnumeric python and your PyQt process over
the wire (pickle is your friend here). You do nastier / fancier /
experimental things in gnumeric and live stuff in PyQt. Also, you
probably enter orders from gnumeric and optionally render the order
state in your PyQt live component... although the IB's Trader
Workstation generally illustrates the state of everything you'd need
to see live

I think this would end up being more powerful and easier than trying
to get a spreadsheet to be live. And since python makes "over the
wire" pretty easy, there's not a huge amount to do for IPC.

In previous posts, there are references to the Twisted IbPy
implementation I wrote. Also, I built / maintain the twisted Qt
reactor. If you're going to do asynchronous processing, I highly
recommend twisted although its a non-trivial elephant to swallow...
IMHO, its worth it as is Qt (thanks Troy)

-glenn

> Comment: Using GnuPG with Mozilla -http://enigmail.mozdev.org

[Troy Melhase]

I once did a bit of looking. There are python interfaces of various
> maturities although I think I remember more about interfaces in Java
> and possibly CORBA (not something with which one dabbles in lightly,
> particularly with Python)

...

> Therefore, you build python remote access to an IbPy process which can
> be accessed both from your gnumeric python and your PyQt process over
> the wire (pickle is your friend here).

What I'm hearing from this thread is that the choice for RPC between
an arbitrary process (gnumeric, OO) and an IbPy process (your script,
gui, app) is either non-existent or is substandard. And you really
shouldn't have to make a choice, there should be a solid, thought-thru
solution for you.

Every time I've had to construct my own RPC mechanism, I've used a
small web server. The benefits typically outweigh the downsides
(stateless nature) of the protocol. So I think I'll code up an "ibpy
server" and an "ibpy command-line client" for the 0.7.7 release.

The server would expose URLs like this:

/ib/cancelOrder/<orderid>
/ib/reqMktData/<tickid>
/ib/reqHistoricalData/<tickid>

Those above would be available via HTTP POST, and other required
params would have to be sent as part of the request (e.g., in
reqMktData we can't have
/<tickid>/<contractfield1>/<contractfield2>/<etc>, so we collapse
those and only require the key (ticker id) as part of the URL).

These calls would be available via HTTP GET:

/ib/order/<orderid>
/ib/marketdata/<tickerid>
/ib/histdata/<tickid>

In the case of these, they are data-returning calls, meaning that
clients can make requests to fetch the values whenever they like.
These calls would return data encoded as CSV, RSS, or JSON.

As for the command line client, it would operate in one of two modes:
as an IbPy HTTP client or as an TWS library client. In both cases,
the interface would be the same, and it would certainly allow for
integration with any program that can start other processes.
Examples:

$ runibpy --clientid=1 --host=myserver --command=reqMktData
--tickerid=832 --rows=50
$ runibpy --http --host=myserver:8080 --command=reqMktData ...

The only question here is how to deal with compound data structures
like the contract and order objects.

> In previous posts, there are references to the Twisted IbPy
> implementation I wrote. Also, I built / maintain the twisted Qt
> reactor. If you're going to do asynchronous processing, I highly
> recommend twisted although its a non-trivial elephant to swallow...
> IMHO, its worth it as is Qt (thanks Troy)

Does twisted have some kind of WSGI interface or support? I'd be
willing to try that over the basic HTTP servers in the standard
library.

Thoughts?

troy

[glenn]

On Jan 29, 4:55 pm, Troy Melhase <troy.melh...@gmail.com> wrote:
> I once did a bit of looking.  There are python interfaces of various
>
> > maturities although I think I remember more about interfaces in Java
> > and possibly CORBA (not something with which one dabbles in lightly,
> > particularly with Python)
> ...
> > Therefore, you build python remote access to an IbPy process which can
> > be accessed both from your gnumeric python and your PyQt process over
> > the wire (pickle is your friend here).
>
> What I'm hearing from this thread is that the choice for RPC between
> an arbitrary process (gnumeric, OO) and an IbPy process (your script,
> gui, app) is either non-existent or is substandard.  And you really
> shouldn't have to make a choice, there should be a solid, thought-thru
> solution for you.

Wouldn't it be wonderful if everyone thought like you. :-)

>
> Every time I've had to construct my own RPC mechanism, I've used a
> small web server.  The benefits typically outweigh the downsides
> (stateless nature) of the protocol.  So I think I'll code up an "ibpy
> server" and an "ibpy command-line client" for the 0.7.7 release.

I'll write some below, but if you were going to do this, I think it
would be useful to take a step back and think broadly about the
architecture.

If you're gonna expose the interface "over the wire", the server side
language can be independent of the client if you're willing to avoid
python specific mechanisms (pickle)

I had to move away from IbPy on the server side for performance
reasons... when you pull depth and a bunch of instruments, throw some
database stuff and maybe an analysis op or two and you quickly hit
performance limits.

In my case, I went all the way: I implemented an interface to a Java
server using Google Protocol Buffers. The reasons are many and its
arguable whether its the right choice for an IB only system, but Java,
C++ and Python are supported natively.

Unfortunately, another problem quickly became apparent: Protocol
buffers are slow in python due to their implementation (exploit a lot
of reflection... makes it elegant and all, but brings the python up
again). So, its great for lots of python apps, but not data at any
real volume.

However, my system is quite a bit more performance intensive than most
need... particularly since I got a live tick feed which requires Java
or C++ anyways.

>
> The server would expose URLs like this:
>
>     /ib/cancelOrder/<orderid>
>     /ib/reqMktData/<tickid>
>     /ib/reqHistoricalData/<tickid>
>
> Those above would be available via HTTP POST, and other required
> params would have to be sent as part of the request (e.g., in
> reqMktData we can't have
> /<tickid>/<contractfield1>/<contractfield2>/<etc>, so we collapse
> those and only require the key (ticker id) as part of the URL).
>
> These calls would be available via HTTP GET:
>
>     /ib/order/<orderid>
>     /ib/marketdata/<tickerid>
>     /ib/histdata/<tickid>
>
> In the case of these, they are data-returning calls, meaning that
> clients can make requests to fetch the values whenever they like.
> These calls would return data encoded as CSV, RSS, or JSON.

So, the proposal is some kind of REST web services interface. I think
this is a great idea...

But, then we're into why is the server in Python?

Believe me, I do every bit of coding in Python I can (which includes
Numpy, Matplotlib, Sage) but Python has performance issues. So, if
you're gonna build a server using "web stuff", I'm guessing that a
rethink might be in order.

It would likely be possible to generate a Python client side interface
which works similarly to your current "Pythonic" API but with
enhancements to support synchronous clients (i.e. no callbacks). Of
course, the design issues here can get tricky as there are results and
the client may choose to poll which means the server side needs to
queue responses. Not nasty technically, but the use case explosion can
sometimes bite you.

>
> As for the command line client, it would operate in one of two modes:
> as an IbPy HTTP client or as an TWS library client.  In both cases,
> the interface would be the same, and it would certainly allow for
> integration with any program that can start other processes.
> Examples:
>
>     $ runibpy --clientid=1 --host=myserver --command=reqMktData
> --tickerid=832 --rows=50
>     $ runibpy --http --host=myserver:8080 --command=reqMktData ...
>
> The only question here is how to deal with compound data structures
> like the contract and order objects.

you betcha... this is where I spent a great deal of time hacking. It
was more typing really as I took the IB java code, commented it all
out, and typed the Protocol Buffer interface definitions kinda in-
line. It took a few days but there are constructs in protobuf which
help with the more complex structures.

Of course, one could also define an XSD, the schema isn't hugely
nasty, but now people who use IbPy because its simple and Python are
wrestling with a different kind of headache.... wrestling with XML can
be kinda nasty although there are certainly good tools / libraries
etc.

>
> > In previous posts, there are references to the Twisted IbPy
> > implementation I wrote.  Also, I built / maintain the twisted Qt
> > reactor.  If you're going to do asynchronous processing, I highly
> > recommend twisted although its a non-trivial elephant to swallow...
> > IMHO, its worth it as is Qt (thanks Troy)
>
> Does twisted have some kind of WSGI interface or support?  I'd be
> willing to try that over the basic HTTP servers in the standard
> library.

Twisted has a full HTTP server implementation and would also be
helpful in that it can easily handle multiple clients and IB without
requiring threads or all the nastiness which comes with it. Its also
pretty high performance as its architected well and all... but its a
non-trivial beast to swallow... as is asynchronous programming in
general. I highly recommend it, but I'm not sure you should take my
recommendation :-)

>
> Thoughts?

As an aside, one of the advantages of the approach I took with a java
based server, is that I can connect to multiple IB clients all pumping
a bunch of data without issue. This is, again, not the kind of
problem one often finds "in the wild", but it helped me out quite a
bit.

Of course, I eventually gave up, wrote a check, and am getting every
tick for every exchange I trade from another vendor (NxCore). Next
step is to go FIX...

-glenn

>
> troy

[glenn]

So, I had this weird thought last night about IbPy... and the java to
python generator / compiler...

and something occurred to me.

Cython....

I've been playing around with Cython for a while. Its pretty amazing
stuff really and the syntax is very much python-like with a few
exceptions... those exceptions are where the Cython compiler does
things to make python code fast (i.e. where you know things about the
data, you can take the load off the Python interpreter and generate
native code)

The speedup can be dramatic... like 200x. There are a number of
important Cython characteristics but its better to just read about it
(links below)

An important link first: This to a talk William Stein gave to/at
Microsoft which has a little Cython example which gets executed in
real-time (i.e. C/C++ code gets generated and linked into python
dynamically through the sage workbook which is browser based). Of
course, just like generating .pyc files, once this happens in real
life, the .dll or .so files are there so just need to be dynamically
linked at runtime unless there is a source code change.

If you're not a math person, just scroll down to the Cython example

http://sagenb.org/home/pub/202/

Note that if you go ahead and get an account on the Sage machines,
you'll be able to run this all yourself... live. (for the more
adventurous, building Sage is amazingly easy and it is completely self
contained... I could expand greatly but those so inclined should check
out Sage (links below)

But, thats not my point.

My point is that since IbPy uses the J2py compiler (called something
like that) its probably "relatively" easy to generate a little extra
python code in key areas to make it Cythonic (couplea type
specifications)... which will make it operate at native code speeds
yet still be completely python at the user level... just that instead
of the library being interpreted, its native...

no diff to the user at all...

and I need to be clear clear cuz it might not be obvious at first
glance. I'm NOT saying the user interface is Cython, just that the
underlying IB handling library is Cython. The user still codes in
Python (unless you want to use Cython which is another story
altogether and might make a lot of sense for those of us who run into
compute problems... this is sort of like using cPickle instead of
Pickle. Same interface, just runs way faster cuz its compiled C code
instead of Python)

This effort might be amost a freebie given the existence of the code
generator used for IbPy. The java code parsed to generate IbPy has
the information necessary to spew the few key type definitions where
IbPy spends all its cycles (Java is strongly typed), in particular,
the inner loops which do the comms handling and object generation.

Cython homepage: http://www.cython.org/

Sage homepage: http://www.sagemath.org/

-glenn

[merlinson]

Just wanted to say thanks, Glenn. I have been using R, and was
thinking of doing my historical simulations in Python with addons
instead. I don't like R's plotting packages. They work but aren't
very friendly to me. I remember passing by Sage a while back (but it
was probably linux-only then, and not as well developed). I
periodically try to switch over to linux, but something invariably
drags me back to windows. (cough ... games) Someday . . . Now Sage
is looking very nice based on my initial explorations, and it's in
python!

Your suggestion to use PyQt was also very helpful. I was surprised
how many places in my code had ugly workarounds for wxPython that PyQt
fixed. I also got rid of my misadventures with multi-threading. For
what I'm doing, that was completely unnecessary. All I do is put the
quotes in a list when they arrive. Then, when a timer times out a few
times a second, I empty the list, do my simple calculations, send my
orders or requests to TWS, and redraw the screen. I let python worry
about everything else. It never crashes and uses almost no cpu time
this way.

Can't wait to see what Cython develops into.

Best Regards