Live trading

[Ed Bartosh]

Hi,

Does anybody use zipline for live trading? Any ideas where to look in the code to implement this. Any points to the zipline code that supports this? 

It should be possible as Quantopian uses zipline for live trading. However, I was not able to find and points in the documentation regarding this except of the description: "Zipline is a Pythonic algorithmic trading library. It is an event-driven system that supports both backtesting and live-trading."

--

BR,
Ed

[Scott Sanderson]

Hi Ed,

It is **possible** to use Zipline for live trading (and we do indeed use Zipline as the foundation for live trading on Quantopian), but the line you mentioned from the README is probably no longer an accurate reflection of how easy it is to use Zipline for live trading: running a Zipline live algorithm currently requires a fair amount of custom work.  I wouldn't recommend taking on that task unless you're willing to dig into the Zipline source and write some nontrivial extensions.

The main two classes that are relevant to live trading are the Blotter class, which manages placing and filling orders, and the DataPortal class, which provides access to minutely and daily pricing data to various subsystems of Zipline.  Running a Zipline algorithm live essentially requires creating custom subclasses of Blotter and DataPortal.  The Blotter subclass would be responsible for communicating with an brokerage API like the one provided by Interactive Brokers, and the DataPortal subclass would be responsible for providing access to up-to-the-minute pricing data (potentially with fallback to static historical data for history() calls).

In the long term, we'd for it to be easier for people to make these kinds of nontrivial extensions to Zipline (and as an engineer and user of open source software this is a frustrating reply for me to write), but it's hard work to prioritize against all the other things we work on to keep the business running.

Best,

-Scott

[Ed Bartosh]

Hi Scott,

Thank you for the info! It helped a lot!

There is one piece of similar work I've found on github: https://github.com/CarterBain/AlephNull

As far as I can see they modified blotter module. Can you tell if this can be used as a starting point or or it's better to start from scratch?

I understand that the code there is outdated and zipline code changed a lot since then, but still it would be nice if it can be used.

Is there any documentation about Blotter and DataPortal class? Can you provide more details on how you'd implement this?

I've seen a lot of requests on this list to implement brokerage interface. If people are still interested it would be a nice community project to implement this. Anyone is interested to collaborate on this?

Regards,

Ed

 

--
You received this message because you are subscribed to the Google Groups "Zipline Python Opensource Backtester" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[Peter Harrington]

Hi Ed,

I can and would like to help out with this.  I have live traded with IbPy in the past.  

The documentation for DataPortal is pretty good, Blotter could use a little more documentation.

Scott gave some good advice on using those two classes, and then next question is after those get subclassed and the proper methods overridden, where would you plug them in? If you dig into the code Zipline uses to run an algorithm you will find that the heart of it is a TradingAlgorithm, most people will never see this if they are letting Zipline "run" the algorithm for them.  The Zipline run function basically just sets up a TradingAlgorithm and runs it.  (I've included complete code at the bottom of this post showing how to run a backtest on your own, it shows how TradingAlgorithm is set up.)  

TradingAlgorithm is where I would plug in the custom DataPortal, let's call it LiveDataPortal (see line #274 of algorithm.py), you can pass it as an argument.  Now where to plug in the custom Blotter, (let's call it LiveBlotter)?  Guess what?  You also can pass it as an argument to TradingAlgorithm (see line #321 of algorithm.py). 

So now that those questions are answered the real work of getting LiveDataPortal and LiveBlotter to talk to IB would need to happen.  

Like I said I could and would like to help out with this, so think of a project name and we can get started.  

Enter code here...

"""
This code is used to demonstrate how to run a Zipline backtest from within code.  (not using the command line tool)
The reason for doing is using Python to drive the running of multiple backtests.

A good reference is the zipline code that actually runs the backtest:
   _run() in zipline/utils/run_algo.py
"""
from zipline import TradingAlgorithm
from zipline.data.data_portal import DataPortal
from zipline.finance.trading import TradingEnvironment
from zipline.utils.factory import create_simulation_parameters
from zipline.utils.calendars import get_calendar
from zipline.pipeline.loaders import USEquityPricingLoader
from zipline.pipeline.data.equity_pricing import USEquityPricing
from zipline.data.bundles.core import load
from zipline.api import symbol, order  # used in handle_data

import os
import re
from time import time
import pandas as pd

CAPITAL_BASE = 1.0e6


def makeTS(date_str):
    """creates a Pandas DT object from a string"""
    return pd.Timestamp(date_str, tz='utc')


def parse_sqlite_connstr(db_URL):
    """parses out the db connection string (needed to make a TradingEnvironment"""
    _, connstr = re.split(r'sqlite:///', str(db_URL), maxsplit=1,)
    return connstr


def make_choose_loader(pl_loader):
    def cl(column):
        if column in USEquityPricing.columns:
            return pipeline_loader
        raise ValueError("No PipelineLoader registered for column %s." % column)
    return cl


if __name__ == '__main__':

    # load the bundle
    bundle_data = load('quantopian-quandl', os.environ, None)
    cal = bundle_data.equity_daily_bar_reader.trading_calendar.all_sessions
    pipeline_loader = USEquityPricingLoader(bundle_data.equity_daily_bar_reader, bundle_data.adjustment_reader)
    choose_loader = make_choose_loader(pipeline_loader)

    env = TradingEnvironment(asset_db_path=parse_sqlite_connstr(bundle_data.asset_finder.engine.url))

    data = DataPortal(
        env.asset_finder, get_calendar("NYSE"),
        first_trading_day=bundle_data.equity_minute_bar_reader.first_trading_day,
        equity_minute_reader=bundle_data.equity_minute_bar_reader,
        equity_daily_reader=bundle_data.equity_daily_bar_reader,
        adjustment_reader=bundle_data.adjustment_reader,
    )

    start = makeTS("2015-11-01"); end = makeTS("2016-11-01")  # this can go anywhere before the TradingAlgorithm

    def initialize(context):
        pass

    def handle_data(context, data):
        order(symbol('AAPL'), 10)

    # the actual running of the backtest happens in the TradingAlgorithm object
    bt_start = time()
    perf = TradingAlgorithm(
        env=env,
        get_pipeline_loader=choose_loader,
        sim_params=create_simulation_parameters(
            start=start,
            end=end,
            capital_base=CAPITAL_BASE,
            data_frequency='daily',
        ),
        **{
            'initialize': initialize,
            'handle_data': handle_data,
            'before_trading_start': None,
            'analyze': None,
        }
    ).run(data, overwrite_sim_params=False,)
    bt_end = time()

    print perf.columns
    print perf['portfolio_value']

    print "The backtest took %0.2f seconds to run." % (bt_end - bt_start)
    print("all done boss")

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[Ed Bartosh]

Hi Peter,

Let's get started. I've created project on github and invited you: https://github.com/organizations/zipline-live/

Please, try to commit your code there.

Regards,

Ed

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[Joseph Sadove]

Hi Ed,

I am confronted with the option of doing something similar. I take it this is a personal git account and you can only permission "collaborators" and that means read/write access. Is there any chance you could add me, although I have no intention of committing. 
Alternatively, are you going to put this under an org and permit pulls when it's ready/close-to-ready to be tested?
And, anything you can say about a timeline on this? I'd be happy to help test it.

Thanks, Joe

[Ed Bartosh]

Hi Josef,

I've added you to the project. Feel free to create your own branch there.

> And, anything you can say about a timeline on this?

Not much, I'm afraid. I'm going to look at it this weekend. We'll see how it goes.

Regards,

Ed

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[Joseph Sadove]

Hi Ed,

I saw and was already taking a look around.

Very nice of you, thanks so much.

Cheers, Joe

[John Walsh]

Hi Ed,

I'm also interested in pursuing live trading with the Zipline framework, and am just now looking into what the best way to go about it is.  Would you mind adding me to the project as well?  I'd be interested in collaborating on this.

Do you intend to record data as it happens so that you can look back at historic data for things like moving averages? One thing I'm still trying to work out is if I'll need to modify the bar reader classes as well.

Best regards,

John

[Ed Bartosh]

Hi John,

I've sent an invitation. Welcome to the project!

Yes, recording historic data should be taken care of. Otherwise data.history API will not work properly.

Regards,

Ed 

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[fenghe liu]

Hi Ed,

Can you please add me as well. Would like to collaborate and contribute.

Thanks!

[Gabriel Moncarz]

Hi Guys,

   Can you please also give me access to the project? I am interested to paper-trade with Zipline. I'll collaborate if I can.

Thanks.

[Stuart Clark]

Hi Ed

Likewise if you can grant me access as well please?  Very interested in taking zipline further with paper/live trading with IB

Thanks, Stuart

[B Holmes]

Me too pls Ed, be v interested to contribute.

Thanks, Brendan

[Mayur Bakhai]

Hi Ed,

I'm interested in live trading with zipline. I'll try my best to contribute as well. Can you please grant me access to the project?

Thanks,

Mayur

[Tibor Kiss]

Hi Ed,

I've made an IB connector to pyalgotrade in 2013 and running it live since.

I'm considering using zipline for upcoming work given it supports live trading.

I'd be interested to see what's available today and possibly extend it if needed.

Could you please add me to zipline-live? My github account is 'tibkiss'.

Thanks,

Tibor

[Sven Haadem]

Hi Ed,

I been working on my own to make zipline live. If you accept more contributors I would love to work on a joint project instead. 

Thanks,

Sven (https://github.com/Haadem)

[Vitaly Ermilov]

Hi, Ed

I'm very interested in the topic, and I am able to invest quite a lot of my time and provide my skills to contribute to the project. I am good at deep understanding of code. Phd in Computer science.

I'd be happy to join the project.

 

Best Regards

Vitaly Ermilov

[Ed Bartosh]

Hi Vitaly,

Welcome to the project! I've just sent you an invitation. Please, create your own branch and contribute code there.

Regards,

Ed

--

You received this message because you are subscribed to the Google Groups "Zipline Python Opensource Backtester" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[Peter Harrington]

I have completed analysis and initial design.  Please see the folder zipline/live in my branch: 

https://github.com/zipline-live/zipline/tree/ph/zipline/live

The analysis notes are here: 

https://github.com/zipline-live/zipline/blob/ph/zipline/live/analysis.txt

There is a test bench here: 

https://github.com/zipline-live/zipline/blob/ph/zipline/live/live_demo.py

This is a simple strategy trading one equity, it doesn't work now, but if you comment out line 119 it will work as a simulation.  

My goal is to get this working on minute data then get pipeline working with daily data. 

With the large group of talented people here we should be able to get this working quickly.  

On Tuesday, May 2, 2017 at 10:52:05 PM UTC-7, Ed Bartosh wrote:

Hi Vitaly,

Welcome to the project! I've just sent you an invitation. Please, create your own branch and contribute code there.

Regards,

Ed

2017-05-03 0:41 GMT+03:00 Vitaly Ermilov <vitaly....@gmail.com>:

Hi, Ed

I'm very interested in the topic, and I am able to invest quite a lot of my time and provide my skills to contribute to the project. I am good at deep understanding of code. Phd in Computer science.

I'd be happy to join the project.

 

Best Regards

Vitaly Ermilov

--
You received this message because you are subscribed to the Google Groups "Zipline Python Opensource Backtester" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[Frédéric Fortier]

I'm interested in contributing to this project if you guys still require help. I see that you are already far along. I'm familiar with the zipline codebase. I'm also exploring the viability of using zipline to trade cryptocurrencies. If it makes sense in context of the project, I could attempt to build a custom bundle and broker client for Bitfinex. 

[Ed Bartosh]

Hi Frederic,

I've just sent you an invitation. Please, create your own branch if you want to contribute code.

As for Bitfinex client, I don't have strong opinion here. Personally I'd like to see IB interface working first, but I'm not against supporting other brokers.

What other project members think about it?

Regards,

Ed

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[Peter Harrington]

I think that writing a loader and broker client for Bitfinex would be great.   

I have seen multiple people want to trade BTC.   

[Frédéric Fortier]

Ok great! I'll keep monitoring the project until the IB broker client implementation is a bit more settled before forking it for Bitfinex. If you need help with the IB broker itself, feel free to ask. I could probably write unit tests or other supporting components.

[Tibor Kiss]

Hi Peter,

Thanks for sharing!

I have also made some progress recently and I believe it is time to bring it to public. I've made the following code changes to make live trading possible:

 * Created RealtimeClock which is a drop-in replacement for MinuteSimulationClock. This is required to have the handle_data called.

 * Created TWS Connection based on IbPy (own version, will share later. IbPy2 should work too)

 * Almost completed the Account & Portfolio download feature using TWS Connection (needs to be tested and extended a bit)

 * Command line argument handling (--live-trading and --tws-connection options)

What's still needed:

 * Live Account & Portfolio validation (some fields are not loaded yet)

 * Live Order support: Blotter extension 

 * Live Data support: DataPortal (get_spot_value, get_adjusted) extension

 * Pickle support to store algorithm state 

 

My branch:

https://github.com/zipline-live/zipline/tree/tibor_live

I was aware that you are also working on this and my efforts might result in duplicated code but I decided to do it myself as it is a good learning exercise.

I believe it would be time to join our efforts and tackle this together in a more organized way.

What do you think?

- Tibor

[alex.wh...@gmail.com]

Hi Ed,

I was working with Zipline for quite a while in the past and now I want to make it work for the paper and live trading. So I was looking for some information on what is the best approach for that and found this thread. It looks like you guys are working on the exact same thing! So I'll be really glad to join your project and help you with it.

Cheers.

[Peter Harrington]

Tibor, 

Wow that is some excellent work.  I like that you even wrote tests. 

I totally agree that we should combine efforts and tackle this in an organized way.  
I think we should have a branch called "develop" that we will use to combine our changes in the future.  We both should create branches based on develop, we we have changes we can create a pull request and someone else can review/merge.   

Secondly, we should come to some agreement on the design, I know this may change as we write the code and discover new things.  Probably using the Wiki is a good place to start, I can think of two pages: 1. A todo list of the minimum required things we need to do before we can trade, this can also have Nice to have features, such as BrokerClient for a broker in China or a DataLoader for FactSet, etc.   We also could enable the "issues" tab on the Github page and use that for tracking things we need to implement.  Do you have an opinion on which way works better?  2. The second page that best belongs on the Wiki is the Design: how do the classes fit together, what classes are new, etc.  

I think that is the best way to get organized, and combine efforts.  Any thoughts?  

Peter

[Tibor Kiss]

+ zipline (it was removed accidentally)

Thanks Peter!

I have created the branch (named it 'live', I hope you don't mind) and the first TODOs under GitHub Project.

I did not find a way to add assignee field to the notes created under project. Also, there is no way to report issues atm.

I suppose you or Ed has admin access. Could you please try to enable those features?

I'll start with the refactoring and fire up a PR hopefully by EOD.

On Sat, May 13, 2017 at 12:45 AM, Peter Harrington <peter.b.h...@gmail.com> wrote:

I have the wiki pages up:

https://github.com/zipline-live/zipline/wiki

https://github.com/zipline-live/zipline/wiki/Design

I agree on all those things points.  I don't see that you missed anything.  

On Fri, May 12, 2017 at 8:33 AM, Tibor Kiss <tibor...@gmail.com> wrote:

Hello Peter,

+1 on branching / PRs & Wiki, great idea!

I read through on your design (analysis.txt) and it is pretty close what I have in mind.

It is a good starting point for the design doc in the wiki. Could you please create an entry based on it?

Before we can merge our work to the new branch we need to agree on some small differences:

 - I like your abstraction of BrokerClient / IBBrokerClient. I will split up my TWSConnection class accordingly

 - I think my RealtimeClock removes the need of TradingAlgorithm.on_dt_changed() caller?

 - TradingAlgorithmLive: Currently I have directly modified the original class. It is better to create a separate live-specific class (which builds on top of TradingAlgorithm). I'll move my changes into TradingAlgorithmLive. 

 - Need of BlotterLive: I'm not completely sure about this. It might be enough to override TradingAlgorithm's order(), cancel(), etc.

Did I miss anything?

Thanks,

Tibor

--
You received this message because you are subscribed to a topic in the Google Groups "Zipline Python Opensource Backtester" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zipline/KHKI1PZx08I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to zipline+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Kiss Tibor

+36 70 275 9863
tibor...@gmail.com

--

Kiss Tibor

+36 70 275 9863
tibor...@gmail.com

[Tibor Kiss]

Hi Ed,

I think it would be very beneficial to have CI setup (travis / appveyor / coveralls) before we start merging into 'live' branch. To make it happen we'd need zipline-live github user to sign up to those services and register the active branch.

Could you please help with this?

Thanks,

Tibor

[Ed Bartosh]

Hi guys,

It's great to see more activity in the project! Thank you for this!

Peter, I gave you admin rights, so please go ahead and change settings as needed.

PS: It would be good if you can create issues that other team members can pick up.

Regards,

Ed

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[Peter Harrington]

I've added the issues tab: https://github.com/zipline-live/zipline/issues

Working on setting up Travis/Coveralls/Appveyor

I did not see a way to add owners to the TO-DO board.  Under the Menu -> show activity you can see who moved/created/edited cards.  

To unsubscribe from this group and all its topics, send an email to zipline+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Kiss Tibor

+36 70 275 9863
tibor...@gmail.com

--

Kiss Tibor

+36 70 275 9863
tibor...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Zipline Python Opensource Backtester" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[Tibor Kiss]

Regarding owners of 'tasks' under projects board: we can drop issues on the board, those will have owners associated.

What I initially created on the board are notes, which cannot have owners.

Will convert my notes onto issues soon.

[Tibor Kiss]

Folks,

Peter has successfully enabled the CI tools for the project and we've merged the first PR.

We have a project board to track progress here: https://github.com/zipline-live/zipline/projects/1

Most of the issues are depending on issue #4, what I'll hopefully finish by EOD Thursday.

We have one task which currently does not depend on TWS connection and it is important for live execution:

https://github.com/zipline-live/zipline/issues/6

It is not so hard to implement, well testable and brings great value.

Anyone willing to volunteer? 

Thanks,

Tibor

[Tibor Kiss]

A quick update from the last two weeks: 

Zipline-Live is now able to connect to Interactive Broker's TWS, load account and portfolio information

and route normal orders (market, limit, stop, sl) to the broker. 

In other words: We're half way there, a simple buy-and-hold strategy could be executed on premise.

Currently the following tasks are under development:

 - Store the algo state locally (Peter)

 - Get the windows build clean (Ed)

 - IB realtime data support (Tibor)

All this work is accessible on our 'live' branch: https://github.com/zipline-live/zipline/tree/live

We need your help:

If you are interested please give it a try, test it & tinker with it.  Any feedback is greatly appreciated!

Join us on Slack if you have any questions / comments / requests:

https://join.slack.com/ziplinelive-live/shared_invite/MTg5MDQ3MzY5MTUzLTE0OTYxMjA1NzYtNmIzZjRmMWE5ZQ

Hope to see you soon!

- Tibor

[Tibor Kiss]

Sorry - the Slack URL was incorrect in my mail.

The correct invite is: https://join.slack.com/zipline-live/shared_invite/MTg4OTUwODgyOTEyLTE0OTYxMjE5ODctMDBhM2ZhZGFiNw

On Tue, May 30, 2017 at 7:19 AM, Tibor Kiss <tibor...@gmail.com> wrote:

A quick update from the last two weeks: 

Zipline-Live is now able to connect to Interactive Broker's TWS, load account and portfolio information

and route normal orders (market, limit, stop, sl) to the broker. 

In other words: We're half way there, a simple buy-and-hold strategy could be executed on premise.

Currently the following tasks are under development:

 - Store the algo state locally (Peter)

 - Get the windows build clean (Ed)

 - IB realtime data support (Tibor)

All this work is accessible on our 'live' branch: https://github.com/zipline-live/zipline/tree/live

We need your help:

If you are interested please give it a try, test it & tinker with it.  Any feedback is greatly appreciated!

Join us on Slack if you have any questions / comments / requests:

https://join.slack.com/zipline-live/shared_invite/MTg4OTUwODgyOTEyLTE0OTYxMjE5ODctMDBhM2ZhZGFiNw

[Ed Bartosh]

Hi Tibor,

Than you for the update!

Can you provide simple example of running simple algo using IB broker for people to play with it?

Regards,

Ed

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[Tibor Kiss]

Sure, running the buy_and_hold.py could be done as follows:

 - Enable TWS API Connection (for my case it is port 8888).

 - Clone the repo, switch to live branch and install with all the dependencies

 - Start zipline with --data-frequency minute --broker ib --broker-uri localhost:7500:1234

With commands:

$ git clone https://github.com/zipline-live/zipline zipline-live

$ cd zipline-live

$ git checkout live

$ virtualenv -p python2 venv

$ . ./venv/bin/activate

$ pip install -r etc/requirements_live_ib.txt

$ pip install -e .

$ zipline run -f zipline/examples/buy_and_hold.py --data-frequency minute --broker ib --broker-uri localhost:8888:1234

[2017-05-30 19:40:22.695194] INFO: IB Broker: Connecting: localhost:8888:1234

Server Version: 76

TWS Time at connection:20170530 15:40:22 EST

....

Cheers,

Tibor

[Douglas S]

Hiya,

Very interested in joining this too! Would love to contribute if I can and if you still accept contributors.

My github account is doggie52.

Thanks in advance,

Douglas

[Ed Bartosh]

Hi Douglas,

Welcome to the project! Yes, we do accept and appreciate contributions.

2 all existing members(16 atm): We really do! Please, don't hesitate to play with the code, create issues, join zipline-live.slack.com etcetera.

Regards,

Ed

--

BR,
Ed

[Tibor Kiss]

Welcome to the project, Douglas!

We are almost ready with the core functionality, which enables non-pipeline apps to be live traded. 

Missing pieces currently being worked on are these:

https://github.com/zipline-live/zipline/issues/6

https://github.com/zipline-live/zipline/issues/8

If you are looking for a particular task to contribute to then please take a look here: 

https://github.com/zipline-live/zipline/issues/26

Medium term we should investigate how can we elegantly support the pipeline api. e.g: Implementing the stock universe 

Other than that new ideas / features are also welcome!

Thanks, 

Tibor

[Mike Ellertson]

Hello Ed,

I'm also quite interested in this topic. I've developed a stock prediction system. I need to test some parts of it using Zipline. I'd like to launch Zipline from either the command line, passing some arguments, or via another Python script. 

I was looking into using a Zipline extension, but then I came across this thread. Would I also be able to join you and your team on Slack?

Best regards,

Mike Ellertson

[Ed Bartosh]

Hi Mike,

I've just sent you an invitation.

Welcome to the project!

Regards,

Ed

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[Mike Ellertson]

Thanks!

--

Best regards,

Mike Ellertson
Email: mdell...@gmail.com

[Richard Frank]

Hey all,

This project is awesome! We (at Quantopian) are excited that you have been adding to zipline what we haven't.

Related to the ongoing need to merge from upstream and the discussions of merging back to upstream, I've been thinking about how we might improve the maintenance workflow for zipline-live and its interaction with zipline.

My hope is that you/we can separate the changes currently in zipline-live into 1. extension points in zipline itself, and 2. a repo with just the broker integration code, which can hook into those extension points.

The extension points would be merged back to quantopian/zipline. Then as long as zipline has passing tests for the extension points, zipline-live should be able to maintain just the IB integration code without continuously merging from upstream or fearing that upstream will break the integration without notice. It also doesn't need to be a fork of zipline anymore, just have zipline as a dependency.

On start, zipline runs `~/.zipline/extensions.py`, which is where folks currently register their custom data bundles. This mechanism should be amenable to custom brokers (or other extensions) if the extension points are made explicit. Just to be clear, these extension points don't exist yet, but we'd work together to build them, now that you've been figuring out the necessary changes.

What do you think?

Thanks,

Rich

[Tibor Kiss]

Hi Rich,

Thanks for the warm words! 

+1 on the idea of merging our adapters to Zp and moving the broker specific code to a separate repo.

Currently all the tests are passing, should not be a big deal to make this separation. 

Please note, that we're not completely ready:

State persistence is still WiP -- that might need additional interfacing in Zp to support scheduled functions.

I'd be happy to hear how you are doing this in your live environment.

If you like we can set up hangouts session to discuss the details.

Best,

Tibor

To unsubscribe from this group and all its topics, send an email to zipline+unsubscribe@googlegroups.com.

[Ed Bartosh]

Hi Richard,

Thank you for your encouragement and nice words about the project!

I was hoping to merge *all* our code into zipline at some point. Having IB support in zipline mainline repo is better from my point of view than having it in a separate project. However, we can have both things too, which is even better. I like your analogy with support of data bundles in zipline. Without having yahoo and quandl bundles it would be much harder for people to understand how to implement custom bundles. I'd propose to do the same with brokers - implement interface to load custom brokers and have IB (and hopefully couple of others) as a working reference implementation.

Thoughts?

Regards,

Ed

--

BR,
Ed

[Richard Frank]

Hi Tibor and Ed,

Thanks for your replies and thoughts.

Tibor - Happy to discuss scheduled functions and such. A hangout should work - shall we arrange the details outside the public email thread?

Ed - I'm glad we're at least on the same page about not having to maintain an entire separate fork, and I like the idea of IB as a reference implementation.

In recent Quantopian discussions, we've been thinking we'd keep integrations with paid services outside the main repo, since they're more difficult for us to maintain. (For example: https://github.com/quantopian/zipline/pull/1716#pullrequestreview-39552166 ) That's still where I lean, but we haven't discussed IB in particular in this context, so I'll gather some others' thoughts as well. We do want to make extensions easy to discover from the zipline docs, with clear instructions on installation, so the user experiences for  core and extension are as similar as possible.

Let me know what you think.

-Rich

[Peter Harrington]

+1 for moving zipline-live to a separate repo that has Zipline as a dependency.  

Thanks!

Best regards,

Mike Ellertson
Email: mdell...@gmail.com

--
You received this message because you are subscribed to the Google Groups "Zipline Python Opensource Backtester" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[Vincent Yip]

Hi Ed,

May I also join and contribute to this great project? (you guys rock!) I used to build a trading system using Kotlin/Java by integrating with the IB API. Hope I can contribute to make Zipline the best Python back-testing and live-trading library. Thanks.

My GitHub account is vincentyip

Regards,
Vincent

[Ed Bartosh]

Hi Vincent,

Welcome to the project!

I've just sent you an invitation.

Please, don't hesitate to pick up any of open issues. This one is most important: https://github.com/zipline-live/zipline/issues/26

Regards,

Ed

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[George Wang]

Hi Ed and team,

May I join and contribute to this project as well? My GitHub ID is grandtiger.

Hopefully we can make zipline-live the best Python library for back-testing and live-trading with IB.

Thanks,

George

[Mark Fink]

Hi there, I am new to zipline-live and have a few questions... I understand zipline and how it runs backtests as fast as possible so one can analyse results after. live trading needs a different operation model so I like to know how that works.

1) I guess zipline-live still works on data-bundles. You still use `ingest` to update bundle data, or is there some continuous mode?

2) Does zipline need to run continuously? I want to run my trading algorithm on `crontab` like once per hour. Is that possible?

[Tibor Kiss]

Hi Mark,

Sorry for the delayed answer.

1) Data bundles are needed. 

What I do is ingest at the beginning of the day, then start the algo which runs throughout the day.

We have an open item about continuously ingesting the data for historical purposes:

https://github.com/zipline-live/zipline/issues/26

2) Continuous running is required if you make use of context variable. 

This will change once we completely implement state persistence, which is underway:

Peter created an initial version what I have picked up and fixed some of the bugs.

Currently I'm working on the scheduled function persistence part - the last piece of the puzzle. 

The issue is tracked here: https://github.com/zipline-live/zipline/issues/6

If you have further question I'd suggest to use our slack channel to have a more direct conversation:

https://join.slack.com/zipline-live/shared_invite/MTg4OTUwODgyOTEyLTE0OTYxMjE5ODctMDBhM2ZhZGFiNw

Thanks,

Tibor

--
You received this message because you are subscribed to a topic in the Google Groups "Zipline Python Opensource Backtester" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zipline/KHKI1PZx08I/unsubscribe.

To unsubscribe from this group and all its topics, send an email to zipline+unsubscribe@googlegroups.com.

[SCOTT DAVID POST]

I have managed to successfully run a few simple algorithms on my local computer but am running into errors when importing ETFs through either the symbol() or sid() functions - including SPY. I've tried to figure this out but can't, has anyone else ran into this problem/fixed it?

On Sunday, February 12, 2017 at 3:12:41 AM UTC-8, Ed Bartosh wrote:

Hi,

Does anybody use zipline for live trading? Any ideas where to look in the code to implement this. Any points to the zipline code that supports this? 

It should be possible as Quantopian uses zipline for live trading. However, I was not able to find and points in the documentation regarding this except of the description: "Zipline is a Pythonic algorithmic trading library. It is an event-driven system that supports both backtesting and live-trading."

--

BR,
Ed

[Tibor Kiss]

Hi Scott,

Unfortunately the quandl based bundles does not contain ETF related data for some reason.

Initially I was using Yahoo Finance to bridge this gap, but recently Yahoo has changed it's public API

which broke ingestion (and Zipline's benchmark comparison too -- but that's fixed).

As a workaround one can use Google based data downloader to ingest ETF and stock data. 

Unfortunately SPY (and other symbols) has a missing bar for 2009-08-11 00:00:00, which makes ingestion of SPY impossible. 

To overcome this obstacle one can use VOO as an alternative to SPY, that symbol does not have such limitation.

I've posted the google downloader I'm using:

https://gist.github.com/tibkiss/e00ba8a9d762f699fb58190cd1c1418e

You need to put the content of the gist into ~/.zipline/extension.py  

We can move this discussion to Slack if you need further assistance.

Happy weekend,

Tibor

--

[Brian Wagener]

Would love to see support for Crypto Currency Exchanges like Bitfinex or GDAX.  I would recommend using the library https://github.com/ccxt-dev/ccxt which provides a wrapper around "85 bitcoin/altcoin exchanges "  What do you think?

[Ed Bartosh]

Hi Brian,

I think it's a good idea and fits nicely into the scope of zipline-live

I'd suggest you to discuss your idea on the project mailing list or slack 

Regards,

Ed

--

You received this message because you are subscribed to the Google Groups "Zipline Python Opensource Backtester" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[Albert Vonpupp]

I *really* second that.

Recently I started using ccxt and I have been in touch with @kroitor, he is an amazingly responsive person so I am pretty sure ccxt is the right way to go and I would highly recommend it.

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[Tibor Kiss]

Hello,

Feel free to open an issue in github and start working on this. We welcome PRs for both bugs and features too!

If you share your github credentials we will be happy to assign this feature request to you - to avoid duplicate work.

Thanks,

Tibor

--
You received this message because you are subscribed to a topic in the Google Groups "Zipline Python Opensource Backtester" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zipline/KHKI1PZx08I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to zipline+unsubscribe@googlegroups.com.

[Albert Vonpupp]

I am totally new to zipline, in fact I registered on the group yesterday and that was my first message.

I want to learn more about zipline and and go though the tutorials. I might be just too noob to contribute but I would love to. Could you please give me any advice on where to start to be able to add crypto functionality support to zipline?

Thanks.

To unsubscribe from this group and all its topics, send an email to zipline+u...@googlegroups.com.

[Tibor Kiss]

Welcome to zipline-live then :)

In order to add new brokerage support one needs to implement the broker interface:

https://github.com/zipline-live/zipline/blob/master/zipline/gens/brokers/broker.py

One working example is IB:

https://github.com/zipline-live/zipline/blob/master/zipline/gens/brokers/ib_broker.py

If you join slack (accessible thru http://www.zipline-live.io/contact) we can provide more guidance.

Thanks,

Tibor

To unsubscribe from this group and all its topics, send an email to zipline+unsubscribe@googlegroups.com.

[ccbttn]

Hi Peter,

I used your framework to run Zipline algo, but it seems the schedule function within initialize() does not work as expected. 

For example if you schedule a rebalance as of 30 Mins after market open. and within the rebalance function call 

print(get_datetime('US/Eastern').time())

it always returns 16:00:00. but since before_trading_strat is specified in TradingAlgorithm, if you call the same gettime function this one returns the correct time 08:45:00

do you happen to have the same issue?

Thanks.

Peter Harrington於 2017年3月20日星期一 UTC-4上午11時04分23秒寫道：

Hi Ed,

I can and would like to help out with this.  I have live traded with IbPy in the past.  

The documentation for DataPortal is pretty good, Blotter could use a little more documentation.

Scott gave some good advice on using those two classes, and then next question is after those get subclassed and the proper methods overridden, where would you plug them in? If you dig into the code Zipline uses to run an algorithm you will find that the heart of it is a TradingAlgorithm, most people will never see this if they are letting Zipline "run" the algorithm for them.  The Zipline run function basically just sets up a TradingAlgorithm and runs it.  (I've included complete code at the bottom of this post showing how to run a backtest on your own, it shows how TradingAlgorithm is set up.)  

TradingAlgorithm is where I would plug in the custom DataPortal, let's call it LiveDataPortal (see line #274 of algorithm.py), you can pass it as an argument.  Now where to plug in the custom Blotter, (let's call it LiveBlotter)?  Guess what?  You also can pass it as an argument to TradingAlgorithm (see line #321 of algorithm.py). 

So now that those questions are answered the real work of getting LiveDataPortal and LiveBlotter to talk to IB would need to happen.  

Like I said I could and would like to help out with this, so think of a project name and we can get started.  

Enter code here...

"""
This code is used to demonstrate how to run a Zipline backtest from within code.  (not using the command line tool)
The reason for doing is using Python to drive the running of multiple backtests.

A good reference is the zipline code that actually runs the backtest:
   _run() in zipline/utils/run_algo.py
"""
from zipline import TradingAlgorithm
from zipline.data.data_portal import DataPortal
from zipline.finance.trading import TradingEnvironment
from zipline.utils.factory import create_simulation_parameters
from zipline.utils.calendars import get_calendar
from zipline.pipeline.loaders import USEquityPricingLoader
from zipline.pipeline.data.equity_pricing import USEquityPricing
from zipline.data.bundles.core import load
from zipline.api import symbol, order  # used in handle_data

import os
import re
from time import time
import pandas as pd

CAPITAL_BASE = 1.0e6


def makeTS(date_str):
    """creates a Pandas DT object from a string"""
    return pd.Timestamp(date_str, tz='utc')


def parse_sqlite_connstr(db_URL):
    """parses out the db connection string (needed to make a TradingEnvironment"""
    _, connstr = re.split(r'sqlite:///', str(db_URL), maxsplit=1,)
    return connstr


def make_choose_loader(pl_loader):
    def cl(column):
        if column in USEquityPricing.columns:
            return pipeline_loader
        raise ValueError("No PipelineLoader registered for column %s." % column)
    return cl


if __name__ == '__main__':

    # load the bundle
    bundle_data = load('quantopian-quandl', os.environ, None)
    cal = bundle_data.equity_daily_bar_reader.trading_calendar.all_sessions
    pipeline_loader = USEquityPricingLoader(bundle_data.equity_daily_bar_reader, bundle_data.adjustment_reader)
    choose_loader = make_choose_loader(pipeline_loader)

    env = TradingEnvironment(asset_db_path=parse_sqlite_connstr(bundle_data.asset_finder.engine.url))

    data = DataPortal(
        env.asset_finder, get_calendar("NYSE"),
        first_trading_day=bundle_data.equity_minute_bar_reader.first_trading_day,
        equity_minute_reader=bundle_data.equity_minute_bar_reader,
        equity_daily_reader=bundle_data.equity_daily_bar_reader,
        adjustment_reader=bundle_data.adjustment_reader,
    )

    start = makeTS("2015-11-01"); end = makeTS("2016-11-01")  # this can go anywhere before the TradingAlgorithm

    def initialize(context):
        pass

    def handle_data(context, data):
        order(symbol('AAPL'), 10)

    # the actual running of the backtest happens in the TradingAlgorithm object
    bt_start = time()
    perf = TradingAlgorithm(
        env=env,
        get_pipeline_loader=choose_loader,
        sim_params=create_simulation_parameters(
            start=start,
            end=end,
            capital_base=CAPITAL_BASE,
            data_frequency='daily',
        ),
        **{
            'initialize': initialize,
            'handle_data': handle_data,
            'before_trading_start': None,
            'analyze': None,
        }
    ).run(data, overwrite_sim_params=False,)
    bt_end = time()

    print perf.columns
    print perf['portfolio_value']

    print "The backtest took %0.2f seconds to run." % (bt_end - bt_start)
    print("all done boss")

On Thursday, February 16, 2017 at 1:31:30 PM UTC-8, Ed Bartosh wrote:

Hi Scott,

Thank you for the info! It helped a lot!

There is one piece of similar work I've found on github: https://github.com/CarterBain/AlephNull

As far as I can see they modified blotter module. Can you tell if this can be used as a starting point or or it's better to start from scratch?

I understand that the code there is outdated and zipline code changed a lot since then, but still it would be nice if it can be used.

Is there any documentation about Blotter and DataPortal class? Can you provide more details on how you'd implement this?

I've seen a lot of requests on this list to implement brokerage interface. If people are still interested it would be a nice community project to implement this. Anyone is interested to collaborate on this?

Regards,

Ed

 

2017-02-14 16:18 GMT+00:00 Scott Sanderson <ssand...@quantopian.com>:

Hi Ed,

It is **possible** to use Zipline for live trading (and we do indeed use Zipline as the foundation for live trading on Quantopian), but the line you mentioned from the README is probably no longer an accurate reflection of how easy it is to use Zipline for live trading: running a Zipline live algorithm currently requires a fair amount of custom work.  I wouldn't recommend taking on that task unless you're willing to dig into the Zipline source and write some nontrivial extensions.

The main two classes that are relevant to live trading are the Blotter class, which manages placing and filling orders, and the DataPortal class, which provides access to minutely and daily pricing data to various subsystems of Zipline.  Running a Zipline algorithm live essentially requires creating custom subclasses of Blotter and DataPortal.  The Blotter subclass would be responsible for communicating with an brokerage API like the one provided by Interactive Brokers, and the DataPortal subclass would be responsible for providing access to up-to-the-minute pricing data (potentially with fallback to static historical data for history() calls).

In the long term, we'd for it to be easier for people to make these kinds of nontrivial extensions to Zipline (and as an engineer and user of open source software this is a frustrating reply for me to write), but it's hard work to prioritize against all the other things we work on to keep the business running.

Best,

-Scott

On Sunday, February 12, 2017 at 6:12:41 AM UTC-5, Ed Bartosh wrote:

Hi,

Does anybody use zipline for live trading? Any ideas where to look in the code to implement this. Any points to the zipline code that supports this? 

It should be possible as Quantopian uses zipline for live trading. However, I was not able to find and points in the documentation regarding this except of the description: "Zipline is a Pythonic algorithmic trading library. It is an event-driven system that supports both backtesting and live-trading."

--

BR,
Ed

--
You received this message because you are subscribed to the Google Groups "Zipline Python Opensource Backtester" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zipline+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[Giulio]

Great work guys, 

zipline-live is something that was really missing, 

I see in the repo that the branch has diverged quite substantially from the original repo from quantopian. 

Any plans to reconcile the code? I think a git rebase shouldn't give too many troubles, there's just a few conflicts. 

Thanks, 

Giulio

To unsubscribe from this group and all its topics, send an email to zipline+u...@googlegroups.com.

[Peter Bakker]

We have done a full reconciliation and are busy with live testing. The rebase was quite a bit of work as a lot of internals needed to be redone. We will be following all Q releases.

If you want to see the work go to : https://github.com/shlomikushchi/zipline/releases

When its all OK we will merge it with zipline-live

cheers, Peter

[Martin Frstr]

Hi Peter,

thanks for the information. Is there any place, where the progress can be tracked?

[Peter Bakker]

I cannot integrate the source in the main repo so watching the releases on Shlomi's repo is best.

Peter

'Martin Frstr' via Zipline Python Opensource Backtester wrote on 14/12/18 07:09:

--

Sent from Postbox

[ajjcoppola]

Peter,

Trying to get the Shlomi release to work, (installed using "python -m pip install -e .[ib]" inside the release,  and keep getting the following error...any ideas???

Also, I had to add a start date to get this far.

Any ideas?

alan

(env_hca_live) ubuntu@ip-172-30-2-125:~/hca/zipline-live-algos$ zipline run -f ~/hca/zipline-live-algos/demo.py --state-file ~/hca/zipline-live-algos/demo.state --realtime-bar-target ~/hca/zipline-live-algos/realtime-bars/ --broker ib --broker-uri localhost:4001:1232 --bundle quantopian-quandl --data-frequency minute --start '2019-02-15'

[2019-02-15 21:29:58.391641] INFO: IB Broker: Connecting: localhost:4001:1232

Server Version: 76

TWS Time at connection:20190215 21:29:57 UTC

[2019-02-15 21:29:58.477454] INFO: IB Broker: [2104] Market data farm connection is OK:usfarm.nj (-1)

[2019-02-15 21:29:58.477693] INFO: IB Broker: [2104] Market data farm connection is OK:cashfarm (-1)

[2019-02-15 21:29:58.477861] INFO: IB Broker: [2104] Market data farm connection is OK:usfarm (-1)

[2019-02-15 21:29:58.478009] INFO: IB Broker: [2106] HMDS data farm connection is OK:euhmds (-1)

[2019-02-15 21:29:58.478155] INFO: IB Broker: [2106] HMDS data farm connection is OK:fundfarm (-1)

[2019-02-15 21:29:58.478296] INFO: IB Broker: [2106] HMDS data farm connection is OK:ushmds (-1)

[2019-02-15 21:29:58.478379] DEBUG: IB Broker: Execution details completed for request 0

[2019-02-15 21:29:58.699393] INFO: IB Broker: Managed accounts: ['DU********']

[2019-02-15 21:29:58.799943] INFO: IB Broker: Local-Broker Time Skew: 0 days 00:00:01

[2019-02-15 21:29:59.143852] WARNING: Loader: Refusing to download new benchmark data because a download succeeded at 2019-02-15 20:41:17.372976+00:00.

[2019-02-15 21:29:59.174610] WARNING: Loader: Refusing to download new treasury data because a download succeeded at 2019-02-15 20:41:19.812946+00:00.

[2019-02-15 21:29:59.213635] INFO: Blotter Live: Initialized blotter_live

[2019-02-15 21:29:59.214509] INFO: Live Trading: initialization done

Traceback (most recent call last):

  File "/home/ubuntu/anaconda3/envs/env_hca_live/bin/zipline", line 11, in <module>

    load_entry_point('zipline-live', 'console_scripts', 'zipline')()

  File "/home/ubuntu/anaconda3/envs/env_hca_live/lib/python3.6/site-packages/click/core.py", line 664, in __call__

    return self.main(*args, **kwargs)

  File "/home/ubuntu/anaconda3/envs/env_hca_live/lib/python3.6/site-packages/click/core.py", line 644, in main

    rv = self.invoke(ctx)

  File "/home/ubuntu/anaconda3/envs/env_hca_live/lib/python3.6/site-packages/click/core.py", line 991, in invoke

    return _process_result(sub_ctx.command.invoke(sub_ctx))

  File "/home/ubuntu/anaconda3/envs/env_hca_live/lib/python3.6/site-packages/click/core.py", line 837, in invoke

    return ctx.invoke(self.callback, **ctx.params)

  File "/home/ubuntu/anaconda3/envs/env_hca_live/lib/python3.6/site-packages/click/core.py", line 464, in invoke

    return callback(*args, **kwargs)

  File "/home/ubuntu/hca/zipline/zipline/__main__.py", line 113, in _

    return f(*args, **kwargs)

  File "/home/ubuntu/hca/zipline/zipline/__main__.py", line 357, in run

    stop_execution_callback=None

  File "/home/ubuntu/hca/zipline/zipline/utils/run_algo.py", line 247, in _run

    'script': algotext,

  File "/home/ubuntu/hca/zipline/zipline/algorithm_live.py", line 256, in run

    daily_stats = super(self.__class__, self).run(*args, **kwargs)

  File "/home/ubuntu/hca/zipline/zipline/algorithm.py", line 652, in run

    for perf in self.get_generator():

  File "/home/ubuntu/hca/zipline/zipline/algorithm.py", line 628, in get_generator

    return self._create_generator(self.sim_params)

  File "/home/ubuntu/hca/zipline/zipline/algorithm_live.py", line 147, in _create_generator

    TradingAlgorithm._create_generator(self, self.sim_params)

  File "/home/ubuntu/hca/zipline/zipline/algorithm.py", line 595, in _create_generator

    self._create_clock(),

  File "/home/ubuntu/hca/zipline/zipline/algorithm_live.py", line 140, in _create_clock

    execution_id=self.sim_params._execution_id,

AttributeError: 'SimulationParameters' object has no attribute '_execution_id'