Can use pipeline with bar data from yahoo in zipline evn ?

[Kang Hua]

HI: 

     I try manually run zipline algo  - date from load_bars_from_yahoo , pipeline with dataframeloader . but failed 

     below is my code ,very simple ,    it seem get_pipeline_loader not use right !  ( customfactor compute function receive nan input  -- print("---",today,assets,highs,lows)     #highs ,lows is nan)

     please give me some advices . thansk 

from zipline import TradingAlgorithm

from zipline.api import (

    attach_pipeline,

    date_rules,

    order_target_percent,

    pipeline_output,

    record,

    schedule_function,

    symbol,

)

from zipline.pipeline import Pipeline

from zipline.pipeline.factors import RSI

from zipline.pipeline.data import USEquityPricing

from zipline.pipeline.data import Column  

from zipline.pipeline.data import DataSet

from zipline.pipeline.engine import SimplePipelineEngine  

from zipline.pipeline.loaders.frame import DataFrameLoader  

from   itertools import chain

import numpy as np

#import pandas.io.data as web

import pandas_datareader.data as web

from   pandas.stats.api import ols

import pandas as pd

import math

import pytz

from datetime import timedelta, date, datetime

dates = pd.date_range('2014-01-01', '2014-01-31')

my_stock = ['AMD', 'CERN']

test_start = datetime(2014, 1, 1, 0, 0, 0, 0, pytz.utc)

test_end   = datetime(2014, 1, 31, 0, 0, 0, 0, pytz.utc)

  

from zipline.utils.factory import load_bars_from_yahoo

panel = load_bars_from_yahoo(stocks=my_stock, start=test_start, end=test_end)  

loaders = {  

    USEquityPricing.open: DataFrameLoader(USEquityPricing.open, panel.minor_xs('open')),  

    USEquityPricing.high: DataFrameLoader(USEquityPricing.high, panel.minor_xs('high')),  

    USEquityPricing.low: DataFrameLoader(USEquityPricing.low, panel.minor_xs('low')),  

    USEquityPricing.close: DataFrameLoader(USEquityPricing.close, panel.minor_xs('close')),  

    USEquityPricing.volume: DataFrameLoader(USEquityPricing.volume, panel.minor_xs('volume'))  

}

def my_dispatcher(column):  

    return loaders[column]  

    

class TenDayRange(CustomFactor):

    inputs = [USEquityPricing.high, USEquityPricing.low]

    window_length = 2

    def compute(self, today, assets, out, highs, lows):

        print("---",today,assets,highs,lows)     #highs ,lows is nan

        from numpy import nanmin, nanmax

        highest_highs = nanmax(highs, axis=0)

        lowest_lows = nanmin(lows, axis=0)

        out[:] = highest_highs - lowest_lows

def make_pipeline():

    tdr = TenDayRange()

    return Pipeline(

        columns={

            'longs': tdr.top(2),

            'shorts': tdr.bottom(2),

        },

    )

def rebalance(context, data):

    # Pipeline data will be a dataframe with boolean columns named 'longs' and

    # 'shorts'.

    pipeline_data = context.pipeline_data

    all_assets = pipeline_data.index

    print("-----------------",pipeline_data,all_assets)

def initialize(context):

    attach_pipeline(make_pipeline(), 'my_pipeline')

    schedule_function(rebalance, date_rules.every_day())

def handle_data(context, data):

    pass

def before_trading_start(context, data):

    context.pipeline_data = pipeline_output('my_pipeline')

    pass

if __name__ == '__main__':         

    

    

    algo = TradingAlgorithm(initialize=initialize, 

                            handle_data=handle_data,

                            before_trading_start=before_trading_start,

                            get_pipeline_loader=my_dispatcher,

                            capital_base = 100000)

    results = algo.run(panel)

[Ed Bartosh]

Hi Kang,

Loading data with  load_bars_from_yahoo is outdated approach. It's better to use yahoo bundle instead. you can read about it here: http://www.zipline.io/bundles.html#yahoo-bundle-factories

Here is your stripped down and cleaned code that works with yahoo-bundle:

from zipline.api import attach_pipeline, pipeline_output

from zipline.api import date_rules, schedule_function, get_datetime

from zipline.pipeline import Pipeline

from zipline.pipeline.factors import CustomFactor

from zipline.pipeline.data import USEquityPricing

class TenDayRange(CustomFactor):

    inputs = [USEquityPricing.high, USEquityPricing.low]

    window_length = 2

    def compute(self, today, assets, out, highs, lows):

        #print("---", today, assets, highs, lows)  #highs ,lows is nan

        from numpy import nanmin, nanmax

        highest_highs = nanmax(highs, axis=0)

        lowest_lows = nanmin(lows, axis=0)

        out[:] = highest_highs - lowest_lows

def make_pipeline():

    tdr = TenDayRange()

    return Pipeline(

        columns={

            'longs': tdr.top(2),

            'shorts': tdr.bottom(2),

        },

    )

def initialize(context):

    attach_pipeline(make_pipeline(), 'my_pipeline')

    schedule_function(rebalance, date_rules.every_day())

def before_trading_start(context, data):

    context.pipeline_data = pipeline_output('my_pipeline')

def rebalance(context, data):

    # Pipeline data will be a dataframe with boolean columns named 'longs' and

    # 'shorts'.

    pipeline_data = context.pipeline_data

    print get_datetime()

    for asset in pipeline_data.index:

        print '   ', asset.symbol, data.current(asset, 'price'), pipeline_data['longs'][asset], pipeline_data['shorts'][asset]

You can run it this way:

$ zipline run -f test6.py -s 2014-01-01 -e 2014-01-31 -b yahoo-bundle -o /dev/null

2014-01-02 21:00:00+00:00

    AMD 3.95 True True

    CERN 55.07 True True

2014-01-03 21:00:00+00:00

    AMD 4.0 True True

    CERN 54.7 True True

2014-01-06 21:00:00+00:00

    AMD 4.13 True True

    CERN 54.2 True True

2014-01-07 21:00:00+00:00

    AMD 4.18 True True

    CERN 55.259 True True

2014-01-08 21:00:00+00:00

    AMD 4.18 True True

    CERN 54.32 True True

2014-01-09 21:00:00+00:00

    AMD 4.09 True True

    CERN 55.02 True True

2014-01-10 21:00:00+00:00

    AMD 4.17 True True

    CERN 55.18 True True

2014-01-13 21:00:00+00:00

    AMD 4.13 True True

    CERN 53.83 True True

2014-01-14 21:00:00+00:00

    AMD 4.3 True True

    CERN 55.07 True True

2014-01-15 21:00:00+00:00

    AMD 4.47 True True

    CERN 55.52 True True

2014-01-16 21:00:00+00:00

    AMD 4.38 True True

    CERN 55.389 True True

2014-01-17 21:00:00+00:00

    AMD 4.18 True True

    CERN 55.74 True True

2014-01-21 21:00:00+00:00

    AMD 4.17 True True

    CERN 55.9 True True

2014-01-22 21:00:00+00:00

    AMD 3.67 True True

    CERN 57.02 True True

2014-01-23 21:00:00+00:00

    AMD 3.62 True True

    CERN 56.49 True True

2014-01-24 21:00:00+00:00

    AMD 3.47 True True

    CERN 54.709 True True

2014-01-27 21:00:00+00:00

    AMD 3.41 True True

    CERN 54.139 True True

2014-01-28 21:00:00+00:00

    AMD 3.54 True True

    CERN 54.9 True True

2014-01-29 21:00:00+00:00

    AMD 3.48 True True

    CERN 54.049 True True

2014-01-30 21:00:00+00:00

    AMD 3.48 True True

    CERN 56.27 True True

2014-01-31 21:00:00+00:00

    AMD 3.43 True True

    CERN 56.889 True True

[2017-04-16 19:42:54.778222] INFO: Performance: Simulated 21 trading days out of 21.

[2017-04-16 19:42:54.778380] INFO: Performance: first open: 2014-01-02 14:31:00+00:00

[2017-04-16 19:42:54.778475] INFO: Performance: last close: 2014-01-31 21:00:00+00:00 

Regards,

Ed

--
You received this message because you are subscribed to the Google Groups "Zipline Python Opensource Backtester" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[Kang Hua]

thanks your reply . with yahoo bundle , I can do same thing . 

but with get_pipeline_loader , we can not do it ?

在 2017年4月17日星期一 UTC+8上午3:43:48，Ed Bartosh写道：

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[Ed Bartosh]

Hi,

> but with get_pipeline_loader , we can not do it ?

Theoretically we can, but why would we need that?

BTW, you code runs just fine if I add import for CustomFactor. The USEquityPricing data is empty, but this is another story.

Regards,

Ed

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[Kang Hua]

HI
    command - zipline ingest -b xxx (include many symbols)  can skip error stock data  ,and continue to run ?


在 2017年4月17日星期一 UTC+8下午5:23:10，Ed Bartosh写道：

[fva...@quantopian.com]

Hi Kang,

I'm not sure I understand your problem in your most recent post.

If you run zipline ingest or zipline ingest -b quantopian-quandl you'll download a bundle with certain stocks.

You can also create a custom yahoo bundle in which you choose the equity data you'd like to download

[Kang Hua]

HI fva:

    thanks your reply 

    My mean is :Yahoo service not stable ,some stock data can not download sometimes ! so hopet to zipline ingest can skip these error

   A new question : 

   can i implement customfactor like marketcap ，but it's input from custom fundamental data(eg. for china exchange) not using morningstar. I can use globe data for this goal, but it is not elegance.

   fundamental data can not use ingest bundle way ?

class MarketCap(CustomFactor):

    inputs= [morningstar.valuation.market_cap]

    window_length= 1

    def compute(self, today, assets, out, market_cap):

         out[:] = market_cap 

在 2017年4月23日星期日 UTC+8下午12:51:38，fva...@quantopian.com写道：