Error trying to load bundle data

Dave Gilbert

unread,

Oct 27, 2016, 7:28:41 AM10/27/16

to Zipline Python Opensource Backtester

I have registered and successfully ingested a data bundle, but can't load the bundle data :

from zipline.data.bundles import load
bundle_data = load('etf-bundle')

result is

---------------------------------------------------------------------------
InvalidRequestError                       Traceback (most recent call last)
<ipython-input-28-f920ad063485> in <module>()
      1 from zipline.data.bundles import load
----> 2 bundle_data = load('etf-bundle')

G:\Anaconda3\lib\site-packages\zipline\data\bundles\core.py in load(name, environ, timestamp)
    514         return BundleData(
    515             asset_finder=AssetFinder(
--> 516                 asset_db_path(name, timestr, environ=environ),
    517             ),
    518             equity_minute_bar_reader=BcolzMinuteBarReader(

G:\Anaconda3\lib\site-packages\zipline\assets\assets.py in __init__(self, engine)
    146     def __init__(self, engine):
    147         self.engine = engine
--> 148         metadata = sa.MetaData(bind=engine)
    149         metadata.reflect(only=asset_db_table_names)
    150         for table_name in asset_db_table_names:

G:\Anaconda3\lib\site-packages\zipline\assets\assets.py in __init__(self, engine)
    147         self.engine = engine
    148         metadata = sa.MetaData(bind=engine)
--> 149         metadata.reflect(only=asset_db_table_names)
    150         for table_name in asset_db_table_names:
    151             setattr(self, table_name, metadata.tables[table_name])

G:\Anaconda3\lib\site-packages\sqlalchemy\sql\schema.py in reflect(self, bind, schema, views, only, extend_existing, autoload_replace, **dialect_kwargs)
   3648                         'Could not reflect: requested table(s) not available '
   3649                         'in %s%s: (%s)' %
-> 3650                         (bind.engine.url, s, ', '.join(missing)))
   3651                 load = [name for name in only if extend_existing or
   3652                         name not in current]

InvalidRequestError: Could not reflect: requested table(s) not available in sqlite:///C:\Users\scuba/.zipline\data\etf-bundle\2016-10-27T11;16;23.777396\assets-5.sqlite: (futures_exchanges, futures_contracts, equity_symbol_mappings, futures_root_symbols, equities, asset_router, version_info)

Can someone hep please?

Ed Bartosh

unread,

Oct 27, 2016, 8:16:42 AM10/27/16

to Dave Gilbert, Zipline Python Opensource Backtester

Hi Dave,

How did you ingest your etf-bundle?

Regards,

Ed

--
You received this message because you are subscribed to the Google Groups "Zipline Python Opensource Backtester" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

Dave Gilbert

unread,

Oct 27, 2016, 8:29:28 AM10/27/16

to Ed Bartosh, Zipline Python Opensource Backtester

zipline ingest -b etf-bundle

after registering the bundle via editing .zipline\extension.py

Rgds, Dave

Ed Bartosh

unread,

Oct 27, 2016, 8:38:06 AM10/27/16

to Dave Gilbert, Zipline Python Opensource Backtester

You're using custom bundle, right?

Try to do the same with any of zipline bundles (quandl, yahoo). If it works then the problem is in the code of your custom bundle.

--

BR,
Ed

Dave Gilbert

unread,

Oct 27, 2016, 8:46:11 AM10/27/16

to Ed Bartosh, Zipline Python Opensource Backtester

no I'm using the Yahoo bundle factory. I've used it successfully for a small list of securities. This bundle has 327 etfs. There were no errors either registering or ingesting the bundle.

from zipline.data.bundles.yahoo import yahoo_equities

equities = {'VHT', 'IEO', 'VFH', 'IYJ', 'GWL', 'CWB', 'VTWO', 'BSCI', 'UNL', 'IEZ', 'FXY', 'XLV', 'PSCC', 'TLO', 'INKM', 'SCHF', 'OUNZ', 'IYH', 'KBWB', 'XLI', 'SST', 'IGV', 'VBK', 'BZF', 'SCHB', 'IEF', 'IYW', 'IWO', 'SCHP', 'AMU', 'EFA', 'AMLP', 'PSCT', 'IWB', 'VNQ', 'OIL', 'GSG', 'SBIO', 'XTN', 'EWI', 'WDTI', 'AFK', 'PSCU', 'SCHZ', 'ITA', 'FHLC', 'XLP', 'IYR', 'ITE', 'XLK', 'VO', 'IDU', 'IWN', 'EWP', 'MCHI', 'EWQ', 'PSCD', 'BNO', 'OILX', 'DCNG', 'EIRL', 'FLRT', 'CEW', 'IHE', 'SCHO', 'EPU', 'BNDS', 'ONEK', 'FYT', 'USV', 'IPE', 'TWOK', 'FXCH', 'FXS', 'RWR', 'PKB', 'PSCH', 'EWY', 'GSP', 'BKLN', 'XES', 'VDE', 'VONE', 'CNCR', 'SLV', 'FXA', 'SLQD', 'PXQ', 'FMAT', 'FIDU', 'JNK', 'SCHH', 'VEA', 'PBJ', 'FNY', 'EPRF', 'CSJ', 'DIA', 'GYLD', 'UUP', 'BCM', 'SPY', 'JKK', 'TUZ', 'XRT', 'PXMV', 'FXC', 'SCHA', 'PSCI', 'HYG', 'PSCF', 'IWF', 'USAG', 'EWS', 'IYE', 'VTV', 'GREK', 'JKE', 'FTLS', 'IYF', 'IXJ', 'XHB', 'RFG', 'DBS', 'FDIS', 'EEM', 'HYXE', 'PBS', 'JKL', 'PSK', 'CVY', 'IVOO', 'IYZ', 'FENY', 'PJP', 'VCSH', 'VTHR', 'EZU', 'IEFA', 'TIPZ', 'SNLN', 'XHE', 'VIXY', 'AOM', 'PSCE', 'IAI', 'FNK', 'MDY', 'XLU', 'VV', 'IEMG', 'UDN', 'IWP', 'AGG', 'FNCL', 'ICVT', 'ARKG', 'IVV', 'EWW', 'IWV', 'DWIN', 'GAZ', 'VTI', 'XTL', 'IEI', 'VB', 'VTWV', 'SCHM', 'XLFS', 'IWS', 'VTWG', 'FTA', 'PGX', 'CUPM', 'CPER', 'JJC', 'SLY', 'PFXF', 'VBR', 'REZ', 'QAI', 'FSTA', 'YLD', 'IWD', 'FXE', 'IYC', 'WBIH', 'IWR', 'EWC', 'AGF', 'DBA', 'CMBS', 'GSC', 'IHF', 'XLE', 'ICN', 'MGK', 'XHS', 'EWU', 'DIRT', 'VOX', 'XME', 'MNA', 'SCHE', 'TIP', 'PNQI', 'CSCR', 'XLF', 'IJK', 'CYB', 'IJR', 'GMM', 'VPU', 'MLPI', 'MLPB', 'EDV', 'PEJ', 'VIS', 'XSD', 'AMJ', 'PSCM', 'PWV', 'PGF', 'SGOL', 'SCPB', 'QQQ', 'ITB', 'EZA', 'EWA', 'VCR', 'LNGR', 'IJJ', 'UNG', 'FYC', 'THRK', 'WBII', 'AOK', 'PFF', 'AMUB', 'PPA', 'BSCH', 'VAW', 'USO', 'SOXX', 'GAL', 'PMR', 'IYM', 'PXE', 'PSI', 'KRE', 'FTEC', 'FUTY', 'VOO', 'EWG', 'TLH', 'BND', 'IYK', 'FDN', 'VONG', 'ZROZ', 'XAR', 'PCEF', 'IYT', 'EGPT', 'ERUS', 'IAU', 'VIXM', 'XLRE', 'KCE', 'TLT', 'YYY', 'EWM', 'USCI', 'AOA', 'KBE', 'EIS', 'SCHX', 'XLY', 'IYLD', 'MDIV', 'FXF', 'PXSV', 'IJH', 'XPH', 'JKI', 'XWEB', 'SCHR', 'SCHV', 'DBC', 'VWO', 'SHY', 'VOE', 'BKF', 'SIVR', 'PXJ', 'INDA', 'VGT', 'VUG', 'IJT', 'OIIL', 'GLD', 'COMT', 'IAT', 'DBEF', 'XOP', 'VDC', 'IYG', 'FREL', 'KBWR', 'EWJ', 'SCHG', 'JKH', 'IHI', 'SLVO', 'QABA', 'FCOM', 'FXB', 'FXSG', 'ITOT', 'XLB', 'VOT', 'SBV', 'MGV', 'IWM', 'AOR', 'IBB'}

register(

'etf_bundle',

yahoo_equities(equities),

)

Dave

Ed Bartosh

unread,

Oct 27, 2016, 9:12:08 AM10/27/16

to Dave Gilbert, Zipline Python Opensource Backtester

Then you should register your bundle first. Python doesn't know anything about ~/.zipline/extensions.py, I'm afraid.

This should work theoretically:

from zipline.data.bundles.yahoo import register, load, yahoo_equities

equities = {'VHT', 'IEO', 'VFH', 'IYJ', 'GWL', 'CWB', 'VTWO', 'BSCI', 'UNL', 'IEZ', 'FXY', 'XLV', 'PSCC', 'TLO', 'INKM', 'SCHF', 'OUNZ', 'IYH', 'KBWB', 'XLI', 'SST', 'IGV', 'VBK', 'BZF', 'SCHB', 'IEF', 'IYW', 'IWO', 'SCHP', 'AMU', 'EFA', 'AMLP', 'PSCT', 'IWB', 'VNQ', 'OIL', 'GSG', 'SBIO', 'XTN', 'EWI', 'WDTI', 'AFK', 'PSCU', 'SCHZ', 'ITA', 'FHLC', 'XLP', 'IYR', 'ITE', 'XLK', 'VO', 'IDU', 'IWN', 'EWP', 'MCHI', 'EWQ', 'PSCD', 'BNO', 'OILX', 'DCNG', 'EIRL', 'FLRT', 'CEW', 'IHE', 'SCHO', 'EPU', 'BNDS', 'ONEK', 'FYT', 'USV', 'IPE', 'TWOK', 'FXCH', 'FXS', 'RWR', 'PKB', 'PSCH', 'EWY', 'GSP', 'BKLN', 'XES', 'VDE', 'VONE', 'CNCR', 'SLV', 'FXA', 'SLQD', 'PXQ', 'FMAT', 'FIDU', 'JNK', 'SCHH', 'VEA', 'PBJ', 'FNY', 'EPRF', 'CSJ', 'DIA', 'GYLD', 'UUP', 'BCM', 'SPY', 'JKK', 'TUZ', 'XRT', 'PXMV', 'FXC', 'SCHA', 'PSCI', 'HYG', 'PSCF', 'IWF', 'USAG', 'EWS', 'IYE', 'VTV', 'GREK', 'JKE', 'FTLS', 'IYF', 'IXJ', 'XHB', 'RFG', 'DBS', 'FDIS', 'EEM', 'HYXE', 'PBS', 'JKL', 'PSK', 'CVY', 'IVOO', 'IYZ', 'FENY', 'PJP', 'VCSH', 'VTHR', 'EZU', 'IEFA', 'TIPZ', 'SNLN', 'XHE', 'VIXY', 'AOM', 'PSCE', 'IAI', 'FNK', 'MDY', 'XLU', 'VV', 'IEMG', 'UDN', 'IWP', 'AGG', 'FNCL', 'ICVT', 'ARKG', 'IVV', 'EWW', 'IWV', 'DWIN', 'GAZ', 'VTI', 'XTL', 'IEI', 'VB', 'VTWV', 'SCHM', 'XLFS', 'IWS', 'VTWG', 'FTA', 'PGX', 'CUPM', 'CPER', 'JJC', 'SLY', 'PFXF', 'VBR', 'REZ', 'QAI', 'FSTA', 'YLD', 'IWD', 'FXE', 'IYC', 'WBIH', 'IWR', 'EWC', 'AGF', 'DBA', 'CMBS', 'GSC', 'IHF', 'XLE', 'ICN', 'MGK', 'XHS', 'EWU', 'DIRT', 'VOX', 'XME', 'MNA', 'SCHE', 'TIP', 'PNQI', 'CSCR', 'XLF', 'IJK', 'CYB', 'IJR', 'GMM', 'VPU', 'MLPI', 'MLPB', 'EDV', 'PEJ', 'VIS', 'XSD', 'AMJ', 'PSCM', 'PWV', 'PGF', 'SGOL', 'SCPB', 'QQQ', 'ITB', 'EZA', 'EWA', 'VCR', 'LNGR', 'IJJ', 'UNG', 'FYC', 'THRK', 'WBII', 'AOK', 'PFF', 'AMUB', 'PPA', 'BSCH', 'VAW', 'USO', 'SOXX', 'GAL', 'PMR', 'IYM', 'PXE', 'PSI', 'KRE', 'FTEC', 'FUTY', 'VOO', 'EWG', 'TLH', 'BND', 'IYK', 'FDN', 'VONG', 'ZROZ', 'XAR', 'PCEF', 'IYT', 'EGPT', 'ERUS', 'IAU', 'VIXM', 'XLRE', 'KCE', 'TLT', 'YYY', 'EWM', 'USCI', 'AOA', 'KBE', 'EIS', 'SCHX', 'XLY', 'IYLD', 'MDIV', 'FXF', 'PXSV', 'IJH', 'XPH', 'JKI', 'XWEB', 'SCHR', 'SCHV', 'DBC', 'VWO', 'SHY', 'VOE', 'BKF', 'SIVR', 'PXJ', 'INDA', 'VGT', 'VUG', 'IJT', 'OIIL', 'GLD', 'COMT', 'IAT', 'DBEF', 'XOP', 'VDC', 'IYG', 'FREL', 'KBWR', 'EWJ', 'SCHG', 'JKH', 'IHI', 'SLVO', 'QABA', 'FCOM', 'FXB', 'FXSG', 'ITOT', 'XLB', 'VOT', 'SBV', 'MGV', 'IWM', 'AOR', 'IBB'}

register('etf_bundle', yahoo_equities(equities))

bundle = load('etf_bundle')

Regards,

Ed

Dave Gilbert

unread,

Oct 27, 2016, 10:02:43 AM10/27/16

to Ed Bartosh, Zipline Python Opensource Backtester

On Thu, Oct 27, 2016 at 3:12 PM, Ed Bartosh <bar...@gmail.com> wrote:

from

zipline.data.bundles.yahoo import register, load, yahoo_equities

equities = {'VHT', 'IEO', 'VFH', 'IYJ', 'GWL', 'CWB', 'VTWO', 'BSCI', 'UNL', 'IEZ', 'FXY', 'XLV', 'PSCC', 'TLO', 'INKM', 'SCHF', 'OUNZ', 'IYH', 'KBWB', 'XLI', 'SST', 'IGV', 'VBK', 'BZF', 'SCHB', 'IEF', 'IYW', 'IWO', 'SCHP', 'AMU', 'EFA', 'AMLP', 'PSCT', 'IWB', 'VNQ', 'OIL', 'GSG', 'SBIO', 'XTN', 'EWI', 'WDTI', 'AFK', 'PSCU', 'SCHZ', 'ITA', 'FHLC', 'XLP', 'IYR', 'ITE', 'XLK', 'VO', 'IDU', 'IWN', 'EWP', 'MCHI', 'EWQ', 'PSCD', 'BNO', 'OILX', 'DCNG', 'EIRL', 'FLRT', 'CEW', 'IHE', 'SCHO', 'EPU', 'BNDS', 'ONEK', 'FYT', 'USV', 'IPE', 'TWOK', 'FXCH', 'FXS', 'RWR', 'PKB', 'PSCH', 'EWY', 'GSP', 'BKLN', 'XES', 'VDE', 'VONE', 'CNCR', 'SLV', 'FXA', 'SLQD', 'PXQ', 'FMAT', 'FIDU', 'JNK', 'SCHH', 'VEA', 'PBJ', 'FNY', 'EPRF', 'CSJ', 'DIA', 'GYLD', 'UUP', 'BCM', 'SPY', 'JKK', 'TUZ', 'XRT', 'PXMV', 'FXC', 'SCHA', 'PSCI', 'HYG', 'PSCF', 'IWF', 'USAG', 'EWS', 'IYE', 'VTV', 'GREK', 'JKE', 'FTLS', 'IYF', 'IXJ', 'XHB', 'RFG', 'DBS', 'FDIS', 'EEM', 'HYXE', 'PBS', 'JKL', 'PSK', 'CVY', 'IVOO', 'IYZ', 'FENY', 'PJP', 'VCSH', 'VTHR', 'EZU', 'IEFA', 'TIPZ', 'SNLN', 'XHE', 'VIXY', 'AOM', 'PSCE', 'IAI', 'FNK', 'MDY', 'XLU', 'VV', 'IEMG', 'UDN', 'IWP', 'AGG', 'FNCL', 'ICVT', 'ARKG', 'IVV', 'EWW', 'IWV', 'DWIN', 'GAZ', 'VTI', 'XTL', 'IEI', 'VB', 'VTWV', 'SCHM', 'XLFS', 'IWS', 'VTWG', 'FTA', 'PGX', 'CUPM', 'CPER', 'JJC', 'SLY', 'PFXF', 'VBR', 'REZ', 'QAI', 'FSTA', 'YLD', 'IWD', 'FXE', 'IYC', 'WBIH', 'IWR', 'EWC', 'AGF', 'DBA', 'CMBS', 'GSC', 'IHF', 'XLE', 'ICN', 'MGK', 'XHS', 'EWU', 'DIRT', 'VOX', 'XME', 'MNA', 'SCHE', 'TIP', 'PNQI', 'CSCR', 'XLF', 'IJK', 'CYB', 'IJR', 'GMM', 'VPU', 'MLPI', 'MLPB', 'EDV', 'PEJ', 'VIS', 'XSD', 'AMJ', 'PSCM', 'PWV', 'PGF', 'SGOL', 'SCPB', 'QQQ', 'ITB', 'EZA', 'EWA', 'VCR', 'LNGR', 'IJJ', 'UNG', 'FYC', 'THRK', 'WBII', 'AOK', 'PFF', 'AMUB', 'PPA', 'BSCH', 'VAW', 'USO', 'SOXX', 'GAL', 'PMR', 'IYM', 'PXE', 'PSI', 'KRE', 'FTEC', 'FUTY', 'VOO', 'EWG', 'TLH', 'BND', 'IYK', 'FDN', 'VONG', 'ZROZ', 'XAR', 'PCEF', 'IYT', 'EGPT', 'ERUS', 'IAU', 'VIXM', 'XLRE', 'KCE', 'TLT', 'YYY', 'EWM', 'USCI', 'AOA', 'KBE', 'EIS', 'SCHX', 'XLY', 'IYLD', 'MDIV', 'FXF', 'PXSV', 'IJH', 'XPH', 'JKI', 'XWEB', 'SCHR', 'SCHV', 'DBC', 'VWO', 'SHY', 'VOE', 'BKF', 'SIVR', 'PXJ', 'INDA', 'VGT', 'VUG', 'IJT', 'OIIL', 'GLD', 'COMT', 'IAT', 'DBEF', 'XOP', 'VDC', 'IYG', 'FREL', 'KBWR', 'EWJ', 'SCHG', 'JKH', 'IHI', 'SLVO', 'QABA', 'FCOM', 'FXB', 'FXSG', 'ITOT', 'XLB', 'VOT', 'SBV', 'MGV', 'IWM', 'AOR', 'IBB'}

register('etf_bundle', yahoo_equities(equities))
bundle = load('etf_bundle')

thanks, this works with one small change : need to import load from zipline.data.bundles (not from

zipline.data.bundles.yahoo).

When I try to use the data I'm getting "symbol not found" error for quite a few - only thing I've noticed is that the symbols are 4 rather than 3 characters (eg 'MLPB','OIIL','LNGR','OILX','XWEB','XLFS','ICVT','CNCR','DWIN','FREL',). Quite painstaking to find them all one at a time. Is there a way of easily getting all the symbols from the bundle object?

Rgds,

Dave

Dave Gilbert

unread,

Oct 27, 2016, 10:40:27 AM10/27/16

to Ed Bartosh, Zipline Python Opensource Backtester

Ed,

so on further investigation, it seems that 'symbol not found' occurs if the start date of the simulation in which I'm using the data is earlier than the earliest date for which data for that symbol is available, the error is triggered. In this case, the error message is misleading and, I'd suggest, should rather be a warning rather than a fatal error, otherwise it's really painful to weed out the securities with insufficient data.

Rgds,

Dave

Misc

unread,

Oct 28, 2016, 10:39:48 PM10/28/16

to Zipline Python Opensource Backtester, scub...@gmail.com

Hi,

I've ingested the bundle successfully and it appears in the command line and i can see it the file manager.

for some reason when i try to access the bundle from jupyter notebook it doesnt recognize the bundle and throws the following error: UnknownBundle: No bundle registered with the name 'my-yahoo-equities-bundle2'

It was actually working earlier until my PC did some updates, and now it doesnt work which makes no sense.

thanks,

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
BR,
Ed

Richard P

unread,

Oct 29, 2016, 7:20:22 AM10/29/16

to zip...@googlegroups.com

See if the solution in this post helps.

https://www.quantopian.com/posts/newbie-with-zipline#57debaab153bf1148c000b11

Richard

Dave Gilbert

unread,

Oct 30, 2016, 6:24:25 AM10/30/16

to Zipline Python Opensource Backtester

I'm still struggling with this. I have done the following :-

1. First tried without registering bundle 'fund_bundle' in extension.py

IN:

!zipline bundles

OUT:

etf_bundle 2016-10-30 07:20:30.523369
etf_bundle 2016-10-29 13:44:58.776082
etf_bundle 2016-10-28 05:22:53.139970
quandl 2016-10-25 10:04:06.739478
quantopian-quandl 2016-10-29 13:24:44.019124
yahoo_bundle 2016-10-25 14:05:26.257039
yahoo_bundle 2016-10-25 13:16:19.252465

IN:

funds = set([
        'VTSMX','VTSAX','VFINX','VFIAX','VIVAX','VVIAX','VIGRX','VIGAX','VIMSX','VIMAX','VMGIX',
        'VMGMX','VMVIX','VMVAX','NAESX','VSMAX','VISVX','VSIAX','VISGX','VSGAX','BRSIX','VGSIX',
        'VGSLX','VGTSX','VTIAX','VTMGX','VTMGX','VEIEX','VEMAX','VEURX','VEUSX','VPACX','VPADX',
        'VTRIX','EFV','VFSVX','VBMFX','VBTLX','VFISX','VFIRX','VFITX','VUSTX','VUSUX','VIPSX','VAIPX',
        'VWSTX','VWSUX','VWITX','VWIUX','VWLTX','VWLUX','VICSX','VWESX','VWETX','VWEHX','VWEAX','VFSTX',
        'VFSUX','PIGLX','VGPMX','PCRI'
        ])

from zipline.data.bundles.yahoo import register, yahoo_equities

register('fund_bundle',  yahoo_equities(funds))

OUT:

<function zipline.data.bundles.yahoo.yahoo_equities.<locals>.ingest>

IN:

!zipline bundles

OUT:

etf_bundle 2016-10-30 07:20:30.523369
etf_bundle 2016-10-29 13:44:58.776082
etf_bundle 2016-10-28 05:22:53.139970
quandl 2016-10-25 10:04:06.739478
quantopian-quandl 2016-10-29 13:24:44.019124
yahoo_bundle 2016-10-25 14:05:26.257039
yahoo_bundle 2016-10-25 13:16:19.252465

fund_bundle failed to register...

If I try to load the bundle :

IN:

from zipline.data.bundles import load
bundle_data = load('fund_bundle')

OUT:

---------------------------------------------------------------------------
InvalidRequestError                       Traceback (most recent call last)

<ipython-input-7-266004c14999> in <module>()


      1 from zipline.data.bundles import load

----> 2 bundle_data = load('fund_bundle')



G:\Anaconda3\lib\site-packages\zipline\data\bundles\core.py in

load(name, environ, timestamp)


    514         return BundleData(
    515             asset_finder=AssetFinder(

--> 516                 asset_db_path(name, timestr, environ=environ),
    517             ),
    518             equity_minute_bar_reader=BcolzMinuteBarReader(

G:\Anaconda3\lib\site-packages\zipline\assets\assets.py in

__init__(self, engine)


    146     def __init__(self, engine):
    147         self.engine = engine

--> 148         metadata = sa.MetaData(bind=engine)
    149         metadata.reflect(only=asset_db_table_names)
    150         for table_name in asset_db_table_names:

G:\Anaconda3\lib\site-packages\zipline\assets\assets.py in

__init__(self, engine)


    147         self.engine = engine
    148         metadata = sa.MetaData(bind=engine)

--> 149         metadata.reflect(only=asset_db_table_names)
    150         for table_name in asset_db_table_names:
    151             setattr(self, table_name, metadata.tables[table_name])

G:\Anaconda3\lib\site-packages\sqlalchemy\sql\schema.py in

reflect(self, bind, schema, views, only, extend_existing, autoload_replace, **dialect_kwargs)


   3648                         'Could not reflect: requested table(s) not available '
   3649                         'in %s%s: (%s)' %

-> 3650                         (bind.engine.url, s, ', '.join(missing)))
   3651                 load = [name for name in only if extend_existing or
   3652                         name not in current]

InvalidRequestError: Could not reflect: requested table(s) not available in sqlite:///C:\Users\scuba/.zipline\data\fund_bundle\2016-10-30T09;11;42.667560\assets-5.sqlite: (futures_root_symbols, futures_contracts, equity_symbol_mappings, equities, version_info, futures_exchanges, asset_router)

So this didn't work

2. Start again and try to register via extension.py

added the following to extension.py

funds = set(['VTSMX','VTSAX','VFINX','VFIAX','VIVAX','VVIAX','VIGRX','VIGAX','VIMSX','VIMAX','VMGIX','VMGMX','VMVIX',
	'VMVAX','NAESX','VSMAX','VISVX','VSIAX','VISGX','VSGAX','BRSIX','VGSIX','VGSLX','VGTSX','VTIAX','VTMGX','VTMGX',
	'VEIEX','VEMAX','VEURX','VEUSX','VPACX','VPADX','VTRIX','EFV','VFSVX','VBMFX','VBTLX','VFISX','VFIRX','VFITX','VUSTX'
	,'VUSUX','VIPSX','VAIPX','VWSTX','VWSUX','VWITX','VWIUX','VWLTX','VWLUX','VICSX','VWESX','VWETX','VWEHX','VWEAX',
	'VFSTX','VFSUX','PIGLX','VGPMX','PCRIX'])

register(
    'fund_bundle', 
    yahoo_equities(equities),
)

restarted ipython notebook, then

IN:

!zipline bundles

OUT:

etf_bundle 2016-10-30 07:20:30.523369
etf_bundle 2016-10-29 13:44:58.776082
etf_bundle 2016-10-28 05:22:53.139970
fund_bundle <no ingestions>
quandl 2016-10-25 10:04:06.739478
quantopian-quandl 2016-10-29 13:24:44.019124
yahoo_bundle 2016-10-25 14:05:26.257039
yahoo_bundle 2016-10-25 13:16:19.252465

fund_bundle registered successfully. So far, so good

Ingest fund_bundle:

zipline ingest -b fund_bundle

IN:

etf_bundle 2016-10-30 07:20:30.523369
etf_bundle 2016-10-29 13:44:58.776082
etf_bundle 2016-10-28 05:22:53.139970
fund_bundle 2016-10-30 10:03:42.188459
quandl 2016-10-25 10:04:06.739478
quantopian-quandl 2016-10-29 13:24:44.019124
yahoo_bundle 2016-10-25 14:05:26.257039
yahoo_bundle 2016-10-25 13:16:19.252465

Looking good...

bundle_data = load('fund_bundle')

but

OUT:

---------------------------------------------------------------------------
UnknownBundle                             Traceback (most recent call last)
<ipython-input-2-266004c14999> in <module>()


      1 from zipline.data.bundles import load

----> 2 bundle_data = load('fund_bundle')



G:\Anaconda3\lib\site-packages\zipline\data\bundles\core.py in

load(name, environ, timestamp)
    511         if timestamp is None:
    512             timestamp = pd.Timestamp.utcnow()
--> 513         timestr = most_recent_data(name, timestamp, environ=environ)


    514         return BundleData(
    515             asset_finder=AssetFinder(

G:\Anaconda3\lib\site-packages\zipline\data\bundles\core.py in most_recent_data(bundle_name, timestamp, environ)
    466         """
    467         if bundle_name not in bundles:
--> 468             raise UnknownBundle(bundle_name)
    469 
    470         try:

UnknownBundle: No bundle registered with the name 'fund_bundle'

So I try your "fix"

IN:

funds = set(['VTSMX','VTSAX','VFINX','VFIAX','VIVAX','VVIAX','VIGRX','VIGAX','VIMSX','VIMAX','VMGIX','VMGMX','VMVIX',
	'VMVAX','NAESX','VSMAX','VISVX','VSIAX','VISGX','VSGAX','BRSIX','VGSIX','VGSLX','VGTSX','VTIAX','VTMGX','VTMGX',
	'VEIEX','VEMAX','VEURX','VEUSX','VPACX','VPADX','VTRIX','EFV','VFSVX','VBMFX','VBTLX','VFISX','VFIRX','VFITX','VUSTX'
	,'VUSUX','VIPSX','VAIPX','VWSTX','VWSUX','VWITX','VWIUX','VWLTX','VWLUX','VICSX','VWESX','VWETX','VWEHX','VWEAX',
	'VFSTX','VFSUX','PIGLX','VGPMX','PCRIX'])

register(
    'fund_bundle', 
    yahoo_equities(equities),
)

OUT:

<function zipline.data.bundles.yahoo.yahoo_equities.<locals>.ingest>

IN:

!zipline bundles

OUT:

etf_bundle 2016-10-30 07:20:30.523369
etf_bundle 2016-10-29 13:44:58.776082
etf_bundle 2016-10-28 05:22:53.139970
fund_bundle 2016-10-30 10:03:42.188459
quandl 2016-10-25 10:04:06.739478
quantopian-quandl 2016-10-29 13:24:44.019124
yahoo_bundle 2016-10-25 14:05:26.257039
yahoo_bundle 2016-10-25 13:16:19.252465

Again, looks good, but:

OUT:

---------------------------------------------------------------------------
InvalidRequestError                       Traceback (most recent call last)

<ipython-input-7-266004c14999> in <module>()


      1 from zipline.data.bundles import load

----> 2 bundle_data = load('fund_bundle')



G:\Anaconda3\lib\site-packages\zipline\data\bundles\core.py in

load(name, environ, timestamp)


    514         return BundleData(
    515             asset_finder=AssetFinder(

--> 516                 asset_db_path(name, timestr, environ=environ),
    517             ),
    518             equity_minute_bar_reader=BcolzMinuteBarReader(

G:\Anaconda3\lib\site-packages\zipline\assets\assets.py in

__init__(self, engine)


    146     def __init__(self, engine):
    147         self.engine = engine

--> 148         metadata = sa.MetaData(bind=engine)
    149         metadata.reflect(only=asset_db_table_names)
    150         for table_name in asset_db_table_names:

G:\Anaconda3\lib\site-packages\zipline\assets\assets.py in

__init__(self, engine)


    147         self.engine = engine
    148         metadata = sa.MetaData(bind=engine)

--> 149         metadata.reflect(only=asset_db_table_names)
    150         for table_name in asset_db_table_names:
    151             setattr(self, table_name, metadata.tables[table_name])

G:\Anaconda3\lib\site-packages\sqlalchemy\sql\schema.py in

reflect(self, bind, schema, views, only, extend_existing, autoload_replace, **dialect_kwargs)


   3648                         'Could not reflect: requested table(s) not available '
   3649                         'in %s%s: (%s)' %

-> 3650                         (bind.engine.url, s, ', '.join(missing)))
   3651                 load = [name for name in only if extend_existing or
   3652                         name not in current]

InvalidRequestError: Could not reflect: requested table(s) not available in sqlite:///C:\Users\scuba/.zipline\data\fund_bundle\2016-10-30T10;03;42.188459\assets-5.sqlite: (equities, futures_root_symbols, futures_exchanges, asset_router, equity_symbol_mappings, version_info, futures_contracts)

@Ed, @Richard,

There definitely appears to be a problem with the documented ways of registering, ingesting and using bundles.

Quite possibly, I'm missing something, but please help...

Regards,

Dave

Message has been deleted

Richard P

unread,

Oct 30, 2016, 9:15:09 AM10/30/16

to zip...@googlegroups.com

It's got me stumped. Just curious, if you follow every step on my blog post

http://www.prokopyshen.com/create-custom-zipline-data-bundle

and using only a couple of yahoo fund symbols and a short time period to keeps things simple, and not using a notebook for the moment, do those instructions work for you?

If so, then I would slowly add in more complexity/variation in the direction you want to go to see where this is tilting for you.

Richard

Ed Bartosh

unread,

Oct 30, 2016, 11:32:51 AM10/30/16

to Dave Gilbert, Zipline Python Opensource Backtester

Hi Dave,

I also tried my "fix" again and again it worked just fine for me.

Here is what I did step by step:

1. Prepared ~/.zipline/extension.py with the following content:

from zipline.data.bundles import register, load, yahoo_equities

funds = set(['VTSMX','VTSAX','VFINX','VFIAX','VIVAX','VVIAX','VIGRX','VIGAX','VIMSX','VIMAX','VMGIX','VMGMX','VMVIX',

'VMVAX','NAESX','VSMAX','VISVX','VSIAX','VISGX','VSGAX','BRSIX','VGSIX','VGSLX','VGTSX','VTIAX','VTMGX','VTMGX',

'VEIEX','VEMAX','VEURX','VEUSX','VPACX','VPADX','VTRIX','EFV','VFSVX','VBMFX','VBTLX','VFISX','VFIRX','VFITX','VUSTX'

,'VUSUX','VIPSX','VAIPX','VWSTX','VWSUX','VWITX','VWIUX','VWLTX','VWLUX','VICSX','VWESX','VWETX','VWEHX','VWEAX',

'VFSTX','VFSUX','PIGLX','VGPMX','PCRIX'])

register(

'fund_bundle',

yahoo_equities(funds),

)

2. Injested fund_bundle:

$ zipline ingest -b fund_bundle

Merging daily equity files: [-----------------------------------#] 59] 100%

Merging daily equity files: [####################################]

Downloading Yahoo adjustment data: [####################################] 100%

3. Put the following code into test.py:

#!/usr/bin/env python

from zipline.data.bundles import register, load, yahoo_equities

funds = set(['VTSMX','VTSAX','VFINX','VFIAX','VIVAX','VVIAX','VIGRX','VIGAX','VIMSX','VIMAX','VMGIX','VMGMX','VMVIX',

'VMVAX','NAESX','VSMAX','VISVX','VSIAX','VISGX','VSGAX','BRSIX','VGSIX','VGSLX','VGTSX','VTIAX','VTMGX','VTMGX',

'VEIEX','VEMAX','VEURX','VEUSX','VPACX','VPADX','VTRIX','EFV','VFSVX','VBMFX','VBTLX','VFISX','VFIRX','VFITX','VUSTX'

,'VUSUX','VIPSX','VAIPX','VWSTX','VWSUX','VWITX','VWIUX','VWLTX','VWLUX','VICSX','VWESX','VWETX','VWEHX','VWEAX',

'VFSTX','VFSUX','PIGLX','VGPMX','PCRIX'])

register( 'fund_bundle', yahoo_equities(funds))

bundle_data = load('fund_bundle')

symbols = set(str(asset.symbol) for asset in bundle_data.asset_finder.retrieve_all(bundle_data.asset_finder.equities_sids))

print symbols

print symbols == funds

4. Run it and see expected proof that bundle is loaded:

$ ./test.py

set(['NAESX', 'VFSUX', 'VVIAX', 'VFISX', 'VGSIX', 'VTSAX', 'VMVAX', 'VGPMX', 'VSIAX', 'BRSIX', 'VIMAX', 'VISGX', 'VIGAX', 'VPACX', 'VIPSX', 'VFINX', 'VUSTX', 'VEUSX', 'VEMAX', 'VTSMX', 'VIMSX', 'VWESX', 'VSMAX', 'VWSUX', 'VWEHX', 'PIGLX', 'VMVIX', 'VTRIX', 'VWIUX', 'VWLUX', 'VAIPX', 'VSGAX', 'VICSX', 'VMGMX', 'VFSTX', 'VFIRX', 'VFITX', 'EFV', 'VBTLX', 'VGSLX', 'VUSUX', 'VGTSX', 'VISVX', 'PCRIX', 'VWEAX', 'VTMGX', 'VEURX', 'VWSTX', 'VEIEX', 'VIGRX', 'VIVAX', 'VWETX', 'VPADX', 'VFIAX', 'VMGIX', 'VWLTX', 'VFSVX', 'VTIAX', 'VBMFX', 'VWITX'])

True

5. Checked zipline version:

$ python -c 'import zipline; print zipline.__version__'

1.0.2

I hope it helps

Regards,

Ed

2016-10-30 12:32 GMT+02:00 Dave Gilbert <scub...@gmail.com>:

sorry, end of post got messed up..

OUT:

etf_bundle 2016-10-30 07:20:30.523369
etf_bundle 2016-10-29 13:44:58.776082
etf_bundle 2016-10-28 05:22:53.139970
fund_bundle 2016-10-30 10:03:42.188459
quandl 2016-10-25 10:04:06.739478
quantopian-quandl 2016-10-29 13:24:44.019124
yahoo_bundle 2016-10-25 14:05:26.257039
yahoo_bundle 2016-10-25 13:16:19.252465

BUT..

IN:

bundle_data = load('fund_bundle')

OUT:

---------------------------------------------------------------------------


InvalidRequestError                       Traceback (most recent call last)

<ipython-input-7-266004c14999> in

<module>()
      1 from zipline.data.bundles import load

----> 2 bundle_data = load('fund_bundle')

InvalidRequestError: Could not reflect: requested table(s) not available in sqlite:///C:\Users\scuba/.zipline\data\fund_bundle\2016-10-30T10;03;42.188459\assets-5.sqlite: (equities, futures_root_symbols, futures_exchanges, asset_router, equity_symbol_mappings, version_info, futures_contracts)



@Ed, @Richard,

the documented process seems very brittle and I'm struggling...

Please help

Regards,
Dave

--
You received this message because you are subscribed to the Google Groups "Zipline Python Opensource Backtester" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

Dave Gilbert

unread,

Oct 30, 2016, 4:04:20 PM10/30/16

to Zipline Python Opensource Backtester

Hi Ed,

so here's what I've now done

1. created a new conda virtual environment with python 2.7.12 and zipline 1.0.2

(zipline) E:\NOTEBOOKS\QUANTOPIAN>conda list

# packages in environment at G:\Anaconda3\envs\zipline:

#

alembic 0.7.7 py27_0 quantopian

bcolz 0.12.1 np110py27_0 quantopian

bottleneck 1.0.0 np110py27_1

cachetools 1.1.6 py27_0 quantopian

click 6.6 py27_0

contextlib2 0.5.3 py27_0

cyordereddict 0.2.2 py27_0 quantopian

cython 0.25.1 py27_0

decorator 4.0.10 py27_0

empyrical 0.2.1 py27_0 quantopian

intervaltree 2.1.0 py27_0 quantopian

logbook 0.12.5 py27_0 quantopian

mako 1.0.4 py27_0

markupsafe 0.23 py27_2

mkl 11.3.3 1

multipledispatch 0.4.9 py27_0

networkx 1.11 py27_0

numexpr 2.6.0 np110py27_0

numpy 1.10.4 py27_2

pandas 0.17.1 np110py27_0

pandas-datareader 0.2.1 py27_0

patsy 0.4.1 py27_0

pip 8.1.2 py27_0

python 2.7.12 0

python-dateutil 2.5.3 py27_0

pytz 2016.7 py27_0

requests 2.11.1 py27_0

requests-file 1.4 py27_0

scipy 0.17.1 np110py27_1

setuptools 27.2.0 py27_1

six 1.10.0 py27_0

sortedcontainers 1.4.4 py27_0 quantopian

sqlalchemy 1.1.3 py27_0

statsmodels 0.6.1 np110py27_1

toolz 0.8.0 py27_0

vs2008_runtime 9.00.30729.1 2

wheel 0.29.0 py27_0

zipline 1.0.2 np110py27_0 quantopian

2. Prepared ~/.zipline/extension.py with the following content:

from zipline.data.bundles import register, load, yahoo_equities

funds = set(['VTSMX','VTSAX','VFINX','VFIAX','VIVAX','VVIAX','VIGRX','VIGAX','VIMSX','VIMAX','VMGIX','VMGMX','VMVIX',

'VMVAX','NAESX','VSMAX','VISVX','VSIAX','VISGX','VSGAX','BRSIX','VGSIX','VGSLX','VGTSX','VTIAX','VTMGX','VTMGX',

'VEIEX','VEMAX','VEURX','VEUSX','VPACX','VPADX','VTRIX','EFV','VFSVX','VBMFX','VBTLX','VFISX','VFIRX','VFITX','VUSTX'

,'VUSUX','VIPSX','VAIPX','VWSTX','VWSUX','VWITX','VWIUX','VWLTX','VWLUX','VICSX','VWESX','VWETX','VWEHX','VWEAX',

'VFSTX','VFSUX','PIGLX','VGPMX','PCRIX'])

register(

'fund_bundle',

yahoo_equities(funds),

)

3. ingested fund_bundle

Microsoft Windows [Version 10.0.14393]

C:\WINDOWS\system32>activate zipline

(zipline) C:\WINDOWS\system32>e:

(zipline) E:\>cd notebooks\quantopian

(zipline) E:\NOTEBOOKS\QUANTOPIAN>zipline ingest -b fund_bundle

Merging daily equity files: [-----------#------------------------] 59] 100%

Merging daily equity files: [####################################]

G:\Anaconda3\envs\zipline\lib\site-packages\zipline\assets\asset_writer.py:192: FutureWarning: sort(columns=....) is deprecated, use sort_values(by=.....)

row = group.sort('end_date').iloc[-1]

Downloading Yahoo adjustment data: [####################################] 100%

4. Created test.py

E:\NOTEBOOKS\QUANTOPIAN>type test.py

from zipline.data.bundles import register, load, yahoo_equities

funds = set(['VTSMX','VTSAX','VFINX','VFIAX','VIVAX','VVIAX','VIGRX','VIGAX','VIMSX','VIMAX','VMGIX','VMGMX','VMVIX',

'VMVAX','NAESX','VSMAX','VISVX','VSIAX','VISGX','VSGAX','BRSIX','VGSIX','VGSLX','VGTSX','VTIAX','VTMGX','VTMGX',

'VEIEX','VEMAX','VEURX','VEUSX','VPACX','VPADX','VTRIX','EFV','VFSVX','VBMFX','VBTLX','VFISX','VFIRX','VFITX','VUSTX'

,'VUSUX','VIPSX','VAIPX','VWSTX','VWSUX','VWITX','VWIUX','VWLTX','VWLUX','VICSX','VWESX','VWETX','VWEHX','VWEAX',

'VFSTX','VFSUX','PIGLX','VGPMX','PCRIX'])

register( 'fund_bundle', yahoo_equities(funds))

bundle_data = load('fund_bundle')

symbols = set(str(asset.symbol) for asset in bundle_data.asset_finder.retrieve_all(bundle_data.asset_finder.equities_sids))

print (symbols)

print (symbols == funds)

E:\NOTEBOOKS\QUANTOPIAN>

4. Run test.py

E:\NOTEBOOKS\QUANTOPIAN>python test.py

G:\Anaconda3\lib\site-packages\pandas\tseries\index.py:755: PerformanceWarning: Non-vectorized DateOffset being applied to Series or DatetimeIndex

PerformanceWarning)

G:\Anaconda3\lib\site-packages\pandas\tseries\index.py:755: PerformanceWarning: Non-vectorized DateOffset being applied to Series or DatetimeIndex

PerformanceWarning)

G:\Anaconda3\lib\site-packages\pandas\tseries\index.py:755: PerformanceWarning: Non-vectorized DateOffset being applied to Series or DatetimeIndex

PerformanceWarning)

Traceback (most recent call last):

File "test.py", line 15, in <module>

bundle_data = load('fund_bundle')

File "G:\Anaconda3\lib\site-packages\zipline\data\bundles\core.py", line 516, in load

asset_db_path(name, timestr, environ=environ),

File "G:\Anaconda3\lib\site-packages\zipline\assets\assets.py", line 148, in __init__

metadata = sa.MetaData(bind=engine)

File "G:\Anaconda3\lib\site-packages\zipline\assets\assets.py", line 149, in __init__

metadata.reflect(only=asset_db_table_names)

File "G:\Anaconda3\lib\site-packages\sqlalchemy\sql\schema.py", line 3650, in reflect

(bind.engine.url, s, ', '.join(missing)))

sqlalchemy.exc.InvalidRequestError: Could not reflect: requested table(s) not available in sqlite:///C:\Users\scuba/.zipline\data\fund_bundle\2016-10-30T19;50;02.949000\assets-5.sqlite: (version_info, futures_contracts, equities, futures_root_symbols, equity_symbol_mappings, asset_router, futures_exchanges)

E:\NOTEBOOKS\QUANTOPIAN>

very frustrating!

Regards,

Dave

Ed Bartosh

unread,

Oct 30, 2016, 4:23:11 PM10/30/16

to Dave Gilbert, Zipline Python Opensource Backtester

Hi Dave,

> very frustrating!

Indeed! But from other point of view it's business as usual, I'd say. It's a software and it has bugs as any other software.

Back to the problem.

Looks like you are facing this issue: https://github.com/quantopian/zipline/issues/1316

Proposed workaround is to install latest zipline: conda install -c https://conda.anaconda.org/Quantopian/label/ci -c quantopian zipline, ingest your bundle again and run your code.

I hope this will fix it. If it doesn't - try to find similar issue here https://github.com/quantopian/zipline/issues and comment on it. If you don't find anything similar - create your own issue.

Cheers,

Ed

2016-10-30 22:04 GMT+02:00 Dave Gilbert <scub...@gmail.com>:

--

You received this message because you are subscribed to the Google Groups "Zipline Python Opensource Backtester" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

Dave Gilbert

unread,

Oct 30, 2016, 4:25:08 PM10/30/16

to Zipline Python Opensource Backtester

Hi Ed,

this appears to be a zipline bug - see issue #1316

I ingested the bundle again, as suggested, and it works.

Guess I should have checked for bugs earlier..

regards,

Dave

Ed Bartosh

unread,

Oct 30, 2016, 4:29:21 PM10/30/16

to Dave Gilbert, Zipline Python Opensource Backtester

Hi Dave,

Did you install latest zipline or it worked for you with 1.0.2?

Regards,

Ed

--

You received this message because you are subscribed to the Google Groups "Zipline Python Opensource Backtester" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

Dave Gilbert

unread,

Oct 31, 2016, 4:27:53 AM10/31/16

to Zipline Python Opensource Backtester

Hi Ed,

I'm using zipline 1.0.2

Rgds,

Dave

Reply all

Reply to author

Forward