Unable to load NSE bundle from Quandl

477 views
Skip to first unread message

Apoorv Vishnoi

unread,
Oct 16, 2017, 2:13:15 PM10/16/17
to Zipline Python Opensource Backtester
Hi All,

Thanks to all who help on this forum!

I am new to Quantopian and loved paying with US stocks. I also wanted to try out Indian stocks. The daily data for the same is available on quandl at https://www.quandl.com/data/NSE-National-Stock-Exchange-of-India

I copied and modified the WIki Quandl file in zipline/data/bundles. Made the changes in the extension file, as mentioned in some earlier posts and ran the bundle but I am getting the following errors:

(35) C:\Users\Apoorv\Documents> zipline ingest -b mynse
Traceback (most recent call last):
  File "C:\Users\Apoorv\Miniconda3\envs\35\Scripts\zipline-script.py", line 11, in <module>
    load_entry_point('zipline==1.1.1', 'console_scripts', 'zipline')()
  File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\click\core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\click\core.py", line 697, in main
    rv = self.invoke(ctx)
  File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\click\core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\click\core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\click\core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\zipline\__main__.py", line 312, in ingest
    show_progress,
  File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\zipline\data\bundles\core.py", line 451, in ingest
    pth.data_path([name, timestr], environ=environ),
TypeError: 'set' object is not callable

 Requesting your kind help. I have also attached the extension and nse file created in  zipline/data/bundles.
 

extension.py
nse.py

Sai Boorlagadda

unread,
Nov 10, 2017, 11:28:26 PM11/10/17
to Zipline Python Opensource Backtester
Hello Apoorv,

It looks like you dont have to use extension.py if registering a bundle with @bundle("nse").
So I was able to download with below setup:

- Dont use extension.py.
- Change data/bundles/__init__.py to include "from . import nse"
- Few fixes to your nse.py 
       * include 'NSE' to metadata_url 
       * adding 1 day in Dataframe using .add 
       * few symbols are different between metadata and daily data (eg: CNX_FINANCE vs CNXFINANCE), so ignored the exception after 5 retires
       * and avoid processing splits & dividends.

59c59,62
<                 pass
---
>                 raise ValueError(
>                     'Failed to download metadata page %d after %d'
>                     ' attempts.' % (page_number, retries),
>                 )
124,125c128,129
<     data['exchange'] = 'NSE'
<     data['auto_close_date'] = data['end_date'].add(pd.Timedelta(days=1))
---
>     data['exchange'] = 'QUANDL'
>     data['auto_close_date'] = data['end_date'] + pd.Timedelta(days=1)
158c162
---
250,251c254,255
<         #_update_splits(splits, asset_id, raw_data)
<         #_update_dividends(dividends, asset_id, raw_data)
---
>         _update_splits(splits, asset_id, raw_data)
>         _update_dividends(dividends, asset_id, raw_data)

Sai
Message has been deleted

Apoorv Vishnoi

unread,
Dec 6, 2017, 6:37:49 AM12/6/17
to Zipline Python Opensource Backtester
Hello Sai,

Thanks a lot for taking out time to resolve this. I had lost hope on this and just checked that you have posted a reply today!  I am very sorry for the delayed response.
I incorporated the changes you mentioned in the data/bundles/__init__.py & nse.py (please find attached) however I am still getting the below errors. 
Requesting your advice.

 C:\Users\Apoorv\Documents>zipline ingest -b nse
Downloading NSE metadata:   [####################################]
Merging daily equity files:  [------------------------------------]
Traceback (most recent call last):
  File "C:\Users\Apoorv\Miniconda3\envs\35\Scripts\zipline-script.py", line 11, in <module>
    load_entry_point('zipline==1.1.1', 'console_scripts', 'zipline')()
  File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\click\core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\click\core.py", line 697, in main
    rv = self.invoke(ctx)
  File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\click\core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\click\core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\click\core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\zipline\__main__.py", line 312, in ingest
    show_progress,
  File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\zipline\data\bundles\core.py", line 451, in ingest
    pth.data_path([name, timestr], environ=environ),
  File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\zipline\data\bundles\nse.py", line 307, in nse_bundle
    show_progress=show_progress,
  File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\zipline\data\us_equity_pricing.py", line 257, in write
    return self._write_internal(it, assets)
  File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\zipline\data\us_equity_pricing.py", line 319, in _write_internal
    for asset_id, table in iterator:
  File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\click\_termui_impl.py", line 259, in next
    rv = next(self.iter)
  File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\zipline\data\us_equity_pricing.py", line 248, in <genexpr>
    (sid, self.to_ctable(df, invalid_data_behavior))
  File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\zipline\data\bundles\nse.py", line 256, in gen_symbol_data
    raw_data = raw_data.reindex(
AttributeError: 'NoneType' object has no attribute 'reindex'


regards
Apoorv
- show quoted text -
__init__.py
nse.py

Sai Boorlagadda

unread,
Dec 29, 2017, 2:57:41 AM12/29/17
to Zipline Python Opensource Backtester
Hey Aproov,

File "C:\Users\Apoorv\Miniconda3\envs\35\lib\site-packages\zipline\data\bundles\nse.py", line 256, in gen_symbol_data
    raw_data = raw_data.reindex(
AttributeError: 'NoneType' object has no attribute 'reindex'

From the stack trace it looks like its failing to download raw data and so raw_data is NoneType. 
Are you using your API key while downloading? 
or try printing raw_data.tail(1) to make sure raw_data is not null 
or It could be that it didnt find symbol data (eg: may be you should lookup using CNXFINANCE instead of CNX_FINANCE etc)

try like this in case if you want to use your API key:

shell> QUANDL_API_KEY=<api_key> zipline ingest -b nse

Also I see that your nse.py does not comment out writing split data (so in case if you succeed it might throw exception to save split data).
323,326c309,312
<     #adjustment_writer.write(
<     #    splits=pd.concat(splits, ignore_index=True),
<     #    dividends=pd.concat(dividends, ignore_index=True),
<     #)

PS: zipline downloads all symbols as CSVs before it merges and writes an sqllite database, so I recommend to delete .zipline from your home directory until you have a successful run.

Sai

Apoorv Vishnoi

unread,
Jan 3, 2018, 4:50:46 AM1/3/18
to Zipline Python Opensource Backtester
Thanks Sai,

Commenting out the code to write split data worked and I was able to ingest the data.

However,  as you mentioned, ingestion was getting stuck for it was unable to find symbol data such as CNX_100. To tackle this I created a new bundle called nse50 with only top 50 stocks of nifty.
I did this by making following changes to nse.py:

changed line 23
from 
excluded_symbols = frozenset({'TEST123456789'})
to 
excluded_symbols = frozenset({'ADANIPORTS', 'AMBUJACEM', 'ASIANPAINT', 'AUROPHARMA', 'AXISBANK',
       'BAJAJ_AUTO', 'BAJFINANCE', 'BPCL', 'BHARTIARTL', 'INFRATEL',
       'BOSCHLTD', 'CIPLA', 'COALINDIA', 'DRREDDY', 'EICHERMOT', 'GAIL',
       'HCLTECH', 'HDFCBANK', 'HEROMOTOCO', 'HINDALCO', 'HINDPETRO',
       'HINDUNILVR', 'HDFC', 'ITC', 'ICICIBANK', 'IBULHSGFIN', 'IOC',
       'INDUSINDBK', 'INFY', 'KOTAKBANK', 'LT', 'LUPIN', 'MM', 'MARUTI',
       'NTPC', 'ONGC', 'POWERGRID', 'RELIANCE', 'SBIN', 'SUNPHARMA', 'TCS',
       'TATAMOTORS', 'TATASTEEL', 'TECHM', 'UPL', 'ULTRACEMCO', 'VEDL',
       'WIPRO', 'YESBANK', 'ZEEL'})

line 121
from 
    data = data[~data.symbol.isin(excluded_symbols)]
to 
    data = data[data.symbol.isin(excluded_symbols)]

This worked and the data ingestion was successful. Attaching nse50.py & __init__.py
However, when I ran a small test code(ZipTest.py, attached) to test the bundle I got the following error:



(zip) C:\Users\Apoorv\Documents>zipline run -f C:\\ZipTest.py --bundle nse50 -s 2017-04-04 -e 2017-05-05
[2018-01-03 09:31:09.925928] INFO: Loader: Cache at C:\Users\Apoorv/.zipline\data\SPY_benchmark.csv does not have data from 1990-01-02 00:00:00+00:00 to 2017-12-29 00:00:00+00:00.

[2018-01-03 09:31:09.925928] INFO: Loader: Downloading benchmark data for 'SPY' from 1989-12-29 00:00:00+00:00 to 2017-12-29 00:00:00+00:00
[2018-01-03 09:31:13.596222] WARNING: Loader: Still don't have expected data after redownload!
Traceback (most recent call last):
  File "C:\Users\Apoorv\Miniconda3\envs\zip\Scripts\zipline-script.py", line 11, in <module>
    load_entry_point('zipline==1.1.1', 'console_scripts', 'zipline')()
  File "C:\Users\Apoorv\Miniconda3\envs\zip\lib\site-packages\click\core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\Apoorv\Miniconda3\envs\zip\lib\site-packages\click\core.py", line 697, in main
    rv = self.invoke(ctx)
  File "C:\Users\Apoorv\Miniconda3\envs\zip\lib\site-packages\click\core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\Apoorv\Miniconda3\envs\zip\lib\site-packages\click\core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\Apoorv\Miniconda3\envs\zip\lib\site-packages\click\core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "C:\Users\Apoorv\Miniconda3\envs\zip\lib\site-packages\zipline\__main__.py", line 97, in _
    return f(*args, **kwargs)
  File "C:\Users\Apoorv\Miniconda3\envs\zip\lib\site-packages\click\decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "C:\Users\Apoorv\Miniconda3\envs\zip\lib\site-packages\zipline\__main__.py", line 240, in run
    environ=os.environ,
  File "C:\Users\Apoorv\Miniconda3\envs\zip\lib\site-packages\zipline\utils\run_algo.py", line 179, in _run
    overwrite_sim_params=False,
  File "C:\Users\Apoorv\Miniconda3\envs\zip\lib\site-packages\zipline\algorithm.py", line 709, in run
    for perf in self.get_generator():
  File "C:\Users\Apoorv\Miniconda3\envs\zip\lib\site-packages\zipline\gens\tradesimulation.py", line 225, in transform
    for capital_change_packet in once_a_day(dt):
  File "C:\Users\Apoorv\Miniconda3\envs\zip\lib\site-packages\zipline\gens\tradesimulation.py", line 183, in once_a_day
    midnight_dt)
  File "C:\Users\Apoorv\Miniconda3\envs\zip\lib\site-packages\zipline\data\data_portal.py", line 1165, in get_splits
    (seconds,)).fetchall()
sqlite3.OperationalError: no such table: SPLITS


Requesting your advice yet again.

Regards
Apoorv
ZipTest.py
__init__.py
nse50.py
Reply all
Reply to author
Forward
0 new messages