Problem with ingesting Norgate data

[Vincent Roye]

Hi,

I followed step by step: https://pypi.org/project/zipline-norgatedata/

The only difference is that I am using zipline 1.4.1 since I can't find the Conda package for 1.3.0

I'm receiving this error when I try to ingest the AAPL equity:

(zipline) C:\Users\OS>zipline ingest -b norgatedata-aapl
[2020-11-29 05:27:14.636076] INFO: Norgate Data: NorgateData package v1.0.53: Init complete
[2020-11-29 05:27:15.082371] INFO: Norgate Data: Zipline_norgatedata package v1.1.46: Init complete
Traceback (most recent call last):
  File "F:\anaconda3\envs\zipline\Scripts\zipline-script.py", line 6, in <module>
    from zipline.__main__ import main
  File "F:\anaconda3\envs\zipline\lib\site-packages\zipline\__init__.py", line 29, in <module>
    from .utils.run_algo import run_algorithm
  File "F:\anaconda3\envs\zipline\lib\site-packages\zipline\utils\run_algo.py", line 29, in <module>
    from zipline.extensions import load
  File "F:\anaconda3\envs\zipline\lib\site-packages\zipline\extensions.py", line 11, in <module>
    from zipline_norgatedata import (
  File "F:\anaconda3\envs\zipline\lib\site-packages\zipline_norgatedata\__init__.py", line 2, in <module>
    from .pipelines import *
  File "F:\anaconda3\envs\zipline\lib\site-packages\zipline_norgatedata\pipelines.py", line 3, in <module>
    from zipline.api import sid, symbol
ImportError: cannot import name 'sid'

Does that come from the incompatibility with 1.4.1? Is there any workaround to solve that problem?

Best Regards,

Vincent

[Brandon]

Hi Vincent, 

Not too sure about v1.4.1, but i have been using v1.3 and it works well. 

I just tried to install zipline v1.3 and it works. 

The command at the command prompt for my virtual environment is "conda install -c Quantopian zipline=1.3"

[Vincent Roye]

Hi Brandon,

I solved the problem just by commenting out:

from zipline.api import sid, symbol 

Thanks,

Vincent

[Brandon Foo]

Hi Vincent, do you mean you are now able to run zipline v1.4.1 successfully to perform backtest?

--
You received this message because you are subscribed to a topic in the Google Groups "Zipline Python Opensource Backtester" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/zipline/aiBPQWo56Rk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to zipline+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/zipline/5c9a69aa-a62d-40cd-99b7-b2ed6aa407e3n%40googlegroups.com.

--

Best wishes,

Brandon

[Vincent Roye]

Hi Brandon,

Yes, I am. I'm using Zipline v1.4.1 and the latest version of the norgate data library.

 

--

Vincent Roye

[Brandon Foo]

Nice. Did you have to implement any changes to any of the py script from zipline v1.4.1 to get around any potential bugs or does it work seamlessly? 

Also, what operating system are you using?

--

Best wishes,

Brandon

[Vincent Roye]

I just had to remove:

"from zipline.api import sid, symbol "

from: File "F:\anaconda3\envs\zipline\lib\site-packages\zipline_norgatedata\pipelines.py"

(My bad, the update was in the zipline_norgatedata library, not zipline)

I'm using Windows 10 Pro

--

Vincent Roye

[Brandon Foo]

Hi Vincent, 

Thanks. 

I am seeing the below error when trying to run a ipynb script on zipline v1.4.1. The same script works fine in zipline 1.3.0

KeyError: 'volume'
Do you or anyone in this group know how to resolve this issue?

--

Best wishes,

Brandon

[Richard Dale]

Zipline-norgatedata v1.2.1 has been released and is compatible with Zipline v1.4.1.  Some interesting changes allow you to mix futures and equities assets within a bundle, plus benchmarks can be handled from the bundle itself.

However, there are a number of issues within the ZIpline code that need patching to resolve these sorts of cryptic (!) error messages.

I hjave documented them here:

https://pypi.org/project/zipline-norgatedata/

(see the "Zipline Issues that require patching")

Specifically, patches are required to:

a) Handle reading of futures data in general

b) Handle backtesting on futures markets not hardcoded into Zipline (since the volatility is not defined for them)

c) Handle multiple asset types within a bundle (i.e. futures + equities)

d) Increase dates for backtesting (trading_calendars is hardcoded to 1990 for stocks and 2000 for futures)

The patches are pretty simple, but given these projects are owned by Quantopian and there was no discussion about any sort of handover to the Open Source community.

Also, I've updated the source code from Andreas Clenow's book to handle Zipline v1.4.1.  The updated examples are here:

http://norgatedata.com/book-examples/trading-evolved/NorgateDataTradingEvolvedExamples.zipline141.zip

Cheers,

Richard.

[Vincent Roye]

Hi Richard,

Is there any documentation about this functionality?

c) Handle multiple asset types within a bundle (i.e. futures + equities) 

I would be interested to see how that works

Best Regards,

Vincent

[Richard Dale]

Hi Vincent,

We just modified our routines (and patched/fixed Zipline) to handle a mixed bundle correctly.  The patches against Zipline v1.4.1 to handle this are shown here:

https://pypi.org/project/zipline-norgatedata/

For example, you can ingest a futures bundle and have an index (or ETF) that you reference for other uses (benchmarking, parking unused funds into a bond ETF, activating certain systems due to market regimes on an index etc.).  Also, futures assets get root symbols, plus other metadata like auto close dates prior to first notice date/expiry, tick size, exchange etc. 

Best regards,
Richard Dale.
Norgate Data
- Premium quality Stock, Futures and Foreign Exchange Data -
NorgateData.com

To view this discussion on the web visit https://groups.google.com/d/msgid/zipline/153c974d-4cfb-48d6-902c-cb47dfeca4c7n%40googlegroups.com.

[Final Fantasy]

Hi Richard,

that is very helpful, thank you.  Why not create a fork of the zipline repo with your patches?  That would make it easier for folks to get going with zipline.

Sacha