Backtest using BTC data from csv file

[Pierre villenave]

I'm new to Zipline and really just trying to figure out how to feed algos with csv data. I have read through the forum and tried several attempts, but so far I have not been able to use the data efficiently.

I'm looking into BTC, have a csv file with daily prices. The dataset looks like this:

                              Open     High      Low    Close        Volume  \

2017-08-19 00:00:00+00:00  4100.00  4188.00  3900.00  4100.00  5.881403e+07   

2017-08-18 00:00:00+00:00  4260.47  4368.00  3964.96  4100.00  7.322403e+07   

2017-08-17 00:00:00+00:00  4361.99  4480.00  4167.21  4276.50  6.322860e+07   

2017-08-16 00:00:00+00:00  4154.99  4379.78  3926.06  4378.84  5.419433e+07   

2017-08-15 00:00:00+00:00  4320.95  4400.00  3800.00  4155.67  1.046514e+08   

                                 Price  

2017-08-19 00:00:00+00:00  4028.205284  

2017-08-18 00:00:00+00:00  4180.171091  

2017-08-17 00:00:00+00:00  4338.694675  

2017-08-16 00:00:00+00:00  4193.426713  

2017-08-15 00:00:00+00:00  4101.447155  

When I run the below code, I get this error: 'the label [2017-08-18 00:00:00+00:00] is not in the [index]'

import pytz

import pandas as pd

from datetime import datetime

import zipline

from zipline.api import order, symbol, record, order_target

from zipline.algorithm import TradingAlgorithm

def initialize(context):

    pass

def handle_data(context, data):

    pass

data = pd.read_csv("bitcoin1.csv", header=0, index_col=0, names=['Open', 'High', 'Low', 'Close', 'Volume', 'Price'], parse_dates=True)

data.index=data.index.tz_localize(pytz.UTC)

algo_obj = TradingAlgorithm(initialize=initialize, handle_data=handle_data, data_frequency='daily')

perf_manual = algo_obj.run(data)

Any help would be great. Both regarding the error and further resources with examples for backtest of cryptocurrencies. 

Thanks!
Pierre

[Ed Bartosh]

Hi Pierre,

Can you show your .csv file here?

You seem to use old approach in your code. It might or might not work, but it would be better to consider using data bundles: http://www.zipline.io/bundles.html

Regards,

Ed

--
You received this message because you are subscribed to the Google Groups "Zipline Python Opensource Backtester" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[Pierre-Adrien Villenave]

Sure. Please find attached.

On Mon, Aug 21, 2017 at 5:23 AM, Ed Bartosh <bar...@gmail.com> wrote:

Hi Pierre,

Can you show your .csv file here?

You seem to use old approach in your code. It might or might not work, but it would be better to consider using data bundles: http://www.zipline.io/bundles.html

Regards,

Ed

[Ed Bartosh]

Hi Pierre,

There seem to be at least 3 issues with your data and code:

- dataframe index must be reverted as data in your csv is in the opposite order

- you may need to create simulation parameters to specify start and end dates and pass them to TradingAlgorithm constructor and to algo.run

- you probably want to use custom calendar as default zipline calendar(NYSE) differs from cryptocurrency market calendar.

Hope it helps,

Regards,

Ed

--

BR,
Ed

[Pierre-Adrien Villenave]

Hi Ed,

Thanks for your response. I was able to work on the first two but I am struggling to see how to custom the calendar. Would you mind providing some insight?

Best,

Pierre

On Tue, Aug 22, 2017 at 4:30 AM, Ed Bartosh <bar...@gmail.com> wrote:

Hi Pierre,

There seem to be at least 3 issues with your data and code:

- dataframe index must be reverted as data in your csv is in the opposite order

- you may need to create simulation parameters to specify start and end dates and pass them to TradingAlgorithm constructor and to algo.run

- you probably want to use custom calendar as default zipline calendar(NYSE) differs from cryptocurrency market calendar.

Hope it helps,

Regards,

Ed

[Ed Bartosh]

Hi Pierre,

You may find it interesting to read this thread https://groups.google.com/forum/#!searchin/zipline/24$2F7$20Trading$20Environment%7Csort:relevance/zipline/WZKRhHLVh6M/-sNkl34iBQAJ

There are good examples of custom calendar and usage of data bundles.

Regards,

Ed

--

BR,
Ed

[Pierre villenave]

Hi Ed,

First, thank you very much for your help, and sorry for the delay. I needed to spend more time understanding how zipline functions. Now, I think I am in a good place to start making progress. I was looking at your csvdir module and would like to use it, however, I did not figure out how to do so it. I was especially struggling with this part: 

register('custom-csvdir-bundle',
	csvdir_equities(sys.environ['CSVDIR'],
	['daily', 'minute']))

Could you provide an example of what should 'custom-csvdir-bundle' and sys.environ['CSVDIR'] be? In additional I did not understand the notes saying: 

Environment variable CSVDIR must contain path to the directory with the
	following structure:
	daily/<symbol>.cvs files
	minute/<symbol>.csv files
	for each symbol.
	"""

Would you mind providing some details on its meaning?

Finally, I have looked into the Exchange_Calendar_Poliniex in the link you provided. But, I'm not sure I understand how to use it with your csvdir module. I'm sorry as I'm sure these questions are very basic but I am stuck at this point..

Sure. Please find attached.

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

--

BR,
Ed

--

BR,
Ed

[Ed Bartosh]

Hi Pierre,

You can see an example of using csvdir bundle here: https://groups.google.com/forum/#!searchin/zipline/csvdir%7Csort:relevance/zipline/GbnwZO-Gbn4/bLIn7e4GDgAJ

> Finally, I have looked into the Exchange_Calendar_Poliniex in the link you provided. But, I'm not sure I understand how to use it with your csvdir module.

To use calendar with the bundle you need to register it using register_calendar and then pass calendar name to register API. Here is an approximate example. It's just to show you the idea. I didn't test if it works.

from zipline.utils.calendars import register_calendar

from zipline.utils.calendars.exchange_calendar_poloniex import POLONIEXExchangeCalendar

from zipline.data.bundles import register

from zipline.data.bundles.csvdir import csvdir_equities

register_calendar('POLONIEX', POLONIEXExchangeCalendar())

register('csvdir-poloniex', csvdir_equities(['daily']),

            calendar_name='POLONIEX')

>  I'm sorry as I'm sure these questions are very basic but I am stuck at this point..

I'd not say this is basic. You're trying to use custom calendar with the custom bundle. It's not covered by zipline docs and not a lot of people had success with. You need to be able to debug zipline code if you want to make it working.

Regards,

Ed

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[Pierre-Adrien Villenave]

Hi Ed!

Thank you for your response. 

I followed the steps on the link you sent: https://groups.google.com/forum/#!searchin/zipline/csvdir%7Csort:relevance/zipline/GbnwZO-Gbn4/bLIn7e4GDgAJ

I was trying to test the module with AAPL data to simplify the process and not have to deal with registering a calendar (for now) and I kept running into the same error message: 

PierreAsMacBook:~ Pierre$ CSVDIR=./csvdir/ zipline ingest -b csvdir

/Users/Pierre/anaconda/lib/python2.7/site-packages/zipline/__main__.py:50: UserWarning: Failed to load extension: '/Users/Pierre/.zipline/extension.py'

'module' object has no attribute 'environ'

  os.environ,

Error: No bundle registered with the name u'csvdir'

In case it is useful, you can find below my steps and different attempts. 

$ find ./csvdir

/csvdir

./csvdir/.DS_Store

./csvdir/daily

./csvdir/daily/.DS_Store

./csvdir/daily/AAPL.csv 

Then, I copied and pasted the following lines into ~/zipline/extension.py

from zipline.data.bundles import register, csvdir_equities

register('csvdir', csvdir_equities(['daily', 'minute'])) 

Finally, when I tried to ingest my data using csvdir bundle:

$ CSVDIR=./csvdir/ zipline ingest -b csvdir

I got the same error message 

PierreAsMacBook:~ Pierre$ CSVDIR=./csvdir/ zipline ingest -b csvdir

/Users/Pierre/anaconda/lib/python2.7/site-packages/zipline/__main__.py:50: UserWarning: Failed to load extension: '/Users/Pierre/.zipline/extension.py'

invalid syntax (extension.py, line 3)

  os.environ,

Error: No bundle registered with the name u'csvdir'

I also tried this $ 

CSVDIR=./csvdir/daily/AAPL.csv zipline ingest -b csvdir and got the same message. 

Later I tried to change the code in ~zipline/extension.py to: register('csvdir', csvdir_equities(sys.environ['CSVDIR', ['daily'])) and received the same message. 

It is clear that I'm doing something wrong, though I'm not sure why it would not find a registered bundle 'csvdir' when I clearly have it coded in ~/zipline/extension.py 

Once again THANK YOU for your help!

On Sun, Sep 10, 2017 at 5:57 AM, Ed Bartosh <bar...@gmail.com> wrote:

Hi Pierre,

You can see an example of using csvdir bundle here: https://groups.google.com/forum/#!searchin/zipline/csvdir%7Csort:relevance/zipline/GbnwZO-Gbn4/bLIn7e4GDgAJ

> Finally, I have looked into the Exchange_Calendar_Poliniex in the link you provided. But, I'm not sure I understand how to use it with your csvdir module.

To use calendar with the bundle you need to register it using register_calendar and then pass calendar name to register API. Here is an approximate example. It's just to show you the idea. I didn't test if it works.

from zipline.utils.calendars import register_calendar

from zipline.utils.calendars.exchange_calendar_poloniex import POLONIEXExchangeCalendar

from zipline.data.bundles import register

from zipline.data.bundles.csvdir import csvdir_equities

register_calendar('POLONIEX', POLONIEXExchangeCalendar())

register('csvdir-poloniex', csvdir_equities(['daily']),

            calendar_name='POLONIEX')

>  I'm sorry as I'm sure these questions are very basic but I am stuck at this point..

I'd not say this is basic. You're trying to use custom calendar with the custom bundle. It's not covered by zipline docs and not a lot of people had success with. You need to be able to debug zipline code if you want to make it working.

Regards,

Ed

[Ed Bartosh]

Hi Pierre,

Can you show your extension.py?

According to this error: "invalid syntax (extension.py, line 3)" you have some error on line 3. This is probably the reason why your bundle is not registered.

Regards,

Ed

--

BR,
Ed

[Pierre villenave]

Hi Ed,

Sure, this my extension.py:

from zipline.data.bundles import csvdir_equities, register

       

register('csvdir', csvdir_equities(['daily','minute']))

It is also mentioning os.envrion, which I saw in the csvdir_equities module. Could it be the problem?

Best,

Pierre

On Wednesday, September 13, 2017 at 3:56:27 AM UTC-4, Ed Bartosh wrote:

Hi Pierre,

[Ed Bartosh]

Hi Pierre,

I can't reproduce this in my environment. It could be that there is a bug in the code.

I'd suggest to apply latest version of csvdir code from here: https://github.com/quantopian/zipline/pull/1860/commits/c1dbb5c798bbfc8637f9563605cd4453f82d847f

Then comment everything related to csvdir in your ~/.zipline/extension.py and try to ingest your data as explained here:

https://github.com/quantopian/zipline/issues/174#issuecomment-331836005

Regards,

Ed

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed