timezone issue with cvs data reader, tz_convert?

[dylan.p...@gmail.com]

Hello, 

I'm trying to get minute data custom bundles from cvs files to work in zipline 1.0.2, python 2.7, OSX Sierra 10.12.  I've combined the suggestions from Ed B and Richard P to make something that works for me.

First, thanks for that, I've been trying to get cvs minute data working with bundles for some time on the side, and it's really good to see the ingestion working.  This is my first post here, but I've been reading and studying posts for quite some time.

The problem i've having is in I think the timezone info, but I'm not sure.  The simplest initial csv data I ingest is from google, so it looks like:

bash-3.2$ head ../csvdir/AAL.csv 

ts,o,h,l,c,v

2016-10-05 09:31:00+00:00,37.76,37.82,37.75,37.82,11000.0

2016-10-05 09:32:00+00:00,37.58,37.76,37.55,37.76,16475.0

2016-10-05 09:33:00+00:00,37.485,37.56,37.45,37.53,20764.0

2016-10-05 09:34:00+00:00,37.62,37.64,37.485,37.485,42301.0

2016-10-05 09:35:00+00:00,37.63,37.64,37.57,37.62,28904.0

2016-10-05 09:36:00+00:00,37.667,37.68,37.6,37.63,73997.0

2016-10-05 09:37:00+00:00,37.66,37.71,37.64,37.65,22911.0

2016-10-05 09:38:00+00:00,37.65,37.66,37.59,37.66,16055.0

2016-10-05 09:39:00+00:00,37.7,37.73,37.65,37.65,11555.0

bash-3.2$ 

The ingestion works much along the same lines as your csvdir and viacsv codes from your recent posts.  And the result is an ingested bundle.  

bash-3.2$ zipline bundles

quandl <no ingestions>

quantopian-quandl <no ingestions>

sharkwave-m 2016-10-26 15:50:07.242376

bash-3.2$ pwd

Note the timezone is not explicitly addressed in those csv ingestion routines, and the google data is clearly US/Eastern.  The problem I'm having in a simple algo to just test buy orders is (please excuse the full dump, but it's best):

bash-3.2$  zipline run -f test_sharkwave.py --bundle sharkwave-m -s 2016-10-06 -e 2016-10-25

get_datetime 2016-10-06 20:00:00+00:00 iNumEqt= 160

Ordering 0.00625  of  AAL

Traceback (most recent call last):

  File "//anaconda/bin/zipline", line 11, in <module>

    load_entry_point('zipline==1.0.2', 'console_scripts', 'zipline')()

  File "/anaconda/lib/python2.7/site-packages/click/core.py", line 716, in __call__

    return self.main(*args, **kwargs)

  File "/anaconda/lib/python2.7/site-packages/click/core.py", line 696, in main

    rv = self.invoke(ctx)

  File "/anaconda/lib/python2.7/site-packages/click/core.py", line 1060, in invoke

    return _process_result(sub_ctx.command.invoke(sub_ctx))

  File "/anaconda/lib/python2.7/site-packages/click/core.py", line 889, in invoke

    return ctx.invoke(self.callback, **ctx.params)

  File "/anaconda/lib/python2.7/site-packages/click/core.py", line 534, in invoke

    return callback(*args, **kwargs)

  File "/anaconda/lib/python2.7/site-packages/zipline/__main__.py", line 97, in _

    return f(*args, **kwargs)

  File "/anaconda/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func

    return f(get_current_context(), *args, **kwargs)

  File "/anaconda/lib/python2.7/site-packages/zipline/__main__.py", line 240, in run

    environ=os.environ,

  File "/anaconda/lib/python2.7/site-packages/zipline/utils/run_algo.py", line 180, in _run

    overwrite_sim_params=False,

  File "/anaconda/lib/python2.7/site-packages/zipline/algorithm.py", line 688, in run

    for perf in self.get_generator():

  File "/anaconda/lib/python2.7/site-packages/zipline/gens/tradesimulation.py", line 220, in transform

    for capital_change_packet in every_bar(dt):

  File "/anaconda/lib/python2.7/site-packages/zipline/gens/tradesimulation.py", line 133, in every_bar

    handle_data(algo, current_data, dt_to_use)

  File "/anaconda/lib/python2.7/site-packages/zipline/utils/events.py", line 184, in handle_data

    dt,

  File "/anaconda/lib/python2.7/site-packages/zipline/utils/events.py", line 203, in handle_data

    self.callback(context, data)

  File "/anaconda/lib/python2.7/site-packages/zipline/algorithm.py", line 459, in handle_data

    self._handle_data(self, data)

  File "test_sharkwave.py", line 11, in handle_data

    order_target_percent(S,fW)

  File "/anaconda/lib/python2.7/site-packages/zipline/utils/api_support.py", line 51, in wrapped

    return getattr(get_algo_instance(), f.__name__)(*args, **kwargs)

  File "/anaconda/lib/python2.7/site-packages/zipline/utils/api_support.py", line 120, in wrapped_method

    return method(self, *args, **kwargs)

  File "/anaconda/lib/python2.7/site-packages/zipline/algorithm.py", line 1983, in order_target_percent

    style=style)

  File "/anaconda/lib/python2.7/site-packages/zipline/utils/api_support.py", line 120, in wrapped_method

    return method(self, *args, **kwargs)

  File "/anaconda/lib/python2.7/site-packages/zipline/algorithm.py", line 1916, in order_target_value

    target_amount = self._calculate_order_value_amount(asset, target)

  File "/anaconda/lib/python2.7/site-packages/zipline/algorithm.py", line 1332, in _calculate_order_value_amount

    self.trading_client.current_data.current(asset, "price")

  File "zipline/_protocol.pyx", line 124, in zipline._protocol.check_parameters.__call__.assert_keywords_and_call (zipline/_protocol.c:2816)

  File "zipline/_protocol.pyx", line 328, in zipline._protocol.BarData.current (zipline/_protocol.c:4246)

  File "/anaconda/lib/python2.7/site-packages/zipline/data/data_portal.py", line 407, in get_spot_value

    return self._get_daily_data(asset, field, session_label)

  File "/anaconda/lib/python2.7/site-packages/zipline/data/data_portal.py", line 600, in _get_daily_data

    asset, found_dt, "close"

  File "/anaconda/lib/python2.7/site-packages/zipline/data/dispatch_bar_reader.py", line 84, in get_value

    return r.get_value(sid, dt, field)

  File "/anaconda/lib/python2.7/site-packages/zipline/data/us_equity_pricing.py", line 707, in get_value

    ix = self.sid_day_index(sid, day)

  File "/anaconda/lib/python2.7/site-packages/zipline/data/us_equity_pricing.py", line 675, in sid_day_index

    offset = day_loc - self._calendar_offsets[sid]

KeyError: Equity(0, symbol=u'AAL', asset_name='', exchange=u'YAHOO', start_date=Timestamp('2016-10-05 09:31:00+0000', tz='UTC'), end_date=Timestamp('2016-10-25 16:00:00+0000', tz='UTC'), first_traded=None, auto_close_date=Timestamp('2016-10-26 16:00:00+0000', tz='UTC'), exchange_full=u'YAHOO')

bash-3.2$

So, the exchange I set is 'YAHOO', which may be the problem (Ed B, I didn't find code for register_calendar_alias, can you clarify?), but the timestamps and timezones seem mismatched.

In any case, thanks in advance, getting tired of beating my head against the bricks alone, thought to try posting.

Dylan

[Richard P]

Hello Dylan -

Not sure I see the issue from the stack trace, but have a look here

http://www.prokopyshen.com/Use-minute-bar-data-in-zipline-custom-data-bundle

and in custom module search down for this comment:

# csv time stamp is in EST, but pandas doesnt know that yet so tell it

Following that are some gyrations to tell pandas the timezone the csv is in, then convert to UTC for zipline, then convert that to the Naive date format expected by the ingest routine.

Richard

--
You received this message because you are subscribed to the Google Groups "Zipline Python Opensource Backtester" group.
To unsubscribe from this group and stop receiving emails from it, send an email to zipline+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[dylan.p...@gmail.com]

Ah, thanks Richard! I'll have a look, must have missed that. D

[dylan.p...@gmail.com]

Hi Richard, 

That got the timestamps apparently correct in my bundle, but the simple buy test doesn't seem to work with a similar error.   With the same command, the top of the error dump looks the same, but:

.

.

.

  File "/anaconda/lib/python2.7/site-packages/zipline/data/us_equity_pricing.py", line 675, in sid_day_index

    offset = day_loc - self._calendar_offsets[sid]

KeyError: Equity(0, symbol=u'AAL', asset_name='', exchange=u'YAHOO', start_date=Timestamp('2016-10-05 13:31:00+0000', tz='UTC'), end_date=Timestamp('2016-10-25 20:00:00+0000', tz='UTC'), first_traded=None, auto_close_date=Timestamp('2016-10-26 20:00:00+0000', tz='UTC'), exchange_full=u'YAHOO')

The algo is basically your test of viacsv:

bash-3.2$ cat test_sharkwave.py 

from zipline.api import order_target_percent,get_datetime

def initialize(context):

    pass

def handle_data(context,data):

    iNumEqt=len(data)

    print "get_datetime",get_datetime(),"iNumEqt=",iNumEqt

    fW=1.0/iNumEqt

    for S in data:

       print "Ordering",fW," of ",S.symbol

       order_target_percent(S,fW)

bash-3.2$ 

Thanks!

[Ed Bartosh]

Hi Dylan,

> Ed B, I didn't find code for register_calendar_alias, can you clarify?

It's in calendar_utils.py as far as I can see: https://github.com/quantopian/zipline/blob/master/zipline/utils/calendars/calendar_utils.py

I'm using zipline 1.0.2 if it matters.

Regards,

Ed

--
You received this message because you are subscribed to the Google Groups "Zipline Python Opensource Backtester" group.

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[dylan.p...@gmail.com]

Hi Ed, 

Yes sorry, I sent that too quick, found it after posting.  I must have misspelled the regex in the grep.

Best,

Dylan

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[dylan.p...@gmail.com]

Actually I'll correct myself on my self-correction :-).  I grep'd the anaconda version and correctly did not find it.  But, it IS on the GitHub version, which is apparently most current (though arguably questionable in stability).  I've been switching between these. Perhaps I should stick with the GitHub version. 

[Ed Bartosh]

Hi Dylan,

Make sure you're not using master branch as it's actively changing, can have bugs, etc.

Use latest release version 1.0.2 [1], which is by definition more stable.

[1] https://github.com/quantopian/zipline/tree/1.0.2

Regards,

Ed

To unsubscribe from this group and stop receiving emails from it, send an email to zipline+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

BR,
Ed

[dylan.p...@gmail.com]

Note, I set up in a virtualenv (with I thought everything needed) and tried "pip install git+https://github.com/quantopian/zip...@1.0.2" but that failed with a huge list of warnings and a final error on numpy 

.

.

.

      File "/var/folders/lr/93y3_7rj6pn0r43jfbzkxq8w0000gp/T/easy_install-l36YnS/numpy-1.11.2/numpy/distutils/fcompiler/gnu.py", line 76, in gnu_version_match

    ValueError: A valid Fortran version was not found in this string:

    couldn't understand kern.osversion `16.1.0'

    4.5.1

oh my head aches...

[dylan.p...@gmail.com]

BTW, Thanks, I have the minute data working and I'm fully functional.  Huge thanks!  Best, Dylan

[Jordan Sitkin]

Hi Dylan,

I am running into the same issue! I would paste the full stacktrace but it looks almost identical to yours. Error is:

KeyError: Equity(4, symbol=u'AAPL', asset_name='', exchange=u'QQ', start_date=Timestamp('2016-01-04 14:31:00+0000', tz='UTC'), end_date=Timestamp('2017-02-28 20:59:00+0000', tz='UTC'), first_traded=None, auto_close_date=Timestamp('2017-03-01 20:59:00+0000', tz='UTC'), exchange_full=u'QQ')

I also have imported minute level data and am trying to run a similar example algo as a sanity check. I've reinstalled zipline via conda and verified that I'm running version 1.0.2. I'm unsure why you didn't find the offending line when grepping your conda-installed local version, as I see it in the source of the 1.0.2 release (https://github.com/quantopian/zipline/blob/1.0.2/zipline/data/us_equity_pricing.py#L675).

I wonder if there was something else you figured out that solved the problem. I appreciate any advice! Thanks

[Dylan Brennan]

Hi Jordan,

I did fix it and have been developing quite a lot since, so I'll have to go back and remind myself.  I am using basically the conda 1.0.2 version, and I did find that line in the grep, I just had a mix up.   Long story short yes it's been working well since.  Maybe you could post a bit more detail on what you're doing and I might spot the culprit.  

Best,

Dylan