Pandas Intraday Time Series plots

[briant57]

Hi,

I am having difficulty sorting out how to do any intra-day plot of time series data in matplotlib or pandas.  I have looked through the documentation and also purchased the Python_for_Data_Analysis ebook but it does not seem covered in either.  There are many references to daily samplings.

I can produce images by using the DataFrame.plot or Series.plot function.  This results in a well laid out plot, except for the hours outside of my timeseries (in the example of exchange hours for financial data) are included in the axis and interpolated.  If I give it the criteria use_index=False, the plot looks as I would expect but I look my Timestamp index and it becomes an integer index as I would receive if I used matplotlib/pyplot.  

I am looking for direction on this and to recommend an inclusion of this topic in either the references and documentation.  Pandas is great and I love learning and working with it.  

[Chang She]

Hi, thanks for the bug report. Currently the time series plotting assumes a continuous x-axis with no gaps, which is why you see the interpolation behavior. And as the name suggests, setting use_index=False means the date information contained in the index is lost. I made a github issue to fix this (https://github.com/pydata/pandas/issues/1466). Please leave any additional feedback there.

Thanks,

Chang

--
Chang She

Lambda Foundry

[briant57]

I have been watching and have saw that the Pull Request has been closed.  I downloaded the latest Dev pandas-0.8.0.dev-87d6da1.win-amd64-py2.7.exe but I still have the same issue when plotting and do not see any new parameters/options in the plot() function.  Has this been included in the version listed above?  Thx

[Wes McKinney]

Chang, correct me if I'm wrong, but my understanding is that the data
needs to be resampled to a regular frequency in order for gaps to
appears. Otherwise (irregular data) it's not at all obvious where data
should be interpolated (or not interpolated). You could add some "gap
inference" (e.g. treat data points at least X time units apart as a
gap) though this would be guesswork in a lot of cases; definitely
possible but the code is a bit hairy. I'd really like to get the final
release out today if possible and leave further tweaks here til 0.8.1
or beyond.

- Wes

[briant57]

If it helps, a typical financial time series user would have regularly sampled data during market hours (varied from 6-23 hrs a day).  The gaps would typically be standard on weekdays, but problems could occur with irregular gaps for holidays and weekends.

I think Wes' idea is clever because the user would know that their minimum overnight gap would be x-hours, and all other gaps >= x could be removed and not interpolated.  

I hope this provides a good example.

[Chang She]

Yeah, the conversion to the regularized frequency will insert NaN in the gaps so that matplotlib knows to not interpolate there.

For now the preferred way is to do something like: 

data.asfreq('5Min').plot()

I just made a github issue regarding gap interpolation here: https://github.com/pydata/pandas/issues/1510

Could do user supplied explicit gap definition, or try and be clever and guess a regular frequency.

[Dale Jung]

Are we talking about completely removing the gaps, or just preventing interpolation? AFAIK matplotlib won't automatically remove the gaps with/without interpolation.

[briant57]

I am talking about the graph labeled no gaps.jpg  I believe that is what most people working with financial timeseries will be looking for.

[Chang She]

the latest changes in pandas removes the interpolation but leaves the gaps.

no gaps (other than omitting weekends) is currently not possible but would love to get more feedback and continue the discussion on https://github.com/pydata/pandas/issues/1476

[Dale Jung]

Try importing the attached file and using it per the notebook.

[Michael Foley]

Dale,

I've downloaded the charting.py file you posted, and that's exactly what I've been looking for.  Is there any way to use charting.fplot() and plot separate y axes for two Series with the same DatetimeIndex?  I noticed that there's no longer functionality for the secondary_y argument in fplot(), and I've been trying to come up with a clever workaround all day.

[Dale Jung]

Whoa. A post from 2012. I was confused at first. The following might be useful to you. 

https://github.com/dalejung/ts-charting

Notebook:

http://nbviewer.ipython.org/github/dalejung/ts-charting/blob/master/notebooks/ohlc.ipynb

The second y-axis can be named. So you can have multiple y-axes. While I don’t find multiple y-axes useful in conveying information, I will use it to keep data dense while I’m exploring haphazardly. 

-- 
Dale Jung

--
You received this message because you are subscribed to the Google Groups "PyData" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pydata+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jeff Reback]

dale if u want to add the notebook as a link into the pandas cookbook (with a hit of explanation)

would be great - pls submit a pull-request

[Alex Rothberg]

Was any solution to plotting multi-day time series with intra day data such that the plot skipped the overnight "gap" ever added to Pandas?