update data bundle with latest data on daily basis

91 views
Skip to first unread message

Justin Lent

unread,
Aug 24, 2016, 5:23:20 PM8/24/16
to Zipline Python Opensource Backtester, guil...@stratifi.com
Hey all,

My colleague (cc: Guillaume) and I have a fairly large data bundle which takes on order of 6 hours to load 5 years of data when originally ingesting it into the bundle.  However, as we get in data updates after each day's market close, we would like to append this new data to our existing previously ingested zipline data bundle. What is the preferred method to accomplish this within the data bundle framework?

-Best,
Justin / Guillaume

Scott Sanderson

unread,
Aug 29, 2016, 4:22:28 PM8/29/16
to Zipline Python Opensource Backtester, guil...@stratifi.com
Hi Justin!

Sorry for the long response delay.  There isn't currently a built-in system for updating an existing bundle in-place. though I could certainly imagine an API for doing so in the future.

Probably your best bet right now would be to make your bundle writer try to load the data written by a previous `ingest` call.  You can get your previously written data by doing something like:

    >>> from zipline.data.bundles import load
  >>> bundle_data = load('my-bundle-name')

By default, that should give you a BundleData object containing most recently-written version of the bundle named 'my-bundle-data'.  `load` also takes an optional `timestamp` argument, which you can use to request data from a specific ingestion.  The BundleData object is a namedtuple with `asset_finder`, `equity_minute_bar_reader`, `daily_bar_reader`, and `adjustment_reader` as attributes, which you can use to get your previously-written data.

Hope that helps,
-Scott

Justin Lent

unread,
Aug 29, 2016, 8:10:23 PM8/29/16
to Zipline Python Opensource Backtester, guil...@stratifi.com
Thanks for this explanation Scott - it helps quite a bit.  I think we'll know how to take it from here using this concept of getting previously written data and then appending to it.

-Justin
Reply all
Reply to author
Forward
0 new messages