Download !FULL! Hourly Stock Data
Sumiko Fagnoni
I am looking for long time historical intraday day data on the S&P500 composite for a time horizon like 10 years with a - for example 10-minutes tick - or prices for call/put options on the S&P500 index itself.
What I tried so far:I checked Bloomberg Terminal and also contacted their Help Desk. They do have intraday data, but not on such a long time basis. They do offer 140 days to export to Excel and 240 days to view in terminal. That's it.I also checked Datastream 5.1, but they don't seem to offer intraday data at all...I am familiar with the Oxford-Man database, which is pretty nice, but I do need the raw data. I already saw the OptionMetrics database, is there any alternative? Maybe Macrobond?
AlgoSeek.com : Intraday data back to 2007 for US Equities, Futures and Options. So you can get S&P 500 data. Intraday they have tick, 1 sec, 1 min and 5 min OHLC bars. Institutional data and well priced.
QuantGo.com : New firm where clients rent access to data by the month from multiple data vendors but have to use Amazon AWS cloud computers on QuantGo's platform. Very affordable if you are can/willing to work with virtual cloud computers.
Tickdata.com : Global Equities, Futures and Options. Intraday bar data from 2004. Consolidated tape since 1993. Have tick and bar data with 1 sec/1 min/5 min OHLC. Institutional quality data but very expensive.
You can get minutely as-traded prices for all US securities on Quantopian, for free. You can't download the original data, but you can query it, analyze it, and do your research within a hosted IPython notebook on the website.
For pure academic work, I also think the T&Q (Trade and Quotations) database has what you are after. It is available through WRDS (Wharton Research Data Services). Again, very not free unless your uni pays for it for you. This database in particular is looked on very favourably by academic journals as there exists a full-time academic position at the NYSE simply to maintain this database and make sure it is fit for use in a research environment.
Every exchange in North America sells their historical data. NYSE and Nasdaq are the one's I'm most familiar with. They will sell you the data on a one off instance, ie no monthly fees, just a single flat rate.
CQG Data Factory offers decades of historical data online. Order and download accurate, top-quality data from over 60 exchanges worldwide.Access over 20 years of End Of Day market data and over 7 years of intraday data, including Time & Sales (tick data), intraday bar data, and trade volume. Additional data going back to the 1930s is also available.
If you are looking for historical data there are a few websites I can think of that provide it. www.eodata.com provides 30 days of historical EOD data for free in various text formats. Additionally, you can purchase up to 90 months of historical intraday data as well. An added benefit is that they cover US Options, Mutual Funds, Currencies, and Commodities.
The choice between exchange and data vendor is largely a matter of cost-to-symbols desired tradeoff. For a small number of symbols, a vendor package that gives you say, 1000 stock tickers and 50 options roots etc. is usually priced more competitively than the exchanges. On the other hand, buying directly from the exchange is useful if your goal is to acquire all the symbols. There's other trade-offs if you care about the cleanliness of the data a lot, but I will reserve that for another time since I think it's unlikely conditioned that you're only interested in the S&P 500 composite.
Don't know if you are looking for the options or futures, but if you are looking for S&P 500 futures, Portara have 1-min intraday data back to 1987. If you need a higher resolution, they have the level 1 tick data with bids and asks. They have sample data on their S&P 500 futures page.
FirstRate Data is a leading provider of high-resolution intraday stock market, crypto, futures, options and FX data. We source our historical stock data directly from major exchanges and fully adjust for both splits and dividends. Futures, options and ETF datasets are also sourced from co-located servers in major exchanges.
All datasets are rigorously tested for accuracy and completeness.
Our historical intraday data solutions are research-ready and used by traders, hedge funds and academic institutions. We offer 1-minute, 5-minute, 30-minute, 1-hour, and 1-day intraday stock data as well as intraday futures, options, ETFs, and FX data going back 15 years, and tick data going back 10 years.
As of December 2023 we offer historical 1-minute, 5-minute, 30-minute, and 1-hour intraday bars as well as daily bar data for 7494 stock tickers (including over 400 delisted tickers) starting 2005. All tickers listed on NYSE and Nasdaq with market capitalizations above $25M are included, as well as all tickers in Dow Jones Industrials, S&P500, Nasdaq100 and Russell 3000 indices (tickers previously included in the indices as well as delisted tickers are also included). For ETFs, we provide historical 1-minute data for the most active 850 tickers back to 2005.
Each 1-minute bar has OHLCV (open/high/low/close/volume) data which is aggregated from trades executed on major exchanges as well as four dark pools.
For tick data, each tick contains the timestamp, trade price, volume and exchange code (please see our stock tick data page for details on tick data.
Our options data offering covers options on over 5800 US stocks and indices. The data is arranged into option chains which list all the options for a given ticker in order of expiry and strike.
Traded option prices, as well as bid/ask quotes, and full Greeks (delta, gamma, vetga, theta, rho) for each option are provided.
For full details and samples, visit our Historical Options Data product page.
For futures, we carry 1-minute, 5-minute, 30-minute, 1-hour, and 1-day historical bars as well as daily bar data for the most active 130 contracts (as of December 2023) starting back to 2007.
We provide both individual futures contracts as well as a continuous futures series with prices adjusted for the price gaps from rolling contracts (this series is best suited to long timeframe backtesting of futures trading strategies).
We offer data by both individual ticker and by bundles. Bundles aggregate multiple intraday data sets for a specific niche - for example our Complete Stocks Bundle contains all 7000+ tickers we carry.
We provide two methods for accessing the data:
- Web download : customers are provided with a unique customer download page with links to zip archives for the data.
- API access (bundles only) : a full-featured API is available for customers that wish to access the data via programmatic script or from statistics packages such as Studio R. (note the historical data API serves zip archives of data files and not raw data).
Data can easily be converted to TradeStation, MetaStock, NinjaTrader, AmiBroker, Wealth-Lab formats, and for use in many other popular trading software applications as well as popular analysis tools/languages such as python, pandas and R / R Studio. Files for stocks and ETFs include out-of-hours trades.
For minute-by-minute data, bars with zero volumes (ie no trades) are excluded to reduce filesizes.
For efficient downloading we offer an API for downloading data in our bundles.
So my question is twofold--what's a simple, elegant way to quickly ingest data for a series of stocks into R, and how do I interpret the time stamping on the Google/Yahoo files that I would be using?
So downloading and standardizing the data ended up being more much of a bear than I figured it would--about 150 lines of code. The problem is that while Google provides the past 50 training days of data for all exchange-traded stocks, the time stamps within the days are not standardized: an index of '1,' for example could either refer to the first of second time increment on the first trading day in the data set. Even worse, stocks that only trade at low volumes only have entries where a transaction is recorded. For a high-volume stock like APPL that's no problem, but for low-volume small caps it means that your series will be missing much if not the majority of the data. This was problematic because I need all the stock series to lie neatly on to of each other for the analysis I'm doing.
and changing the stock ticker at the end will give you the past 50 days of trading days on 1/2-hourly increment. POSIX time stamps, very helpfully decoded by @geektrader, appear in the timestamp column at 3-week intervals. Though the timestamp indexes don't invariably correspond in a convenient 1:1 manner (I almost suspect this was intentional on Google's part) there is a pattern. For example, for the half-hourly series that I looked at the first trading day of ever three-week increment uniformly has timestamp indexes running in the 1:15 neighborhood. This could be 1:13, 1:14, 2:15--it all depends on the stock. I'm not sure what the 14th and 15th entries are: I suspect they are either daily summaries or after-hours trading info. The point is that there's no consistent pattern you can bank on.The first stamp in a training day, sadly, does not always contain the opening data. Same thing for the last entry and the closing data. I found that the only way to know what actually represents the trading data is to compare the numbers to the series on Google maps. After days of futiley trying to figure out how to pry a 1:1 mapping patter from the data, I settled on a "ballpark" strategy. I scraped APPL's data (a very high-volume traded stock) and set its timestamp indexes within each trading day as the reference values for the entire market. All days had a minimum of 13 increments, corresponding to the 6.5 hour trading day, but some had 14 or 15. Where this was the case I just truncated by taking the first 13 indexes. From there I used a while loop to essentially progress through the downloaded data of each stock ticker and compare its time stamp indexes within a given training day to the APPL timestamps. I kept the overlap, gap-filled the missing data, and cut out the non-overlapping portions.
df19127ead