Finance data on Yahoo

[root]

Over the past two weeks Yahoo has rewritten their web site:
https://finance.yahoo.com
so that pages within that site are no longer accessed the
way they have been for many years.

For example, you used to be able to get key statistics via:
http://finance.yahoo.com/q/ks?s=msft
using msft for example.

This same information is now accessed via:
http://finance.yahoo.com/quote/msft/key-statistics

It seems this new page cannot be accessed except by qualified
browsers such as Chrome, Firefox, or Opera.


I normally use my own browser which started out as w3m
and my browser can't access the new page. Neither can
lynx, nor wget or curl.

If I fetch the page through Chrome (say) and save the
resulting html file then lynx and my browser can read
the page. Yahoo is not responsive to queries and I
am hoping someone might be able to suggest a work-around
so I can fetch the data.

Thanks.

[Sarcastic Fringehead]

On Mon, 01 May 2017 22:42:13 +0000, root wrote:

> Over the past two weeks Yahoo has rewritten their web site:
> https://finance.yahoo.com so that pages within that site are no longer
> accessed the way they have been for many years.
>
> For example, you used to be able to get key statistics via:
> http://finance.yahoo.com/q/ks?s=msft using msft for example.
>
> This same information is now accessed via:
> http://finance.yahoo.com/quote/msft/key-statistics

This works well for me:
http://www.circlemud.org/~jelson/software/quote

[Allodoxaphobia]

I've been using cURL to fetch quotes via
http://finance.yahoo.com/quotes/^DJI,^IXIC,^GSPC,LLY,HON,HPQ?ltr=1&bypass=true
and it, too, went snafu on Monday.
(The "^DJI,^IXIC,^GSPC," are (were) "symbols" for DJIA, NASDAQ, S&P500.)

Dropping that URL into either Chrome or Firefox here results in a totally
blank screen produced by a 3,000 line html/css/javascript hodgepodge.

Yahoo Finance had (in the past) an API for fetching stock data in CSV
format. I wonder if that went snafu, too? I'll have to re-research
that information.

Jonesy
--
Marvin L Jones | Marvin | W3DHJ.net | linux
38.238N 104.547W | @ jonz.net | Jonesy | FreeBSD
* Killfiling google & XXXXbanter.com: jonz.net/ng.htm

[Rich]

It still works (at the moment) for some of your quotes:

$ curl "http://download.finance.yahoo.com/d/quotes.csv?s=^DJI+^IXIC+^GSPC+LLY+HON+HPQ&f=sl1c1t1d1v&e=.csv"
"^DJI",N/A,N/A,N/A,N/A,N/A
"^IXIC",6091.60,+44.00,"5:15pm","5/1/2017",1521734789
"^GSPC",2388.33,+4.13,"4:35pm","5/1/2017",1893003482
"LLY",81.82,-0.24,"4:01pm","5/1/2017",2397236
"HON",130.90,-0.24,"4:03pm","5/1/2017",3164456
"HPQ",18.93,+0.11,"4:03pm","5/1/2017",7958624

[root]

Thanks, but getting a simple price quote is different from
the key-statistics. Key-statistics includes over 91 lines of
data about the economic condition of the company.

finance.yahoo.com used to allow you to get income statements,
balance sheets, and cash flow statements for the various
companies.

[root]

The bypass=true trick worked for exactly one week for me. Using Firefox
or Chrome you can still get what you want (except for ^DJI which is a
puzzle to me). Use this URL:

http://finance.yahoo.com/quotes/^IXIC+LLY+HON+..../view/v1

>
> Yahoo Finance had (in the past) an API for fetching stock data in CSV
> format. I wonder if that went snafu, too? I'll have to re-research
> that information.

You can still get the price quotes, at least as of yesterday.
>
> Jonesy

[root]

Thanks for responding, but as I mentioned in a previous response simple price
quotes are not the problem. Yahoo used to provide an ensemble of financial
reports for every stock currently traded. Take, for example, the balance
sheet. This URL used to give you the annual balance sheet:

http://finance.yahoo.com/q/bs?s=$stock\&annual
where $stock is the ticker symbol. As you can see the element /q/ indicates
it is coming from the quote section of yahoo, but the bs? tells it to
provide the balance sheet.

Under the new yahoo, the bs command becomes:

http://finance.yahoo.com/quote/$stock/balance-sheet.

Firefox and Chrome can bring up this page for ibm as an example.

http://finance.yahoo.com/quote/ibm/balance-sheet

As I am writing this I find that wget can fetch this page but lynx
and w3m cannot. Similarly the key-statistics page I mentioned above
and be accessed by wget. So Yahoo is changing its site as we speak.

I have a workaround now, I can fetch the pages with wget and
strip out the data.

[Henrik Carlqvist]

On Tue, 02 May 2017 08:31:30 +0000, root wrote:
> The bypass=true trick worked for exactly one week for me. Using Firefox
> or Chrome you can still get what you want (except for ^DJI which is a
> puzzle to me). Use this URL:
>
> http://finance.yahoo.com/quotes/^IXIC+LLY+HON+..../view/v1

To me it seems as if those http-urls at finance.yahoo.com gets redirected
to https urls instead. For a browser like Firefox or Chrome https is no
problem, but I use the old and unmaintained software qtstalker
( http://qtstalker.sourceforge.net/ ) for technical analysis.

Having the source to qtstalker I was able to update the URLs and
recompile the Yahoo quote plugin. Unfortunately the QUrlOperator class in
qt used to download data from URLs does not seem to support https.

My solution to this problem was to modify the URLs in Yahoo.cpp to http
URLs looking like this:
http://localhost/~henca/https/url.php/chart.finance.yahoo.com/table.csv?..

And writing a simple php script which fetches data using https from any
URL and delivers the fetched data by http:

url.php:
-8<---------------
<?

if(strlen($_SERVER["QUERY_STRING"]))
$extra="?" . $_SERVER["QUERY_STRING"];
else
$extra="";

$ch=curl_init();
curl_setopt($ch, CURLOPT_URL, "https:/" . $_SERVER["PATH_INFO"] . $extra);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch,CURLOPT_FOLLOWLOCATION,true); // follow http redirect
curl_setopt($ch,CURLOPT_SSL_VERIFYPEER,true); //temporary ignore
certificate error for easier testing
$data=curl_exec($ch);
curl_close($ch);
?>
-8<---------------

regards Henrik
--
The address in the header is only to prevent spam. My real address is:
hc351(at)poolhem.se Examples of addresses which go to spammers:
root@localhost postmaster@localhost

[root]

["Followup-To:" header set to alt.os.linux.slackware.]

Henrik Carlqvist <Henrik.C...@deadspam.com> wrote:

>> http://finance.yahoo.com/quotes/^IXIC+LLY+HON+..../view/v1
>
> To me it seems as if those http-urls at finance.yahoo.com gets redirected
> to https urls instead. For a browser like Firefox or Chrome https is no
> problem, but I use the old and unmaintained software qtstalker
> ( http://qtstalker.sourceforge.net/ ) for technical analysis.
>
> Having the source to qtstalker I was able to update the URLs and
> recompile the Yahoo quote plugin. Unfortunately the QUrlOperator class in
> qt used to download data from URLs does not seem to support https.
>
> My solution to this problem was to modify the URLs in Yahoo.cpp to http
> URLs looking like this:
> http://localhost/~henca/https/url.php/chart.finance.yahoo.com/table.csv?..

^^^^^^




Thanks for responding Henrik.

Both lynx and my w3m can handle https/http. This is not the problem.
The long standing method for downloading some data from Yahoo described
here:
http://www.jarloo.com/yahoo_finance/

has not changed. The URL which you show for fetching .csv data
is functional. I have marked the change in URL above. I use
ichart instead of chart, real-chart also works, I suppose for
real time data. One big problem is that fetching stock quotes
via the ichart.finance.yahoo takes about three times as long
as it did previously. Quotes from google run substantially
faster but they require reformatting of the date and they fail
to find data for many stocks.

You can get quotes and many other reports directly from nasdaq.

I have managed work-arounds for most of the Yahoo reports using
google.

[jrg]

I know you're running via console, but FWIW, on my 64 bit desktop using
e-17 wm and firefox, yahoo is near useless now and NASDAQ ain't far
behind. Sometime in the recent past, yahoo changed their layout and I
don't know what else but the net effect has been abominable. Takes
30-40 seconds for new quote screen to paint and it doesn't indicate what
its doing anywhere - just sits there until. Then starts popping up
videos all over (I've just disabled flash and that helps) but it seems
that yahoo isn't the best source of anything anymore - I commend your
fortitude but have to wonder if maybe you should rethink using that
site. And google has become useless for finding news - google bombs
abound - seems like everyone with a computer can become a financial guru
today.

[root]

jrg <conn...@cox.net> wrote:
>> I have managed work-arounds for most of the Yahoo reports using
>> google.
>>
>>
> I know you're running via console, but FWIW, on my 64 bit desktop using
> e-17 wm and firefox, yahoo is near useless now and NASDAQ ain't far
> behind. Sometime in the recent past, yahoo changed their layout and I
> don't know what else but the net effect has been abominable. Takes
> 30-40 seconds for new quote screen to paint and it doesn't indicate what
> its doing anywhere - just sits there until. Then starts popping up
> videos all over (I've just disabled flash and that helps) but it seems
> that yahoo isn't the best source of anything anymore - I commend your
> fortitude but have to wonder if maybe you should rethink using that
> site. And google has become useless for finding news - google bombs
> abound - seems like everyone with a computer can become a financial guru
> today.
>
>

Thanks for responding.

I don't understand what Yahoo is doing/has done: If they don't want to
provide the service why not just shut everything down?

On the other hand, google finance seems to be playing games to limit
the usefulness of their data. Google provides in one chunk quarterly/annual
reports for income, balance and cash-flow. However the format changes
between stocks so it is nearly impossible to align stocks together
for comparison. Google finance doesn't cover all the exchanges
and neither does Nasdaq.

I have two machines running side-by-side. One runs chrome and a
page is open to Yahoo quotes stocks of interest to me. That
page is very well behaved and provides (nearly) real-time
data. I'm not a day trader, but I watch what is going on.

Don't get me started on how difficult it is to work with
data coming from the St. Louis Fed.

[jrg]

On 05/08/2017 06:31 AM, root wrote:

<snip>

> I don't understand what Yahoo is doing/has done: If they don't want to
> provide the service why not just shut everything down?

<snip>

Probably something to do with the security problems plus an ongoing
attempt to scale down/up to make themselves salable...

> Don't get me started on how difficult it is to work with
> data coming from the St. Louis Fed.

OT
Whats with the St.Louis Fed?
Are they the gadfly one?

[root]

The SLF maintains a trove of economic and financial data:
https://research.stlouisfed.org/fred2/

I have over 625,000 time series from there. You can visit
their site and browse over the "popular" offerings.

If you find the stuff interesting bounce back and I
will tell you how I work with the data.

[Lynx user]

Try changing the user-agent setting in Lynx to one of the qualified
browsers. This seems to work for the yahoo.com page mentioned.

In this case, I used:
Mozilla/5.0 (Windows NT 6.1; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0

You'll get the warning:
Warning: User-Agent string does not contain "Lynx" or "L_y_n_x"!
when you exit Lynx.

[root]

Thanks for responding.

It didn't seem to work for me. I'm not sure which specific Yahoo
URL you tried. I tried this:
lynx -useragent="Mozilla/5.0 (Windows NT 6.1; WOW64; rv:47.0) Gecko/20100101 Firefox/47.0" -dump https://finance.yahoo.com/quotes/tvty+tyl+ulta+v+vale+vsto+wmb+wpx+x/view/v1

And I got no useful output.

[root]

["Followup-To:" header set to alt.os.linux.slackware.]

I pursued your suggestion further, and changing the User-Agent does allow me
to access the Yahoo reports such as Income-Statement

That has been a great help to me, thanks very much for the suggestion.

[Robert Riches]

As I think has been mentioned in other branches of this thread,
Yahoo appears to have dropped reporting the Dow Jones 30. Is
there some other symbol, perhaps an ETF, that is based on the Dow
(perhaps scaled by some constant of proportionality) and is
sufficient for a simple chart?

By the way, as of a few days ago, the CSV historical data now
requires fetching an HTML page and supplying a 'crumb' to
download the CSV.

Thanks.

--
Robert Riches
spamt...@jacob21819.net
(Yes, that is one of my email addresses.)

[Henrik Carlqvist]

On Fri, 19 May 2017 04:13:25 +0000, Robert Riches wrote:
> By the way, as of a few days ago, the CSV historical data now requires
> fetching an HTML page and supplying a 'crumb' to download the CSV.

That probably explains why my fix to qtstalker together with the http-
https-proxy no longer works...

[root]

Robert Riches <spamt...@jacob21819.net> wrote:
>
> As I think has been mentioned in other branches of this thread,
> Yahoo appears to have dropped reporting the Dow Jones 30. Is
> there some other symbol, perhaps an ETF, that is based on the Dow
> (perhaps scaled by some constant of proportionality) and is
> sufficient for a simple chart?
>
> By the way, as of a few days ago, the CSV historical data now
> requires fetching an HTML page and supplying a 'crumb' to
> download the CSV.
>
> Thanks.
>

You can still get csv data for the current day but that is unreliable.
If a given stock is not traded on that day you will miss an entry.
After 2017-05-12 Yahoo has delisted a number (something like 200)
stocks that were traded on on the previous Friday.

As you have noticed, to be sure of updating your stocks you
have to download the historical data. Formerly you could
specify starting/ending dates when requesting historical
data, but that is no longer true. Now, unless you use
a browser to mark MAX, you get the last 100 days of
data.

I have written a small C program to fetch the current historical
data and pull out the lines that fill in the missing data since
the last recorded trade. As another nuisance, the historical
data has changed the date form from, for example, 2017-05-15
to May 5, 2017.

With the new system you can start updating your historical
records about half an hour after the market closes. Formerly
it took five hours or more for the current (historical) data
to be made available.

You have to be careful with the current historical data:
the format sometimes changes the position between the
closing price and the actual closing price. Formerly
the data was Date, Open, High, Low, Close, Volume,
Actual Close (not considering splits). Now the
Close and Actual Close are sometimes reversed. So far this
has only happened when I used a browser to fetch the
data.

If you do use a browser you know that the URL for fetching
all the data has two "magic" numbers and does not contain
the stock symbol. If we could figure out the two magic
numbers we could fetch historical data as before the
change.

The first two magic numbers I encountered happened to
differ by 365*86,400 which is the number of seconds
in a year. I thought I was onto something but that
was just a fluke.

Thanks for responding.

[root]

Henrik Carlqvist <Henrik.C...@deadspam.com> wrote:
> On Fri, 19 May 2017 04:13:25 +0000, Robert Riches wrote:
>> By the way, as of a few days ago, the CSV historical data now requires
>> fetching an HTML page and supplying a 'crumb' to download the CSV.
>
> That probably explains why my fix to qtstalker together with the http-
> https-proxy no longer works...
>
> regards Henrik

Tell me Henrik, do you really use technical analysis?

[Henrik Carlqvist]

On Fri, 19 May 2017 14:15:32 +0000, root wrote:
> Tell me Henrik, do you really use technical analysis?

Yes I do. Not to decide which stocks to buy or sell, but to choose the
day for placing a bid.

[Robert Riches]

I'm still struggling to get historical prices for a given
symbol. This page comes up in a browser, with a 'Download Data'
button:

https://finance.yahoo.com/quote/MU/history?p=MU

That button goes to something of this form:

https://query1.finance.yahoo.com/v7/finance/download/MU?period1=1492574329&period2=1495166329&interval=1d&events=history&crumb=...

where the crumb=... at the end looks like a hash of some kind.

When using wget or similar method, the only occurrence of
'Download Data' from the first URL is in something that looks
like a complex JSON structure. It appears the second URL is
manufactured by JavaScript, AJAX, and so forth.

If I copy the second URL, I can point the browser at it but wget
or similar method get a 401 unauthorized error. The site must be
using cookies and/or the user-agent string.

Are you getting any closer to downloading historical prices for
previous days without using a full-blown browser?

Thanks,

[root]

Robert Riches <spamt...@jacob21819.net> wrote:
>
> I'm still struggling to get historical prices for a given
> symbol. This page comes up in a browser, with a 'Download Data'
> button:
>
> https://finance.yahoo.com/quote/MU/history?p=MU
>
> That button goes to something of this form:
>
> https://query1.finance.yahoo.com/v7/finance/download/MU?period1=1492574329&period2=1495166329&interval=1d&events=history&crumb=...

Your site differs from the one I use:

https://finance.yahoo.com/quote/VLO/history?period1=378979200&period2=1495004400&interval=1d&filter=history&frequency=1d

I find stuff from yahoo by trying google searches in a form such
as this:

inurl:finance.yahoo.com inurl:history inurl:ibm

none of these searches brings up query1.finance.yahoo.com, how did you
find that source?

>
> where the crumb=... at the end looks like a hash of some kind.
>
> When using wget or similar method, the only occurrence of
> 'Download Data' from the first URL is in something that looks
> like a complex JSON structure. It appears the second URL is
> manufactured by JavaScript, AJAX, and so forth.

>
> If I copy the second URL, I can point the browser at it but wget
> or similar method get a 401 unauthorized error. The site must be
> using cookies and/or the user-agent string.

I tried starting tcpdump before using a browser to fetch the complete
record but I couldn't find anything useful in the dump which would
allow me to fetch the data directly.

>
> Are you getting any closer to downloading historical prices for
> previous days without using a full-blown browser?
>
> Thanks,

No, I am no closer to fetching the complete history for a stock
without a browser.
>

[Robert Riches]

On 2017-05-24, root <NoE...@home.org> wrote:
> Robert Riches <spamt...@jacob21819.net> wrote:
>>
>> I'm still struggling to get historical prices for a given
>> symbol. This page comes up in a browser, with a 'Download Data'
>> button:
>>
>> https://finance.yahoo.com/quote/MU/history?p=MU
>>
>> That button goes to something of this form:
>>
>> https://query1.finance.yahoo.com/v7/finance/download/MU?period1=1492574329&period2=1495166329&interval=1d&events=history&crumb=...
>
> Your site differs from the one I use:
>
> https://finance.yahoo.com/quote/VLO/history?period1=378979200&period2=1495004400&interval=1d&filter=history&frequency=1d
>
> I find stuff from yahoo by trying google searches in a form such
> as this:
>
> inurl:finance.yahoo.com inurl:history inurl:ibm
>
> none of these searches brings up query1.finance.yahoo.com, how did you
> find that source?

Let's see if I can remember/repeat the steps to get there:

Start at https://www.yahoo.com/

Click 'Finance', which goes to https://finance.yahoo.com/

Enter a symbol ('MU' or 'ADBE' or 'RHT') in the 'Quote
Lookup' box, which goes to
https://finance.yahoo.com/quote/MU?p=MU
https://finance.yahoo.com/quote/ADBE?p=ADBE
https://finance.yahoo.com/quote/RHT?p=RHT

(Wait a considerable time for all the ads and other fluff to
load.)

Click on 'Historical Data', which goes to
https://finance.yahoo.com/quote/MU/history?p=MU
https://finance.yahoo.com/quote/ADBE/history?p=ADBE
https://finance.yahoo.com/quote/RHT/history?p=RHT

(Again, wait a while for all the ads and other fluff to
load.)

There's a 'Download Data' link/button. That loads the CSV
file via a URL with the crumb=... magic number.

For me, it appears the crumb=... magic number is the same for all
symbols I tried. Its purpose appears to be to ensure I had
opportunity to see all the ads before I downloaded the CSV.

>> where the crumb=... at the end looks like a hash of some kind.
>>
>> When using wget or similar method, the only occurrence of
>> 'Download Data' from the first URL is in something that looks
>> like a complex JSON structure. It appears the second URL is
>> manufactured by JavaScript, AJAX, and so forth.
>
>>
>> If I copy the second URL, I can point the browser at it but wget
>> or similar method get a 401 unauthorized error. The site must be
>> using cookies and/or the user-agent string.
>
> I tried starting tcpdump before using a browser to fetch the complete
> record but I couldn't find anything useful in the dump which would
> allow me to fetch the data directly.

Ahhhh, excellent method. I would not have thought to try that.

>> Are you getting any closer to downloading historical prices for
>> previous days without using a full-blown browser?
>>
>> Thanks,
>
>
> No, I am no closer to fetching the complete history for a stock
> without a browser.

Neither am I. In fact, last night I had a setback. I had
planned to fetch the .../history?p=... HTML and extract the URL
from the 'Download Data' link. However, that link and its URL
are not in the HTML but evidently come from JavaScript or AJAX or
something similar. I might have to resort to using a browser and
xmacroplay to automate the mouse clicks and keystrokes. That
will be inconvenient if it comes to that.

Thanks and HTH