"Cookie-Crumb"

Bert Mariani

unread,

May 29, 2017, 12:51:36 PM5/29/17

to The Ring Programming Language

Hello Mahmoud

Yahoo changed the format of the URL that would fetch historical stock data.

They now use a "Cookie-Crumb" in their website that is sent to the browser to be able to fetch and download the data.

I am using Chrome browser

If I copy the exact "url" it will work in another tab in the browser and return data.

If I copy the exact "url" into Ring and use the "cStr = download($url)" it will fail with "Invalid Cookie"

Is there a way in Ring, that can connect to the Yahoo site, get the "Cookie-Crumb", and then use it is a valid "Crumb" as part of the URL to make requests to Yahoo for the stock data ?

See attached screen captures for the manual steps used to get the "Cookie-Crumb" copy

=================

This Fails inside Ring ... Works inside browser (Google Chrome)

$url = "https://query1.finance.yahoo.com/v7/finance/download/AMZN?period1=1464538017&period2=1496074017&interval=1wk&events=history&crumb=zLCRKpYFBdW"

cStr = download($url) ### <<<== URL

================

Returns ERROR : Msg Output

{

"finance": {

"error": {

"code": "Unauthorized",

"description": "Invalid cookie"

}

===============

This used to Work, before Yahoo started using "Cookie-Crumb" format

$symbol = "AAPL"

$urlbase = "http://ichart.yahoo.com/table.csv?s="

$urlbaseType = "&a=" + $MonthStart + "&b=" + $DayStart + "&c=" + $YearStart + "&d=" + $MonthEnd + "&e=" + $DayEnd + "&f=" + $YearEnd + "&g=" + $DayWeekMonth + "&ignore=.csv"

$url = $urlbase + $symbol + $urlbaseType

cStr= download($url)

write("download.txt",cStr)

===============

Steps to manually get download of stock data

Copy-Paste this link in the browser. It will show AMZN - Amazon data

https://finance.yahoo.com/quote/AMZN/history?period1=1464538017&period2=1496074017&interval=1wk&filter=history&frequency=1wk

Right-Click the "Download Data" button

Select "Copy link address" (It has a new url with the "Cookie-Crumb" at the end,)

https://query1.finance.yahoo.com/v7/finance/download/AMZN?period1=1464538017&period2=1496074017&interval=1wk&events=history&crumb=zLCRKpYFBdW

Paste in in another Tab

Data is downloaded to a csv file.

Mahmoud Fayed

unread,

May 29, 2017, 2:25:43 PM5/29/17

to The Ring Programming Language

Hello Bert

(1) In this topic you will find the problem and the solution

https://stackoverflow.com/questions/44030983/yahoo-finance-url-not-working

(2) In this chapter in Ring documentation - You will find how to use RingLibCurl

http://ring-lang.sourceforge.net/doc1.3/libcurl.html

Greetings,

Mahmoud

Bert Mariani

unread,

Jun 3, 2017, 9:54:31 PM6/3/17

to The Ring Programming Language

Hello Mahmoud

I need your help on this.

Trying to fetch Yahoo historical stock data using Curl_Easy with Cookie and Crumb returned by Yahoo

The data return is to StdOut in the Dos Window.

- How can I capture the data output to a String or File ?

- The initial data "get request" returns thousands of bytes in Html format

- I need to capture the "CrumbStore":{"crumb":"a07fAG9jmUy"} and extract the Crumb

- The Cookie is returned to the Cookie.txt File

- I manual built for now, the $URL + Crumb

- The 2nd Curl_Easy request to Yahoo with the Cookie, URL and Crumb, returns the stock data to StdOut.

- Same question: how do I get the data returned in a String or File ?

- You can use the Cookie and Crumb in the code "as is". It should work.

- Before I would use "Download($URL) function

- This no longer returns anything .... is it a question of Cookie and Crumb ?

- Can a parameter be added so that it would imitate the CurlEasy ... to get the stock data ?

Greetings

Bert Mariani

=====================

OUTPUT 1st Part

<!DOCTYPE html><html id="atomic" class="NoJs desktop" lang="en-US"><head prefix="og: http://ogp.me/ns#"><script>window.performance && window.performance.mark && window.performance.mark('PageStart');</script><meta charset="utf-8"/><title>AMZN Historical Prices | Amazon.com, Inc. Stock - Yahoo Finance</title><meta name="keywords" content="AMZN, Amazon.com, Inc

...

... ..... "CrumbStore":{"crumb":"a07fAG9jmUy"}

...

URL Call: Result

016B4928

CURLcode

0

================

OUTPUT 2nd Part

Date,Open,High,Low,Close,Adj Close,Volume

2016-06-06,726.500000,731.500000,714.210022,717.909973,717.909973,13256700

2016-06-13,714.010010,722.559998,699.179993,706.390015,706.390015,17564300

2016-06-20,713.500000,722.119995,692.200012,698.960022,698.960022,18532700

================

OUTPUT 3rd Part

Back from download:

{

"finance": {

"error": {

"code": "Unauthorized",

"description": "Invalid cookie"

}

=========== END ===============

=====================

INPUT Code

load "libcurl.ring"

###------------------------------

### Get Request¶

fh = Fopen("fileYahooData.out","w")

See nl + nl +"===== GET REQUEST ====="+ nl +nl

### Part 1 --- Get Crumb and Cookie -----------------------------------------

curl = curl_easy_init() ### >>> HANDLE >>> 01006BD0 CURL 0

curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1) ### No output

curl_easy_setopt(curl, CURLOPT_COOKIEJAR, "cookies.txt") ### No output, Result to File

curl_easy_setopt(curl, CURLOPT_COOKIEFILE, "cookies.txt") ### No output, Result to File

curl_easy_setopt(curl, CURLOPT_URL, "https://finance.yahoo.com/quote/AMZN/history") ### No output

# curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, curl_write) ### ??? How do you do this

# curl_easy_setopt(curl, CURLOPT_WRITEDATA, &write_result)) ### ???

### >>> HTML Data to STDOUT Window

### Find: CrumbStore in StdOut, Cookie is in cookies.txt

### "CrumbStore":{"crumb":"a07fAG9jmUy"}

result = curl_easy_perform(curl) ### >>> 005F4250 CURLcode 0

See nl +nl +"===== URL Call: Result ======="+nl ; See result ; See nl +"============"+ nl +nl

### Part 2 --- Send URL with Crumb, and Cookie -----------------------------------------

myCrumb = "a07fAG9jmUy"

myCookie = "16n00i9cj0nj1&b=3&s=vq"

$url = "https://query1.finance.yahoo.com/v7/finance/download/AMZN?period1=1464726832&period2=1496262832&interval=1wk&events=history&crumb="+ myCrumb

$url2 = "https://query1.finance.yahoo.com/v7/finance/download/NVDA?period1=1464811233&period2=1496347233&interval=1wk&events=history&crumb="+ myCrumb

### Send URL and Cookie to Yahoo to fetch stock history data

### Cookie and AMZN

curl_easy_setopt(curl, CURLOPT_COOKIE, myCookie);

curl_easy_setopt(curl, CURLOPT_URL, $url);

curl_easy_perform(curl)

### NVDA

curl_easy_setopt(curl, CURLOPT_URL, $url2);

curl_easy_perform(curl)

### Part 3 --- Try Download() -----------------------------------------

See nl +"===== TRY Dowload(): =========="+ nl

cStr = download($url) ### <<<== URL

write("download.txt", cStr) ### Write data to file

See nl +"===== Back from download: ====="+ nl

See cStr

curl_easy_cleanup(curl) ### No output

See nl +"=========== END ==============="+ nl

AA-RingCurl-Test-Yahoo2.ring

Mahmoud Fayed

unread,

Jun 3, 2017, 10:35:01 PM6/3/17

to ring...@googlegroups.com

Hello Bert

To save the output from RingLibCurl to a String
use

curl_easy_perform_silent()

Example

load "libcurl.ring"

curl = curl_easy_init()

curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1)
curl_easy_setopt(curl, CURLOPT_URL, "http://ring-lang.sf.net")

cOutput = curl_easy_perform_silent(curl)

See "Output:" + nl
see cOutput

curl_easy_cleanup(curl)

To play with the output (The String)
use SubStr()

* To find a substring position
* To Get Substring

See Documentation : http://ring-lang.sourceforge.net/doc1.3/strings.html#substr-function

Example
# cHTML contains the HTML file
# Get the text starting from what we want
NewStr = substr(cHTML, substr(cHTML, '"CrumbStore":{"crumb":"' ) )

Then continue using SubStr() to remove text around what you want.
It will be simple in two lines of code.

Greetings,
Mahmoud

Bert Mariani

unread,

Jun 5, 2017, 12:27:42 PM6/5/17

to The Ring Programming Language

Hello Mahmoud

Thank you for the suggestions on using:

cOutput = curl_easy_perform_silent(curl)

The code uses the Cookie and Crumb provided by Yahoo

It is now able to fetch the historical stock data, after Yahoo changed their interface.

=================

OUTPUT

====== Output: AMZN ======

Date,Open,High,Low,Close,Adj Close,Volume

2017-05-01,927.799988,954.400024,927.799988,934.150024,934.150024,18182800

2017-05-08,940.950012,962.789978,939.210022,961.349976,961.349976,14594300

2017-05-15,958.729980,970.059998,944.119995,959.840027,959.840027,20453700

2017-05-22,964.000000,999.000000,962.900024,995.780029,995.780029,15795000

2017-05-29,996.510010,1001.200012,982.159973,994.619995,994.619995,7176200

====== Output: NVDA ======

Date,Open,High,Low,Close,Adj Close,Volume

2017-05-01,104.739998,106.849998,102.559998,103.860001,103.860001,42820600

2017-05-08,104.339996,130.429993,102.309998,127.889999,127.889999,156110500

2017-05-15,129.559998,138.220001,127.050003,136.000000,136.000000,141350300

2017-05-22,137.770004,145.279999,135.710007,141.839996,141.839996,93030500

2017-05-29,143.699997,147.000000,142.050003,144.350006,144.350006,46924200

On Monday, May 29, 2017 at 12:51:36 PM UTC-4, Bert Mariani wrote:

AA-RingCurl-Test-Yahoo6.ring

Mahmoud Fayed

unread,

Jun 5, 2017, 6:10:46 PM6/5/17

to The Ring Programming Language

Hello Bert

Very Nice Sample

Please

(1) Add it to this folder
https://github.com/ring-lang/ring/tree/master/samples/other

(2) Update this application, So we can use it again
https://github.com/ring-lang/ring/tree/master/applications/getquoteshistory

Keep up the good work :D

Greetings,
Mahmoud

Reply all

Reply to author

Forward