Google Trends Data and Mathematica
jroge
Trends? I am able to download the CSV files from Google and then load
into Mathematica easily enough, but I would rather skip the
intermediate steps and directly pull the data from Google into my
Mathematica notebook. I think where I am getting stuck is that Google
requires authentication and programming that into Mathematica is a bit
beyond my skill sets. Any thoughts?
Jon
Szabolcs Horvát
If you don't mind fiddling with Python a bit, you could use this script:
http://www.juiceanalytics.com/openjuice/programmatic-google-trends-api/
Chris Osborn
Have you tried "Import?" It allows you to specify a URL; perhaps it
will suffice.
Chris
telefunkenvf14
I was trying to do something similar just the other day... However,
the examples in Mathematica's documentation, concerning using web
services to access Google, are broken. I believe Chris Osborn's
suggestion of using Import[] should work.
I'll play around this weekend trying to write some functions
corresponding to the google search url options (more info here:
http://www.our-picks.com/archives/2007/01/30/google-search-urls-revealed-or-how-to-create-your-own-search-url/).
Maybe then we/you can help adapt the code for the Trends searches.
I generally find that getting data into Mathematica can be a bit
cumbersome--probably because I've been spoiled/ruined by pointy-clicky
my whole life. Also, if you have Excel 2007 you should be able to do
the same type of thing. (Querying and automatically having Excel pull
the data. Actually read something about customizing Google queries for
Excel the other day. Bookmarked it somewhere...)
jroge
> On Oct 7, 6:08 am, jroge <jr...@mac.com> wrote:
>
> > I am looking for some advice for directly importing data from Google
> > Trends? I am able to download the CSV files from Google and then load
> > into Mathematica easily enough, but I would rather skip the
> > intermediate steps and directly pull the data from Google into my
> > Mathematica notebook. I think where I am getting stuck is that Google
> > requires authentication and programming that into Mathematica is a bit
> > beyond my skill sets. Any thoughts?
>
> > Jon
>
> I was trying to do something similar just the other day... However,
> the examples in Mathematica's documentation, concerning using web
> services to access Google, are broken. I believe Chris Osborn's
> suggestion of using Import[] should work.
>
> I'll play around this weekend trying to write some functions
.our-picks.com/archives/2007/01/30/google-search-urls-revea...).
> Maybe then we/you can help adapt the code for the Trends searches.
>
> I generally find that getting data into Mathematica can be a bit
> cumbersome--probably because I've been spoiled/ruined by pointy-clicky
> my whole life. Also, if you have Excel 2007 you should be able to do
> the same type of thing. (Querying and automatically having Excel pull
> the data. Actually read something about customizing Google queries for
> Excel the other day. Bookmarked it somewhere...)
Import is certainly the right function, it is actually the one I was
using. The roadblock I am running into the user authentication. Google
requires you to first log-in to you account . Where I am getting stuck
is the proper way to represent the user credentials when I run Import.
The Wolfram Blog post on tweeting from mathematica is the closest
documentation I can find, but that is an apples to oranges scenario.
Jonah
I've been working on a way of importing data using mathematica's java
functionality (using apache's Httpclient). It worked until a couple
weeks ago. It still gives a successful http code response when
requesting data but that data is not read correctly. It seems not to
be recognizing the data format. I'm not sure what changed... maybe
google updated something on its webpage. I've been distracted from
working on it lately but I'd like to figure out what's going on. If
anybody has any suggestions on how to fix it I'd definitely appreciate
the help. Don't forget to change the PASSWORD and EMAIL to correlate
to your account.
<< Jlink`
passwordattribute = "PASSWORD"; (*google account password*)
emailattribute =
"EM...@gmail.com";(*google email*)
(*Initiate Http client*)
client = JavaNew["org.apache.commons.httpclient.HttpClient"];
client@getHostConfiguration[]@setHost["www.google.com", 443, "https"]
(*prepare Request Parameters*)
gat3t = JavaNew["org.apache.commons.httpclient.NameValuePair", "GA3T",
"ouVrvynsses"];
galx = JavaNew["org.apache.commons.httpclient.NameValuePair", "GALX",
"ouVrvynQwdf"];
continue =
JavaNew["org.apache.commons.httpclient.NameValuePair", "continue",
"http://www.google.com/insights/search/#"];
nui = JavaNew["org.apache.commons.httpclient.NameValuePair", "nui",
"1"];
hl = JavaNew["org.apache.commons.httpclient.NameValuePair", "hl",
"en-US"];
rmshown =
JavaNew["org.apache.commons.httpclient.NameValuePair", "rmShown",
"1"];
persistent =
JavaNew["org.apache.commons.httpclient.NameValuePair",
"PersistentCookie", "yes"];
accountId =
JavaNew["org.apache.commons.httpclient.NameValuePair", "Email",
emailattribute];
pass = JavaNew["org.apache.commons.httpclient.NameValuePair",
"Passwd", passwordattribute];
service =
JavaNew["org.apache.commons.httpclient.NameValuePair", "service",
"trends"];
nameValuePairs = {gat3t, galx, continue, nui, hl, rmshown,
persistent, accountId, pass, service };
postMethod =
JavaNew["org.apache.commons.httpclient.methods.PostMethod",
"https://www.google.com/accounts/ServiceLoginBoxAuth"];
postMethod@setRequestBody[nameValuePairs]
client@executeMethod[postMethod]
postMethod@releaseConnection[]
(*make request for data*)
insights[query_String] := (
redirect =
JavaNew["org.apache.commons.httpclient.methods.GetMethod",
"http://www.google.com/insights/search/overviewReport?q=" <>
query <> "&cmpt=q&content=1&export=1"];
client@executeMethod[redirect];
(*get data and drop first 5 lines "header"*)
(* data = Drop[ImportString[redirect@getResponseBodyAsStream[]],
5];*)
data=Drop[ImportString[redirect@getResponseBodyAsString[]],5];
insightDateAndData = {DateList[
StringSplit[#[[1]], " - "][[2]]], #[[2]]} & /@ (Take[
data, (Position[data, {""}][[1]])[[1]] - 1]);
redirect@releaseConnection[];
Return[insightDateAndData]
)