Data Gathering : imacro and firefox

[Angus Sinclair]

I would like to "scrape" some data from google finance to use as
training data in a stat440 course project. I have tried to find the
high-frequency data I need elsewhere, and this appears to be my only option.

I have an "imacro" ( a scripting plugin for firefox) that does exactly
what I need. The problem is I need it to run all day (9:30am to 4pm with
1 minute polling). Sometimes i dont wake up in time to start it on my
home computer, and this causes incomplete data.

I tried installing the imacro plugin on my cs account so I could run a
"screen" session and just basically cron it to run automatically.
However, the installation process failed.

I was wondering, is there a way I can install the imacros plugin? As I
said, its for research/data gathering purposes. If the plugin was
installed on my account, I believe I can use command line support to run
the macro as necessary.

Thanks for any help/ideas/input.

Angus Sinclair

[Derek Shaw]

Scrapping through firefox is a really, really bad idea... and a
technical support forum is probably not the best way to ask for help
with scrapping issues... but it is an interesting technical problem.

For a quick hack, I would suggest using cron run wget:

*/5 * * * * /usr/bin/wget http://www.skichicopee.com/webcam/camera.jpg
-q -O `/bin/date +\%Y-\%0m-\%0d-\%0k-\%0M`.jpg

(followed by a well written perl/ack/python/etc script to extract the
relevant information)

However, if you are looking for stock information with one minute
resolutions, I'm sure that there is service out there that has all of
the historical information available.

-Derek