Scraper Tool

[flu...@gmail.com]

Hi everyone,

like so many others, my camera has been stolen about 2 days ago, today I used the Flickr-Scraper for a few hours an was able to scan nearly 7500 Pictures.
I'd now like to run the scraper tool for a longer period,thus I'd like to ask if there is a commandline version of the tool available?
(my server runs linux 64bit)

regards,
Florian

[Matt Burns]

I'm currently in the final stages of developing such a tool. I'll let you know when it's ready for general release.

Thanks for your interest and I hope it helps find your camera!

Matt

[fluetke]

hi, 

its been a while since the last post.

I'm currently using an ugly hack to get flickr scrapr to work on my server:

 * Installed Xrdp, started flickr scraper (and a script that takes screenshots every 15minutes) in an rdp session, then firewalled it so nobody gets through.

works fine so far.

Florian

[Matt Burns]

Hi Florian,

Thanks for helping. I'll email you an early-release command line version that I'm sure you'll find much easier (and hopefully faster).

Matt

[fluetke]

Hi Matt, 

thanks for emailing the early version, looks good so far. 

yodaTime is spitting out parseExceptions, but this doesn't look like a real problem.

I'm currently working on my bachelors degree in computer science, so if you need help testing and developing the scraper, 

just drop me a line and i'll try to help as best as I can.

Florian

[fluetke]

I think I'll take a look at the source on google code and see what i can do.

[Matt Burns]

Thanks,

You're right, the exceptions are nothing to worry about. Basically, if an image is found with a date in an illegal format, then it is aborted and processing continues. It just doesn't look that pretty!

To help you get started, the source is separated into several components. That jar is built from scf-flickrtools. Simply type "ant jar" to build it (assuming ant is installed / on your path).

It in turn uses another component, the scf-api, which is a java library for communicating with my web services / live database. It basically wraps the public api so that it can be invoked in nice java :) The other key advantage is that it pre-validates the request so as to save time and bandwidth. For example, better validation could be applied to the date in the RequestValidator. There is a corresponding test class that gives you an idea of what is validated.

I think all of the components are stand alone eclipse projects. Once checked out, just import them into eclipse. Most have ant build scripts too.

I should probably write this sort of stuff up properly on some wiki documentation pages...

I did my Software Engineering degree ~10 years ago (eek, when did I get old?) but am happy to help you out however I can. If you're looking for final year project, a distributed, de-centralised image-crawling web spider would be a great (and useful) project...

Thanks again,

Matt

[Håkan Lindgren]

ant jar

[Tariq Mageed]