Anyone with WebCrawling experience
Pek
I was wondering if anyone here has experience writing and working with
web crawlers. I'd like to talk to you and get some insight on parsing
the data.
Thanks
Duane Johnson
Tom Printy
-Tom
Matthew Walsh
scraping a few pages? Or indexing the web?
Matt
Pek
I have a social media monitoring platform SaaS which currently
monitors the microblogs - Twitter, Facebook, Buzz. I'd like to make
this more full featured and monitor the social web. This is where the
crawlers come in. Say your a brand like Starbucks. It's not just
conversations about Starbucks on Twitter and Facebook, but I'd like to
uncover blog posts, comments, etc about Starbucks and have a way to
rank them. I hope that's a better picture.
On May 25, 10:53 am, Matthew Walsh <wals...@purdue.edu> wrote:
> "experience writing and working with web crawlers"
>
> If you add a little context, we may be able to help more. Do you mean
> scraping a few pages? Or indexing the web?
>
> Matt
>
>
>
> On Tue, May 25, 2010 at 5:45 PM, Tom Printy <tpri...@gmail.com> wrote:
> > I have done a lot in Perl........ It can be a really difficult task unless
> > there is an RSS feed or something like that.
>
> > -Tom
>
Matthew Walsh
You probably don't want to crawl the whole web: consider running your
queries through Yahoo BOSS (and lookup the Bing counterpart)
http://en.wikipedia.org/wiki/Yahoo!_Search_BOSS
Matt
Abuna Demoz
I've got experience doing large scale web crawling. Its a
interesting, but time consuming endeavor. In the long term, its
better to write your own crawler and have your own db. But depending
on the size of your team, that could take a year by itself. If you're
in the prototyping stage, Matt's advice to use search API's definitely
makes the most sense.
-Abuna
Jonathan Nelson
work. My language of choice is Python, and my screen scraping tools
include the mechanize, BeautifulSoup, twill. I've also been
researching http://scrapy.org/ for my next project.
Crawling random blogs and comments is going to be a Hard Problem
(TM). If you can pick a site like Blogger.com, and decide to scrape
that, it becomes a bit easier, but you're still talking about scraping
and indexing millions of web pages. You might be better off setting
up Google alerts for the specific terms that you're looking for, and
then scraping the results that come up and indexing those.
Mind you, if you're well funded, and don't mind spinning up a bunch of
EC2 instances to do that kind of crawling, that becomes a big issue.
Indexing those results rapidly becomes a Big Data problem. If you
think of a web page being 20kb of text x 1,000,000 blog posts a day ==
20GB of data a day that you're just crawling and then indexing. After
8 months, you have a Terabyte of data that you're having to deal with
and move around.
You're also probably looking at using Cassandra, or Hadoop/Hbase as a
back end for a data store. You're also looking at using a lot of map/
reduce to do the indexing for you.
It sounds like an interesting problem, however. I'm tackling a
similar problem in that I want to search and index financial news for
http://Newsley.com. I'm going to be pulling RSS feeds, and running
them through an http://OpenCalais.com api, however.
I'd love to hear more about what you're doing.
Best,
Jonathan
Zishan Ahmad
--
LinkedIn - http://www.linkedin.com/in/zishanahmad
Twitter - http://www.twitter.com/zishanahmad
Schedule a meeting - http://meetwith.me/zishan