Re: [storm-user] [Storm-newbie] Vertical Crawler using Storm

[Ken Krugler]

Hi Nishant,

On May 14, 2013, at 5:00pm, Nishant Shah wrote:

Hello,

I have gone through the basic examples on the storm wiki and want to try out implemeting a vertical crawler using Storm. I have read many posts talking too vaguely on this topic. I wanted to know if anyone is willing to share some information regarding this that they have tried ? May be some blog post or code. I am hoping to convince my boss to switch to a storm framework from an existing sequential framework.

Any help would be greatly appreciated.

There's a very rough/broken chunk of code at https://github.com/bixo/bixo-storm

And a fork that's been stripped down/cleaned up a bit at https://github.com/DigitalPebble/storm-crawler

And I know that six weeks ago Enno posted that he was planning to open source a Storm-based crawler:

What else.. we use S3 to store the fetched content. This was a bit tricky because you want to "stream" into S3 if you want to use decently large file size (which is a good idea because small files are pain to use later with EMR etc.) & be still memory efficient. This can be done by using the multipart upload feature of S3, but it's a bit tricky to use. This code will be included in the crawler code we are planning to open source.

But I haven't seen anything since then.

-- Ken

--------------------------

Ken Krugler

+1 530-210-6378

http://www.scaleunlimited.com

custom big data solutions & training

Hadoop, Cascading, Cassandra & Solr

[Nishant Shah]

Thanks for the reply.

But now I have run into another snag. Turns out there is license issue around zeroMQ as it is LGPL3. So now I have to wait till the storm-netty is usable in production.

Nishant

On Tuesday, 14 May 2013 17:00:49 UTC-7, Nishant Shah wrote:

Hello,

I have gone through the basic examples on the storm wiki and want to try out implemeting a vertical crawler using Storm. I have read many posts talking too vaguely on this topic. I wanted to know if anyone is willing to share some information regarding this that they have tried ? May be some blog post or code. I am hoping to convince my boss to switch to a storm framework from an existing sequential framework.

Any help would be greatly appreciated.

Thanks.

Nishant

[Ken Krugler]

On May 15, 2013, at 2:52pm, Nishant Shah wrote:

Thanks for the reply.

But now I have run into another snag. Turns out there is license issue around zeroMQ as it is LGPL3. So now I have to wait till the storm-netty is usable in production.

Interesting - but there's no issue with using Linux and GPL? :)

ZeroMQ is installed as native code, so it's just like any of the other zillions of packages on a Linux distro.

-- Ken

On Tuesday, 14 May 2013 17:00:49 UTC-7, Nishant Shah wrote:

Hello,

I have gone through the basic examples on the storm wiki and want to try out implemeting a vertical crawler using Storm. I have read many posts talking too vaguely on this topic. I wanted to know if anyone is willing to share some information regarding this that they have tried ? May be some blog post or code. I am hoping to convince my boss to switch to a storm framework from an existing sequential framework.

Any help would be greatly appreciated.

Thanks.

Nishant

[Michael Rose]

ZMQ isn't linked into the program and should be safe to use.

--
You received this message because you are subscribed to the Google Groups "storm-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to storm-user+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

--
Michael Rose (@Xorlev)
Senior Platform Engineer, FullContact
mic...@fullcontact.com

[Nishant Shah]

What do you mean it's not linked into the program ? Also, if we use storm and don't publish the code, would it violate the LGPL v3 license ?

[Krishna Gade]

afaik, you should be fine, lots of people use storm and not
necessarily publish their source.

-- Krishna
@krishnagade