LWP::Parallel concerns

[cha...@lonemerchant.com]

Maybe I'm wrong, but if I were to use LWP::Parallel to parse a remote
site for a few hours, then couldn't this be possibly interpreted as a
Denial of Service? And if could be interpreted as a possibly Denial of
Service attack, what could I do to possibly avoid it?

[Ben Morrow]

Quoth cha...@lonemerchant.com:

By default LWP::Parallel won't make more than 5 requests to any given
host at a time. If you are worried that even this many over a sustained
period would be considered abuse, you could reduce it with ->max_req; or
you could use ->on_connect to do some more sophisticated rate-limiting.

Ben

--
The Earth is degenerating these days. Bribery and corruption abound.
Children no longer mind their parents, every man wants to write a book,
and it is evident that the end of the world is fast approaching.
Assyrian stone tablet, c.2800 BC b...@morrow.me.uk

[cha...@lonemerchant.com]

On May 25, 10:39 am, Ben Morrow <b...@morrow.me.uk> wrote:
> Quoth cha...@lonemerchant.com:
>
> > Maybe I'm wrong, but if I were to use LWP::Parallel to parse a remote
> > site for a few hours,  then couldn't this be possibly interpreted as a
> > Denial of Service? And if could be interpreted as a possibly Denial of
> > Service attack, what could I do to possibly avoid it?
>
> By default LWP::Parallel won't make more than 5 requests to any given
> host at a time. If you are worried that even this many over a sustained
> period would be considered abuse, you could reduce it with ->max_req; or
> you could use ->on_connect to do some more sophisticated rate-limiting.
>

Okay. Thanks.

[A. Sinan Unur]

Ben Morrow <b...@morrow.me.uk> wrote in
news:lqkog5-...@osiris.mauzo.dyndns.org:

> Quoth cha...@lonemerchant.com:
>> Maybe I'm wrong, but if I were to use LWP::Parallel to parse a remote
>> site for a few hours, then couldn't this be possibly interpreted as
>> a Denial of Service? And if could be interpreted as a possibly Denial
>> of Service attack, what could I do to possibly avoid it?
>
> By default LWP::Parallel won't make more than 5 requests to any given
> host at a time. If you are worried that even this many over a
> sustained period would be considered abuse,

On the other hand, this would not help at all if what you are doing is
against the terms of use the web site and use scraping to re-construct a
data set to which you do not have commercial use rights.

If you then go and set up a commercial operation of any sort using the
data you obtained that way, well, I am no lawyer, but I think a decent
case could be made against you.

Sinan

--
A. Sinan Unur <1u...@llenroc.ude.invalid>
(remove .invalid and reverse each component for email address)

comp.lang.perl.misc guidelines on the WWW:
http://www.rehabitation.com/clpmisc/