Retry policy

[Alberto]

Looking at this presentation:
http://www.slideshare.net/kazuho/q4m-microblogcon-presentation

Slide 39 :)

I'm in the same situation, a queue of urls that I should retry.

But I have one doubt, I want to keep a max-retry policy.

I was thinking about having a column as a counter and increase this
counter in the rescheduler code, before inserting to the request
queue, or discard the row when max-retry has been reached.

Is this a good method? Or I'm missing a better way :)

[Kazuho Oku]

Hi,

Yes, I think using a counter is a good way to enforce max-retry policy.

The example crawler included in the Q4M distribution
(http://kazuho.31tools.com/svn/q4m/tags/0.8.3/examples/crawler/crawler.pl)
does that as well (by using a constant named $max_retries).

2008/10/10 Alberto <albsu...@gmail.com>:

--
Kazuho Oku

[Alberto]

Following your case, I see that you open one connection (mysql) for
each thread.

We will have arround 150 threads, do you think it's better to have a
unique connection on the main loop/thread, and dispatch a row to each
http thread? Or open 150 connections to the mysql server.

On 13 oct, 07:36, "Kazuho Oku" <kazuho...@gmail.com> wrote:
> Hi,
>
> Yes, I think using a counter is a good way to enforce max-retry policy.
>
> The example crawler included in the Q4M distribution
> (http://kazuho.31tools.com/svn/q4m/tags/0.8.3/examples/crawler/crawler.pl)
> does that as well (by using a constant named $max_retries).
>

> 2008/10/10 Alberto <albsusc...@gmail.com>: