How to avoid 503 error

[Hao Zhang]

Dear all,

I am new to common crawl, and I want to use the index server to check if a couple of links was crawled by common crawl. But I have got 503 response when I use multithreading methods to check. How can I avoid it?

Thank you.

Best regards,

Hao

[Sebastian Nagel]

Hi Hao,

please use less threads and slow down until all requests succeed.

It's a single server, it can only answer few requests in parallel.
Please behave cooperative!

There are a couple of other options if you cannot wait. Please read
the instructions on
http://index.commoncrawl.org/

Thanks,
Sebastian

> --
> You received this message because you are subscribed to the Google Groups "Common Crawl" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> common-crawl...@googlegroups.com <mailto:common-crawl...@googlegroups.com>.
> To post to this group, send email to common...@googlegroups.com
> <mailto:common...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/common-crawl.
> For more options, visit https://groups.google.com/d/optout.

[Hao Zhang]

Hi, Sebastian,

Thank you very much for your reply!

Best regards,

Hao

在 2018年3月7日星期三 UTC+1下午2:19:45，Sebastian Nagel写道：

Hi Hao,

please use less threads and slow down until all requests succeed.

It's a single server, it can only answer few requests in parallel.
Please behave cooperative!

There are a couple of other options if you cannot wait. Please read
the instructions on
  http://index.commoncrawl.org/

Thanks,
Sebastian


On 03/07/2018 02:06 PM, Hao Zhang wrote:
> Dear all,
>
> I am new to common crawl, and I want to use the index server to check if a couple of links was
> crawled by common crawl. But I have got 503 response when I use multithreading methods to check. How
> can I avoid it?
>
> Thank you.
>
> Best regards,
> Hao
>
> --
> You received this message because you are subscribed to the Google Groups "Common Crawl" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> common-crawl...@googlegroups.com <mailto:common-crawl+unsub...@googlegroups.com>.