limit the number of threads per site or host

[Roscoe]

Hello,

When using many threads with many sites, is there a way to limit the
number of threads per site or host?

Thanks!

[Maxim Zakharov]

Hi,

There is no such limit. But you can use CrawlDelay command to set a
period between consecutive hits to the site site.
See http://www.dataparksearch.org/dpsearch-indexcmd.en.html#CRAWLDELAY_CMD

--
http://www.dataparksearch.org/ - an open source search engine.