Optimus Caching -- URLs that are not in the sitemap

[Bhishan Hemrajani]

Hi Optimus Caching Group,

Does Optimus crawl my URLs and find other URLs on the website to crawl? Ideally I would like Optimus to crawl all URLs that the crawler can find through the sitemap, that are **not** blocked by robots.txt.

Please let me know if Optimus can do this, or if there is another tool that is out there that can do this. Thank you,

Bhishan Hemrajani