I wanted to introduce you to
Kimono Labs, a tool we've created that makes most webscraping, including crawling, a fast and easy visual process. We'd love to help in anyway we can as you guys move forward, and our product is free for life for most users (as long as you're not scraping more 20M pages/mo).
Let me know if you have any questions or need help.