Crawler not crawling a few pages, crawls everything else

[VP]

I have the crawler setup in my app and it works great, but have an issue with just couple urls. It does not crawl these pages, the only difference I see between these couple pages and other pages which are crawled is these have a lot of sub links on these pages, about 800-1000.

Can you please help, with any settings that I can update to see if these would get crawled.

Thanks.

[DigitalPebble]

Please use stack overflow and give details about your setup e.g store urls in elastic search etc...

Thanks 

--
You received this message because you are subscribed to the Google Groups "DigitalPebble" group.
To unsubscribe from this group and stop receiving emails from it, send an email to digitalpebbl...@googlegroups.com.
To post to this group, send email to digita...@googlegroups.com.
Visit this group at https://groups.google.com/group/digitalpebble.
For more options, visit https://groups.google.com/d/optout.

[vishal....@gmail.com]

Sure, I will post this on SO. Can you please let me know what details would you need from the setup?