I have a spider that I created that use splash and it seems to never complete, ie it runs for two days and thenI finally stop it.
I have the following settings for my spider:
SPIDER_MIDDLEWARES = {
'scrapy_splash.SplashDeduplicateArgsMiddleware': 100
}
DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter'
Can someone provide some advice on how I should debug the issue?
Thanks
Sean