project not working after upgrading to 0.16

[shadow]

Hi, 

I've been working on a complicated data scraping project for several months, the development are mainly based on 0.14.x stable and 0.15.x unstable. Last time I run it's around early Oct, all project runs flawlessly. 

But after today I upgraded to 0.16, it just printed endless ERROR, not doing anything. The only thing I can do is actually to `kill -9 `

Then I tried downgrading to 0.14.4, it runs back to normal.

the error msg is like follows:

2012-11-09 00:39:33+0800 [scrapy] INFO: Scrapy 0.16.1 started (bot: mybot)

2012-11-09 00:39:33+0800 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetConsole, CloseSpider, CoreStats, SpiderState, FailLogger

2012-11-09 00:39:33+0800 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, RandomUserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats

2012-11-09 00:39:33+0800 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware

2012-11-09 00:39:33+0800 [scrapy] DEBUG: Enabled item pipelines: sanitizePipeline

2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 

[Pablo Hoffman]

I would recommend disabling all components (extensions, middlewares, pipelines, etc) and start enabling them one by one until you find the problematic one. At least, that would give you a better idea of where to chase the problem.

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/scrapy-users/-/cFDeCuMnd-4J.
To post to this group, send email to scrapy...@googlegroups.com.
To unsubscribe from this group, send email to scrapy-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/scrapy-users?hl=en.