project not working after upgrading to 0.16

27 views
Skip to first unread message

shadow

unread,
Nov 8, 2012, 11:54:29 AM11/8/12
to scrapy...@googlegroups.com
Hi, 
I've been working on a complicated data scraping project for several months, the development are mainly based on 0.14.x stable and 0.15.x unstable. Last time I run it's around early Oct, all project runs flawlessly. 
But after today I upgraded to 0.16, it just printed endless ERROR, not doing anything. The only thing I can do is actually to `kill -9 `
Then I tried downgrading to 0.14.4, it runs back to normal.

the error msg is like follows:
2012-11-09 00:39:33+0800 [scrapy] INFO: Scrapy 0.16.1 started (bot: mybot)
2012-11-09 00:39:33+0800 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetConsole, CloseSpider, CoreStats, SpiderState, FailLogger
2012-11-09 00:39:33+0800 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, RandomUserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats
2012-11-09 00:39:33+0800 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2012-11-09 00:39:33+0800 [scrapy] DEBUG: Enabled item pipelines: sanitizePipeline
2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 2012-11-09 00:39:34+0800 [-] ERROR: 

Pablo Hoffman

unread,
Nov 12, 2012, 1:37:59 PM11/12/12
to scrapy...@googlegroups.com
I would recommend disabling all components (extensions, middlewares, pipelines, etc) and start enabling them one by one until you find the problematic one. At least, that would give you a better idea of where to chase the problem.


--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/scrapy-users/-/cFDeCuMnd-4J.
To post to this group, send email to scrapy...@googlegroups.com.
To unsubscribe from this group, send email to scrapy-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/scrapy-users?hl=en.

Reply all
Reply to author
Forward
0 new messages