Problem with crawling ssl website

817 views
Skip to first unread message

Morad Edwar

unread,
Jan 5, 2015, 6:33:50 AM1/5/15
to scrapy...@googlegroups.com
Hello,
I'm trying to crawl ssl website but i am stuck with this error in the spider and scrapy shell :


2015-01-05 11:35:04+0200 [ecsonlinestore2] ERROR: Error downloading <GET https://ecsonlinestore.com/>: [<twisted.python.failure.Failure <class 'OpenSSL.SSL.Error'>>]

I think it's because the "SNI Support" which is needed by the website, is there any plugin to solve this problem??
Scrapy Version : 0.20.2
Python Version : 2.7.6

Nicolás Alejandro Ramírez Quiros

unread,
Jan 5, 2015, 9:49:34 AM1/5/15
to scrapy...@googlegroups.com

Travis Leleu

unread,
Jan 5, 2015, 12:26:48 PM1/5/15
to scrapy...@googlegroups.com
Morad -

It appears like the site you're scraping has an older configuration on their SSL that means the CA chain doesn't work as expected.  I'd look into how you can turn off SSL validation / strict checking so it doesn't try to validate the site identity.  After all, you probably don't care about the security component, you just want the data!

On Mon, Jan 5, 2015 at 6:49 AM, Nicolás Alejandro Ramírez Quiros <nrami...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users...@googlegroups.com.
To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.

Reply all
Reply to author
Forward
0 new messages