Getting 'sslv3 alert illegal parameter' on some https connections

[void_main]

Howdy!
I am new to Scrapy. I am trying to retrieve some numeric data from a
website, let's call it targetsite.com for the sake of discusion.
Unfortunately I am getting the following error for https call:
(...) [targetsite.com] ERROR: Crawling <https://www.targetsite.com/
account/login.action>: [Failure instance: Traceback (failure with no
frames): <class 'OpenSSL.SSL.Error'>: [('SSL routines',
'SSL23_GET_SERVER_HELLO', 'sslv3 alert illegal parameter')]
Scrapy seems to work for http and even other https requests (like
mail.google.com).

Can't google any hints and even this group has only one related to
sslv3 post.
Tried to do some basic troubleshooting and it seems that Curl gets the
same error. However -3 parameter fixes Curl's trouble.

Is there anything am I missing in Scrapy?
I will appreciate any help,
void_main

Environment
Ubutu: 10.04
Scrapy: 0.8
Python: 2.6.5

[Shane Evans]

Hi,

It seems that the server you're connecting to accepts only the SSLv3
protocol (since it worked when you forced curl to use SSLv3) and twisted
(which scrapy uses) defaults to SSLv23.

I saw some more info on this with urllib: http://bugs.python.org/issue11220

Scrapy uses twisted's ClientContextFactory:
http://dev.scrapy.org/browser/scrapy/core/downloader/handlers/http.py#L32
which seems to use SSLv23:
http://twistedmatrix.com/trac/browser/tags/releases/twisted-9.0.0/twisted/internet/ssl.py#L127

Can you try your own ContextFactory that uses SSL.SSLv3_METHOD and see
if that fixes your problem?

I think it would be nice to allow users to specify this on the Request
object, which would default to using a scrapy setting (SSLv3). Anyone
want to make a patch? :)

Cheers,

Shane

[Shane Evans]

I opened a defect:
http://dev.scrapy.org/ticket/315

[void_main]

Shane,
Thanks a bunch for your thoughts.
Can you give me a specific example (a code snippet?) what should I do
with ContextFactory?
I will be happy to try...
Irek

On Mar 24, 6:25 am, Shane Evans <shane.ev...@gmail.com> wrote:
> Hi,
>
> It seems that the server you're connecting to accepts only the SSLv3
> protocol (since it worked when you forced curl to use SSLv3) and twisted
> (which scrapy uses) defaults to SSLv23.
>
> I saw some more info on this with urllib:http://bugs.python.org/issue11220
>

> Scrapy uses twisted's ClientContextFactory:http://dev.scrapy.org/browser/scrapy/core/downloader/handlers/http.py...
> which seems to use SSLv23:http://twistedmatrix.com/trac/browser/tags/releases/twisted-9.0.0/twi...