Proxy support

[Jey]

I think this has been posted here earlier, and was not resolved
because of lack of reproducible usecase. The proxy support problem
can be seen using the BloggerExample.py found in gdata.py-1.0.9.

When run, from within a proxy setup, I get this error, after the
connection has been established.
USE-CASE:
>>> os.environ['https_proxy']='http://wwwgate0.mot.com:1080'
>>> os.environ['proxy-password']='passwd'
>>> os.environ['proxy-username']='userid'
>>> import BloggerExample
>>> s=BloggerExample.BloggerExample('****@gmail.com','passwd')

ERROR:
-------------------------------------
response: HTTP/1.0 200 Connection established
Proxy-Agent: NetCache NetApp/5.6.2R2

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "BloggerExample.py", line 60, in __init__
self.service.ProgrammaticLogin()
File "d:\work\sodium\trunk\core\3rdparty\gdata.py-1.0.9\src\gdata
\service.py", line 299, in ProgrammaticLogin
(auth_connection, uri) = self._PrepareConnection(AUTH_SERVER_HOST)
File "d:\work\sodium\trunk\core\3rdparty\gdata.py-1.0.9\src\atom
\service.py", line 171, in _PrepareConnection
ssl = socket.ssl(p_sock, None, None)
File "c:\python25\lib\socket.py", line 74, in ssl
return _realssl(sock, keyfile, certfile)
socket.sslerror: (8, 'EOF occurred in violation of protocol')
-------------------------------------

Hope this usecase helps resolving the issue. Quick web search seems
to indicate that the gmail authentication has known issues. Is this a
symptom of any such known problem?

-thanks
Jey

[Jey]

Thanks to Scott Melton, replacing the new line after base64 encoding,
atleast pushes the problem further down to a different area. ( I will
post it separately)
[Re: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/301740]

ie, Making the following change addresses the socket problem:
user_pass=base64.encodestring('%s:%s' % (proxy_username,
proxy_password))
user_pass=user_pass.replace('\n','')

Do you guys want to consider this patch for
gdata.py-1.0.9/src/atom/service.py file ?

-Jey

On Nov 9, 4:40 pm, Jey <jey.mich...@gmail.com> wrote:
> I think this has been posted here earlier, and was not resolved
> because of lack of reproducible usecase. The proxy support problem
> can be seen using the BloggerExample.py found in gdata.py-1.0.9.
>
> When run, from within a proxy setup, I get this error, after the
> connection has been established.
> USE-CASE:
>
> >>> os.environ['https_proxy']='http://wwwgate0.mot.com:1080'
> >>> os.environ['proxy-password']='passwd'
> >>> os.environ['proxy-username']='userid'
> >>> import BloggerExample

> >>> s=BloggerExample.BloggerExample('*...@gmail.com','passwd')

[Jey]

Adding the missing
user_pass=user_pass.strip()
in atom/service.py reveals this new problem with gdata client library.

-------------------------------------------------------------------------------------------------------------

Traceback (most recent call last):

File "BloggerExampleTest.py", line 8, in <module>
s=BloggerExample.BloggerExample('*********@gmail.com','******')
File "BloggerExample.py", line 63, in __init__
feed = self.service.Get('/feeds/default/blogs')

File "d:\work\sodium\trunk\core\3rdparty\gdata.py-1.0.9\src\gdata

\service.py", line 519, in Get
server_response = atom.service.AtomService.Get(self, uri,
extra_headers)

File "d:\work\sodium\trunk\core\3rdparty\gdata.py-1.0.9\src\atom

\service.py", line 268, in Get
url_params, escape_params)

File "d:\work\sodium\trunk\core\3rdparty\gdata.py-1.0.9\src\atom

\service.py", line 224, in _CreateConnection
connection.endheaders()
File "c:\python25\lib\httplib.py", line 856, in endheaders
self._send_output()
File "c:\python25\lib\httplib.py", line 728, in _send_output
self.send(msg)
File "c:\python25\lib\httplib.py", line 695, in send
self.connect()
File "c:\python25\lib\httplib.py", line 663, in connect
socket.SOCK_STREAM):
socket.gaierror: (11001, 'getaddrinfo failed')

-Jey

> > Jey- Hide quoted text -
>
> - Show quoted text -

[Jey]

I figured the cause for the last error (oversight by me)

But, there was another caveat in the atom-python-client library.

In, gdata.py-1.0.9/src/atom/service.py, _PrepareConnection method does
a
self.skip_host = True
(after the workaround-code for the secured-proxy connection)

I suspect there is some background for adding that there. In the
scenario that I am going through, I need the Python-httplib to put the
second header ("Host:") in the connection. This happens by default in
Python-httplib, but the skip_host is deliberately added in the atom
library to address some other issues? This addition breaks the ssl
connection via proxy.

So, In summary...in the file atom/service.py
1) "user_pass=user_pass.strip()" needs to be added, for sure.
2) "self.skip_host = True" causes problem. Fixing this, needs some
input from the authors.

cheers,
-Jey

On Nov 9, 6:06 pm, Jey <jey.mich...@gmail.com> wrote:
> Adding the missing
> user_pass=user_pass.strip()
> in atom/service.py reveals this new problem with gdata client library.
>

> ---------------------------------------------------------------------------­----------------------------------
>

[Jeff S]

Thank you for all of your hard work debugging this Jey, great
findings.

On Nov 10, 9:17 am, Jey <jey.mich...@gmail.com> wrote:
> I figured the cause for the last error (oversight by me)
>
> But, there was another caveat in the atom-python-client library.
>
> In, gdata.py-1.0.9/src/atom/service.py, _PrepareConnection method does
> a
> self.skip_host = True
> (after the workaround-code for the secured-proxy connection)
>
> I suspect there is some background for adding that there. In the
> scenario that I am going through, I need the Python-httplib to put the
> second header ("Host:") in the connection. This happens by default in
> Python-httplib, but the skip_host is deliberately added in the atom
> library to address some other issues? This addition breaks the ssl
> connection via proxy.
>
> So, In summary...in the file atom/service.py
> 1) "user_pass=user_pass.strip()" needs to be added, for sure.

I'm a bit wary of adding this, what if the password in question ends
with a space. It is true that it should not end in a newline
character, but I would rather leave this up to the application using
the service object. Does this sound reasonable?

> 2) "self.skip_host = True" causes problem. Fixing this, needs some
> input from the authors.

I agree, I think it was Takashi who had initially written this, but
perhaps I'm not recalling correctly. There have been a couple of
changes to the proxy code. Takashi, do you recall the rationale for
self.skil_host = True? Would it hurt anything if we set the default to
False?

>
> cheers,
> -Jey
>

Happy coding,

Jeff

[Takashi MATSUO]

Thank you all,

Unfortunately, I've forgotten the rationale for self.skil_host = True.
But I will try to remember the rationale, and will test for False default value.

Please wait for a moment...

Regards,

--
Takashi Matsuo
matsuo....@gmail.com

[Takashi MATSUO]

Hello Jey, Jeff,

I think I've got it :)

If skip_host is set to False, our HTTPConnection send 'localhost' as
the value for Host header. So, I had to set it to True.

But I should have set additional_headers attribute properly for actual
host header. I'm sorry I've forgotton.

Jey, would you like to try this fix bellow?

----
Index: src/atom/service.py
===================================================================
--- src/atom/service.py (revision 219)
+++ src/atom/service.py (working copy)
@@ -169,6 +169,7 @@
connection.sock=fake_sock
full_uri = partial_uri
self.skip_host = True
+ self.additional_headers['Host'] = server

else:
connection = httplib.HTTPSConnection(server, port)
----

--
Takashi Matsuo
matsuo....@gmail.com

[Takashi MATSUO]

Hello, guys

At last, I think I've found more simple solution.
There is no need for skip_host attribute from the beggining...

Jey, would you like to try this fixation? This is a diff against svn 219.
This works fine at least for me.

---- svn diff

Index: src/atom/service.py
===================================================================
--- src/atom/service.py (revision 219)
+++ src/atom/service.py (working copy)

@@ -52,8 +52,6 @@
ssl = False
# If debug is True, the HTTPConnection will display debug information
debug = False
- # Used in connection put requests if a proxy is configured.
- skip_host = False

def __init__(self, server=None, additional_headers=None):
"""Creates a new AtomService client.
@@ -165,10 +163,9 @@
fake_sock = httplib.FakeSocket(p_sock, ssl)

# Initalize httplib and replace with the proxy socket.
- connection = httplib.HTTPConnection('localhost')
+ connection = httplib.HTTPConnection(server)
connection.sock=fake_sock
full_uri = partial_uri
- self.skip_host = True

else:
connection = httplib.HTTPSConnection(server, port)

@@ -200,7 +197,7 @@

full_uri = BuildUri(uri, url_params, escape_params)
(connection, full_uri) = self._PrepareConnection(full_uri)
- connection.putrequest(http_operation, full_uri, skip_host=self.skip_host)
+ connection.putrequest(http_operation, full_uri)

if isinstance(self.additional_headers, dict):
for header in self.additional_headers:

----

--
Takashi Matsuo
matsuo....@gmail.com

[Takashi MATSUO]

Hello again,

The patch I sent was wrong. I've attached correct one.
Please try this.

Regards,

--
Takashi Matsuo
matsuo....@gmail.com

[Jey]

Hi Takashi:

The patch works well from within the proxy setup. I was going to
suggest removing the skip_host as well. Nice to see that going
away :-)

thanks,
Jey

On Nov 13, 4:53 am, "Takashi MATSUO" <matsuo.taka...@gmail.com> wrote:
> Hello again,
>
> The patch I sent was wrong. I've attached correct one.
> Please try this.
>
> Regards,
>

> > On Nov 13, 2007 12:07 PM, Takashi MATSUO <matsuo.taka...@gmail.com> wrote:
> > > Hello Jey, Jeff,
>
> > > I think I've got it :)
>
> > > If skip_host is set to False, our HTTPConnection send 'localhost' as
> > > the value for Host header. So, I had to set it to True.
>
> > > But I should have set additional_headers attribute properly for actual
> > > host header. I'm sorry I've forgotton.
>
> > > Jey, would you like to try this fix bellow?
>
> > > ----
> > > Index: src/atom/service.py
> > > ===================================================================
> > > --- src/atom/service.py (revision 219)
> > > +++ src/atom/service.py (working copy)
> > > @@ -169,6 +169,7 @@
> > > connection.sock=fake_sock
> > > full_uri = partial_uri
> > > self.skip_host = True
> > > + self.additional_headers['Host'] = server
>
> > > else:
> > > connection = httplib.HTTPSConnection(server, port)
> > > ----
>

> > > > matsuo.taka...@gmail.com
>
> > > --
> > > Takashi Matsuo
> > > matsuo.taka...@gmail.com
>
> > --
>
> > Takashi Matsuo
> > matsuo.taka...@gmail.com
>
> --
> Takashi Matsuo
> matsuo.taka...@gmail.com
>
> gdata-python-proxy-problem.diff
> 1KDownload- Hide quoted text -

[Jey]

On Nov 12, 5:25 pm, Jeff S <j...@google.com> wrote:

> On Nov 10, 9:17 am, Jey <jey.mich...@gmail.com> wrote:

> > So, In summary...in the file atom/service.py
> > 1) "user_pass=user_pass.strip()" needs to be added, for sure.
>
> I'm a bit wary of adding this, what if the password in question ends
> with a space. It is true that it should not end in a newline
> character, but I would rather leave this up to the application using
> the service object. Does this sound reasonable?

Hi Jeff:

I agree with your point about the strip()... Originally I was
suggesting to remove the trailing newline, but then found the strip()
call in the existing code (atom/service.py) and followed the same.

But, there is still a problem. The fix cannot sit outside the
library, I am afraid. Its the base64 encoding thats adding the
newline, and this happens within the library. Perhaps, the removal of
trailing newline after base64 encoding, at both those places, is what
is needed.

-Jey

[Takashi MATSUO]

Hi Jey,

I'm glad it works for you too. I'm afraid my code had annoyed you much.
I'm sorry about that.
I swear that I will write codes more carefully.

Thank you again, for your report.

--
Takashi Matsuo
matsuo....@gmail.com

[Jeff S]

Hi Jey,

I have checked in Takashi's diff to subversion. I found the area you
were talking about where a base64 encoded string needs to be stripped
of a trailing newline so here is a diff you can try for that issue:

Index: service.py
===================================================================
--- service.py (revision 220)
+++ service.py (working copy)
@@ -132,10 +132,10 @@
proxy_username = os.environ.get('proxy-username')
proxy_password = os.environ.get('proxy-password')
if proxy_username:
- user_pass=base64.encodestring('%s:%s' % (proxy_username,
- proxy_password))
- proxy_authorization = ('Proxy-authorization: Basic '+
user_pass
- + '\r\n')
+ user_auth = base64.encodestring('%s:%s' % (proxy_username,
+ proxy_password))
+ proxy_authorization = ('Proxy-authorization: Basic %s\r\n'
% (
+ user_auth.strip()))
else:
proxy_authorization = ''
proxy_connect = 'CONNECT %s:%s HTTP/1.0\r\n' % (server,port)

Please let me know if this fixes it.

Thank you,

Jeff

On Nov 13, 1:55 pm, "Takashi MATSUO" <matsuo.taka...@gmail.com> wrote:
> Hi Jey,
>
> I'm glad it works for you too. I'm afraid my code had annoyed you much.
> I'm sorry about that.
> I swear that I will write codes more carefully.
>
> Thank you again, for your report.
>

> matsuo.taka...@gmail.com

[Jey]

Back from vacation... :-)
Yes, I double checked again. Your fix works from within the proxy
environment, Jeff.

Also, since we are discussing about this proxy stuff...
Are a lot of other components depend on these names 'proxy-username' &
'proxy-password' ?
If its not too late to change it, ,I would suggest replacing them with
underscores. Something like 'proxy_username' & 'proxy_password'
These hyphens are a pain, especially as environ variable data....

cheers,
-Jey

PS: Takashi... I was not annoyed. Cool :-)

[Jeff S]

Thank you Jey,

I've checked this in as revision 223.

Happy coding,

Jeff