urllib2 - My POST Request just isn't working right

[Gregory Piñero]

Hi guys,

I'm trying to write a program to get FedEx shipping rates, and I'm
following their API as outlined in this document:
http://www.fedex.com/us/solutions/wis/pdf/fsm_directmanual.pdf

I need specifically to make a POST request to their server as shown on
page 12. However I continually get server error 500, and they FedEx
support advised me that this means my request is incorrect somewhere.

I put the code I"m using at the end of this email. Unfortunately I
don't know how to "see" the request the code generates either so I
can't paste that here.

Any help would be greatly appriciated.

--
Gregory Piñero
Chief Innovation Officer
Blended Technologies
(www.blendedtechnologies.com)

Code pasted below (contains sample info so it won't work for you):

import urllib
import urllib2

FedEx_API_URL="https://gatewaybeta.fedex.com:443/GatewayDC"

xml_request="""<?xml version="1.0" encoding="UTF-8" ?>
<FDXSubscriptionRequest xmlns:api="http://www.fedex.com/fsmapi"
xmlns:xsi="http://www.w3.org/2001/XMLSchemainstance"
xsi:noNamespaceSchemaLocation="FDXSubscriptionRequest.xsd">
<RequestHeader>
<CustomerTransactionIdentifier>String</CustomerTransactionIdentifier>
<AccountNumber>123456789</AccountNumber>
</RequestHeader>
<Contact>
<PersonName>Jim Smith</PersonName>
<CompanyName>Creative Widgets</CompanyName>
<Department>Shipping</Department>
<PhoneNumber>5405559900</PhoneNumber>
<PagerNumber>9999999999</PagerNumber>
<FaxNumber>5405559901</FaxNumber>
<E-MailAddress>j...@abc.com</E-MailAddress>
</Contact>
<Address>
<Line1>123 Main Street</Line1>
<Line2>1st Floor</Line2>
<City>Anycity</City>
<StateOrProvinceCode>VA</StateOrProvinceCode>
<PostalCode>24060</PostalCode>
<CountryCode>US</CountryCode>
</Address>
</FDXSubscriptionRequest>"""

values = {'request':xml_request}
headers = { 'Referer' : 'YourCompany',
'Host':'https://gatewaybeta.fedex.com/GatewayDC',
'Accept':'image/gif, image/jpeg, image/pjpeg, text/plain,
text/html, */*',
'Content-Type':'image/gif'
}

data = urllib.urlencode(values)
req = urllib2.Request(FedEx_API_URL, data, headers)
try:
handle = urllib2.urlopen(req)
except IOError, e:
if hasattr(e, 'reason'):
print 'We failed to reach a server.'
print 'Reason: ', e.reason
print e.info()
elif hasattr(e, 'code'):
print 'The server couldn\'t fulfill the request.'
print 'Error code: ', e.code
print e.info()
else:
the_page = handle.read()
print the_page

[Don't Code Faster Than You Can See]

(blogger.com gives the same error if you don't .strip() your password
before using it to connect)

i noticed one spelling mistake here:

> headers = { 'Referer' : 'YourCompany',

'Referrer'

could it be that simple?

[Eric Nieuwland]

Hi Greg,

> values = {'request':xml_request}

> headers = { 'Referer' : 'YourCompany',

> 'Host':'https://gatewaybeta.fedex.com/GatewayDC',
> 'Accept':'image/gif, image/jpeg, image/pjpeg, text/plain,
> text/html, */*',
> 'Content-Type':'image/gif'
> }

Could it be Host? It should be "gatewaybeta.fedex.com" AFAIKT

--eric

[Fuzzyman]

That would certainly be an error. *Also*, urllib2 adds the host header
for you, no need to set it.

All the best,

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

[Gregory Piñero]

Ok, I tried the changes you guys suggested but same error still:
1. Corrected spelling of referrer.
2. Don't specify host.

Here is what they say my request should look like:
-------------------------------------------------------------------------
POST /GatewayDC HTTP/1.0
Referer: YourCompanyNameGoesHere
Host: SSLserver.fedex.com

Accept: image/gif, image/jpeg, image/pjpeg, text/plain, text/html, */*

Content-Type: image/gif Content-length: %d

Your FedEx Transaction
-------------------------------------------------------------------------
Note that Host here is ficticious. My host should be correct. And
they have referrer mispelled here so who knows which to use!

Here are some suspicions I have:

1 . One thing that is weird is that the data part just says "Your
FedEx Transaction". I know I put the XML here, but I also notice that
urllib2 is putting this as:

request=<xml ....

Can I have my xml placed here without that something= part? That
don't specify what parameter name I should use.

2. urllib2 seems to create my request with \n\r line endings. Should
it maybe just be \n? And if so, how to change that?

Thanks again,

Greg

[Gregory Piñero]

Correction:

-------------------------------------------------------------------------
POST /GatewayDC HTTP/1.0
Referer: YourCompanyNameGoesHere
Host: SSLserver.fedex.com

Accept: image/gif, image/jpeg, image/pjpeg, text/plain, text/html, */*

[David Wahler]

Gregory Piñero wrote:
> Ok, I tried the changes you guys suggested but same error still:
> 1. Corrected spelling of referrer.
> 2. Don't specify host.
>
> Here is what they say my request should look like:
> -------------------------------------------------------------------------
> POST /GatewayDC HTTP/1.0
> Referer: YourCompanyNameGoesHere
> Host: SSLserver.fedex.com
> Accept: image/gif, image/jpeg, image/pjpeg, text/plain, text/html, */*
> Content-Type: image/gif Content-length: %d
>
> Your FedEx Transaction
> -------------------------------------------------------------------------
> Note that Host here is ficticious. My host should be correct. And
> they have referrer mispelled here so who knows which to use!
>

"Referer" is the correct header name. The HTTP specification spells it
incorrectly, so "Referrer" is the proper English spelling but invalid
in the context of HTTP. This is for historical reasons, and is not
likely to change.

> Here are some suspicions I have:
>
> 1 . One thing that is weird is that the data part just says "Your
> FedEx Transaction". I know I put the XML here, but I also notice that
> urllib2 is putting this as:
>
> request=<xml ....
>
> Can I have my xml placed here without that something= part? That
> don't specify what parameter name I should use.

Are you sure it's supposed to be URL-encoded? Try just sending the XML
data directly as part of the request.

-- David

[Eric Nieuwland]

Don't forget to substitute the actual Content-length for %d!

--eric

On 1 feb 2006, at 17:34, Gregory Piñero wrote:

> Correction:
> -----------------------------------------------------------------------

> --
> POST /GatewayDC HTTP/1.0
> Referer: YourCompanyNameGoesHere
> Host: SSLserver.fedex.com

> Accept: image/gif, image/jpeg, image/pjpeg, text/plain, text/html, */*

> Content-Type: image/gif
> Content-length: %d
>
> Your FedEx Transaction
> -----------------------------------------------------------------------
> --

> --
> http://mail.python.org/mailman/listinfo/python-list
>

[Fredrik Lundh]

Gregory Pińero wrote:

> 1 . One thing that is weird is that the data part just says "Your
> FedEx Transaction". I know I put the XML here, but I also notice that
> urllib2 is putting this as:
>
> request=<xml ....
>
> Can I have my xml placed here without that something= part? That
> don't specify what parameter name I should use.

you told it to do that:

values = {'request':xml_request}
data = urllib.urlencode(values)

so if you don't want it, all you have to do is to pass in the transaction
xml instead of the urlencoded form data:

req = urllib2.Request(FedEx_API_URL, xml_request, headers)

> 2. urllib2 seems to create my request with \n\r line endings. Should
> it maybe just be \n? And if so, how to change that?

"\r\n", more likely. and that's how things should be.

</F>

[Never Code Faster Than You Can See]

Gregory Pińero wrote:
> Ok, I tried the changes you guys suggested but same error still:
> 1. Corrected spelling of referrer.

errr, the other chap said the INcorrect spelling is correct for http.
sorry about that.

> Here is what they say my request should look like:
> -------------------------------------------------------------------------
> POST /GatewayDC HTTP/1.0
> Referer: YourCompanyNameGoesHere
> Host: SSLserver.fedex.com
> Accept: image/gif, image/jpeg, image/pjpeg, text/plain, text/html, */*
> Content-Type: image/gif Content-length: %d
>
> Your FedEx Transaction
> -------------------------------------------------------------------------
> Note that Host here is ficticious. My host should be correct. And
> they have referrer mispelled here so who knows which to use!

one oddity: they say here "SSL"
ie, a secure connection.

do you have to type in a password for your a/c when you look this info
up manually on the web?

if so, you probably need a secure connection in the api too.

e.g. (copied out of my blogger code):
conn= httplib.HTTPSConnection(host)
conn.set_debuglevel(1)
conn.request("POST",path,body,headers)
response=conn.getresponse()
print response.read()

WHERE:
headers = { "Content-type" : "application/xml"
,"Authorization" : basicstring
,"Host" : host
}

WHERE:
username="YOURACCOUNT"
pwd="YOURPWD"
host="SSLserver.fedex.com"
path="ANYTHINGAPPROPRIATE"
pwdDigest= base64.encodestring(username+":"+pwd).strip()
basicstring = "Basic "+ pwdDigest

altho chances are you'll have to fiddle with basicstring

[Gregory Piñero]

Thanks for all the help guys, it finally worked! I don't know if I
ever would have figured this out on my own...

skipping the url encoding is what did the trick. I didn't end up
needing to do anything special for ssl. I guess urllib2 just handles
that itself.

Thanks again,

-Greg