403 error

[jparicka]

Guys,

I am getting the following 403 error:

[ jparicka dev ~/social ] python test_digg_useragent.py
headers
Date: Sat, 07 Nov 2009 04:54:27 GMT
Server: Apache
X-Powered-By: PHP/5.2.9-digg8
Cache-control: private
X-Digg-Api-Version: 1.0
Accept-Ranges: bytes
Content-Length: 90
Connection: close
Content-Type: application/json

file
['{"error":{"timestamp":1257569667,"message":"HTTP User-Agent header
required","code":1029}}']

When running the following code in python 2.4.3:

from urllib import FancyURLopener
from random import choice

user_agents = [
'Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.8.1.11) Gecko/
20071127 Firefox/2.0.0.11',
'Opera/9.25 (Windows NT 5.1; U; en)',
'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR
1.1.4322; .NET CLR 2.0.50727)',
'Mozilla/5.0 (compatible; Konqueror/3.5; Linux) KHTML/3.5.5 (like
Gecko) (Kubuntu)',
'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.12) Gecko/
20070731 Ubuntu/dapper-security Firefox/1.5.0.12',
'Lynx/2.8.5rel.1 libwww-FM/2.14 SSL-MM/1.4.1 GNUTLS/1.2.9'
]

class MyOpener(FancyURLopener, object):
version = choice(user_agents)

myopener = MyOpener()
myopener.retrieve('http://services.digg.com/search/stories?
query=frankincense%20tree&sort=digg_count-desc&appkey=http%3A%2F
%2Fwww.beepl.com&type=json', 'out.json.txt')

print 'headers'
print myopener.open('http://services.digg.com/search/stories?
query=frankincense%20tree&sort=digg_count-desc&appkey=http%3A%2F
%2Fwww.beepl.com&type=json').headers

print 'file'
print myopener.open('http://services.digg.com/search/stories?
query=frankincense%20tree&sort=digg_count-desc&appkey=http%3A%2F
%2Fwww.beepl.com&type=json').readlines()

I am confident I have set the User-Agent correctly.

Thank you for your help.

Jan P.

[Bill Shupp]

I can't duplicate this. All 3 of your queries work for me if I supply
a User-Agent header. Can you supply the request headers instead of
the response headers?

Regards,

Bill Shupp

[Jan Paricka]

[ jparicka dev ~/social ] python test_digg_useragent.py

connect: (services.digg.com, 80)

send: 'GET /search/stories?query=frankincense%20tree&sort=digg_count-desc&appkey=http%3A%2F%2Fwww.beepl.com&type=json HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: services.digg.com\r\nConnection: close\r\nUser-agent: Python-urllib/2.4\r\n\r\n'

Thanks,

Jan

--
Jan Paricka

Let's stay in touch via LinkedIn:
http://www.linkedin.com/in/jparicka

[Bill Shupp]

On Nov 6, 2009, at 10:02 PM, Jan Paricka wrote:

[ jparicka dev ~/social ] python test_digg_useragent.py

connect: (services.digg.com, 80)

send: 'GET /search/stories?query=frankincense%20tree&sort=digg_count-desc&appkey=http%3A%2F%2Fwww.beepl.com&type=json HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: services.digg.com\r\nConnection: close\r\nUser-agent: Python-urllib/2.4\r\n\r\n'

Thanks,

Jan

Try using User-Agent instead of User-agent.

Regards,

Bill

[Jan Paricka]

Bill,

That didn't help either.   I suspect dodgy python installation .. or I don't know.   :-(

Thanks for your help anyway.

Cheers,

Jan P.

[Steve Williams]

At 10:02 PM 11/6/2009, Jan Paricka wrote:
>[ jparicka dev ~/social ] python test_digg_useragent.py

>connect: (<http://services.digg.com>services.digg.com, 80)
>send: 'GET
>/search/stories?query=frankincense%20tree&sort=digg_count-desc&appkey=http%3A%2F%<http://2Fwww.beepl.com>2Fwww.beepl.com&type=json

>HTTP/1.1\r\nAccept-Encoding: identity\r\nHost:

><http://services.digg.com>services.digg.com\r\nConnection:

>close\r\nUser-agent: Python-urllib/2.4\r\n\r\n'

Jan, can you share the source for that test program? I don't see
native methods in urllib to emit those diagnostics.

I tried something similar on my FreeBSD server running Python
2.5.4. I tried this at the Python prompt:

import urllib
urllib.urlopen("http://services.digg.com/search/stories?query=frankincense%20tree&sort=digg_count-desc&appkey=http%3A%2F%2Fwww.beepl.com&type=json").readlines()

I did not get an error response, but rather a list of stories, as expected.

Looking at the urllib source code, it appears the User-Agent header was:

User-Agent: Python-urllib/1.17

[Jan Paricka]

[ jparicka dev ~ ] cat abcd.py 

import urllib

print urllib.urlopen("http://services.digg.com/search/stories?query=frankincense%20tree&sort=digg_count-desc&appkey=http%3A%2F%2Fwww.beepl.com&type=json").readlines()

[ jparicka dev ~ ] python abcd.py 

['{"error":{"timestamp":1258939383,"message":"HTTP User-Agent header required","code":1029}}']

[ jparicka dev ~ ] uname -a

Linux dev.nrds.co.uk 2.6.18-92.1.22.el5 #1 SMP Tue Dec 16 11:57:43 EST 2008 x86_64 x86_64 x86_64 GNU/Linux

[ jparicka dev ~ ] echo tryied both python 2.4.3 and 2.6.4 centos 5.2 rightscale

tryied both python 2.4.3 and 2.6.4 centos 5.2 rightscale

I have deployed python 2.6.4 across all servers specificly to tackle this problem.

I tried to set the user-agent (and User-agent) manually, checked on the other "dummy" end, worked perfectly.  I'm having this problem with digg API only.   :-(

I very much appreciate your help on this

Jan Paricka

--

You received this message because you are subscribed to the Google Groups "Digg API" group.
To post to this group, send email to dig...@googlegroups.com.
To unsubscribe from this group, send email to diggapi+u...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/diggapi?hl=.

[Steve Williams]

At 05:26 PM 11/22/2009, Jan Paricka wrote:
>import urllib
>print

>urllib.urlopen("<http://services.digg.com/search/stories?query=frankincense%20tree&sort=digg_count-desc&appkey=http%3A%2F%2Fwww.beepl.com&type=json>http://services.digg.com/search/stories?query=frankincense%20tree&sort=digg_count-desc&appkey=http%3A%2F%2Fwww.beepl.com&type=json").readlines()

Ugh. That worked in my environment.

Let me see whether there's more information on our side.

[Jan Paricka]

Steve .. any more thoughts on this problem?

I tried manually setting up the User-agent but that didn't help either.

:-(

Thank you,

Jan

--

You received this message because you are subscribed to the Google Groups "Digg API" group.
To post to this group, send email to dig...@googlegroups.com.
To unsubscribe from this group, send email to diggapi+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/diggapi?hl=.

[Steve Williams]

At 05:14 PM 12/3/2009, Jan Paricka wrote:
>Steve .. any more thoughts on this problem?

We found your requests in the server logs, but the logs don't capture
enough detail to tell us what's different from my requests.

Here's a suggestion: Can you try your request through our public
proxy and let us know the result? Here's how to do it:

http://apidoc.digg.com/BasicConcepts#DiggsPublicAPIProxy

[Jan Paricka]

Steve, I think I am giving up on digg.  

And it's a shame as I do not feel this is my problem since everything else but digg works for us .. and believe me, we did some failry sophisticated stuff over at this end.

I spend endless amount of hours on this - and it's leading nowhere and I hate the idea of having a proxy sitting somewhere in the middle (as if the digg wasn't slow enough).

I tried vanilla bitnami machines, .. etc, .. all hosted on gogrid .. but had no luck.

I don't know.

It's working from my Mac but that's about it really.

Is anyone actually doing something with digg api?   I find it hard to believe.....

I still do appreciate your help with this.

Cheers,

Jan

--

You received this message because you are subscribed to the Google Groups "Digg API" group.
To post to this group, send email to dig...@googlegroups.com.
To unsubscribe from this group, send email to diggapi+u...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/diggapi?hl=en.

[Jan Paricka]

No luck.

PSA (Please see attached)

Kind regards,

Jan

[jparicka@25358_2_42578_205369:~] cat abcd.py

import urllib, urllib2

USER_AGENT = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)'

# To use Digg's public API proxy, just change the URL of any API request to http://digg.com/tools/services.

#address = "http://services.digg.com/search/stories?query=frankincense%20tree&sort=digg_count-desc&appkey=http%3A%2F%2Fwww.beepl.com&type=json"

address = "http://digg.com/tools/services/search/stories?query=frankincense%20tree&sort=digg_count-desc&appkey=http%3A%2F%2Fwww.beepl.com&type=json"

request = urllib2.Request(address, None, {'User-Agent':USER_AGENT} )

urlfile = urllib2.urlopen(request)

    

page = urlfile.read(200000)

print page

urlfile.close()

[jparicka@25358_2_42578_205369:~] 

[Steve Williams]

At 06:31 PM 12/4/2009, Jan Paricka wrote:
>Steve, I think I am giving up on digg.

Sorry to hear that. We are grateful for the time you've put in on this.

>... I hate the idea of having a proxy sitting somewhere in the

>middle (as if the digg wasn't slow enough).

For what it's worth, Digg's public API proxy won't add much of a
delay, as it calls the API right inside the data center. Its purpose
is to cache responses for identical requests from client-side
widgets. That caching may not benefit your server-side application,
but it won't hurt.

>Is anyone actually doing something with digg api? I find it hard
>to believe.....

I hear your frustration, but, yes, of course lots of web sites use
the API. The problem you're seeing would be quite interesting if it
weren't so frustrating.

[Steve Williams]

You didn't say what response you got from the API request via the
proxy, but I see an error in your request URL.

At 07:06 PM 12/4/2009, Jan Paricka wrote:
>address =
>"<http://digg.com/tools/services/search/stories?query=frankincense%20tree&sort=digg_count-desc&appkey=http%3A%2F%2Fwww.beepl.com&type=json>http://digg.com/tools/services/search/stories?query=frankincense%20tree&sort=digg_count-desc&appkey=http%3A%2F%2Fwww.beepl.com&type=json"

One more change is required to send the request through Digg's public
API proxy: You must pass the API request path as the encoded value of
the "endPoint" query string argument, not part of the proxy URL
path. Here's a working example. (I used urllib, not urllib2. Not
sure whether that makes a difference):

import urllib
class AppURLopener(urllib.FancyURLopener):
version = "Python-urllib/2.4"
urllib._urlopener = AppURLopener()
print
urllib.urlopen("http://digg.com/tools/services?endPoint=%2Fsearch%2Fstories&query=frankincense%20tree&sort=digg_count-desc&appkey=http%3A%2F%2Fwww.beepl.com&type=json").readlines()

Let me break down that URL for you:

First is the proxy's base URL:

http://digg.com/tools/services

The first argument is the Digg API request path, urlencoded:

?endPoint=%2Fsearch%2Fstories

The remaining arguments are the same as for the direct API request:

&query=frankincense%20tree
&sort=digg_count-desc
&appkey=http%3A%2F%2Fwww.beepl.com
&type=json

Hope that clarifies things a bit.

[Jan Paricka]

Oh, Lord.  I really wish I did not send this email.

I apologize - I think I just needed a little break from this, that's all.

You guys are great and the support I am getting is fantastic.

I am going to try one more thing which I think that may work.

PS - no more emails after 3 AM.

Kind regards,

Jan

On Sat, Dec 5, 2009 at 2:31 AM, Jan Paricka <jpar...@gmail.com> wrote: