limit to amount of itmes you can hit a server?? - beautifulsoup

The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Message from discussion limit to amount of itmes you can hit a server??
Received-SPF: neutral (google.com: 209.85.215.43 is neither permitted nor denied by best guess record for domain of l...@mustbuilddigital.com) client-ip=209.85.215.43;
MIME-Version: 1.0
In-Reply-To: <6d7c0b8e-c619-4455-aea9-faccba74350c@googlegroups.com>
References: <6d7c0b8e-c619-4455-aea9-faccba74350c@googlegroups.com>
Date: Mon, 23 Jul 2012 16:15:32 -0500
Message-ID: <CAHrLguWCNDXxFBvge7U5CQueiXjxhVmj1p_HeJHcYR0L93E...@mail.gmail.com>
Subject: Re: limit to amount of itmes you can hit a server??
From: Link Swanson <l...@mustbuilddigital.com>
To: beautifulsoup@googlegroups.com
Content-Type: multipart/alternative; boundary=f46d040714c5038c3504c585bf1a

--f46d040714c5038c3504c585bf1a
Content-Type: text/plain; charset=ISO-8859-1

Looks like you are being rate limited. If you can determine the time window
of the rate limit, you can build error handling into your urllib2 calls so
that Python will wait until the limit is lifted, or try the call over again
through a proxy server until the proxy gets rate-limited ...

On Mon, Jul 23, 2012 at 3:44 PM, Tom <boot...@gmail.com> wrote:

> Hello,
>         I recently got my code to work hitting a yahoo owned website... (
> www.rivals.com) However after around 1000 hits,  it all the sudden
> crashes or stops and I get this error.....
>
> Traceback (most recent call last):
>   File "C:\Users\Tom\Documents\Python\bs4final.py", line 42, in <module>
>     main()
>   File "C:\Users\Tom\Documents\Python\bs4final.py", line 15, in main
>     page = urllib2.urlopen(request)
>   File "C:\Python27\lib\urllib2.py", line 126, in urlopen
>     return _opener.open(url, data, timeout)
>   File "C:\Python27\lib\urllib2.py", line 406, in open
>     response = meth(req, response)
>   File "C:\Python27\lib\urllib2.py", line 519, in http_response
>     'http', request, response, code, msg, hdrs)
>   File "C:\Python27\lib\urllib2.py", line 444, in error
>     return self._call_chain(*args)
>   File "C:\Python27\lib\urllib2.py", line 378, in _call_chain
>     result = func(*args)
>   File "C:\Python27\lib\urllib2.py", line 527, in http_error_default
>     raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
> HTTPError: HTTP Error 999: Unable to process request at this time -- error
> 999
> >>>
>
> PLEASE tell my that there is a way around this issue... if it is server
> hits related! or is there a limit inside bs4 somehwere?
>
> Thanks,
> Tom
>
> --
> You received this message because you are subscribed to the Google Groups
> "beautifulsoup" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/beautifulsoup/-/czHhafYyEqQJ.
> To post to this group, send email to beautifulsoup@googlegroups.com.
> To unsubscribe from this group, send email to
> beautifulsoup+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/beautifulsoup?hl=en.
>



-- 
Link Swanson
Must Build Digital

--f46d040714c5038c3504c585bf1a
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Looks like you are being rate limited. If you can determine the time window=
 of the rate limit, you can build error handling into your urllib2 calls so=
 that Python will wait until the limit is lifted, or try the call over agai=
n through a proxy server until the proxy gets rate-limited ...=A0<br>
<br><div class=3D"gmail_quote">On Mon, Jul 23, 2012 at 3:44 PM, Tom <span d=
ir=3D"ltr">&lt;<a href=3D"mailto:boot...@gmail.com" target=3D"_blank">booth=
t...@gmail.com</a>&gt;</span> wrote:<br><blockquote class=3D"gmail_quote" sty=
le=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello,<br>=A0=A0=A0=A0=A0=A0=A0 I recently got my code to work hitting a ya=
hoo owned website... (<a href=3D"http://www.rivals.com" target=3D"_blank">w=
ww.rivals.com</a>) However after around 1000 hits,=A0 it all the sudden cra=
shes or stops and I get this error.....<br>
<br>Traceback (most recent call last):<br>=A0 File &quot;C:\Users\Tom\Docum=
ents\Python\bs4final.py&quot;, line 42, in &lt;module&gt;<br>=A0=A0=A0 main=
()<br>=A0 File &quot;C:\Users\Tom\Documents\Python\bs4final.py&quot;, line =
15, in main<br>
=A0=A0=A0 page =3D urllib2.urlopen(request)<br>=A0 File &quot;C:\Python27\l=
ib\urllib2.py&quot;, line 126, in urlopen<br>=A0=A0=A0 return _opener.open(=
url, data, timeout)<br>=A0 File &quot;C:\Python27\lib\urllib2.py&quot;, lin=
e 406, in open<br>
=A0=A0=A0 response =3D meth(req, response)<br>=A0 File &quot;C:\Python27\li=
b\urllib2.py&quot;, line 519, in http_response<br>=A0=A0=A0 &#39;http&#39;,=
 request, response, code, msg, hdrs)<br>=A0 File &quot;C:\Python27\lib\urll=
ib2.py&quot;, line 444, in error<br>
=A0=A0=A0 return self._call_chain(*args)<br>=A0 File &quot;C:\Python27\lib\=
urllib2.py&quot;, line 378, in _call_chain<br>=A0=A0=A0 result =3D func(*ar=
gs)<br>=A0 File &quot;C:\Python27\lib\urllib2.py&quot;, line 527, in http_e=
rror_default<br>
=A0=A0=A0 raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)<br>HTTPE=
rror: HTTP Error 999: Unable to process request at this time -- error 999<b=
r>&gt;&gt;&gt; <br><br>PLEASE tell my that there is a way around this issue=
... if it is server hits related! or is there a limit inside bs4 somehwere?=
<br>
<br>Thanks,<br>Tom<span class=3D"HOEnZb"><font color=3D"#888888"><br>

<p></p>

-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;beautifulsoup&quot; group.<br>
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/d/msg/beautifulsoup/-/czHhafYyEqQJ" target=3D"_blank">https://groups.goo=
gle.com/d/msg/beautifulsoup/-/czHhafYyEqQJ</a>.<br>=20
To post to this group, send email to <a href=3D"mailto:beautifulsoup@google=
groups.com" target=3D"_blank">beautifulsoup@googlegroups.com</a>.<br>
To unsubscribe from this group, send email to <a href=3D"mailto:beautifulso=
up%2Bunsubscribe@googlegroups.com" target=3D"_blank">beautifulsoup+unsubscr=
ibe@googlegroups.com</a>.<br>

For more options, visit this group at <a href=3D"http://groups.google.com/g=
roup/beautifulsoup?hl=3Den" target=3D"_blank">http://groups.google.com/grou=
p/beautifulsoup?hl=3Den</a>.<br>


</font></span></blockquote></div><br><br clear=3D"all"><div><br></div>-- <b=
r>Link Swanson<div>Must Build Digital</div><div><br></div><br>

--f46d040714c5038c3504c585bf1a--
Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
Account Options
