empty reponse body

[Andreas Bloch]

I'm receiving empty response bodies on some sites:

In [1]: fetch('http://www.stvalentinshop.com/all/faguo-linden-camel.html')

In [2]: type(response)

Out[2]: scrapy.http.response.html.HtmlResponse

In [3]: response.body

Out[3]: ''

Doing a simple urlopen with urllib works...

In [9]: response = urllib2.urlopen('http://www.stvalentinshop.com/all/faguo-linden-camel.html').read()

In [10]: response (works)

What it wrong here?

[Rolando Espinoza La Fuente]

The target web server replies with an empty body when the user agent is the scrapy's default. I'm able to see the content by changing the user agent:

$ scrapy shell http://www.stvalentinshop.com/all/faguo-linden-camel.html --set USER_AGENT=googlebot

Regards

Rolando

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users...@googlegroups.com.
To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.

[Andreas Bloch]

Perfect! Thanks.

--
You received this message because you are subscribed to a topic in the Google Groups "scrapy-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scrapy-users/3hhBkwWxcM8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to scrapy-users...@googlegroups.com.