Re: how to call referer from response object?

[Pablo Hoffman]

You should use response.request.headers.get('Referer') - Referer is a request header, not a response one.

On Mon, Jul 9, 2012 at 1:37 PM, fiveoheight <exploding...@gmail.com> wrote:

Hello,

so I want my crawler to find 404 errors and when it figures out that they are invalid, to print a line about the invalid page and where it was linked from.

I tried using response.header.get('Referer') --based on the scrapy's logformatter file using request.header.get('Referer')

unfortunately it seems that the header of referer goes away in the transition from the page as a request to the page as a response that can be parsed. 

is there another way to call the referer of the page being parsed within the parse method? just to print it as a string?

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/scrapy-users/-/gx42L0pMQ_UJ.
To post to this group, send email to scrapy...@googlegroups.com.
To unsubscribe from this group, send email to scrapy-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/scrapy-users?hl=en.