Combining feedzirra with HTTP header checking?

[Scooterrr]

Hi folks -

First, I heart feedzirra, which replaced some spaghetti code - thanks!

Second - in order to improve feed processing time and reduce bandwidth
use, I'd like to combine checking the HTTP request header for last-
modified with feedzirra, so I only download full feeds when I need
them: pseudocode like

header = get_header(feed.url)
if header.last_modified > feed.last_modified
get_full_feed(feed.url)

There may be some way to use the new etag features in Rails 2.3.X, but
I've been able to combine those code samples and descriptions with
checking an external URL. Does anybody have (or could generate) any
samples?

Thanks!

[Paul Dix]

Feedzirra already has support for that. Use the update functionality.
It will put in etag and if-modified-since headers. There's a gist that
shows specific usage here:
http://gist.github.com/132671

Best,
Paul

[Scooterrr]

Thanks. It wasn't clear to me if the updated? actually pulled the
entire feed and parsed, or if it was just using the headers.

-Scott

[Paul Dix]

It uses the headers first. Although not every feed actually looks at
those. So if it receives a response then it uses the url of the most
recent post to compare against the others to see what is new.

To test it out load up 20 feedburner feeds, time doing a bulk get on
those and then time doing an update. Drastic difference.

[Scooterrr]

Sounds good, testing it this am. Do I need to store etags or will
modified_date be enough?

-Scott

[Paul Dix]

it really depends on the feed provider. You're best storing both.

[Scooterrr]

OK, so the problem I'm finding very consistently with this approach is
that updated_feed is not getting the new last_modified date, so unless
I pull feeds cleanly my last_modified dates get out of sync. See
console interaction at http://gist.github.com/188601.

(I think the same problem exists with etags but haven't provided an
example.)

-Scott

[Paul Dix]

Thanks Scott, I'll have to look at that. Can you add an issue on github?

Paul