Your cURL is messed up, OR FeedFetcher is BROKEN, or someone doesn't
know what they are doing and they are LYING to cover their butt ... or
a combination of all of the above!
The tech copied reader support on that email. Go dig it out and see
for yourselves ... and if you can talk the talk, why not reply and
have a nice little chat so you can be set straight on this issue once
and for all, so I don't have to be stuck in the middle of this Charlie
Foxtrot!
Hi, Lissa, sorry you're still having this problem. We know this is
frustrating and we apologize. We looked into this again, perhaps the
following can help everyone:
Here's why the feed isn't updating. Reader tries to ask servers if
their feeds have updated. Reader sends a date. The server says OK or
NO CHANGE. If OK, those servers send a feed. Not all feedreaders ask
if a feed has updated, but Reader's popularity means it's better to
ask to prevent bringing harm to servers or causing bandwidth issues.
In this case, something's different. Reader asks lissakay.com if that
particular feed has updated. Reader sends a date (e.g. the last time
Reader got an OK). The server says OK. But the feed is never sent.
Here's the nerdy part, and why it's been hard for your techs to see:
The curl command that mimics feedfetcher seems to only produce the
error from lissakay.com (e.g. no feed) when the date is ~somewhat~
recent to the current time. We can reliably reproduce this every time,
for example in the following command:
The trick is to keep the if-modified-since date relatively close to
the current time but still in the past. I'd recommend trying to keep
the time within the hour. Basically the server is telling us "hi,
everything's ok" but it's not sending us the feed, so we never get an
update.
We also noticed that on running the curl command, the connection takes
some time to close. I dunno if that information is important, but I
thought I'd pass it along.
I'm sorry for the technical support ping-pong, that's always maddening
to me when it happens. Hopefully this can help.
That just looks like more excuses to me. There's no solution offered
there, just more blame ... while you continue to ignore the big fat
elephant smack dab in the middle of the friggin room!
***ONLY GOOGLE READER*** IS NOT UPDATING MY FEED and it started all of
a sudden with NOTHING being changed on my end!
We have checked everything from every angle imaginable on my end and
found NOTHING.
GOOGLE READER IS BROKEN.
FIX IT!!
And you might want to fix the feeds of all the others that are
starting to notice this issue too!!
> Hi, Lissa, sorry you're still having this problem. We know this is
> frustrating and we apologize. We looked into this again, perhaps the
> following can help everyone:
> Here's why the feed isn't updating. Reader tries to ask servers if
> their feeds have updated. Reader sends a date. The server says OK or
> NO CHANGE. If OK, those servers send a feed. Not all feedreaders ask
> if a feed has updated, but Reader's popularity means it's better to
> ask to prevent bringing harm to servers or causing bandwidth issues.
> In this case, something's different. Reader asks lissakay.com if that
> particular feed has updated. Reader sends a date (e.g. the last time
> Reader got an OK). The server says OK. But the feed is never sent.
> Here's the nerdy part, and why it's been hard for your techs to see:
> The curl command that mimics feedfetcher seems to only produce the
> error from lissakay.com (e.g. no feed) when the date is ~somewhat~
> recent to the current time. We can reliably reproduce this every time,
> for example in the following command:
> The trick is to keep the if-modified-since date relatively close to
> the current time but still in the past. I'd recommend trying to keep
> the time within the hour. Basically the server is telling us "hi,
> everything's ok" but it's not sending us the feed, so we never get an
> update.
> We also noticed that on running the curl command, the connection takes
> some time to close. I dunno if that information is important, but I
> thought I'd pass it along.
> I'm sorry for the technical support ping-pong, that's always maddening
> to me when it happens. Hopefully this can help.
> ***ONLY GOOGLE READER*** IS NOT UPDATING MY FEED and it started all of
> a sudden with NOTHING being changed on my end!
Unfortunately, you're right that Reader is probably one of the few
feedreaders that will encounter this issue. As I mentioned, it's only
because not all feedreaders ask if a feed has updated in this way, but
Reader's popularity means it's better to ask to prevent bringing harm
to servers or causing bandwidth issues.
> We have checked everything from every angle imaginable on my end and
> found NOTHING.
I know this is frustrating, but asking your techs to reproduce that
curl command with that date might help them to figure out why only
that feed isn't returning any information. One thing's for certain,
that feed isn't showing when a recent if-modified-since date is sent
with the request. We've run this command from locations inside and
outside of Google and are getting the same results (e.g. no feed).
> And you might want to fix the feeds of all the others that are
> starting to notice this issue too!!
So far when we've troubleshooted (troubleshot?) other feeds that
haven't been updating, the last-modified-date hasn't been the cause.
Do you know of any reports I'm missing?
Mehh ... forgot that these groups munge email addresses ... that would
have been fake (at) craig.com posting from 122.18.253.72 which is
located in Japan, and you STILL SUCK
Chris has already mentioned most of the relevant points about this
issue, but let me just see if I can paint a picture of the overall
situation, to clarify further.
You can see there how the system is set up: Google's various crawl
services (web search, blog search, etc.) all share a common cache of
crawled content, to minimize the load we put on people's sites.
However, these services do need to verify that the cached copy of a
page is still current. They do that using "if-modified-since"
requests, which are more light-weight than full content requests. In
the case of your feed, this is what happens:
1. Google Reader checks the crawl cache and finds a recent copy of
your feed.
2. Wanting to make sure it's a current copy, Reader sends your server
an if-modified-since request. (Assume for now that your blog hasn't
been updated, since we crawl more often than you post.)
3. Instead of a 304 ("Not Modified") response, your server sends back
a 200 ("OK") with no content.
4. Reader sees the 200 and assumes this must be the new, up-to-date
version of your feed.
5. Reader therefore ignores the cached copy, and "updates" your feed
with the version it got. Unfortunately, that version was empty, so
there's no change to make.
Some important points about this:
* If we had gotten a 304 in step 3 above, Reader would have used the
cached copy, which probably would have had your recent posts in it.
* If the timing works out such that your blog has indeed been updated
since the last time it was crawled and cached by one of our services,
then the 200 would come back with content and we'd update it in Google
Reader. This explains why you do occasionally see updates. They're
infrequent, though, because there's a relatively narrow time window
relative to our crawls during which you can post and have your server
return a correct response.
* Your feed updates fine in BlogLines and other services because they
presumably don't have this same system set up, with the cached copies
and the "if-modified-since" requests. (If they're only running a
single service, there's less of a need to set up a more complex system
like this.)
* The curl command, referenced several times before, will return
different results depending on the date used. If put far enough in the
past, it will return your content, since you've updated since then.
However, our requests generally have a very recent date/time, so it's
more common for there to be no updates.
To sum it all up, it really is a matter of it all coming down to the
200 vs. 304 responses. If you can get your server to consistently sent
304 responses for unmodified content, then the issue will most likely
go away. For reference, the spec for using 304 response codes can be
found here:
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
If you can get this change made and still verify that there are
problems, we'll be happy to look into this again. However, as things
stand now, there is nothing further we can do from our end. I'm sorry
for all the back-and-forth about it, and I know it's a confusing
issue, but hopefully this (admittedly long) explanation helps.
I'm afraid the above responses are going to remain our final answer
until such time as we can verify that your server is correctly
handling if-modified-since requests, at which point we could take
another look. We've spent a large amount of time investigating this
very case-specific issue, and won't have anything more to go on until
that change is made.
Keep in mind that the existence of another feed that apparently works
isn't a refutation of the previous arguments. It's possible that other
factors on your server cause the two feeds to be served differently.
It's also possible, as described above, for updates to feeds to
sometimes work, depending on the exact timing of posting and crawling.
One feed could simply have luckier timing than another.
I'm sorry we can't do anything more for you at this time.
> I'm afraid the above responses are going to remain our final answer
> until such time as we can verify that your server is correctly
> handling if-modified-since requests, at which point we could take
> another look. We've spent a large amount of time investigating this
> very case-specific issue, and won't have anything more to go on until
> that change is made.
> Keep in mind that the existence of another feed that apparently works
> isn't a refutation of the previous arguments. It's possible that other
> factors on your server cause the two feeds to be served differently.
> It's also possible, as described above, for updates to feeds to
> sometimes work, depending on the exact timing of posting and crawling.
> One feed could simply have luckier timing than another.
> I'm sorry we can't do anything more for you at this time.
Unfortunately, this still seems to exhibit the same behavior. You can
see the results of my test below. (I noticed your latest post was from
yesterday, so I used today's date to make sure it should give a 304,
and you can see that it doesn't.)
A couple more points: First, I think you mentioned at one point having
trouble using the curl command shown here. Google Groups does some
line wrapping, which could cause problems if you're just copying and
pasting our example command. Note that we've run this from both
outside and inside Google, and gotten the same results.
Additionally, we've noticed something else which may be a useful clue:
After receiving the 200 OK message and the initial headers, there is a
noticeable delay before the connection is closed, though this should
be instantaneous. This implies that your server may be trying to
return something, but failing internally, so that no content reaches
us. Any such errors may be getting logged, so that's something your
tech support could help you look for.
Note that simply switching to a new server doesn't necessarily help in
and of itself. This seems more like it's an issue in your PHP
software, so if each server is using the same software, there won't be
much difference between the two. What software are you running, and is
there an upgrade available that you could install?
I'm sorry if that last bit sounded contradictory or confusing. I
believe it is still basically consistent with the premise we've had
all along; I simply intended to point out another thing we'd noticed
that might be useful in tracking down the issue. If you'd like
clarification on any specific part of what I wrote, please let me
know.
However, I did notice that the tech support for your hosting company
has CC'd us on some emails to you. The latest one is along the lines
of "all's well that ends well," though I can't see what he's
responding to since your emails to them aren't included. I'm hoping
that means they found something that they were able to get fixed. Is
that the case?
Well, we have gone from one extreme to another. Apparently the
Feedfetcher is set on my RSS feed to continuously monitor? My updates
are now showing up there nearly instantaneously. Even an addition to
an already published post showed up within a couple of minutes. But
the Atom feed is not updating as quickly.
What happened ... actually, the mention of PHP reminded me of
something that happened. Well, several things happened all at once,
late in September. Google Reader made some change in how feeds are
cached for search purposes, as well as some other changes. At the same
time, I was having problems with some software not working as it
should ... ImageMagick was not correctly installed and/or did not have
a dependency installed at the right version level. Tech support tried
to fix that, but it resulted in my getting Server Error 500 problems
when attempting to use ImageMagick. This server runs Phpsuexec and if
file permissions are set too high, it can cause issues. Tech support
did this, that and the other thing to try to correct those issues, but
we finally gave up with no resolution.
I now believe that it was that mucking around with file perms and
other settings in combination with the changes made to Google Reader
that got everything off track.
So, what was the cure? Who knows. Tech support thinks it is because
they disabled the .htaccess file. Apparently, that was the last thing
they did when the problem came clear. (Yeah, I'm snickering) But, at
the same time, I was clearing off a bunch of old files and software
directories from my server, dropping the MySQL databases and importing
only the ones that I still use. Then I deleted and reuploaded my CMS
software. Somewhere in there was the cure for the ills with Google
Reader - which does indeed handle feeds differently than almost all
other aggregators and readers. But I would bet a egg salad sandwich
that it was NOT the .htaccess file. But if tech support is happy in
believing that, more power to them. We achieved the needful. Somehow.
Now next time, don't be so stingy with the information you guys can
see! I really had to rattle the cages to get you all to cough anything
up! If it makes you feel better, I got cranky with the tech support
for the server too ... they weren't paying attention!
> I'm sorry if that last bit sounded contradictory or confusing. I
> believe it is still basically consistent with the premise we've had
> all along; I simply intended to point out another thing we'd noticed
> that might be useful in tracking down the issue. If you'd like
> clarification on any specific part of what I wrote, please let me
> know.
> However, I did notice that the tech support for your hosting company
> has CC'd us on some emails to you. The latest one is along the lines
> of "all's well that ends well," though I can't see what he's
> responding to since your emails to them aren't included. I'm hoping
> that means they found something that they were able to get fixed. Is
> that the case?
Glad to hear it's resolved! I'm sorry that it was a frustrating
process, but I hope you understand that we weren't deliberately trying
to withhold information. It's tricky debugging this sort of thing from
our end because we have no visibility into your server. So we're
limited to describing the effects we see, and hoping your server's
tech support can trace them back to the source of the issue, be
it .htaccess files or anything else.
We don't continuously monitor any feeds, by the way. However, if you
use a pinging service to notify us when you post then the whole update
process (as described previously) will happen much more quickly. So I
assume that's what you've got going here.
