Anyone else not receiving notifications from Blogger.com?

[Waleed]

I've been suspecting issues with blogger.com feeds for a while, but
today I confirmed it.

1. I subscribed to a test blog on blogspot.com and received a
confirmation from the hub.
2. To double check, I went to the subscription details page on
pubsubhubbub.appspot.com and entered the callback and topic, and it
confirmed that the subscription is valid.
3. I wrote a new test post on the blog and waited for the notification
to come. It didn't.

I repeated the test a few times and checked things on my side, and
as far as I can tell, Blogger notifications are not being sent. Anyone
else noticing this?

[Julien Genestoux]

We (http://superfeedr.com) have seen issues with blogger feeds as well, but I just did a bunch of tests 

and I couldn't reproduce :(

What's the feed url that you used?

thanks

[Waleed Abdulla]

Thanks, Julien. 

    The blog I'm testing with is: http://ninuatest.blogspot.com/

    I'm subscribing to it's "self" feed: http://www.blogger.com/feeds/6999034805426555507/posts/default

    I gave you permission to write new posts to that test blog, in case the needed arises.

Waleed

[Andy Dennie]

It was working for me earlier this week, but today I'm not getting updates from my blogspot test blog ( http://fizzbuzztest.blogspot.com/feeds/posts/default), at least not as promptly as I used to. I'll have to see if they arrive later.

-Andy

[Andy Dennie]

... which is really too bad if it doesn't resolve itself, since that was the only mechanism I had for generating on-demand test notifications.

[Tamer Yousef]

I just scanned some blog spot specific blogs and I'm getting pings, but as you can tell the issue with this kind of technology is there is no way to measure, you have to find specific instances where there is a new blog post without notification sent, on a large scale list, it's hard to catch issues unless you have a backup pull strategy.

Waleed: I subscribed to your blog, can you add a test post?

-Tamer

[Andy Dennie]

Sorry, I goofed in my earlier test; my test posts did arrive promptly; I was just looking in the wrong place. 

[Waleed Abdulla]

Tamer,

    I published a few new test posts. Again, no notifications. I do get notifications from blogs I subscribed to in the past, though (i.e. I see pings from pubsubhubbub.appspot.com in my logs). But I'm also getting reports from users who say that their feeds in our app are not updating.

Andy,

    I subscribed to your feed. Can you please publish something?

Regards,

Waleed

[Andy Dennie]

OK, I just posted about 10 seconds ago.
-Andy

Waleed Abdulla wrote:

Tamer,
    I published a few new test posts. Again, no notifications. I do
get notifications from blogs I subscribed to in the past, though (i.e.
I see pings from pubsubhubbub.appspot.com

<http://pubsubhubbub.appspot.com> in my logs). But I'm also getting

[Waleed Abdulla]

Thanks, Andy. No notification yet!  Did you get one?

[Andy Dennie]

Yes, I got it.  Strange.  I'm using Superfeedr, are you?  If not, then it's likely your hub.

[Waleed Abdulla]

I'm using the hub declared in the feed: http://pubsubhubbub.appspot.com/

Could it an issue in Google's hub?

[Tamer Yousef]

I have not got any notifications from Waleed's test blog, that's after waiting for almost 15 minutes after post submission. 

I thought it may be the foreign characters in the blog title is causing some bad behavior, so I created another test blog  http://ninuatestenglish.blogspot.com/ and sent a new post, but no notifications either...

I agree that it's either a delay or some malfunction on the Google appengine "AppEngine-Google".

-Tamer

[Julien Genestoux]

I haven't had time to dive into this, but if Andy got an update thru the Superfeedr hub, it's likely that we've been notified just fine by Google.... 

I'll triple check a bit later.

Thanks,

[Waleed Abdulla]

Thanks, Tamer. 

    I'm glad I wasn't imagining this. This might've been going on for weeks, but I didn't notice it because we have a backup pulling strategy that marks hub subscriptions as "failed to deliver" and switches feeds to old fashioned polling.

Julien,

    Do you depend on PSHB pings only, or do you also pull?

Waleed

[Julien Genestoux]

So, we did get a ping from the Google Hub for that feed... now, I can't tell how long after the publish time it happend.

At Superfeedr, we do take pings into account, and we reduce our polling period when we know we're getting pings. For non Pubsubhubbub feeds, we fetch them once every 15 mins, for PubSubHubbub enabled feeds, we can sometimes go up to once every 12 hours.

This feed was on the once every 12hrs schedule... but we have more systems than just PuSubHubbub to detect updates in feeds, so we may have detected the update from another feed (we have a feed graph that links graphs that should update at the same time).

Thanks

[Waleed Abdulla]

Thank you, Julian. 

    So if I understand correctly, your test is not conclusive because you have other sources that might've sent the ping. Is that accurate? If so, then we have two independent tests (mine and Tamer's) that confirm there is something wrong with Blogger.com's updates. 

    I'm CCing Brett. Is there a proper channel to report this issue other than this list?

[Julien Genestoux]

Waleed,

Exactly, the test is not conclusive in a sense that I can't confirm that we got the notification from Blogger. However, I can't confirm it's broken.

Let's hope that Brett's answer comes quickly :)

Ju

[YC]

Hi there, I'm not sure is this a solution but it's works fine for me.

1.) Use http://ninuatest.blogspot.com/feeds/posts/default instead of
http://www.blogger.com/feeds/6999034805426555507/posts/default
2.) After subscribe your blog at pubsubhubbub.appspot.com, go to
https://pubsubhubbub.appspot.com/publish and publish
http://ninuatest.blogspot.com/feeds/posts/default manually. You will
receive a notification with few feeds inside.

Hope this method works fine for you too.

[Waleed Abdulla]

Thanks YC, 

    As far as I know, the specs state that we should subscribe to the "self" link if it's in the feed. And that's what I do, and it's always worked until recently. 

    I tried the hub suggestion (using the self link) and it worked and I got a notification immediately. So I guess it shows that the hub itself is working, and the problem is on Blogger.com's side. 

Thanks,

Waleed

[ED209]

Hi Folks,

Well, glad we found this thread as we're experiencing the same problems. We have tried with 3 different blogger blogs. For one of those we get notifications but for the other two we do not.

• The subscription appears to be ok (subscription is found and was verified).Checking on https://pubsubhubbub.appspot.com/subscribe brings up https://skitch.com/edleadesign/eby5r/hub-subscription-details-http-edsdemoblog.blogspot.com-feeds-posts-default 

• We tried to subscribe to "self" url and blog "alias" - neither worked (for two of the blogs)

    - http://edsdemoblog.blogspot.com/feeds/posts/default

    - http://www.blogger.com/feeds/2698432277089135744/posts/default

• We tried to publish these topics but they were not found on (https://pubsubhubbub.appspot.com/publish)

Has this issue been acknowledged? Has anyone found the cause or a fix for it?

Thanks!

Ed.

[Roman]

Hi Ed,

There is a known problem in the hub that prevents notifications to go through if all subscribers for a topic have used async verification mode.

Please try subscribing with hub.verify=sync.

Roman.

[ED209]

Thanks for the quick reply Roman, but we already are subscribing with hub.verify=sync

anything else we could try? Any way to debug?

[Roman]

I'll take a look.

Roman.

[Roman]

I figured out what's going on.

The subscriptions in the hub have an asynchronous component even when you are subscribing with hub.verify=sync. Until the asynchronous part finishes, no updates on the topic will be distributed to the subscribers.

There is a huge backlog of asynchronous processing right now and I estimate that it would take two weeks to process it. I nuked the backlog and see that the hub has resumed processing submissions.

Things should work for you once you resubscribe (with hub.verify=sync). Could you verify?

Roman.

[ED209]

Thanks Roman! That has worked, we're getting notifications now :)

What caused that huge backlog? Not just on our 2 blog subscription?

[Roman]

On Fri, Jun 15, 2012 at 4:37 PM, ED209 <edlea...@gmail.com> wrote:

Thanks Roman! That has worked, we're getting notifications now :)

Great!

 

What caused that huge backlog? Not just on our 2 blog subscription?

I suspect a background process that I launched a week ago in order to fix a different problem (namely, that subscribers with hub.verify=sync don't receive notifications). Although with internal throttling that I've put in place (150 QPS) it couldn't have generated so much backlog... I turned the background process off for now; will take a closer look on Monday.

Roman.

[Maxim Lacrima]

Hi!

This is really strange!

In my experience it doesn't matter how you subscribe - synchronously or asynchronously. The verification request from the hub always comes very quickly (usually in a half a second).

In order to receive notifications from the hub regarding blogger feeds you need to subscribe to the link with rel="http://schemas.google.com/g/2005#feed" . In my experience if I subscribe to the feed using 'self' links iyou never get notifications, but if I subscribe using 'http://schemas.google.com/g/2005#feed' I get almost instantaneous updates.

@ED209, it seems that you solved your problem... But what link[@rel] do you use to create subscriptions? If it 'self' then it is really strange, because it doesn't work for me.

--
with regards,
Maxim

[Roman]

On Fri, Jun 15, 2012 at 4:47 PM, Maxim Lacrima <lacrim...@gmail.com> wrote:

Hi!

This is really strange!

In my experience it doesn't matter how you subscribe - synchronously or asynchronously. The verification request from the hub always comes very quickly (usually in a half a second).

The verification request does indeed happen almost instantaneously, but you won't receive content updates unless someone else subscribes to the same topic with hub.verify=sync.

In order to receive notifications from the hub regarding blogger feeds you need to subscribe to the link with rel="http://schemas.google.com/g/2005#feed" . In my experience if I subscribe to the feed using 'self' links iyou never get notifications, but if I subscribe using 'http://schemas.google.com/g/2005#feed' I get almost instantaneous updates.

@ED209, it seems that you solved your problem... But what link[@rel] do you use to create subscriptions? If it 'self' then it is really strange, because it doesn't work for me.

You should use feed URL as Topic. If you can wget the feed, it means you can also subscribe to it.

[Maxim Lacrima]

Hi Roman,

You should use feed URL as Topic. If you can wget the feed, it means you can also subscribe to it.

 

The feed can have several URLs, like this

http://devhubbub.blogspot.com/feeds/posts/default  which correspnds to link[@rel="http://schemas.google.com/g/2005#feed"]

and this

http://www.blogger.com/feeds/2243540998291553416/posts/default which corresponds to link[@rel="self"]

These URLs point to the same feed. If I subscribe to link[@rel="http://schemas.google.com/g/2005#feed"] (and it doesn't matter which 'hub.mode' I use) then I get updates, but if I subscribe to link[@rel="self"] I never get updates.

Is it expected?

--
with regards,
Maxim

[Dmitri Lakachauskis]

On Jun 15, 2012, at 5:47 PM, Maxim Lacrima wrote:

@ED209, it seems that you solved your problem... But what link[@rel] do you use to create subscriptions? If it 'self' then it is really strange, because it doesn't work for me.

I'm not Ed, but I can answer for him. We used the same link you did. 'self' didn't work for us. But as we have figured out already the problem was in notorious backlog Roman successfully paused. So I assume "http://schemas.google.com/g/2005#feed" link should work just fine.

--

Dmitri

[Roman]

Ok, I got it now.

When PubSubHubbub gets notified of topic changes (i.e., when publishing happens), it must fetch the feed and distribute changes to all subscribers on the topic. When you post something on http://devhubbub.blogspot.com, Blogger tells pubsubhubbub.appspot.com that http://devhubbub.blogspot.com/feeds/posts/default has changed. The hub then propagates the changes to all subscribers of http://devhubbub.blogspot.com/feeds/posts/default.

Ideally, the hub should figure out that http://www.blogger.com/feeds/2243540998291553416/posts/default is the same feed and notify its subscribers as well. Feed deduping is a non-trivial problem; pubsubhubbub.appspot.com uses a bunch of heuristics but unfortunately they did't work in this particular case.

The good news is that the 0.4 PubSubHubbub spec avoids this problem by explicitly specifying which feed URL should be used as topic when subscribing and publishing. Things will get less confusing once Blogger and pubsubhubbub.appspot.com adopt it.

Roman.

[YC]

Hi there, anyone else not receiving updates from blogger.com blogs now? 

My web app is not getting notification from all of my subscribed blog in pubsubhubbub.appspot.com.

All of my subscribed blog In Publisher Diagnostics (https://pubsubhubbub.appspot.com/publish) have the following result:

Topic Details - http://devhubbub.blogspot.com/feeds/posts/default

Fetch from domain: BLOCKED

My web app is able to get notification from superfeedr but not pubsubhubbub.appspot.com now. This issue just started today. Anyone know how to fix this?

[Roman]

Hi YC,

This might be related to a problem I've been trying to fix today. Could you post a couple of feed topics that are supposed to work for you but don't?

Roman.

[Mario Menti]

Hi Roman,

we (Twitterfeed) have been seeing quite a few reports from our users today that could be explained by a problem at the appspot hub -
I've just taken a look at one in particular that's just been reported to us now: http://www.spoilertv.com/feeds/posts/default

We look to be subscribed ok, but haven't received any new post notifications since 2am (UTC) this morning.

Thanks,
Mario.

[Mario Menti]

Roman,

further to this, there appear to be a very large number of feeds that have stopped working, we're getting increasing numbers of problem reports, and looking at our logs, it appears the number of new post notifications received today is a very small percentage of what we usually get.

Thanks,
Mario.

On 10/07/2012 16:16, Roman wrote:

[Julien Genestoux]

I'm with Mario here, looks like there is a big problem with the blogger feeds.

We have reduced our polling period down to 15 mins for these as well, but we are being server cached cnotent as well...

Thanks

[Roman]

All feed fetch attempts are currently failing with google.appengine.runtime.apiproxy_errors.ResponseTooLargeError: The response from API call urlfetch.Fetch() was too large. The limit is apparently 1MB or 32MB (it was increased at some point) and the feeds the hub is trying to fetch are way below that.

Nothing recently has changed in the fetching code. I also trying rolling back to the last version (pushed 2 weeks ago) with no success.

I'm still trying to figure out what's going on.

Roman.

[Julien Genestoux]

Thanks Roman! I know your pain, and I'm sending good thoughts!

Julien

[Mario Menti]

I'm seconding the good thoughts!

[Roman]

Turned out there was a configuration problem that started causing all RPC requests with >32kb response to fail after the last AppEngine release. It was fixed around 10:30 UTC yesterday and since then everything looks fine again.

Sorry for the inconvenience.

Roman.

[Mario Menti]

Thanks for the update Roman, glad it's all sorted!

Sent from my iPad

[Roman]

On Tue, Jul 10, 2012 at 10:32 PM, DarkUFO <andyp...@gmail.com> wrote:

I've forwarded the Issue to my Blogger Developer contacts at Google to see if they can help at all.

Are you still seeing problems with blogger feeds? If so, could you post details?

Roman.

[DarkUFO]

All working fine now Roman. My message was posted last night when the problem was still ongoing but for some reason it only just posted :)