Problem with Hub?

[Pankaj Jain]

Is there a problem with the hub? I'm not receiving any notifications but if I manually go and publish a topic, the notifications come through.

For example:

http://www.nytimes.com/services/xml/rss/nyt/start-ups.xml

https://pubsubhubbub.appspot.com/topic-details?hub.url=http%3A%2F%2Fwww.nytimes.com%2Fservices%2Fxml%2Frss%2Fnyt%2Fstart-ups.xml

Or

http://www.google.com/reader/public/atom/user/18179744707000356486/state/com.google/broadcast

https://pubsubhubbub.appspot.com/topic-details?hub.url=http%3A%2F%2Fwww.google.com%2Freader%2Fpublic%2Fatom%2Fuser%2F18179744707000356486%2Fstate%2Fcom.google%2Fbroadcast

Thanks!

[Julien Genestoux]

Have you tried with other hubs? Like the tumblr one? http://tumblr.superfeedr.com/

If it works, then yes the problem is with the hub :)

Julien

[Waleed Abdulla]

I'm receiving pings from pubsubhubbub.appspot.com. Looks like the hub is up. 

[Pankaj Jain]

Ok, I'm not entirely nuts. The problem is persisting not just for my app but also within friendfeed. Take for example, this Google Reader topic: 

http://www.google.com/reader/public/atom/user/18179744707000356486/state/com.google/broadcast

I share items from Google Reader and no notifications hit my FriendFeed account or my app. However, when I go and publish the feed manually, both FriendFeed and my app receive notifications. 

The reason I think it's more than Google Reader is because I'm not receiving notifications for other topics either, e.g. http://www.nytimes.com/services/xml/rss/nyt/start-ups.xml

Any suggestions would be welcome.

Thanks!

Pankaj

[Julien Genestoux]

Pankaj,

this feed uses the pheedo hub. You want to see with pheedo if they have anything wrong with their hub... or again, if you're not doing a mistake yourself, by testing your code with another feed on another hub.

Julien

[Pankaj Jain]

Hi Julien,

When you say "this feed uses the pheedo hub" you're referring to the nytimes feed, right? That still wouldn't explain the Google Reader feed problem. I can also give you 10-15 other feeds that I'm not receiving any notifications from the appspot hub.

I'm going to make a few changes to test with some other hubs but no code has changed on my side, I just stopped receiving notifications unless I manually go and publish the feed(s).

Thanks!

Pankaj

[Julien Genestoux]

Pankaj

The nytimes uses the pheedo hub, so if if the pheedo hub has an issue, then, yes, it will explain the issue with Google Reader.

For the appspot hub, we will have to wait for  Brett from Google, but the fact that 2 unrelated hubs would stop working for you would seem to indicate that they are not the problem, which is why I am advocating that you try testing other hubs as well.

Julien

[Waleed Abdulla]

I don't know if you're having trouble with all the feeds you're working with, or with a small subset! But here is my own personal experience with PSHB in case it helps:

I subscribe to a lot of feeds and I've learned that, in general, PubSubHubbub is not a reliable system. It's a distributed system that involves the publisher, hub, and subscriber communicating over the Internet, which means that any bug or issue at any point in the chain will cause a notification to be lost. Often the issue is when a publisher doesn't send a notification to the hub, and sometimes the hub is to blame, and so on. 

The way I handle it is that I have a slow polling process that fetches feeds once a day. If my polling process fetches a post that's not in my database, then I know something is wrong because that post should have been received already through a PSHB notification. So I mark that hub is "potentially broken", and put that feed on a faster polling process for a month (in the hope that the hub gets fixed in that time), and then repeat the process. 

Waleed

[Julien Genestoux]

Waleed, 

I am not sure why you have issues, but I think it's wrong to say pubsubhubbub is unreliable. there may be issues, but they are only due to implementation issues... not to the protocol. It is hard to debug, but it can be done, and I believe this community has been helpful and have been able to fix issues often times.

Pankaj, I have offered help many times...

Julien

[Tamer Yousef]

Julien:

Yes the implementation that may break some data access points, that's why I have some logic very similar to what Waleed described, which is to have a backup system to supplement the PSHB main notification system. So if for any reason the whole PSHB is down, we have a plan B to turnaround and pull the data without putting stress on hosts.

-Tamer

All in all, it has been working fine for some time with minor issues. 

[Waleed Abdulla]

Julian,

    I'm not referring to the protocol. It's the nature of any distributed system that it suffers from unreliability due to the wide range of moving parts. When you deal with a lot of feeds, those failures start to add up to the point that you have to build a back up plan for them. 

Waleed

[Kieran Farr]

Adding to the mix here, we're having issues getting pubsubhubbub
updates with YouTube (via GData), trying to track where things are
breaking down here...

- We attempted to subscribe to a video channel based on the feed of a
users' most recent videos. The hub we are trying to subscribe to is
the one at "https://pubsubhubbub.appspot.com" with the feed structure
being "http://gdata.youtube.com/feeds/api/users/CHANNEL_NAME/uploads?
v=2".
- We've noticed that while using the publisher diagnostic provided at
https://pubsubhubbub.appspot.com, that changes made to the YouTube
upload feeds are not being pushed to the hub.
- We can manually publish a feed at https://pubsubhubbub.appspot.com/publish,
but the feeds doesn't appear to auto self publish.

Kieran (and team)

> >> On Mon, Sep 12, 2011 at 2:49 AM, Pankaj Jain <pjain...@gmail.com> wrote:
>
> >>> Hi Julien,
> >>> When you say "this feed uses the pheedo hub" you're referring to the
> >>> nytimes feed, right? That still wouldn't explain the Google Reader feed
> >>> problem. I can also give you 10-15 other feeds that I'm not receiving any
> >>> notifications from the appspot hub.
>
> >>> I'm going to make a few changes to test with some other hubs but no code
> >>> has changed on my side, I just stopped receiving notifications unless I
> >>> manually go and publish the feed(s).
>
> >>>  Thanks!
> >>> Pankaj
>
> >>> On Sep 12, 2011, at 2:49 PM, Julien Genestoux wrote:
>
> >>> Pankaj,
> >>> this feed uses the pheedo hub. You want to see with pheedo if they have
> >>> anything wrong with their hub... or again, if you're not doing a mistake
> >>> yourself, by testing your code with another feed on another hub.
> >>> Julien
>

> >>> On Mon, Sep 12, 2011 at 11:11 AM, Pankaj Jain <pjain...@gmail.com>wrote:
>
> >>>> Ok, I'm not entirely nuts. The problem is persisting not just for my app
> >>>> but also within friendfeed. Take for example, this Google Reader topic:
>

> >>>>http://www.google.com/reader/public/atom/user/18179744707000356486/st...