7.4 aggregated Content distribution
Tim Bray
aggregated distribution might be quite a bit different. And also, the
original upstream feed might have used entry/source already (this
makes me nervous about the whole notion of PuSH co-opting <source> for
its own purposes).
I was wondering if you might want to put an extension element here as
a child of feed, before the entries start, in a pubsubhubbub
namespace, saying "the following are aggregated by the hub". You can
do this safely because Atom has MustIgnore on markup it doesn't
recognize (hint hint).
[Um, when I read this section, there's a little voice in the back of
my head shouting "YAGNI!"]
-T
R P
Brett Slatkin
>
> Seems to me that the client model for processing a single vs.
> aggregated distribution might be quite a bit different. And also, the
> original upstream feed might have used entry/source already (this
> makes me nervous about the whole notion of PuSH co-opting <source> for
> its own purposes).
>
> I was wondering if you might want to put an extension element here as
> a child of feed, before the entries start, in a pubsubhubbub
> namespace, saying "the following are aggregated by the hub". You can
> do this safely because Atom has MustIgnore on markup it doesn't
> recognize (hint hint).
This is the first time I've heard someone point this out. I believe
the atom:source element was specifically included in that spec for the
purpose that PubSubHubbub is using it. Bob Wyman seemed to indicate
the same thing too in some other email threads on this list. Could you
clarify how this is "co-opting" the source element?
> [Um, when I read this section, there's a little voice in the back of
> my head shouting "YAGNI!"]
I disagree with "YAGNI" here. Take world-wide RSS traffic. Multiply by
1,000,000. We will need aggregated delivery to fully utilize links.
-Brett
Tim Bray
>> Seems to me that the client model for processing a single vs.
>> aggregated distribution might be quite a bit different. And also, the
>> original upstream feed might have used entry/source already (this
>> makes me nervous about the whole notion of PuSH co-opting <source> for
>> its own purposes).
...
> This is the first time I've heard someone point this out. I believe
> the atom:source element was specifically included in that spec for the
> purpose that PubSubHubbub is using it. Bob Wyman seemed to indicate
> the same thing too in some other email threads on this list. Could you
> clarify how this is "co-opting" the source element?
Well, consider the popular feed at
http://planet.intertwingly.net/atom.xml - it's already an aggregation,
produced by the well-known "planet" software, and makes heavy use of
the <source> element. What happens when PSHB tries to combine this
and several other feeds?
It's arguably a shortcoming of atom:source that it doesn't handle
multiple levels of nesting. But I also think it's a mistake for PSHB
to assume that it's the only link in the aggregation chain.
>> [Um, when I read this section, there's a little voice in the back of
>> my head shouting "YAGNI!"]
>
> I disagree with "YAGNI" here. Take world-wide RSS traffic. Multiply by
> 1,000,000. We will need aggregated delivery to fully utilize links.
You're entitled to your opinion, but I think you should get it working
first and discover your bottlenecks by experience, rather than invent
something to fix a problem you're pretty sure you're going to have.
Following this advice is easier for me than for other people because
repeated humiliation has taught me that I'm not smart enough to
predict where the choke points are going to be in things that approach
internet scale.
Anyhow, the <source> element strikes me as a lousy solution. It's
going to bulk up your feeds pretty severely, unless I'm missing
something. Why not just have a
<pubsubhubbub:divider src="http://wherever..." />
separator element here and there among the entries? Or jam a bunch of
feeds together with multipart/related (works great, lots of
libraries)? -T
Bob Wyman
> It's arguably a shortcoming of atom:source that it
> doesn't handle multiple levels of nesting.
The general issue is that when you copy an entry from any feed document other than that feed document whose metadata is in the entry's atom:source, there is no way to indicate from which feed document you copied the entry unless you insert some extension element. Of course, if you add an extension element, you'll might be breaking a signature... So, you probably don't want to do that.
One line of reasoning says that this is not a big problem. As is often noted, Atom is supposed to be about entries, not feeds. Thus, the *important* feed to associate with an entry is its source feed -- not the feed where you happened to stumble across the entry. Atom, as defined, satisfies these people.
Others will argue that it *is* important to know not only the source feed but *also* where you found the entry. This tends to lead people to want to record "provenance" or insert into an entry a list of feeds and/or locations (entries can live outside a feed) that record all the places that an entry has been found as it is copied around the network. Atom, as defined, does not satisfy these people.
bob wyman
Ravi Pinjala
case. :) For any subscriber not getting multiple updates per minute
(because they're only subscribed to hundreds of feeds, say, and not tens
of thousands), aggregated delivery introduces some complexity to
parsing, and doesn't really give any benefits.
It really depends on how PuSH is used in practice. If the majority of
the bandwidth is used by a few clients subscribing to many feeds, then
aggregated delivery could be really important. If the majority of
bandwidth is going to a lot of clients, each subscribing to few feeds,
then it really won't be. I personally think that once personal news
aggregators support PuSH we'll see a huge shift in the latter direction,
but that's really just guessing on my part.
I still think aggregated distribution should be made optional, and
controlled by an extra parameter when subscribing.
--Ravi
James Holderness
On Oct 21, 10:13 pm, Brett Slatkin <bslat...@gmail.com> wrote:
> > > Seems to me that the client model for processing a single vs.
> > aggregated distribution might be quite a bit different. And also, the
> > original upstream feed might have used entry/source already (this
> > makes me nervous about the whole notion of PuSH co-opting <source> for
> > its own purposes).
>
> the atom:source element was specifically included in that spec for the
> purpose that PubSubHubbub is using it.
"If an atom:entry is copied from one feed into another feed, then
the source atom:feed's metadata MAY be preserved within the
copied entry by adding an atom:source child element, IF IT IS
NOT ALREADY PRESENT IN THE ENTRY" (emphasis mine)
When you casually wipe out an existing source element, you're not
only erasing the true source of the entry - in some cases you're
potentially erasing the actual authorship of the entry too. I
can't believe you think that's acceptable!
The Google Readers "shared items" feeds do the same thing. In
that case you go so far as to add your own author element with
the name "(author unknown)". No, the author wasn't "unknown" -
you just deleted them.
You people need to stop pulling crap like this. It's evil.
Pádraic Brady
Paddy
http://blog.astrumfutura.com
http://www.survivethedeepend.com
OpenID Europe Foundation Irish Representative
From: James Holderness <j4j...@gmail.com>
To: Pubsubhubbub <pubsub...@googlegroups.com>
Sent: Sat, October 24, 2009 12:21:41 PM
Subject: [pubsubhubbub] Re: 7.4 aggregated Content distribution
John Panzer
Tim Bray
> The right thing to do IMHO is to add something like a psh:provenance element
> that is just like atom:source but tracks the most recent context of the
> entry.
The right thing to do IMHO is not aggregate at the PubSubHubbub level
until you've proved that (a) you have to and (b) multipart/related
won't cut it.
-T
Bob Wyman
Actually, the PSHB use case *was* frequently discussed in the Atom WG... The reason for this is that what PSHB does is provide pretty much what FeedMesh was intended to provide - a topic based subset of the content-based system that Pubsub.com provided.
An atom:source, if present in an entry, should never be modified or overwritten by an aggregator. An aggregator should only add atom:source if it isn't already present.
If you want to show provenence, you need to add an extension element. As pointed out in an earlier message, the WG was aware that provenence was useful but we couldn't get consensus on how to record it.
Please, if you define a provenence element, include in the definition a requirement to remove that element prior to canonicalisation when verifying signatures...
bob wyman
On Oct 24, 2009 12:45 PM, "John Panzer" <jpa...@google.com> wrote:Agreed to keeping the tone civil -- assume good intentions until proven otherwise; it can only help.Mea culpa for not noticing this issue with PubSubHubbub. The Atom spec didn't envision this use case and so atom:source is almost, but not quite, what's needed -- thus the confusion is understandable.The right thing to do IMHO is to add something like a psh:provenance element that is just like atom:source but tracks the most recent context of the entry.
On Sat, Oct 24, 2009 at 7:24 AM, Pádraic Brady <padrai...@yahoo.com> wrote: > > Whoa ;). Ton...
John Panzer
multipart/related and batching that may be relevant; I'm skeptical
about ease of subscriber implementation.
--
John Panzer
> jpa...@google.com / abstractioneer.org <http://www.abstractioneer.org/> / @jpanzer
>
>
>
> On Sat, Oct 24, 2009 at 7:24 AM, Pádraic Brady <padrai...@yahoo.com> wrote:
>>
>> Whoa ;). Ton...
>
>
--
Brett Slatkin
Cool. This thread is a great example of peer review. I'll file an issue in the bug tracker to distill this. Thanks everyone!
-Brett
On Oct 24, 2009 6:29 PM, "John Panzer" <jpa...@google.com> wrote:
Yep, the WG envisioned this, the final spec did not. :)
On Saturday, October 24, 2009, Bob Wyman <b...@wyman.us> wrote: > Actually, the PSHB use case *was* ...
> jpa...@google.com / abstractioneer.org <http://www.abstractioneer.org/> / @jpanzer
> > > > On Sat, Oct 24, 2009 at 7:24 AM, Pádraic Brady <padrai...@yahoo.com> wrote: >> >> Whoa ;...