Message from discussion Directory of Feeds?
Received: by 10.100.120.20 with SMTP id s20mr3301005anc.24.1288317287186;
Thu, 28 Oct 2010 18:54:47 -0700 (PDT)
Received: by 10.100.102.9 with SMTP id z9ls840945anb.3.p; Thu, 28 Oct 2010
18:54:46 -0700 (PDT)
Received: by 10.100.194.9 with SMTP id r9mr204761anf.42.1288317286149; Thu, 28
Oct 2010 18:54:46 -0700 (PDT)
Received: by f33g2000yqh.googlegroups.com with HTTP; Thu, 28 Oct 2010 18:54:46
Date: Thu, 28 Oct 2010 18:54:46 -0700 (PDT)
X-HTTP-UserAgent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:22.214.171.124)
Subject: Re: Directory of Feeds?
From: Vivek Venugopalan <venugopalan.vi...@gmail.com>
To: NY Senate Developers Forum <firstname.lastname@example.org>
Content-Type: text/plain; charset=ISO-8859-1
All - I have been lurking on this thread and thought I will pitch in
with my 2c. You can also look at services such as feedburner (http://
www.feedburner.com) that can provide feeds in multiple formats and
also save you the hits as they take the traffic load and poll you
periodically for changes.
On Oct 28, 10:08=A0pm, Bob Wyman <b...@wyman.us> wrote:
> Noel Hidalgo <n...@noneck.org> wrote:
> > please understand we are a handful of
> > developers who are working day and night
> Noel, I would expect nothing else. Somehow, it always seems that the most
> interesting projects are often the ones with the tightest resources!
> > We have a firehose feed at NYSenate.gov/rss <http://nysenate.gov/rss> b=
> it's just a "feed."
> That's not surprising since most folk only provide the old-style polled
> feeds these days and thus suffer propagation latency and the constant loa=
> of many people polling for updates. Hopefully, however, we're moving to a=
> environment which will be more push based -- using technologies like
> PubSubHubbub (PSHB). This should allow people to get feed updates more
> rapidly while reducing the load on publisher's sites. While it can probab=
> be argued that the load issues aren't going to be that significant for a
> site such as yours since the community of those who recognize the value o=
> your data is much smaller than it should be, I think it is very much the
> case that much of what you publish is data that has a good bit of "time
> value" to those who are interested in it and thus propagating timely upda=
> would be a good thing.
> If you had more development resources, you might consider implementing a
> PSHB hub for use by the Senate, or perhaps shared across the entire NY St=
> Legislature. (A number of folk are already working on Drupal and PHP
> extensions to support PSHB.) However, there is a "first step" that you co=
> take in the absence of extra developers and that would be to simply add a
> bit of auto-discovery data to your feed that would allow an existing PSHB
> hub to service your feeds. Please consider taking a look athttp://code.go=
ogle.com/p/pubsubhubbub/wiki/RssFeedsand adding a hub link to
> the <channel/> element in your =A0feed. If you wanted to authorize servic=
> the hub that Google runs on App Engine, you would use:
> <atom:link rel=3D"hub" href=3D"http://pubsubhubbub.appspot.com"/>
> Adding this link won't actually result in any additional load on your sit=
> unless someone actually subscribes to your feed via the hub. From then on=
> no matter how many people subscribe, you would see just the load of a sin=
> You might also consider taking a look at SuperFeedr <http://superfeedr.co=
> who provide a service of running PSHB hubs for people and thus allowing t=
> the benefit of PSHB but without the overhead of development, keeping up o=
> evolving specification changes, etc. see:http://superfeedr.com/documentat=
> > As for why RSSv2? It's what is baked into Drupal 6
> > and no one has asked for ATOM feeds.
> Yes, many people choose RSS for similar reasons. This gets a bit frustrat=
> since there are quite a large number of us who have put a great deal of w=
> into making Atom a much better specified and more capable format than RSS=
> Given that virtually every feed reader handles both RSS and Atom, there
> really isn't any reason to support RSS anymore. (Eventually legacy protoc=
> should be allowed to fade away...) Today, much of the new development in
> syndication formats is focused on Atom, not the "frozen" RSS specificatio=
> and thus folk who go RSS-only are, unfortunately, not able to benefit fro=
> the work that we do. Frankly, I think folk who bundle RSS-only support as
> the default in things like Drupal are not really serving the best interes=
> of their users.
> bob wyman
> On Thu, Oct 28, 2010 at 12:34 PM, Noel Hidalgo <n...@noneck.org> wrote:
> > Hey bob! Thanks for your comments.
> > We have a firehose feed at NYSenate.gov/rss but it's just a "feed." whi=
> > we have discussed pubsubhubub, we have chosen to develop a full feature=
> > that we use internally for our apps and mobile apps.
> > As for why RSSv2? It's what is baked into Drupal 6 and no one has asked=
> > ATOM feeds.
> > Also, please understand we are a handful of developers who are working =
> > and night to make a difference. As I responded to your buzz feed post, =
> > throw the baby out with the bath water. We gladly take feedback and wil=
> > work to incorporate it into our development cycle, but understand we ha=
> > limited resources.
> > Thanks.
> > --
> > Sent from my Apple Newton
> > noel hidalgo
> > irc | twitter - @noneck
> > voice | sms - +1.646.867.2263
> > On Oct 27, 2010, at 20:57, Graylin Kim <graylin....@gmail.com> wrote:
> > Thanks for the detailed answer Bob! I'm really enjoying the links as we=
> > Its clear that you've either thought a good deal about this or done thi=
> > kind of thing before. Are you part of the ActivityStreams team?
> > As far as votes feeds go, is this<http://open.nysenate.gov/legislation/=
search/?term=3Dotype:vote&format=3Drss>what you were looking for? How about=
> > Passage Events<http://open.nysenate.gov/legislation/search/?term=3Dotyp=
> > I don't think the current production API can support Committee Action
> > filters (Nathan or Jared, do you know?) but the API under development s=
> > be more robust.
> > We currently pipe the search box contents (almost) directly into our
> > Lucene <http://lucene.apache.org> powered database which supports a fai=
> > robust query language<http://lucene.apache.org/java/2_3_2/queryparsersy=
> > I've got some pretty poor (but workable) documentation<http://github.co=
m/GraylinKim/nyss_openlegislation/wiki/Searching>of the different object ty=
pes and fields currently supported on the wiki for
> > my more or less completed but collecting dust python library I was writ=
> > for it.
> > As far as fire hoses and push notifications, I know there has been some
> > talk of such things internally but that may not be possible for us at t=
> > current time (for a variety of reasons). Perhaps someone else can provi=
> > better perspective there.
> > If you are willing, I'd certainly enjoy picking your brain about some o=
> > these things in the future. If only for my own enlightenment.
> > ~Graylin
> > P.S. If you have OpenLegislation related questions you can also try and
> > stop by #nyss_openlegislation on FreeNode.net and find us/me (hopefully=
> > there.
> > On Wed, Oct 27, 2010 at 1:42 PM, Bob Wyman < <b...@wyman.us>b...@wyman.=
> >> Graylin Kim < <graylin....@gmail.com>graylin....@gmail.com> wrote:
> >> > what sorts of feeds/data were you looking for?
> >> My primary interest is in "alerting" or publish/subscribe applications=
> >> this particular case, I'm intrigued by the idea of using the recently
> >> announced Google App Engine Matcher API<http://groups.google.com/group=
/google-appengine/browse_thread/thread/...>to build a service that would pr=
ovide real-time prospective search or alerts
> >> for actions, events, etc. that occur in the New York State Senate. The
> >> various statements, press releases and such that seem to make up the b=
> >> the feeds that I've been able to find are, of course, interesting. (i.=
> >> tell me whenever some Senator mentions "No. 6 fuel oil..") However, I =
> >> really like to be able to get feeds that report events or activities l=
> >> votes, committee actions, bill passage events, etc. (I can't find vote
> >> feeds... Do they exist or have I just missed them?) The App Engine Mat=
> >> API permits one to build pretty much arbitrarily complex boolean expre=
> >> that are applied to structured data. Thus, if there is a rich source o=
> >> data, much can be done with it with minimal effort. (And, Matcher scal=
> >> very nicely as well...)
> >> Ideally, it would be possible to create an extension to ActivityStream=
> >> might be called "Legislative ActivityStreams." This format would provi=
> >> means to describe the common activities of a legislature. (i.e. verbs =
> >> "voted", "sponsored", "co-sponsored", "reported from committee",
> >> "made-statement", etc... Given that, I could then create an interface =
> >> Activity Query Language that would allow interested parties to be noti=
> >> whenever interesting Activities were published.
> >> Also, it would be ideal if the feeds were pushed by the New York Senat=
> >> instead of requiring everyone to engage in wasteful polling of the fee=
> >> And, it would be nice if there was a "firehose" feed that aggregated i=
> >> single stream all of the data being published to the other feeds. For
> >> instance, such a firehose feed might be distributed using the
> >> PubSubHubbub <http://code.google.com/p/pubsubhubbub/> (PSHB) protocol
> >> which has become the common protocol for pushing syndication feeds on =
> >> Internet. If the senate was pushing their feeds using PSHB, I could th=
> >> build a subscriber that would consume the firehose feed in real-time f=
> >> PSHB hub and do the necessary subscription matching and alerting with =
> >> trouble by using the AppEngine Matcher.
> >> Just some ideas... I hope that answers your question.
> >> bob wyman
> >> On Wed, Oct 27, 2010 at 2:06 AM, Graylin Kim < <graylin....@gmail.com>
> >> graylin....@gmail.com> wrote:
> >>> On a separate note of curiosity, what sorts of feeds/data were you
> >>> looking for?
> >>> ~Graylin
> >>> On Tue, Oct 26, 2010 at 11:07 PM, Nathan Freitas <<nathanfrei...@gmai=
> >>> nathanfrei...@gmail.com> wrote:
> >>>> We will be providing a more detailed listing of the RSS feeds next w=
> >>>> It will include both feeds from our Drupal-based primary site and th=
> >>>> Legislation system. Most of these are also fairly obvious if you bro=
> >>>> sites yourself.
> >>>> As for ATOM, our goal was to support feed reader libraries and tools=
> >>>> the most comprehensive manner, and RSS fit the bill for that.
> >>>> +Nathan Freitas
> >>>> NYSenate Developer
> >>>> On Oct 27, 2010 8:14 AM, "Bob Wyman" < <bobwy...@gmail.com>
> >>>> bobwy...@gmail.com> wrote:
> >>>> > The developer network site says that there are "hundreds" of feeds
> >>>> > available, but I can't seem to find a listing of those feeds anywh=
> >>>> > Does a directory to the feeds exist?
> >>>> > Also, the site says