Directory of Feeds? - NY Senate Developers Forum

The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Message from discussion Directory of Feeds?
MIME-Version: 1.0
Date: Thu, 28 Oct 2010 18:54:46 -0700 (PDT)
In-Reply-To: <AANLkTimkw53S8WQ1Svy1JUYg=E0e7LXU8Wt_Da7Q3_wL@mail.gmail.com>
References: <13eaa5f6-0ec7-4a83-9edd-554051675fd9@30g2000yql.googlegroups.com>
 <AANLkTimP42xx2pdxfgnSSAGPRn_YTWtj35U-1m7M53cg@mail.gmail.com>
 <AANLkTim8ZBnrzcp7HgedgSmOcD+1XNAtYxnG2zBtMTV9@mail.gmail.com>
 <AANLkTinRFQgL8StGrAMn1LCz2N_ziOcnXnFsKr7b3tsB@mail.gmail.com>
 <AANLkTik+fUx1q3c8A7UN_XeTUzxHEbm3DumUGLRNWgbT@mail.gmail.com>
 <F6D318C3-CA84-4CC0-9248-FCBA402DA7CA@noneck.org> <AANLkTimkw53S8WQ1Svy1JUYg=E0e7LXU8Wt_Da7Q3_wL@mail.gmail.com>
User-Agent: G2/1.0
Message-ID: <e05e0277-54b2-4e2c-861a-6c06f9a7dd7d@f33g2000yqh.googlegroups.com>
Subject: Re: Directory of Feeds?
From: Vivek Venugopalan <venugopalan.vi...@gmail.com>
To: NY Senate Developers Forum <nyss-dev@googlegroups.com>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

All - I have been lurking on this thread and thought I will pitch in
with my 2c.  You can also look at services such as feedburner (http://
www.feedburner.com) that can provide feeds in multiple formats and
also save you the hits as they take the traffic load and poll you
periodically for changes.

On Oct 28, 10:08=A0pm, Bob Wyman <b...@wyman.us> wrote:
> Noel Hidalgo <n...@noneck.org> wrote:
> > please understand we are a handful of
> > developers who are working day and night
>
> Noel, I would expect nothing else. Somehow, it always seems that the most
> interesting projects are often the ones with the tightest resources!
>
> > We have a firehose feed at NYSenate.gov/rss <http://nysenate.gov/rss> b=
ut
>
> it's just a "feed."
> That's not surprising since most folk only provide the old-style polled
> feeds these days and thus suffer propagation latency and the constant loa=
d
> of many people polling for updates. Hopefully, however, we're moving to a=
n
> environment which will be more push based -- using technologies like
> PubSubHubbub (PSHB). This should allow people to get feed updates more
> rapidly while reducing the load on publisher's sites. While it can probab=
ly
> be argued that the load issues aren't going to be that significant for a
> site such as yours since the community of those who recognize the value o=
f
> your data is much smaller than it should be, I think it is very much the
> case that much of what you publish is data that has a good bit of "time
> value" to those who are interested in it and thus propagating timely upda=
tes
> would be a good thing.
>
> If you had more development resources, you might consider implementing a
> PSHB hub for use by the Senate, or perhaps shared across the entire NY St=
ate
> Legislature. (A number of folk are already working on Drupal and PHP
> extensions to support PSHB.) However, there is a "first step" that you co=
uld
> take in the absence of extra developers and that would be to simply add a
> bit of auto-discovery data to your feed that would allow an existing PSHB
> hub to service your feeds. Please consider taking a look athttp://code.go=
ogle.com/p/pubsubhubbub/wiki/RssFeedsand adding a hub link to
> the <channel/> element in your =A0feed. If you wanted to authorize servic=
e by
> the hub that Google runs on App Engine, you would use:
>
> <atom:link rel=3D"hub" href=3D"http://pubsubhubbub.appspot.com"/>
>
> Adding this link won't actually result in any additional load on your sit=
e
> unless someone actually subscribes to your feed via the hub. From then on=
,
> no matter how many people subscribe, you would see just the load of a sin=
gle
> subscriber.
>
> You might also consider taking a look at SuperFeedr <http://superfeedr.co=
m/>,
> who provide a service of running PSHB hubs for people and thus allowing t=
hem
> the benefit of PSHB but without the overhead of development, keeping up o=
n
> evolving specification changes, etc. see:http://superfeedr.com/documentat=
ion
>
> > As for why RSSv2? It's what is baked into Drupal 6
> > and no one has asked for ATOM feeds.
>
> Yes, many people choose RSS for similar reasons. This gets a bit frustrat=
ing
> since there are quite a large number of us who have put a great deal of w=
ork
> into making Atom a much better specified and more capable format than RSS=
...
> Given that virtually every feed reader handles both RSS and Atom, there
> really isn't any reason to support RSS anymore. (Eventually legacy protoc=
ols
> should be allowed to fade away...) Today, much of the new development in
> syndication formats is focused on Atom, not the "frozen" RSS specificatio=
n
> and thus folk who go RSS-only are, unfortunately, not able to benefit fro=
m
> the work that we do. Frankly, I think folk who bundle RSS-only support as
> the default in things like Drupal are not really serving the best interes=
ts
> of their users.
>
> bob wyman
>
> On Thu, Oct 28, 2010 at 12:34 PM, Noel Hidalgo <n...@noneck.org> wrote:
> > Hey bob! Thanks for your comments.
>
> > We have a firehose feed at NYSenate.gov/rss but it's just a "feed." whi=
le
> > we have discussed pubsubhubub, we have chosen to develop a full feature=
d API
> > that we use internally for our apps and mobile apps.
>
> > As for why RSSv2? It's what is baked into Drupal 6 and no one has asked=
 for
> > ATOM feeds.
>
> > Also, please understand we are a handful of developers who are working =
day
> > and night to make a difference. As I responded to your buzz feed post, =
don't
> > throw the baby out with the bath water. We gladly take feedback and wil=
l
> > work to incorporate it into our development cycle, but understand we ha=
ve
> > limited resources.
>
> > Thanks.
>
> > --
> > Sent from my Apple Newton
>
> > noel hidalgo
> > irc | twitter - @noneck
> > voice | sms - +1.646.867.2263
>
> > On Oct 27, 2010, at 20:57, Graylin Kim <graylin....@gmail.com> wrote:
>
> > Thanks for the detailed answer Bob! I'm really enjoying the links as we=
ll.
>
> > Its clear that you've either thought a good deal about this or done thi=
s
> > kind of thing before. Are you part of the ActivityStreams team?
>
> > As far as votes feeds go, is this<http://open.nysenate.gov/legislation/=
search/?term=3Dotype:vote&format=3Drss>what you were looking for? How about=
 for Bill
> > Passage Events<http://open.nysenate.gov/legislation/search/?term=3Dotyp=
e:action%20AND%...>?
> > I don't think the current production API can support Committee Action
> > filters (Nathan or Jared, do you know?) but the API under development s=
hould
> > be more robust.
>
> > We currently pipe the search box contents (almost) directly into our
> > Lucene <http://lucene.apache.org> powered database which supports a fai=
rly
> > robust query language<http://lucene.apache.org/java/2_3_2/queryparsersy=
ntax.html>.
> > I've got some pretty poor (but workable) documentation<http://github.co=
m/GraylinKim/nyss_openlegislation/wiki/Searching>of the different object ty=
pes and fields currently supported on the wiki for
> > my more or less completed but collecting dust python library I was writ=
ing
> > for it.
>
> > As far as fire hoses and push notifications, I know there has been some
> > talk of such things internally but that may not be possible for us at t=
he
> > current time (for a variety of reasons). Perhaps someone else can provi=
de
> > better perspective there.
>
> > If you are willing, I'd certainly enjoy picking your brain about some o=
f
> > these things in the future. If only for my own enlightenment.
>
> > ~Graylin
>
> > P.S. If you have OpenLegislation related questions you can also try and
> > stop by #nyss_openlegislation on FreeNode.net and find us/me (hopefully=
)
> > there.
>
> > On Wed, Oct 27, 2010 at 1:42 PM, Bob Wyman < <b...@wyman.us>b...@wyman.=
us>wrote:
>
> >> Graylin Kim < <graylin....@gmail.com>graylin....@gmail.com> wrote:
>
> >> > what sorts of feeds/data were you looking for?
>
> >> My primary interest is in "alerting" or publish/subscribe applications=
. In
> >> this particular case, I'm intrigued by the idea of using the recently
> >> announced Google App Engine Matcher API<http://groups.google.com/group=
/google-appengine/browse_thread/thread/...>to build a service that would pr=
ovide real-time prospective search or alerts
> >> for actions, events, etc. that occur in the New York State Senate. The
> >> various statements, press releases and such that seem to make up the b=
ulk of
> >> the feeds that I've been able to find are, of course, interesting. (i.=
e.
> >> tell me whenever some Senator mentions "No. 6 fuel oil..") However, I =
would
> >> really like to be able to get feeds that report events or activities l=
ike
> >> votes, committee actions, bill passage events, etc. (I can't find vote
> >> feeds... Do they exist or have I just missed them?) The App Engine Mat=
cher
> >> API permits one to build pretty much arbitrarily complex boolean expre=
ssions
> >> that are applied to structured data. Thus, if there is a rich source o=
f
> >> data, much can be done with it with minimal effort. (And, Matcher scal=
es
> >> very nicely as well...)
>
> >> Ideally, it would be possible to create an extension to ActivityStream=
s<http://activitystrea.ms/> that
> >> might be called "Legislative ActivityStreams." This format would provi=
de a
> >> means to describe the common activities of a legislature. (i.e. verbs =
like
> >> "voted", "sponsored", "co-sponsored", "reported from committee",
> >> "made-statement", etc... Given that, I could then create an interface =
or
> >> Activity Query Language that would allow interested parties to be noti=
fied
> >> whenever interesting Activities were published.
>
> >> Also, it would be ideal if the feeds were pushed by the New York Senat=
e
> >> instead of requiring everyone to engage in wasteful polling of the fee=
ds.
> >> And, it would be nice if there was a "firehose" feed that aggregated i=
nto a
> >> single stream all of the data being published to the other feeds. For
> >> instance, such a firehose feed might be distributed using the
> >> PubSubHubbub <http://code.google.com/p/pubsubhubbub/> (PSHB) protocol
> >> which has become the common protocol for pushing syndication feeds on =
the
> >> Internet. If the senate was pushing their feeds using PSHB, I could th=
en
> >> build a subscriber that would consume the firehose feed in real-time f=
rom a
> >> PSHB hub and do the necessary subscription matching and alerting with =
little
> >> trouble by using the AppEngine Matcher.
>
> >> Just some ideas... I hope that answers your question.
>
> >> bob wyman
>
> >> On Wed, Oct 27, 2010 at 2:06 AM, Graylin Kim < <graylin....@gmail.com>
> >> graylin....@gmail.com> wrote:
>
> >>> On a separate note of curiosity, what sorts of feeds/data were you
> >>> looking for?
>
> >>> ~Graylin
>
> >>> On Tue, Oct 26, 2010 at 11:07 PM, Nathan Freitas <<nathanfrei...@gmai=
l.com>
> >>> nathanfrei...@gmail.com> wrote:
>
> >>>> We will be providing a more detailed listing of the RSS feeds next w=
eek.
> >>>> It will include both feeds from our Drupal-based primary site and th=
e Open
> >>>> Legislation system. Most of these are also fairly obvious if you bro=
wse the
> >>>> sites yourself.
>
> >>>> As for ATOM, our goal was to support feed reader libraries and tools=
 in
> >>>> the most comprehensive manner, and RSS fit the bill for that.
>
> >>>> +Nathan Freitas
> >>>> NYSenate Developer
> >>>> On Oct 27, 2010 8:14 AM, "Bob Wyman" < <bobwy...@gmail.com>
> >>>> bobwy...@gmail.com> wrote:
> >>>> > The developer network site says that there are "hundreds" of feeds
> >>>> > available, but I can't seem to find a listing of those feeds anywh=
ere.
> >>>> > Does a directory to the feeds exist?
>
> >>>> > Also, the site says
>
> ...
>
Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
Account Options
