Directory of Feeds?

[Bob Wyman]

The developer network site says that there are "hundreds" of feeds
available, but I can't seem to find a listing of those feeds anywhere.
Does a directory to the feeds exist?

Also, the site says that the feeds all use the legacy RSS V2.0 format.
Is there some reason that you don't use the more capable and modern
Atom format?

bob wyman

[Nathan Freitas]

We will be providing a more detailed listing of the RSS feeds next week. It will include both feeds from our Drupal-based primary site and the Open Legislation system. Most of these are also fairly obvious if you browse the sites yourself.

As for ATOM, our goal was to support feed reader libraries and tools in the most comprehensive manner, and RSS fit the bill for that.

+Nathan Freitas
NYSenate Developer

[Graylin Kim]

On a separate note of curiosity, what sorts of feeds/data were you looking for?

~Graylin

[Bob Wyman]

Graylin Kim <grayl...@gmail.com> wrote:

> what sorts of feeds/data were you looking for?

My primary interest is in "alerting" or publish/subscribe applications. In this particular case, I'm intrigued by the idea of using the recently announced Google App Engine Matcher API to build a service that would provide real-time prospective search or alerts for actions, events, etc. that occur in the New York State Senate. The various statements, press releases and such that seem to make up the bulk of the feeds that I've been able to find are, of course, interesting. (i.e. tell me whenever some Senator mentions "No. 6 fuel oil..") However, I would really like to be able to get feeds that report events or activities like votes, committee actions, bill passage events, etc. (I can't find vote feeds... Do they exist or have I just missed them?) The App Engine Matcher API permits one to build pretty much arbitrarily complex boolean expressions that are applied to structured data. Thus, if there is a rich source of data, much can be done with it with minimal effort. (And, Matcher scales very nicely as well...)

Ideally, it would be possible to create an extension to ActivityStreams that might be called "Legislative ActivityStreams." This format would provide a means to describe the common activities of a legislature. (i.e. verbs like "voted", "sponsored", "co-sponsored", "reported from committee", "made-statement", etc... Given that, I could then create an interface or Activity Query Language that would allow interested parties to be notified whenever interesting Activities were published.

Also, it would be ideal if the feeds were pushed by the New York Senate instead of requiring everyone to engage in wasteful polling of the feeds. And, it would be nice if there was a "firehose" feed that aggregated into a single stream all of the data being published to the other feeds. For instance, such a firehose feed might be distributed using the PubSubHubbub (PSHB) protocol which has become the common protocol for pushing syndication feeds on the Internet. If the senate was pushing their feeds using PSHB, I could then build a subscriber that would consume the firehose feed in real-time from a PSHB hub and do the necessary subscription matching and alerting with little trouble by using the AppEngine Matcher.

Just some ideas... I hope that answers your question.

bob wyman

[Graylin Kim]

Thanks for the detailed answer Bob! I'm really enjoying the links as well.

Its clear that you've either thought a good deal about this or done this kind of thing before. Are you part of the ActivityStreams team?

As far as votes feeds go, is this what you were looking for? How about for Bill Passage Events? I don't think the current production API can support Committee Action filters (Nathan or Jared, do you know?) but the API under development should be more robust.

We currently pipe the search box contents (almost) directly into our Lucene powered database which supports a fairly robust query language. I've got some pretty poor (but workable) documentation of the different object types and fields currently supported on the wiki for my more or less completed but collecting dust python library I was writing for it.

As far as fire hoses and push notifications, I know there has been some talk of such things internally but that may not be possible for us at the current time (for a variety of reasons). Perhaps someone else can provide better perspective there.

If you are willing, I'd certainly enjoy picking your brain about some of these things in the future. If only for my own enlightenment.

~Graylin

P.S. If you have OpenLegislation related questions you can also try and stop by #nyss_openlegislation on FreeNode.net and find us/me (hopefully) there.

--
--
You received this message because you are subscribed to the NY Senate Developers Forum group. For more information about this group, visit http://nysenate.gov/developers

[Noel Hidalgo]

Hey bob! Thanks for your comments. 

We have a firehose feed at NYSenate.gov/rss but it's just a "feed." while we have discussed pubsubhubub, we have chosen to develop a full featured API that we use internally for our apps and mobile apps.

As for why RSSv2? It's what is baked into Drupal 6 and no one has asked for ATOM feeds. 

Also, please understand we are a handful of developers who are working day and night to make a difference. As I responded to your buzz feed post, don't throw the baby out with the bath water. We gladly take feedback and will work to incorporate it into our development cycle, but understand we have limited resources.

Thanks. 

--

Sent from my Apple Newton

noel hidalgo

irc | twitter - @noneck

voice | sms - +1.646.867.2263

[Bob Wyman]

Noel Hidalgo <no...@noneck.org> wrote:

> please understand we are a handful of

> developers who are working day and night

Noel, I would expect nothing else. Somehow, it always seems that the most interesting projects are often the ones with the tightest resources! 

> We have a firehose feed at NYSenate.gov/rss but it's just a "feed."

That's not surprising since most folk only provide the old-style polled feeds these days and thus suffer propagation latency and the constant load of many people polling for updates. Hopefully, however, we're moving to an environment which will be more push based -- using technologies like PubSubHubbub (PSHB). This should allow people to get feed updates more rapidly while reducing the load on publisher's sites. While it can probably be argued that the load issues aren't going to be that significant for a site such as yours since the community of those who recognize the value of your data is much smaller than it should be, I think it is very much the case that much of what you publish is data that has a good bit of "time value" to those who are interested in it and thus propagating timely updates would be a good thing.

If you had more development resources, you might consider implementing a PSHB hub for use by the Senate, or perhaps shared across the entire NY State Legislature. (A number of folk are already working on Drupal and PHP extensions to support PSHB.) However, there is a "first step" that you could take in the absence of extra developers and that would be to simply add a bit of auto-discovery data to your feed that would allow an existing PSHB hub to service your feeds. Please consider taking a look at http://code.google.com/p/pubsubhubbub/wiki/RssFeeds and adding a hub link to the <channel/> element in your  feed. If you wanted to authorize service by the hub that Google runs on App Engine, you would use:

<atom:link rel="hub" href="http://pubsubhubbub.appspot.com"/>

Adding this link won't actually result in any additional load on your site unless someone actually subscribes to your feed via the hub. From then on, no matter how many people subscribe, you would see just the load of a single subscriber.

You might also consider taking a look at SuperFeedr, who provide a service of running PSHB hubs for people and thus allowing them the benefit of PSHB but without the overhead of development, keeping up on evolving specification changes, etc. see: http://superfeedr.com/documentation

> As for why RSSv2? It's what is baked into Drupal 6

> and no one has asked for ATOM feeds. 

Yes, many people choose RSS for similar reasons. This gets a bit frustrating since there are quite a large number of us who have put a great deal of work into making Atom a much better specified and more capable format than RSS... Given that virtually every feed reader handles both RSS and Atom, there really isn't any reason to support RSS anymore. (Eventually legacy protocols should be allowed to fade away...) Today, much of the new development in syndication formats is focused on Atom, not the "frozen" RSS specification and thus folk who go RSS-only are, unfortunately, not able to benefit from the work that we do. Frankly, I think folk who bundle RSS-only support as the default in things like Drupal are not really serving the best interests of their users. 

bob wyman

[Bob Wyman]

Graylin Kim <grayl...@gmail.com> wrote:

> Are you part of the ActivityStreams team?

Well, there isn't really a "team"... ActivityStreams is an open protocol that is developed in the IETF style through discussions on a mailing list. I am a member of the list and do contribute from time to time. I also participate in the PubSubHubbub discussions, but my every-day work is focused on the technology behind the Matcher API...

> As far as votes feeds go, is this what you were looking for?

It looks like with a good bit of parsing and link following, I could use that to figure out what the votes were. Or, I could use discovery of a vote in that feed to trigger a lookup using the API.... It would, however, be very nice if the data in the feed was made available with a bit more semantic markup in order to simplify the parsing, make it a bit more resistant to modifications that might be made in the presentation format in the future, and avoid the need to hit your API for details. Something like the HTML Microdata format which is being baked into HTML5 might work very nicely here. 

bob wyman

[Graylin Kim]

So going back to your original statements:

"Thus, if there is a rich source of data, much can be done with it with minimal effort."

If a, preferably ATOM, feed were set up with the data being

"made available with a bit more semantic markup in order to simplify the parsing"

and containing a more complete picture of the event so as to

"avoid the need to hit your API for details"

That would suit your needs (at least for this use case)? 

Do you think future developers looking to work with Senate data would (commonly) adapt a similar data flow? In past work I think the focus has been more on the Web API and less on the feeds on the thought (my thought at least) that it would be the primary developer point of contact. Indeed that's how the Senate's mobile apps and internal application currently retrieve information.

Your thoughts on how to best deliver the information to developers would be very much appreciated.

~Graylin

[Vivek Venugopalan]

All - I have been lurking on this thread and thought I will pitch in
with my 2c. You can also look at services such as feedburner (http://
www.feedburner.com) that can provide feeds in multiple formats and
also save you the hits as they take the traffic load and poll you
periodically for changes.

On Oct 28, 10:08 pm, Bob Wyman <b...@wyman.us> wrote:

> Noel Hidalgo <n...@noneck.org> wrote:
> > please understand we are a handful of
> > developers who are working day and night
>
> Noel, I would expect nothing else. Somehow, it always seems that the most
> interesting projects are often the ones with the tightest resources!
>

> > We have a firehose feed at NYSenate.gov/rss <http://nysenate.gov/rss> but

>
> it's just a "feed."
> That's not surprising since most folk only provide the old-style polled
> feeds these days and thus suffer propagation latency and the constant load
> of many people polling for updates. Hopefully, however, we're moving to an
> environment which will be more push based -- using technologies like
> PubSubHubbub (PSHB). This should allow people to get feed updates more
> rapidly while reducing the load on publisher's sites. While it can probably
> be argued that the load issues aren't going to be that significant for a
> site such as yours since the community of those who recognize the value of
> your data is much smaller than it should be, I think it is very much the
> case that much of what you publish is data that has a good bit of "time
> value" to those who are interested in it and thus propagating timely updates
> would be a good thing.
>
> If you had more development resources, you might consider implementing a
> PSHB hub for use by the Senate, or perhaps shared across the entire NY State
> Legislature. (A number of folk are already working on Drupal and PHP
> extensions to support PSHB.) However, there is a "first step" that you could
> take in the absence of extra developers and that would be to simply add a
> bit of auto-discovery data to your feed that would allow an existing PSHB

> hub to service your feeds. Please consider taking a look athttp://code.google.com/p/pubsubhubbub/wiki/RssFeedsand adding a hub link to

> the <channel/> element in your  feed. If you wanted to authorize service by
> the hub that Google runs on App Engine, you would use:
>
> <atom:link rel="hub" href="http://pubsubhubbub.appspot.com"/>
>
> Adding this link won't actually result in any additional load on your site
> unless someone actually subscribes to your feed via the hub. From then on,
> no matter how many people subscribe, you would see just the load of a single
> subscriber.
>

> You might also consider taking a look at SuperFeedr <http://superfeedr.com/>,

> who provide a service of running PSHB hubs for people and thus allowing them
> the benefit of PSHB but without the overhead of development, keeping up on
> evolving specification changes, etc. see:http://superfeedr.com/documentation
>
> > As for why RSSv2? It's what is baked into Drupal 6
> > and no one has asked for ATOM feeds.
>
> Yes, many people choose RSS for similar reasons. This gets a bit frustrating
> since there are quite a large number of us who have put a great deal of work
> into making Atom a much better specified and more capable format than RSS...
> Given that virtually every feed reader handles both RSS and Atom, there
> really isn't any reason to support RSS anymore. (Eventually legacy protocols
> should be allowed to fade away...) Today, much of the new development in
> syndication formats is focused on Atom, not the "frozen" RSS specification
> and thus folk who go RSS-only are, unfortunately, not able to benefit from
> the work that we do. Frankly, I think folk who bundle RSS-only support as
> the default in things like Drupal are not really serving the best interests
> of their users.
>
> bob wyman
>

> On Thu, Oct 28, 2010 at 12:34 PM, Noel Hidalgo <n...@noneck.org> wrote:
> > Hey bob! Thanks for your comments.
>
> > We have a firehose feed at NYSenate.gov/rss but it's just a "feed." while
> > we have discussed pubsubhubub, we have chosen to develop a full featured API
> > that we use internally for our apps and mobile apps.
>
> > As for why RSSv2? It's what is baked into Drupal 6 and no one has asked for
> > ATOM feeds.
>
> > Also, please understand we are a handful of developers who are working day
> > and night to make a difference. As I responded to your buzz feed post, don't
> > throw the baby out with the bath water. We gladly take feedback and will
> > work to incorporate it into our development cycle, but understand we have
> > limited resources.
>
> > Thanks.
>
> > --
> > Sent from my Apple Newton
>
> > noel hidalgo
> > irc | twitter - @noneck
> > voice | sms - +1.646.867.2263
>

> > On Oct 27, 2010, at 20:57, Graylin Kim <graylin....@gmail.com> wrote:
>
> > Thanks for the detailed answer Bob! I'm really enjoying the links as well.
>
> > Its clear that you've either thought a good deal about this or done this
> > kind of thing before. Are you part of the ActivityStreams team?
>

> > As far as votes feeds go, is this<http://open.nysenate.gov/legislation/search/?term=otype:vote&format=rss>what you were looking for? How about for Bill
> > Passage Events<http://open.nysenate.gov/legislation/search/?term=otype:action%20AND%...>?

> > I don't think the current production API can support Committee Action
> > filters (Nathan or Jared, do you know?) but the API under development should
> > be more robust.
>
> > We currently pipe the search box contents (almost) directly into our

> > Lucene <http://lucene.apache.org> powered database which supports a fairly
> > robust query language<http://lucene.apache.org/java/2_3_2/queryparsersyntax.html>.
> > I've got some pretty poor (but workable) documentation<http://github.com/GraylinKim/nyss_openlegislation/wiki/Searching>of the different object types and fields currently supported on the wiki for

> > my more or less completed but collecting dust python library I was writing
> > for it.
>
> > As far as fire hoses and push notifications, I know there has been some
> > talk of such things internally but that may not be possible for us at the
> > current time (for a variety of reasons). Perhaps someone else can provide
> > better perspective there.
>
> > If you are willing, I'd certainly enjoy picking your brain about some of
> > these things in the future. If only for my own enlightenment.
>
> > ~Graylin
>
> > P.S. If you have OpenLegislation related questions you can also try and
> > stop by #nyss_openlegislation on FreeNode.net and find us/me (hopefully)
> > there.
>

> > On Wed, Oct 27, 2010 at 1:42 PM, Bob Wyman < <b...@wyman.us>b...@wyman.us>wrote:

>
> >> Graylin Kim < <graylin....@gmail.com>graylin....@gmail.com> wrote:
>
> >> > what sorts of feeds/data were you looking for?
>
> >> My primary interest is in "alerting" or publish/subscribe applications. In
> >> this particular case, I'm intrigued by the idea of using the recently

> >> announced Google App Engine Matcher API<http://groups.google.com/group/google-appengine/browse_thread/thread/...>to build a service that would provide real-time prospective search or alerts

> >> for actions, events, etc. that occur in the New York State Senate. The
> >> various statements, press releases and such that seem to make up the bulk of
> >> the feeds that I've been able to find are, of course, interesting. (i.e.
> >> tell me whenever some Senator mentions "No. 6 fuel oil..") However, I would
> >> really like to be able to get feeds that report events or activities like
> >> votes, committee actions, bill passage events, etc. (I can't find vote
> >> feeds... Do they exist or have I just missed them?) The App Engine Matcher
> >> API permits one to build pretty much arbitrarily complex boolean expressions
> >> that are applied to structured data. Thus, if there is a rich source of
> >> data, much can be done with it with minimal effort. (And, Matcher scales
> >> very nicely as well...)
>

> >> Ideally, it would be possible to create an extension to ActivityStreams<http://activitystrea.ms/> that

> >> might be called "Legislative ActivityStreams." This format would provide a
> >> means to describe the common activities of a legislature. (i.e. verbs like
> >> "voted", "sponsored", "co-sponsored", "reported from committee",
> >> "made-statement", etc... Given that, I could then create an interface or
> >> Activity Query Language that would allow interested parties to be notified
> >> whenever interesting Activities were published.
>
> >> Also, it would be ideal if the feeds were pushed by the New York Senate
> >> instead of requiring everyone to engage in wasteful polling of the feeds.
> >> And, it would be nice if there was a "firehose" feed that aggregated into a
> >> single stream all of the data being published to the other feeds. For
> >> instance, such a firehose feed might be distributed using the

> >> PubSubHubbub <http://code.google.com/p/pubsubhubbub/> (PSHB) protocol

> >> which has become the common protocol for pushing syndication feeds on the
> >> Internet. If the senate was pushing their feeds using PSHB, I could then
> >> build a subscriber that would consume the firehose feed in real-time from a
> >> PSHB hub and do the necessary subscription matching and alerting with little
> >> trouble by using the AppEngine Matcher.
>
> >> Just some ideas... I hope that answers your question.
>
> >> bob wyman
>

> >> On Wed, Oct 27, 2010 at 2:06 AM, Graylin Kim < <graylin....@gmail.com>
> >> graylin....@gmail.com> wrote:
>
> >>> On a separate note of curiosity, what sorts of feeds/data were you
> >>> looking for?
>
> >>> ~Graylin
>

> >>> On Tue, Oct 26, 2010 at 11:07 PM, Nathan Freitas <<nathanfrei...@gmail.com>
> >>> nathanfrei...@gmail.com> wrote:
>
> >>>> We will be providing a more detailed listing of the RSS feeds next week.
> >>>> It will include both feeds from our Drupal-based primary site and the Open
> >>>> Legislation system. Most of these are also fairly obvious if you browse the
> >>>> sites yourself.
>
> >>>> As for ATOM, our goal was to support feed reader libraries and tools in
> >>>> the most comprehensive manner, and RSS fit the bill for that.
>
> >>>> +Nathan Freitas
> >>>> NYSenate Developer

> >>>> On Oct 27, 2010 8:14 AM, "Bob Wyman" < <bobwy...@gmail.com>

> >>>> bobwy...@gmail.com> wrote:
> >>>> > The developer network site says that there are "hundreds" of feeds
> >>>> > available, but I can't seem to find a listing of those feeds anywhere.
> >>>> > Does a directory to the feeds exist?
>
> >>>> > Also, the site says
>

> ...
>
> read more »

[Nathan Freitas]

Hey Bob,

Just jumping back into this thread after being on the road for a bit. I
am the project lead on Open Legislation at the Senate, and really
appreciate your response and feedback on this.

Our RSS view was developed with a very user-centric model, specifically
driven by members of Senate staff, journalists, and watchdog groups who
wanted any easy way to subscribe to searches using Google Reader or
other end-user aggregators. This is why there is very little structured
data in the content itself, as it was mostly intended as a brief summary
of content that the user would click-through on into the full item on
the Open Legislation site.

However, we are also very eager to support the alert, pub/sub model for
application developers, and to support standards and services which make
it easy to do so. We have internal drivers for this (adding
push/sms/email notification services for our mobile apps), and are glad
to see external devs like yourself that also interested in using the
data in this matter. As the legislative activity ramps up in January, we
truly do want to open a realtime firehose to developers, and see what
devs can do to make it useful and interesting to their target users.

After our scrum this morning, I think we have decided that we can't fit
too many changes into our next release (1.6 coming out in November), but
that we should be able to provide an enhanced RSS mode or Atom support
in a 1.6.1 update, and also work on integrating with some PubSubHubBub
libraries/services in the meantime, as you proposed.

We'll keep you posted, and please keep the feedback coming.

+Nathan

On 10/28/2010 01:56 PM, Bob Wyman wrote:
> Graylin Kim <grayl...@gmail.com> wrote:
>> Are you part of the ActivityStreams team?
> Well, there isn't really a "team"... ActivityStreams is an open protocol
> that is developed in the IETF style through discussions on a mailing list. I
> am a member of the list and do contribute from time to time. I also
> participate in the PubSubHubbub discussions, but my every-day work is
> focused on the technology behind the Matcher API...
>

>> As far as votes feeds go, is this<http://open.nysenate.gov/legislation/search/?term=otype:vote&format=rss> what

> you were looking for?
> It looks like with a good bit of parsing and link following, I could use
> that to figure out what the votes were. Or, I could use discovery of a vote
> in that feed to trigger a lookup using the API.... It would, however, be
> very nice if the data in the feed was made available with a bit more
> semantic markup in order to simplify the parsing, make it a bit more
> resistant to modifications that might be made in the presentation format in
> the future, and avoid the need to hit your API for details. Something like

> the HTML Microdata <http://www.w3.org/TR/microdata/> format which is being

> baked into HTML5 might work very nicely here.
>
> bob wyman
>
> On Wed, Oct 27, 2010 at 11:57 PM, Graylin Kim <grayl...@gmail.com> wrote:
>
>> Thanks for the detailed answer Bob! I'm really enjoying the links as well.
>>
>> Its clear that you've either thought a good deal about this or done this
>> kind of thing before. Are you part of the ActivityStreams team?
>>

>> As far as votes feeds go, is this<http://open.nysenate.gov/legislation/search/?term=otype:vote&format=rss>what you were looking for? How about for Bill
>> Passage Events<http://open.nysenate.gov/legislation/search/?term=otype:action%20AND%20passed&format=rss>?

>> I don't think the current production API can support Committee Action
>> filters (Nathan or Jared, do you know?) but the API under development should
>> be more robust.
>>
>> We currently pipe the search box contents (almost) directly into our

>> Lucene <http://lucene.apache.org> powered database which supports a fairly
>> robust query language<http://lucene.apache.org/java/2_3_2/queryparsersyntax.html>.
>> I've got some pretty poor (but workable) documentation<http://github.com/GraylinKim/nyss_openlegislation/wiki/Searching>of the different object types and fields currently supported on the wiki for

>> my more or less completed but collecting dust python library I was writing
>> for it.
>>
>> As far as fire hoses and push notifications, I know there has been some
>> talk of such things internally but that may not be possible for us at the
>> current time (for a variety of reasons). Perhaps someone else can provide
>> better perspective there.
>>
>> If you are willing, I'd certainly enjoy picking your brain about some of
>> these things in the future. If only for my own enlightenment.
>>
>> ~Graylin
>>
>> P.S. If you have OpenLegislation related questions you can also try and
>> stop by #nyss_openlegislation on FreeNode.net and find us/me (hopefully)
>> there.
>>
>> On Wed, Oct 27, 2010 at 1:42 PM, Bob Wyman <b...@wyman.us> wrote:
>>
>>> Graylin Kim <grayl...@gmail.com> wrote:
>>>
>>>> what sorts of feeds/data were you looking for?
>>> My primary interest is in "alerting" or publish/subscribe applications. In
>>> this particular case, I'm intrigued by the idea of using the recently

>>> announced Google App Engine Matcher API<http://groups.google.com/group/google-appengine/browse_thread/thread/5462e14c31f44bef/>to build a service that would provide real-time prospective search or alerts

>>> for actions, events, etc. that occur in the New York State Senate. The
>>> various statements, press releases and such that seem to make up the bulk of
>>> the feeds that I've been able to find are, of course, interesting. (i.e.
>>> tell me whenever some Senator mentions "No. 6 fuel oil..") However, I would
>>> really like to be able to get feeds that report events or activities like
>>> votes, committee actions, bill passage events, etc. (I can't find vote
>>> feeds... Do they exist or have I just missed them?) The App Engine Matcher
>>> API permits one to build pretty much arbitrarily complex boolean expressions
>>> that are applied to structured data. Thus, if there is a rich source of
>>> data, much can be done with it with minimal effort. (And, Matcher scales
>>> very nicely as well...)
>>>

>>> Ideally, it would be possible to create an extension to ActivityStreams<http://activitystrea.ms/> that

>>> might be called "Legislative ActivityStreams." This format would provide a
>>> means to describe the common activities of a legislature. (i.e. verbs like
>>> "voted", "sponsored", "co-sponsored", "reported from committee",
>>> "made-statement", etc... Given that, I could then create an interface or
>>> Activity Query Language that would allow interested parties to be notified
>>> whenever interesting Activities were published.
>>>
>>> Also, it would be ideal if the feeds were pushed by the New York Senate
>>> instead of requiring everyone to engage in wasteful polling of the feeds.
>>> And, it would be nice if there was a "firehose" feed that aggregated into a
>>> single stream all of the data being published to the other feeds. For
>>> instance, such a firehose feed might be distributed using the

>>> PubSubHubbub <http://code.google.com/p/pubsubhubbub/> (PSHB) protocol