reginfo.gov API?

Shawn Johnson

unread,

Dec 12, 2014, 10:41:35 AM12/12/14

to us-govern...@googlegroups.com

Does anyone know someone at reginfo.gov that we can pitch about making an API? We'd really benefit from that.

Thanks!

Gray Brooks

unread,

Dec 12, 2014, 11:01:51 AM12/12/14

to Shawn Johnson, <us-government-apis@googlegroups.com>, John Thomas - M1V1E, Connie Jordan - M1V1E, Jeff Kwiecinski - BBI, Jeremy Miller - MVE-C, Julio Baez - MVE-C

Hey Shawn,

I'm looping in some of that reginfo.gov team, who will better know how things look. Can you share some of the use cases for what you'd want to do?

Gray

------------------------------

c - 205.541.2245

Sr. API Strategist

Team 18F - GSA

* /Developer Program

* US Government APIs listserve

On Fri, Dec 12, 2014 at 10:41 AM, Shawn Johnson <shawnjo...@gmail.com> wrote:

Does anyone know someone at reginfo.gov that we can pitch about making an API? We'd really benefit from that.
Thanks!

--
You received this message because you are subscribed to the Google Groups "US Government APIs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to us-government-a...@googlegroups.com.
To post to this group, send email to us-govern...@googlegroups.com.
Visit this group at http://groups.google.com/group/us-government-apis.

Alan deLevie

unread,

Dec 12, 2014, 11:15:29 AM12/12/14

to gray....@gsa.gov, Shawn Johnson, <us-government-apis@googlegroups.com>, John Thomas - M1V1E, Connie Jordan - M1V1E, Jeff Kwiecinski - BBI, Jeremy Miller - MVE-C, Julio Baez - MVE-C

Hi Shawn,

(writing from my personal email, but I'm a dev at 18F)

I've actually written some scrapers for reginfo.gov data a few months back. It's a little difficult at the moment to make them useful to others. But they worked decently well (written in Ruby, using Nokogiri to parse the XML). If you want, I could try to piece together my old code.

Nevertheless, an API for this data (e.g. http://www.reginfo.gov/public/do/PRAXML) would be useful.

Best,

Alan deLevie

Shawn Johnson

unread,

Dec 12, 2014, 1:00:21 PM12/12/14

to us-govern...@googlegroups.com, shawnjo...@gmail.com, john....@gsa.gov, connie...@gsa.gov, jeffrey.k...@gsa.gov, jeremy...@gsa.gov, julio...@gsa.gov, gray....@gsa.gov

We are primarily looking to pull Unified Agenda data. So, I would want to be able to GET a list of publication periods. GET a list of Agencies within a given publication period. List of RINs available for a given agency+publication period. Then ultimately RIN data found on a page like this: http://www.reginfo.gov/public/do/eAgendaViewRule?pubId=201410&RIN=0503-AA55 And, yes, I know there is XML there already :-)

-=Shawn

Alan deLevie

unread,

Dec 12, 2014, 4:23:41 PM12/12/14

to Shawn Johnson, us-govern...@googlegroups.com, John Thomas - M1V1E, Connie Jordan - M1V1E, Jeff Kwiecinski - BBI, Jeremy Miller - MVE-C, Julio Baez - MVE-C, gray....@gsa.gov

Hi Shawn,

Not sure if this fits your use case, but fellow 18F dev (and former Presidential Innovation Fellow), Justin Grevich wrote a Ruby parser for some RegInfo.gov data: https://github.com/GSA/just_inform

Usage:

parser = JustInform.new

# Return an array with the sort value (cost) and InformationCollectionRequest objects

parser.top(10, :cost) =>

[["33,672,000,000", U.S. Individual Income Tax Return], ["8,190,000,004", Rule 10b-10 Confirmation of Securities Transactions (17 C.F.R. 240.10b-10)], ["2,857,465,000", Exchange Act Form 10-K], ["2,727,479,226", America Invents Act Section 10 Patent Fee Adjustments], ["1,801,830,000", Application for a U.S. Passport], ["1,300,147,200", ASSIST Database], ["1,280,341,567", Form SD], ["910,000,000", Online Application for Nonimmigrant Visa], ["791,160,764", Rules for Patent Maintenance Fees], ["772,798,833", Initial Patent Applications]]

Shawn Johnson

unread,

Dec 18, 2014, 1:06:26 PM12/18/14

to us-govern...@googlegroups.com, shawnjo...@gmail.com, john....@gsa.gov, connie...@gsa.gov, jeffrey.k...@gsa.gov, jeremy...@gsa.gov, julio...@gsa.gov, gray....@gsa.gov

Thanks Adam, I think that is a different data set than we are looking for. Nice tool, thanks for sharing!

To unsubscribe from this group and stop receiving emails from it, send an email to us-government-apis+unsub...@googlegroups.com.

To post to this group, send email to us-govern...@googlegroups.com.
Visit this group at http://groups.google.com/group/us-government-apis.

Shawn Johnson

unread,

Dec 18, 2014, 1:09:43 PM12/18/14

to us-govern...@googlegroups.com, shawnjo...@gmail.com, john....@gsa.gov, connie...@gsa.gov, jeffrey.k...@gsa.gov, jeremy...@gsa.gov, julio...@gsa.gov, gray....@gsa.gov

Oops, sorry Alan deLevie, miskeyed there.

On Thursday, December 18, 2014 1:06:26 PM UTC-5, Shawn Johnson wrote:

Thanks Adam, I think that is a different data set than we are looking for. Nice tool, thanks for sharing!

Shawn Johnson

unread,

Nov 13, 2015, 10:42:59 AM11/13/15

to US Government APIs, shawnjo...@gmail.com, john....@gsa.gov, connie...@gsa.gov, jeffrey.k...@gsa.gov, jeremy...@gsa.gov, julio...@gsa.gov, gray....@gsa.gov

Bump... any thoughts on Unified Agenda data as an API - would there be other folks out there who would use it if it were available as an API?

On Friday, December 12, 2014 at 1:00:21 PM UTC-5, Shawn Johnson wrote:

We are primarily looking to pull Unified Agenda data. So, I would want to be able to GET a list of publication periods. GET a list of Agencies within a given publication period. List of RINs available for a given agency+publication period. Then ultimately RIN data found on a page like this: http://www.reginfo.gov/public/do/eAgendaViewRule?pubId=201410&RIN=0503-AA55 And, yes, I know there is XML there already :-)

-=Shawn

--
You received this message because you are subscribed to the Google Groups "US Government APIs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to us-government-apis+unsub...@googlegroups.com.
To post to this group, send email to us-government-apis@googlegroups.com.

Mark Silverberg

unread,

Nov 13, 2015, 11:04:23 AM11/13/15

to Shawn Johnson, US Government APIs, john....@gsa.gov, connie...@gsa.gov, jeffrey.k...@gsa.gov, jeremy...@gsa.gov, julio...@gsa.gov, gray....@gsa.gov

Is there an API or list of XML docs? I wonder if a decoupled API like on AWS' API Gateway or with Lambda could be created where it parsed the XML and put it into a more modern format

Mark

On Fri, Nov 13, 2015 at 10:42 AM, Shawn Johnson <shawnjo...@gmail.com> wrote:

Bump... any thoughts on Unified Agenda data as an API - would there be other folks out there who would use it if it were available as an API?

On Friday, December 12, 2014 at 1:00:21 PM UTC-5, Shawn Johnson wrote:

We are primarily looking to pull Unified Agenda data. So, I would want to be able to GET a list of publication periods. GET a list of Agencies within a given publication period. List of RINs available for a given agency+publication period. Then ultimately RIN data found on a page like this: http://www.reginfo.gov/public/do/eAgendaViewRule?pubId=201410&RIN=0503-AA55 And, yes, I know there is XML there already :-)

-=Shawn

--
You received this message because you are subscribed to the Google Groups "US Government APIs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to us-government-a...@googlegroups.com.
To post to this group, send email to us-govern...@googlegroups.com.

Visit this group at http://groups.google.com/group/us-government-apis.

--

You received this message because you are subscribed to the Google Groups "US Government APIs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to us-government-a...@googlegroups.com.
To post to this group, send email to us-govern...@googlegroups.com.

Visit this group at http://groups.google.com/group/us-government-apis.

--

Mark Silverberg, Socrata Inc.

Health Data Solutions Architect

512-826-7004 | @Skram

Shawn Johnson

unread,

Nov 13, 2015, 1:44:29 PM11/13/15

to US Government APIs, shawnjo...@gmail.com, john....@gsa.gov, connie...@gsa.gov, jeffrey.k...@gsa.gov, jeremy...@gsa.gov, julio...@gsa.gov, gray....@gsa.gov

I just went back to the site and noticed a link I'd not seen before "XML Reports" - links to an XML file for each edition This is actually pretty nice to see. Sounds like a fun project to wrap those up with a lightweight Lambda-type of solution to do something like allow query based on year and spring/fall edition maybe? Ideally the larger XML could be broken down into individual actions, and wrapped with something to allow query by Agency, RIN, anything else?

Also - the site has individual XML files for each Rule as well. It follows a pretty standard URL... so maybe an API is practically there?

Example:

http://www.reginfo.gov/public/do/eAgendaViewRule?pubId=201504&RIN=1801-AA11&operation=OPERATION_EXPORT_XML

For my specific use-case, I'd like to be able to make a request only by RIN, and get the 'most recent' edition for the given RIN.

Anyone else using this data for any applications?

On Friday, November 13, 2015 at 11:04:23 AM UTC-5, Mark Silverberg wrote:

Is there an API or list of XML docs? I wonder if a decoupled API like on AWS' API Gateway or with Lambda could be created where it parsed the XML and put it into a more modern format

Mark

On Fri, Nov 13, 2015 at 10:42 AM, Shawn Johnson <shawnjo...@gmail.com> wrote:

Bump... any thoughts on Unified Agenda data as an API - would there be other folks out there who would use it if it were available as an API?

On Friday, December 12, 2014 at 1:00:21 PM UTC-5, Shawn Johnson wrote:

We are primarily looking to pull Unified Agenda data. So, I would want to be able to GET a list of publication periods. GET a list of Agencies within a given publication period. List of RINs available for a given agency+publication period. Then ultimately RIN data found on a page like this: http://www.reginfo.gov/public/do/eAgendaViewRule?pubId=201410&RIN=0503-AA55 And, yes, I know there is XML there already :-)

-=Shawn

--
You received this message because you are subscribed to the Google Groups "US Government APIs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to us-government-apis+unsub...@googlegroups.com.

To post to this group, send email to us-govern...@googlegroups.com.
Visit this group at http://groups.google.com/group/us-government-apis.

--
You received this message because you are subscribed to the Google Groups "US Government APIs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to us-government-apis+unsub...@googlegroups.com.

To post to this group, send email to us-govern...@googlegroups.com.
Visit this group at http://groups.google.com/group/us-government-apis.

Shawn Johnson

unread,

Feb 4, 2016, 9:23:40 AM2/4/16

to US Government APIs, shawnjo...@gmail.com, john....@gsa.gov, connie...@gsa.gov, jeffrey.k...@gsa.gov, jeremy...@gsa.gov, julio...@gsa.gov, gray....@gsa.gov

I'm revisiting the idea of using a service like AWS Lambda for transforming the XML. A couple of thoughts.

- The XML contains often contains CDATA with embedded HTML in the 'abstract' property - any good examples of representing something like this in JSON?

- The missing piece for us is the lookup of the most recent publication period for a given RIN. So far I don't know of a comprehensive list of publication periods + RIN - so that could involve trying to pass many XML files containing all of the metadata for all of the years - feels like that's starting to get pretty heavy for a lambda, as the list would only change twice a year - seems to make more sense to process the data, then store the result only twice a year. I guess I'm still looking to skip storing the data.

On Friday, November 13, 2015 at 11:04:23 AM UTC-5, Mark Silverberg wrote:

Is there an API or list of XML docs? I wonder if a decoupled API like on AWS' API Gateway or with Lambda could be created where it parsed the XML and put it into a more modern format

Mark

On Fri, Nov 13, 2015 at 10:42 AM, Shawn Johnson <shawnjo...@gmail.com> wrote:

Bump... any thoughts on Unified Agenda data as an API - would there be other folks out there who would use it if it were available as an API?

On Friday, December 12, 2014 at 1:00:21 PM UTC-5, Shawn Johnson wrote:

We are primarily looking to pull Unified Agenda data. So, I would want to be able to GET a list of publication periods. GET a list of Agencies within a given publication period. List of RINs available for a given agency+publication period. Then ultimately RIN data found on a page like this: http://www.reginfo.gov/public/do/eAgendaViewRule?pubId=201410&RIN=0503-AA55 And, yes, I know there is XML there already :-)

-=Shawn

--
You received this message because you are subscribed to the Google Groups "US Government APIs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to us-government-apis+unsub...@googlegroups.com.

To post to this group, send email to us-govern...@googlegroups.com.
Visit this group at http://groups.google.com/group/us-government-apis.

--
You received this message because you are subscribed to the Google Groups "US Government APIs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to us-government-apis+unsub...@googlegroups.com.

To post to this group, send email to us-govern...@googlegroups.com.
Visit this group at http://groups.google.com/group/us-government-apis.

Shawn Johnson

unread,

Feb 4, 2016, 9:30:03 AM2/4/16

to US Government APIs, shawnjo...@gmail.com, john....@gsa.gov, connie...@gsa.gov, jeffrey.k...@gsa.gov, jeremy...@gsa.gov, julio...@gsa.gov, gray....@gsa.gov

Another item here - http://www.reginfo.gov/public/jsp/XML/eAgendaXmlReport.jsp has a list of each publication period's complete agenda XML files. That might be useful in bridging the gap.

On Friday, November 13, 2015 at 11:04:23 AM UTC-5, Mark Silverberg wrote:

Is there an API or list of XML docs? I wonder if a decoupled API like on AWS' API Gateway or with Lambda could be created where it parsed the XML and put it into a more modern format

Mark

On Fri, Nov 13, 2015 at 10:42 AM, Shawn Johnson <shawnjo...@gmail.com> wrote:

Bump... any thoughts on Unified Agenda data as an API - would there be other folks out there who would use it if it were available as an API?

On Friday, December 12, 2014 at 1:00:21 PM UTC-5, Shawn Johnson wrote:

We are primarily looking to pull Unified Agenda data. So, I would want to be able to GET a list of publication periods. GET a list of Agencies within a given publication period. List of RINs available for a given agency+publication period. Then ultimately RIN data found on a page like this: http://www.reginfo.gov/public/do/eAgendaViewRule?pubId=201410&RIN=0503-AA55 And, yes, I know there is XML there already :-)

-=Shawn

--
You received this message because you are subscribed to the Google Groups "US Government APIs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to us-government-apis+unsub...@googlegroups.com.

To post to this group, send email to us-govern...@googlegroups.com.
Visit this group at http://groups.google.com/group/us-government-apis.

--
You received this message because you are subscribed to the Google Groups "US Government APIs" group.

To unsubscribe from this group and stop receiving emails from it, send an email to us-government-apis+unsub...@googlegroups.com.

To post to this group, send email to us-govern...@googlegroups.com.
Visit this group at http://groups.google.com/group/us-government-apis.

Reply all

Reply to author

Forward