Account Options

  1. Sign in
The old Google Groups will be going away soon, but your browser is incompatible with the new version.
Google Groups Home
« Groups Home
Open State Project
Home
+ new post
About this group
Join this group
xml data
Options
There are currently too many topics in this group that display first. To make this topic appear first, remove this option from another topic.
There was an error processing your request. Please try again.
flag
   10 messages - Collapse all  -  Translate all to Translated (View all originals)
The group you are posting to is a Usenet group. Messages posted to this group will make your email address visible to anyone on the Internet.
Your reply message has not been sent.
Your post was successful
 
From:
To:
Cc:
Followup To:
Add Cc | Add Followup-to | Edit Subject
Subject:
Validation:
For verification purposes please type the characters you see in the picture below or the numbers you hear by clicking the accessibility icon. Listen and type the numbers you hear
 
Danielle Ochoa  
View profile  
 More options Jul 26 2012, 3:51 pm
From: Danielle Ochoa <dianeinflor...@gmail.com>
Date: Thu, 26 Jul 2012 12:51:25 -0700 (PDT)
Local: Thurs, Jul 26 2012 3:51 pm
Subject: xml data
Print | Individual message | Show original | Report this message | Find messages by this author

How would I use the api to request xml formatted data?


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
James Turk  
View profile  
 More options Jul 26 2012, 4:08 pm
From: James Turk <james.p.t...@gmail.com>
Date: Thu, 26 Jul 2012 13:08:28 -0700 (PDT)
Local: Thurs, Jul 26 2012 4:08 pm
Subject: Re: xml data
Print | Individual message | Show original | Report this message | Find messages by this author

Our API does not support XML responses, we did for some time as an
experiment but it was not widely used and increased our maintenance
time/costs.  

That said I'd be happy to guide you in the direction of a JSON library that
is suitable for your language of choice.  What programming language are you
using?


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Gmail  
View profile  
 More options Jul 26 2012, 4:22 pm
From: Gmail <dianeinflor...@gmail.com>
Date: Thu, 26 Jul 2012 16:22:04 -0400
Local: Thurs, Jul 26 2012 4:22 pm
Subject: Re: [openstates] Re: xml data
Print | Individual message | Show original | Report this message | Find messages by this author

Well I know this is stupid but I was hoping to store some of the data in MySQL as to not make repetive API calls. Plus thy would let me combine with other data. I am using php to make the request and with the Json response I am decoding and then using XML serializer problem is that is saves te office as obj strings . The goal was to be able to load into SQL.

By the way this is awesome

Sent from my iPhone

On Jul 26, 2012, at 4:08 PM, James Turk <james.p.t...@gmail.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Paul Tagliamonte  
View profile  
 More options Jul 26 2012, 4:35 pm
From: Paul Tagliamonte <paul...@sunlightfoundation.com>
Date: Thu, 26 Jul 2012 13:35:52 -0700 (PDT)
Local: Thurs, Jul 26 2012 4:35 pm
Subject: Re: [openstates] Re: xml data
Print | Individual message | Show original | Report this message | Find messages by this author

On Thursday, July 26, 2012 4:22:04 PM UTC-4, Danielle Ochoa wrote:

> Well I know this is stupid but I was hoping to store some of the data in
> MySQL as to not make repetive API calls. Plus thy would let me combine with
> other data. I am using php to make the request and with the Json response I
> am decoding and then using XML serializer problem is that is saves te
> office as obj strings . The goal was to be able to load into SQL.

Howdy!

So, let's see if I understand you correctly --

You're using PHP to hit our API (in JSON) which your json_decode'ing into
PHP objects, which you re-serialize into XML?

Are you using a PHP script to proxy API requests? I don't fully understand
the question, but let me attempt to answer the second part - the bit about
offices.

So, if I understand you, you're saying the office data doesn't show up with
broken apart address / zipcode / state fields, but as a single string. We
store office data (since it's pretty complex) as a single string - since
decoding this programmatically leads to issues and mistakes, we just store
the raw data for humans to digest on the other end.

In all the cases I could think of (printing a letter, putting the address
into something to map a trip to their office, etc) would involve humans
anyway.

As for your SQL Schema - you could have a table for offices (since they're
1-to-N) - such as as follows (excuse ascii art)

+--------------------------------------------------+
|                   legislators                    |
+--------------------------------------------------+
| leg_id     : VARCHAR(N), PRIMARY_KEY, NOT NULL   |
| first_name : VARCHAR(N)                          |
| ......                                           |
| ......                                           |
+--------------------------------------------------+

+--------------------------------------+
|              offices                 |
+--------------------------------------+
| legislator  : FK(legislator.leg_id)  |
| office      : VARCHAR(N)             |
+--------------------------------------+

However, do keep in mind that we don't use a relational DB in the backend
(enforcing a schema becomes a bit complex if you're dealing with 50+2
legislative bodies) - so you might find stuff missing or unique to a state,

> By the way this is awesome

Thanks, man!

HTH,
  Paul

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Gmail  
View profile  
 More options Jul 26 2012, 4:48 pm
From: Gmail <dianeinflor...@gmail.com>
Date: Thu, 26 Jul 2012 16:48:21 -0400
Local: Thurs, Jul 26 2012 4:48 pm
Subject: Re: [openstates] Re: xml data
Print | Individual message | Show original | Report this message | Find messages by this author

Wow that was a super quick answer!
Plus I'm a chick .
Yes I'm using php script to proxy.

Sent from my iPhone

On Jul 26, 2012, at 4:35 PM, Paul Tagliamonte <paul...@sunlightfoundation.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Gmail  
View profile  
 More options Jul 26 2012, 5:08 pm
From: Gmail <dianeinflor...@gmail.com>
Date: Thu, 26 Jul 2012 17:08:50 -0400
Local: Thurs, Jul 26 2012 5:08 pm
Subject: Re: [openstates] Re: xml data
Print | Individual message | Show original | Report this message | Find messages by this author

My real goal is shoving into an SQL database

Sent from my iPhone

On Jul 26, 2012, at 4:35 PM, Paul Tagliamonte <paul...@sunlightfoundation.com> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Paul Tagliamonte  
View profile  
 More options Jul 26 2012, 5:35 pm
From: Paul Tagliamonte <paul...@sunlightfoundation.com>
Date: Thu, 26 Jul 2012 17:35:16 -0400
Local: Thurs, Jul 26 2012 5:35 pm
Subject: Re: [openstates] Re: xml data
Print | Individual message | Show original | Report this message | Find messages by this author
OK, so, long email - but it's all sorta needed information for what
you're going at here:

First of all, sorry for using "man" - I didn't see your real name
(your name comes up as "Gmail" for me :)), I didn't mean to offend, so
I'm sorry.

OK, now, onto some technical stuff:

We do not (and really can not) place a schema on this data. Designing
a SQL Schema that holds *exactly* what we provide is such a monumental
task that none of the team here at sunlight would be able to get it
done in any reasonable timeframe.

You'd end up with either a *ton* of tables, or a *ton* of NULL records
in each row. Clearly, a bunch of tables is "better", but it's a bear
to maintain.

Now, if you understand exactly what subset of the data you wish to
capture, you can design a schema without too much trouble. Just be
sure none of your fields from our API are "plus" fields (such as
"+foo"), since those are non-standard fields that are provided outside
the core schema.

In my first response I attached a rough ASCII-art schema one might use
(although, as jeremyb pointed out, the FK on the offices table should
be "legislators" not "legislator", thanks jeremyb!) -- so keeping a
subset of this stuff in the DB is pretty easyish.

In fact, the iOS app does something similar with sqlite3, iirc.

If I were designing this (and again, I'd encourage you to use
something that doesn't enforce a schema if possible), I would spec out
exactly what I need, and use that.

What exactly are you trying to capture, I might be able to provide
some more constructive advice (and feel free to wait until you're off
your iPhone, I know how painful it can be to type on an iDevice :) )
-- the more you give us, the more we can give you :)

Cheers,
 Paul

--
Paul Tagliamonte
Software Developer | Sunlight Foundation

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Ray Kiddy  
View profile  
 More options Jul 27 2012, 9:57 pm
From: Ray Kiddy <r...@ganymede.org>
Date: Fri, 27 Jul 2012 18:57:20 -0700
Local: Fri, Jul 27 2012 9:57 pm
Subject: Re: [openstates] xml data
Print | Individual message | Show original | Report this message | Find messages by this author

On Jul 26, 2012, at 2:08 PM, Gmail wrote:

> My real goal is shoving into an SQL database

I have been looking at this data for a while and have been looking at ways to interchange the JSON with SQL, or rather how it can be done but not quite. But there may be value in converting the data that can be converted.

Is your interest in putting it into a MySQL that you just want a way to store it? Or are you interested in doing it because of the structure it would then have?

If you are interested, I am going to start putting the tools that I have already written for this up on http://github.com/rkiddy/LegislatureSQL. If there is anyone who wants to be a committer, let me know. I would be fine with adding anyone who is interested. It seems worth creating a separate project because OpenStates has said before that they are not interested in restricting themselves to an SQL-like database and that is fine. So, this can be a complementary effort.

I am trying to think of when I have seen mention of other projects (with shared code) that use something from OpenStates. I cannot think of any, but it may be that my scans of the list have missed something. Does anyone know of any other projects?

I think it would be useful to look at the OpenStates data and examine it as to its data integrity. This is not to say that the OpenStates effort has not done what they have done for good reason, but you do things one way if you want to get as much data as possible and another way if you want to have some structure on that data.

A project that provides an easy interchange of the data in the the OpenStates JSON would provide, I think, something useful. That data would be accessible to more tools, there being more tools that speak SQL than tools which speak python/JSON. There are also entity schema design tools which could be used to talk to a database.

And maybe it will not be useful. We will see.

cheers - ray


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Dianeinflorida  
View profile  
 More options Jul 27 2012, 10:15 pm
From: Dianeinflorida <dianeinflor...@gmail.com>
Date: Fri, 27 Jul 2012 22:15:54 -0400
Local: Fri, Jul 27 2012 10:15 pm
Subject: Re: [openstates] xml data
Print | Individual message | Show original | Report this message | Find messages by this author

Ray,
Anything I can do to help let me know. I am thinking that it will have to be spilt between 2 tables.

Sent from my iPad

On Jul 27, 2012, at 9:57 PM, Ray Kiddy <r...@ganymede.org> wrote:


 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
Paul Tagliamonte  
View profile  
 More options Jul 27 2012, 10:17 pm
From: Paul Tagliamonte <paul...@sunlightfoundation.com>
Date: Fri, 27 Jul 2012 22:17:11 -0400
Local: Fri, Jul 27 2012 10:17 pm
Subject: Re: [openstates] xml data
Print | Individual message | Show original | Report this message | Find messages by this author

Hiya!

So, we have some pretty great schemas for the data -- even though it can
(sometimes) look a bit mucky, you can mostly count on non-plus'ed fields
being there -- we validate data on import with validictory -- here's the
legislator schema:

https://github.com/sunlightlabs/billy/blob/master/billy/schemas/perso...

As I said before, Greg used SQL for the iOS app, and it seems to work
pretty great there :)

I'd be happy to help you spec out what you need, if you have an idea of
what you (exactly) need :)

Cheers,
  Paul

--
Paul Tagliamonte
Software Developer | Sunlight Foundation

 
You must Sign in before you can post messages.
To post a message you must first join this group.
Please update your nickname on the subscription settings page before posting.
You do not have the permission required to post.
End of messages
« Back to Discussions « Newer topic

Create a group - Google Groups - Google Home - Terms of Service - Privacy Policy
©2013 Google