Re: xml data

[James Turk]

Our API does not support XML responses, we did for some time as an experiment but it was not widely used and increased our maintenance time/costs.  

That said I'd be happy to guide you in the direction of a JSON library that is suitable for your language of choice.  What programming language are you using?

On Thursday, July 26, 2012 3:51:25 PM UTC-4, Danielle Ochoa wrote:

How would I use the api to request xml formatted data?

[Gmail]

Well I know this is stupid but I was hoping to store some of the data in MySQL as to not make repetive API calls. Plus thy would let me combine with other data. I am using php to make the request and with the Json response I am decoding and then using XML serializer problem is that is saves te office as obj strings . The goal was to be able to load into SQL.

By the way this is awesome

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "Open State Project" group.
To view this discussion on the web visit https://groups.google.com/d/msg/fifty-state-project/-/_WLCZr5HS6gJ.
To post to this group, send email to fifty-sta...@googlegroups.com.
To unsubscribe from this group, send email to fifty-state-pro...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/fifty-state-project?hl=en.

[Gmail]

Wow that was a super quick answer!

Plus I'm a chick .

Yes I'm using php script to proxy.

Sent from my iPhone

On Jul 26, 2012, at 4:35 PM, Paul Tagliamonte <pau...@sunlightfoundation.com> wrote:

On Thursday, July 26, 2012 4:22:04 PM UTC-4, Danielle Ochoa wrote:

Well I know this is stupid but I was hoping to store some of the data in MySQL as to not make repetive API calls. Plus thy would let me combine with other data. I am using php to make the request and with the Json response I am decoding and then using XML serializer problem is that is saves te office as obj strings . The goal was to be able to load into SQL.

Howdy!

So, let's see if I understand you correctly -- 

You're using PHP to hit our API (in JSON) which your json_decode'ing into PHP objects, which you re-serialize into XML?

Are you using a PHP script to proxy API requests? I don't fully understand the question, but let me attempt to answer the second part - the bit about offices.

So, if I understand you, you're saying the office data doesn't show up with broken apart address / zipcode / state fields, but as a single string. We store office data (since it's pretty complex) as a single string - since decoding this programmatically leads to issues and mistakes, we just store the raw data for humans to digest on the other end.

In all the cases I could think of (printing a letter, putting the address into something to map a trip to their office, etc) would involve humans anyway.

As for your SQL Schema - you could have a table for offices (since they're 1-to-N) - such as as follows (excuse ascii art)

+--------------------------------------------------+

|                   legislators                    |

+--------------------------------------------------+

| leg_id     : VARCHAR(N), PRIMARY_KEY, NOT NULL   |

| first_name : VARCHAR(N)                          |

| ......                                           |

| ......                                           |

+--------------------------------------------------+

+--------------------------------------+

|              offices                 |

+--------------------------------------+

| legislator  : FK(legislator.leg_id)  |

| office      : VARCHAR(N)             |

+--------------------------------------+

However, do keep in mind that we don't use a relational DB in the backend (enforcing a schema becomes a bit complex if you're dealing with 50+2 legislative bodies) - so you might find stuff missing or unique to a state,

By the way this is awesome

Thanks, man!

 

Sent from my iPhone

On Jul 26, 2012, at 4:08 PM, James Turk <james....@gmail.com> wrote:

Our API does not support XML responses, we did for some time as an experiment but it was not widely used and increased our maintenance time/costs.  

That said I'd be happy to guide you in the direction of a JSON library that is suitable for your language of choice.  What programming language are you using?

On Thursday, July 26, 2012 3:51:25 PM UTC-4, Danielle Ochoa wrote:

How would I use the api to request xml formatted data?

--
You received this message because you are subscribed to the Google Groups "Open State Project" group.
To view this discussion on the web visit https://groups.google.com/d/msg/fifty-state-project/-/_WLCZr5HS6gJ.

To post to this group, send email to fifty-state-project@googlegroups.com.
To unsubscribe from this group, send email to fifty-state-project+unsub...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/fifty-state-project?hl=en.

HTH,

  Paul

 

--
You received this message because you are subscribed to the Google Groups "Open State Project" group.

To view this discussion on the web visit https://groups.google.com/d/msg/fifty-state-project/-/BXSsqyE1BWQJ.

[Gmail]

My real goal is shoving into an SQL database

Sent from my iPhone

On Jul 26, 2012, at 4:35 PM, Paul Tagliamonte <pau...@sunlightfoundation.com> wrote:

On Thursday, July 26, 2012 4:22:04 PM UTC-4, Danielle Ochoa wrote:

Well I know this is stupid but I was hoping to store some of the data in MySQL as to not make repetive API calls. Plus thy would let me combine with other data. I am using php to make the request and with the Json response I am decoding and then using XML serializer problem is that is saves te office as obj strings . The goal was to be able to load into SQL.

Howdy!

So, let's see if I understand you correctly -- 

You're using PHP to hit our API (in JSON) which your json_decode'ing into PHP objects, which you re-serialize into XML?

Are you using a PHP script to proxy API requests? I don't fully understand the question, but let me attempt to answer the second part - the bit about offices.

So, if I understand you, you're saying the office data doesn't show up with broken apart address / zipcode / state fields, but as a single string. We store office data (since it's pretty complex) as a single string - since decoding this programmatically leads to issues and mistakes, we just store the raw data for humans to digest on the other end.

In all the cases I could think of (printing a letter, putting the address into something to map a trip to their office, etc) would involve humans anyway.

As for your SQL Schema - you could have a table for offices (since they're 1-to-N) - such as as follows (excuse ascii art)

+--------------------------------------------------+

|                   legislators                    |

+--------------------------------------------------+

| leg_id     : VARCHAR(N), PRIMARY_KEY, NOT NULL   |

| first_name : VARCHAR(N)                          |

| ......                                           |

| ......                                           |

+--------------------------------------------------+

+--------------------------------------+

|              offices                 |

+--------------------------------------+

| legislator  : FK(legislator.leg_id)  |

| office      : VARCHAR(N)             |

+--------------------------------------+

However, do keep in mind that we don't use a relational DB in the backend (enforcing a schema becomes a bit complex if you're dealing with 50+2 legislative bodies) - so you might find stuff missing or unique to a state,

By the way this is awesome

Thanks, man!

 

Sent from my iPhone

On Jul 26, 2012, at 4:08 PM, James Turk <james....@gmail.com> wrote:

Our API does not support XML responses, we did for some time as an experiment but it was not widely used and increased our maintenance time/costs.  

That said I'd be happy to guide you in the direction of a JSON library that is suitable for your language of choice.  What programming language are you using?

On Thursday, July 26, 2012 3:51:25 PM UTC-4, Danielle Ochoa wrote:

How would I use the api to request xml formatted data?

--
You received this message because you are subscribed to the Google Groups "Open State Project" group.
To view this discussion on the web visit https://groups.google.com/d/msg/fifty-state-project/-/_WLCZr5HS6gJ.

To post to this group, send email to fifty-state-project@googlegroups.com.
To unsubscribe from this group, send email to fifty-state-project+unsub...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/fifty-state-project?hl=en.

HTH,

  Paul

 

--
You received this message because you are subscribed to the Google Groups "Open State Project" group.

To view this discussion on the web visit https://groups.google.com/d/msg/fifty-state-project/-/BXSsqyE1BWQJ.

[Paul Tagliamonte]

OK, so, long email - but it's all sorta needed information for what
you're going at here:

First of all, sorry for using "man" - I didn't see your real name
(your name comes up as "Gmail" for me :)), I didn't mean to offend, so
I'm sorry.

OK, now, onto some technical stuff:

We do not (and really can not) place a schema on this data. Designing
a SQL Schema that holds *exactly* what we provide is such a monumental
task that none of the team here at sunlight would be able to get it
done in any reasonable timeframe.

You'd end up with either a *ton* of tables, or a *ton* of NULL records
in each row. Clearly, a bunch of tables is "better", but it's a bear
to maintain.

Now, if you understand exactly what subset of the data you wish to
capture, you can design a schema without too much trouble. Just be
sure none of your fields from our API are "plus" fields (such as
"+foo"), since those are non-standard fields that are provided outside
the core schema.

In my first response I attached a rough ASCII-art schema one might use
(although, as jeremyb pointed out, the FK on the offices table should
be "legislators" not "legislator", thanks jeremyb!) -- so keeping a
subset of this stuff in the DB is pretty easyish.

In fact, the iOS app does something similar with sqlite3, iirc.

If I were designing this (and again, I'd encourage you to use
something that doesn't enforce a schema if possible), I would spec out
exactly what I need, and use that.

What exactly are you trying to capture, I might be able to provide
some more constructive advice (and feel free to wait until you're off
your iPhone, I know how painful it can be to type on an iDevice :) )
-- the more you give us, the more we can give you :)


Cheers,
Paul

>> To post to this group, send email to fifty-sta...@googlegroups.com.
>> To unsubscribe from this group, send email to
>> fifty-state-pro...@googlegroups.com.
>> For more options, visit this group at
>> http://groups.google.com/group/fifty-state-project?hl=en.
>
>
>
> HTH,
> Paul
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "Open State Project" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/fifty-state-project/-/BXSsqyE1BWQJ.
> To post to this group, send email to fifty-sta...@googlegroups.com.
> To unsubscribe from this group, send email to
> fifty-state-pro...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/fifty-state-project?hl=en.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Open State Project" group.

> To post to this group, send email to fifty-sta...@googlegroups.com.
> To unsubscribe from this group, send email to
> fifty-state-pro...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/fifty-state-project?hl=en.

--
Paul Tagliamonte
Software Developer | Sunlight Foundation

[Ray Kiddy]

On Jul 26, 2012, at 2:08 PM, Gmail wrote:

My real goal is shoving into an SQL database

I have been looking at this data for a while and have been looking at ways to interchange the JSON with SQL, or rather how it can be done but not quite. But there may be value in converting the data that can be converted.

Is your interest in putting it into a MySQL that you just want a way to store it? Or are you interested in doing it because of the structure it would then have?

If you are interested, I am going to start putting the tools that I have already written for this up on http://github.com/rkiddy/LegislatureSQL. If there is anyone who wants to be a committer, let me know. I would be fine with adding anyone who is interested. It seems worth creating a separate project because OpenStates has said before that they are not interested in restricting themselves to an SQL-like database and that is fine. So, this can be a complementary effort.

I am trying to think of when I have seen mention of other projects (with shared code) that use something from OpenStates. I cannot think of any, but it may be that my scans of the list have missed something. Does anyone know of any other projects?

I think it would be useful to look at the OpenStates data and examine it as to its data integrity. This is not to say that the OpenStates effort has not done what they have done for good reason, but you do things one way if you want to get as much data as possible and another way if you want to have some structure on that data.

A project that provides an easy interchange of the data in the the OpenStates JSON would provide, I think, something useful. That data would be accessible to more tools, there being more tools that speak SQL than tools which speak python/JSON. There are also entity schema design tools which could be used to talk to a database.

And maybe it will not be useful. We will see.

cheers - ray

[Dianeinflorida]

Ray,

Anything I can do to help let me know. I am thinking that it will have to be spilt between 2 tables.

Sent from my iPad

To post to this group, send email to fifty-sta...@googlegroups.com.
To unsubscribe from this group, send email to fifty-state-pro...@googlegroups.com.

[Paul Tagliamonte]

On Fri, Jul 27, 2012 at 9:57 PM, Ray Kiddy <r...@ganymede.org> wrote:

On Jul 26, 2012, at 2:08 PM, Gmail wrote:

My real goal is shoving into an SQL database

I have been looking at this data for a while and have been looking at ways to interchange the JSON with SQL, or rather how it can be done but not quite. But there may be value in converting the data that can be converted.

Is your interest in putting it into a MySQL that you just want a way to store it? Or are you interested in doing it because of the structure it would then have?

If you are interested, I am going to start putting the tools that I have already written for this up on http://github.com/rkiddy/LegislatureSQL. If there is anyone who wants to be a committer, let me know. I would be fine with adding anyone who is interested. It seems worth creating a separate project because OpenStates has said before that they are not interested in restricting themselves to an SQL-like database and that is fine. So, this can be a complementary effort.

I am trying to think of when I have seen mention of other projects (with shared code) that use something from OpenStates. I cannot think of any, but it may be that my scans of the list have missed something. Does anyone know of any other projects?

I think it would be useful to look at the OpenStates data and examine it as to its data integrity. This is not to say that the OpenStates effort has not done what they have done for good reason, but you do things one way if you want to get as much data as possible and another way if you want to have some structure on that data.

Hiya!

So, we have some pretty great schemas for the data -- even though it can (sometimes) look a bit mucky, you can mostly count on non-plus'ed fields being there -- we validate data on import with validictory -- here's the legislator schema:

https://github.com/sunlightlabs/billy/blob/master/billy/schemas/person.json

As I said before, Greg used SQL for the iOS app, and it seems to work pretty great there :)

I'd be happy to help you spec out what you need, if you have an idea of what you (exactly) need :)

Cheers, 

  Paul

 

To post to this group, send email to fifty-sta...@googlegroups.com.
To unsubscribe from this group, send email to fifty-state-pro...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/fifty-state-project?hl=en.