An update on officeholder data

[Kolina Yokoyama]

Hi everyone,

I am sharing an update on the status of updated API data, and on the errors that some folks have noticed in the API over the last few weeks due to redistricting and congressional updates.

1/ All federal offices, governors, state legislative officeholder data for the following states are now up to date:

• AK, AL, AR, AS, AZ, CA, CO, CT, DC, DE,
• FL, GA, GU, HI, IA, ID, IL, IN, KS, KY, LA,
• MA, MD, ME, MI, MN, MO, MP, MS, MT,
• NC, ND, NE, NH, NJ, NM, NV, NY,
• OH, OK, OR, PA, PR, RI, SC, SD, TN, TX,
• UT, VA, VI, VT, WA, WI, WV, WY

2/ The errors that you may have seen over the the last few weeks included:

• Out of date districts (e.g., some Morgantown, WV addresses incorrectly reported as being in congressional District 1 instead of District 2).
• Out of date officeholders (e.g., Tom Wolf reported as Governor of Pennsylvania).
• Missing local and state officeholders (e.g., representatives from the New York Assembly were missing)

As of January 30, these errors should now be resolved. Please let us know if you continue to see any related errors.

3/ Contact and social info for Federal and State-level officeholders should be fully updated by early March, including all governors. Mayors and top city officeholders are on track to be updated by end of March, while the rest of local officeholders should be updated by June.

Thank you, 

Kolina on behalf of the Google Civic Information API team

[Thomas Knox]

Kolina - this is super helpful.   Re the state legislative officeholder data, I'm seeing very sparse info in Michigan - is the timeline for more than names of officeholder March? June?  I wasn't 100% sure from your post.

For the details on what I'm seeing, with calls to either of these endpoints:

Representatives: representativeInfoByDivision

Representatives: representativeInfoByAddress

I get really great, detailed info for officials up and down the list, except for state legislators in Michigan.

E.g. I get back short entries under 'officials' like

{
    "name": "Sue Shink",
    "party": "Democratic Party"
},
{
    "name": "Kathy Schmaltz",
    "party": "Republican Party"
},

for any ocdId like

ocd-division/country:us/state:mi/sldu:<number>

ocd-division/country:us/state:mi/sldl:<number>

So we are still missing all the sweet contact data normally under keys like

phones

urls

emails

channels

geocodingSummaries

for those officials.

For officials 'above' and 'below' those divisionIds things look really good!

Question: If there an official body or contact you work with in MI government, would it be helpful for citizens (like me!) to reach out and express how valuable it would be for this data to be shared?

If you're good to do with the data, and it's mainly a matter of capacity to get it into the dataset, is there anything I could to help.  Are you looking for volunteers?  If I can be useful, I'd like to do my part.

Many thanks for all you do!  The Civic Info dataset is super valuable resource!

Best,

-t

[Kolina Yokoyama]

Hi Thomas,

Thanks for reaching out! 

You're right that this data is sparse for now. Most incumbents will still have contact info populated, but new officeholders and incumbents who changed districts don't have updated contact info yet. We're expecting to get that data updated early March and will share here when it's available.

At the moment, we not currently looking for volunteers, but thanks so much for expressing interest! We'll definitely reach out when there are opportunities to contribute. For now, your feedback is invaluable, and we'll be reaching out soon to gather feedback more broadly.

 Please stay tuned, and thanks for your patience!

Kolina on behalf of the Google Civic Information API team

[Thomas Knox]

Thanks Kolina! Got it.  Just in case it helps, there is structured data for MI State Senators available at

https://senate.michigan.gov/senators/SenatorData.json

It lacks socials but does have address, email, phone, and web, and it is officially published data from the state.

Are links to official data like this helpful to your efforts?  Or do you already have processes and contacts with various Secretaries of State to provide info for the dataset?

Again, thanks!

[Kolina Yokoyama]

Hi Thomas,

We do have partnerships and processes to get all this data, but it does take a bit of time for us to get it all together. Thank you for your patience!

Kolina on behalf of the Google Civic Information API team

[Kas Stohr]

Kolina, 

Some time has passed. Wondering if you can give another update on the status of filling in the data post-redistricting for office holders, specifically contact information, profile photos and other meta data. Many kind thanks for all you do. Updates are very helpful! 

[Lynn Demarest]

I'm working on scraping local officeholders in Florida. I also have monthly access to voter records. 

ChatGPT has been enormously helpful parsing the HTML. I then ask it to write SQL INSERTs and viola! 

--
You received this message because you are subscribed to the Google Groups "Google Civic Information API" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-civicinfo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-civicinfo-api/74bec8bf-f674-4d32-9224-00defaf0e5b6n%40googlegroups.com.

--

Lynn Demarest

561 644 0961

ly...@lynndemarest.com 

https://www.linkedin.com/in/lynn-demarest-44428863/

https://www.lynndemarest.com 

https://www.github.com/LynnDemarest

https://smile.amazon.com/The-Soul-Gene-ebook/dp/B008SFRZ34  

[Joe Gillon]

Hi Lynn. I'm doing this in Michigan. From Tampa though and interested in FL data as well. My app is mireps.org. You can see we have local officials. At first, I considered scraping for their data but there'd be so many scrapers and any change to the web pages would require a new scraper. I decided to show authorized users a form and table and have them basically copy and paste the data from the various websites to the app. I did school boards myself, took a bit over an hour. Fortunately, the data is pretty stable. If you put a few volunteers on the task it'd be done in no time. But I'm interested in your ChatGPT idea. Some of these web pages are pretty squirrelly. The data is not sanely identified with id or class attributes. Did you run into sites like that with ChatGPT?

[Lynn Demarest]

I have not had any trouble with ChatGPT's parsing of HTML so far, other than the token limit. 

I've created a little C# app that takes a URL as input, reads the HTML content, lets you edit the HTML content (to reduce the token count), and then passes the clipped HTML to ChatGPT along with a prompt-prefix (below) that tells ChatGPT the CREATE TABLE statement for the table you're trying to add to.

The edited HTML is appended to the prefix and then all of it is sent to ChatGPT, which returns the SQL INSERT statements. For an old guy like me, this is magical. Scraping web pages with selenium or puppeteer is tragic. Of course if ChatGPT doesn't do it right on the first try, you can amend the prompt and that sometimes works. Like if your data says FLORIDA but you want FL, just add that request to the prompt.

Here's the prefix I use (your INCUMBENTS data table may differ) : 

Given the T-SQL table:
CREATE TABLE [dbo].[INCUMBENTS](
[OfficeName] [nvarchar](255) NULL,
[OfficeHolder] [nvarchar](255) NULL,
[Party] [nvarchar](255) NULL,
[Phone] [nvarchar](20) NULL,
[Email] [nvarchar](150) NULL,
[District] [nvarchar](20) NULL,
[Precinct] [nvarchar](20) NULL,
[BodyTypeId] int,
    [CountyCode] [nvarchar(3)]
) ON [PRIMARY]
Parse the following data and create T-SQL INSERT statements to add items to the INCUMBENTS table.

Here's the data: ":

       return promptPrefix;
}
  

To view this discussion on the web visit https://groups.google.com/d/msgid/google-civicinfo-api/1754f9e9-6b11-4456-9c28-3dddaa4c386dn%40googlegroups.com.