Election Metadata & Results repositories

[Derek Willis]

Hello OpenElectioneers!

One of the first things we did in this project was to collect election metadata in a Django admin (many of you helped with this task at NICAR conferences and other gatherings, and we thank you for it!). As the project has evolved, we've used GitHub as the place for our data, which has left our election metadata process decoupled from our conversion work. In practice, that means it has been easy to ignore and is now out of date. So here's what we're going to do: we'll be converting the metadata into flat files (CSV or JSON, maybe both) and posting it in a new repository on GitHub, and retiring the metadata admin site.

There's also the issue of the metadata API, which is used mostly by the openelections-core repository to build results files. Since that process has been used by only a few volunteers (mostly in Wisconsin and Georgia), we're planning on keeping the API but in a more static form, likely served by GitHub pages.

The overall goal is to standardize our data offerings on a single, simple format - likely the format used in the openelections-data-{state} repositories. For those states that have published data in openelections-results-{state} repositories, we're thinking about converting them into the simpler format in separate repositories so we're not maintaining multiple versions. We won't delete any of the -results- repositories, but we will not add to them going forward.

By 2020, we'd like to get to the point where the data conversion process is simpler and less dependent on external resources. It also will save us a little bit of money each month.

All of this will take a bit to accomplish, and we'd love to hear any comments and reactions to this.

Thanks,

Derek Willis

[Erik Paulson]

I think this sounds like a good plan - making the data easier to see and clone on github seems to be low-friction and I think probably easier for folks to directly see what's in the data and how things work end-to-end. I'm guessing that metadata isn't huge and so a git repo of JSON is perfectly reasonable.

I'm not sure that getting rid of -results-{state} repos is a good idea; I think having a simple consistent archive for each state with "results" and no other distractions would make data most usable. I wonder (and maybe I'm just now a bit confused) if maybe combining the -sources-{state} and -data-{state} repos would be easier to understand. Let's say that for each state, there were two repos: -sources-{state} and -results-{state}. -sources could be a state-specific adventure; maybe it's PDFs of scans of tally sheets or decent CSVs and some Python to process those CSV or some combo, but most users of the data products wouldn't look in that repo anyway, they'd just pull -results-{state} and have nicely formatted CSVs. -sources-{state} is only meant to be able to recreate and track whatever steps were necessary to create -results-{state}. 

(Maybe I'm just confused by the current -sources-{state} and -data-{state})

One of the coolest possibilities about the metadata repo is it might become a unofficial home for election identifiers - what do we call the election that happened on November 8th 2016, and if there were special elections that also happened that day, were they the same election? It'd be awesome to have some kind of OCD-like name for that. There are good things and bad things about there not being a central body in the USA that manages elections, but certainly one of the complications is that there's no one who can make canonical IDs.

-Erik

--
You received this message because you are subscribed to the Google Groups "openelections" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openelection...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openelections/89aecfa4-552c-49f8-8000-cc85a7aab8b6%40googlegroups.com.