Difference between 'data' and 'results' directories?

[Alasdair Mackintosh]

Hi,

I'm trying to understand the difference between the files in the 'data' directories and those in the 'results' directories. (E.g. https://github.com/openelections/openelections-data-or and https://github.com/openelections/openelections-results-or.)

It looks as though the 'results' files contain rows from multiple 'data' files. E.g. in results we just have '20161108__or__general__precinct__raw.csv' whereas the data files are one per county. So I assume there's a process to combine them.

However, it also looks as though the 'results' files have a richer schema (defined at  https://github.com/openelections/docs/blob/master/common-fields.md) whereas the 'data' files have a much simpler schema (see http://docs.openelections.net/data-entry.) Where is the additional data in the results files come from?

Thanks,

Alasdair

[Derek Willis]

Hey Alasdair,

Hopefully a good guide to this is the workflow illustrated here: http://docs.openelections.net/guide/

The -data- repositories contain data that needed to be converted from a non machine-readable source or from variable machine-readable sources like differently formatted spreadsheets. That's why they contain fewer fields - they tend to have only what the official sources provide.

The -results- repositories contain data that has been loaded through our processing pipeline, which adds some of the standardized fields like ocd_id and other common fields. The additional data comes from the loader process (here's Oregon's: https://github.com/openelections/openelections-core/blob/dev/openelex/us/or/load.py) and from some common utilities in the openelections-core repository. Details about the election come from our metadata API (http://docs.openelections.net/metadata-api/).

Derek

--
You received this message because you are subscribed to the Google Groups "openelections" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openelections+unsubscribe@googlegroups.com.
To post to this group, send email to openel...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/openelections/00174eb1-d73f-45e9-8006-20e58fc7bf05%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alasdair Mackintosh]

On Monday, August 28, 2017 at 6:54:51 AM UTC-7, Open Elections Team wrote:

Hey Alasdair,

Hopefully a good guide to this is the workflow illustrated here: http://docs.openelections.net/guide/

The -data- repositories contain data that needed to be converted from a non machine-readable source or from variable machine-readable sources like differently formatted spreadsheets. That's why they contain fewer fields - they tend to have only what the official sources provide.

The -results- repositories contain data that has been loaded through our processing pipeline, which adds some of the standardized fields like ocd_id and other common fields. The additional data comes from the loader process (here's Oregon's: https://github.com/openelections/openelections-core/blob/dev/openelex/us/or/load.py) and from some common utilities in the openelections-core repository. Details about the election come from our metadata API (http://docs.openelections.net/metadata-api/).

Hi Derek,

Thanks for the links. I'll read through them.

May I ask why some states don't have the `results' directories populated yet? Is is just that nobody has run the pipeline, or is there some specialised code that still needs to be written? 

Thanks,

Alasdair

[Derek Willis]

Hi Alasdair,

Sure, that's right. States that don't have -results- repositories with any data in them are because we haven't wired up the processing pipeline for them (each state has a directory in the openelections-core repository for this). In many cases, that's because we've had to convert the original source material into CSV files first.

Derek

--

You received this message because you are subscribed to the Google Groups "openelections" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openelections+unsubscribe@googlegroups.com.
To post to this group, send email to openel...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/openelections/96080658-664e-4aab-bc40-005c1b43d472%40googlegroups.com.