How to deal with VA vote inconsistencies?

[Daniel Roberts]

A client noticed issues with votes for a bill in VA and I see that the root issue is that the VA site is listing votes twice, once with just a useful title and another time with generic title and link to legislator votes. The end result is that votes get added twice to openstates data and the one with a useful title has no legislator votes associated with it, making the data unusable.

For example on this page "http://lis.virginia.gov/cgi-bin/legp604.exe?161+sum+HB30" there is listed the vote "04/20/16  House: House rejected Governor's recommendation #23 (34-Y 65-N 1-A)" and then the vote "04/20/16  House: VOTE: REJECTED (34-Y 65-N 1-A)" which appear to be referring to the same vote however only the second one links to legislator votes. 

There does not appear to be a great way to correlate the two other than by order in list but that could be inconsistent from bill to bill. How is this sort of thing normally handled? Does Sunlight contact a state if there is a problem with their data or format? I could do so but would be concerned the state would make updates that would unexpectedly break parsing of all VA bills.

[Tim Showers]

<thinking out loud while we wait for a response from the OS folks.>

Do they consistently capitalize one set? Maybe we could:

For each vote:

If the vote is all caps, AND there's another vote with matching counts on the same day:

ignore the all caps vote

This is narrow enough that it shouldn't be terribly fragile to future updates, but someone should probably just reach out to the state. I'll wait from word from the OS folks but i'm happy to email the VA legislature and ask.

Generally in the scraper code they try to clean up inconsistencies like this programmatically rather than have dupes in the data.

[Andy Lo]

This is something I can look into resolving on the Open States side, but if we can get VA to fix these issues at the source, then that would be ideal.