Progress Update on 2016 Preparations

83 views
Skip to first unread message

Andy Lo

unread,
Jan 15, 2016, 6:27:40 PM1/15/16
to Open State Project
Hello All,

Just posting a quick update on where the preparations for making 2016 data available are.  Most of the states should have their scrapers running and the 2016 data available.  Following are the state scrapers that have outstanding issues:
  1. Alabama - Alabama has been a long-running issue for us.  Still needs eyes.
  2. Iowa - Was running up until a few days ago - will need to check whether this is an upstream issue.
  3. Louisiana - Scrapers are stopped due to incomplete senator data.  However, it looks like Louisiana's session isn't slated to begin for a while.
  4. Nebraska - Currently working on it.
  5. New Jersey - Not updating as their 2016 committee data are still pending.
  6. Rhode Island - Still needs eyes.
  7. Vermont - Working up until a few days ago - it looks like the production scrapers may be blocked.
  8. West Virginia - Still needs eyes.
I haven't been able to get around to too much of the data quality fixes, but if a data quality issue hasn't already been reported via JIRA, you can submit a ticket there. 

Daniel Roberts

unread,
Jan 22, 2016, 2:29:34 PM1/22/16
to Open State Project
What's wrong with Alabama? I can help.

Andy Lo

unread,
Jan 22, 2016, 10:16:13 PM1/22/16
to Open State Project
It looks as though the scraper is currently choking on Senator Randy Davis' profile, which prevents the scraper from running any further.  Although the scrapers seem to have traditionally avoided using default "empty" placeholders in the data, I'm strongly considering going that route to let the scrapers grab what data they can, though I'm reaching out to some former Sunlight staff to see if there are any unforeseen ramifications of doing so.

Daniel Roberts

unread,
Jan 23, 2016, 1:53:49 PM1/23/16
to Open State Project
They are in the middle of a bad-publicity "update" to the legislature's website. Randy Davis' new link is here:

The full list of the lower house is now here: http://www.legislature.state.al.us/aliswww/ISD/House/ALRepresentatives.aspx. It looks like a workable link to the individuals can be reconstructed using the longdesc and alt attributes of the <input> (?) tag.

I can see about working up a pull request to the scraper in the next few days, but it will take me a couple days to get to it.

Andy Lo

unread,
Jan 23, 2016, 2:35:03 PM1/23/16
to Open State Project
Thanks for the heads up.  I can probably whip up a refactor for the new site pretty quickly sometime this weekend, but it looks like the bill data is still coming from AlisonDB.

Tomek Pajor

unread,
Jan 26, 2016, 11:28:27 AM1/26/16
to Open State Project
Hello,

I recently started to use openstates api as we need to obtain all legislators. I can see that for 462 people are missing their office address data. How can I help to fix this issue? I'm sorry if this is not relevant for this thread.

Andy Lo

unread,
Jan 26, 2016, 9:07:59 PM1/26/16
to Open State Project
Hi Tomek,

If you could list a good sample of the legislators that are missing address data, that would help me verify and pinpoint the issue.

Tomek Pajor

unread,
Jan 27, 2016, 11:10:02 AM1/27/16
to Open State Project
Hello Andy,

Actually I was a bit off on my original number. 217 are missing address. For some capitol address is missing, there is district address available. But for now let's focus on the ones that don't have address at all.

CAL000003 Mark Leno MISSING ADDRESS
CAL000009 Loni Hancock MISSING ADDRESS
CAL000316 Hannah-Beth Jackson MISSING ADDRESS
CAL000018 Fran Pavley MISSING ADDRESS
CAL000051 Jim Beall MISSING ADDRESS
CAL000057 Kevin de León MISSING ADDRESS
CAL000062 Cathleen Galgiani MISSING ADDRESS
CAL000083 Jerry Hill MISSING ADDRESS
CAL000087 Jean Fuller MISSING ADDRESS
CAL000102 Ted Gaines MISSING ADDRESS
CAL000107 Bill Monning MISSING ADDRESS
CAL000116 Joel Anderson MISSING ADDRESS
CAL000133 Anthony Cannella MISSING ADDRESS
CAL000141 Ben Hueso MISSING ADDRESS
CAL000155 Ricardo Lara MISSING ADDRESS
CAL000327 Richard D. Roth MISSING ADDRESS
CAL000494 John M. W. Moorlach MISSING ADDRESS
CAL000444 Holly J. Mitchell MISSING ADDRESS
CAL000490 Benjamin Allen MISSING ADDRESS
CAL000469 Janet Nguyen MISSING ADDRESS
CAL000466 Connie M. Leyva MISSING ADDRESS
CAL000154 Mike Morrell MISSING ADDRESS
CAL000475 Patricia C. Bates MISSING ADDRESS
CAL000482 Robert M. Hertzberg MISSING ADDRESS
CAL000495 Steven M. Glazer MISSING ADDRESS
CAL000426 Andy Vidak MISSING ADDRESS
CAL000486 Jeff Stone MISSING ADDRESS
CAL000447 Mike McGuire MISSING ADDRESS
CAL000085 Tom Berryhill MISSING ADDRESS
CAL000022 Bob Huff MISSING ADDRESS
CAL000016 Carol Liu MISSING ADDRESS
CAL000005 Lois Wolk MISSING ADDRESS
CAL000044 Marty Block MISSING ADDRESS
CAL000103 Jim Nielsen MISSING ADDRESS
CAL000049 Ed Hernandez MISSING ADDRESS
CAL000169 Sharon Runner MISSING ADDRESS
CAL000167 Bob Wieckowski MISSING ADDRESS
CAL000149 Richard Pan MISSING ADDRESS
CAL000112 Isadore Hall MISSING ADDRESS
CAL000091 Tony Mendoza MISSING ADDRESS
FLL000253 Travis Hutson MISSING ADDRESS
IAL000066 Jack Drake MISSING ADDRESS
IAL000036 Matt McCoy MISSING ADDRESS
IDL000053 Gary Collins MISSING ADDRESS
IDL000064 Stephen Hartgen MISSING ADDRESS
IDL000101 Jeff Thompson MISSING ADDRESS
IDL000089 John Rusche MISSING ADDRESS
IDL000060 Marc Gibbs MISSING ADDRESS
IDL000040 Linden Bateman MISSING ADDRESS
IDL000070 Phylis King MISSING ADDRESS
IDL000085 Christy Perry MISSING ADDRESS
IDL000212 Mark Nye MISSING ADDRESS
IDL000205 Heather Scott MISSING ADDRESS
IDL000038 Vito Barbieri MISSING ADDRESS
IDL000203 John McCrostie MISSING ADDRESS
IDL000037 Ken Andrus MISSING ADDRESS
IDL000213 Paulette E. Jordan MISSING ADDRESS
IDL000161 Clark Kauffman MISSING ADDRESS
IDL000177 Paul Romrell MISSING ADDRESS
IDL000197 Don Cheatham MISSING ADDRESS
IDL000201 Caroline Nilsson Troy MISSING ADDRESS
IDL000151 Mathew Erpelding MISSING ADDRESS
IDL000189 Rick D. Youngblood MISSING ADDRESS
IDL000168 Steven Miller MISSING ADDRESS
IDL000172 Kelley Packer MISSING ADDRESS
IDL000148 Lance Clow MISSING ADDRESS
IDL000155 Steven Harris MISSING ADDRESS
IDL000190 Ilana Rubel MISSING ADDRESS
IDL000199 Ryan Kerby MISSING ADDRESS
IDL000167 Ron Mendive MISSING ADDRESS
IDL000150 Thomas Dayley MISSING ADDRESS
IDL000191 Patrick McDonald MISSING ADDRESS
IDL000209 Van T. Burtenshaw MISSING ADDRESS
IDL000056 Reed DeMordaunt MISSING ADDRESS
IDL000195 Greg Chaney MISSING ADDRESS
IDL000169 Jason A. Monks MISSING ADDRESS
IDL000082 Joe Palmer MISSING ADDRESS
IDL000159 James Holtzclaw MISSING ADDRESS
IDL000208 Dan Rudolph MISSING ADDRESS
IDL000184 Julie VanOrden MISSING ADDRESS
IDL000204 Melissa Wintrow MISSING ADDRESS
IDL000183 Janet Trujillo MISSING ADDRESS
IDL000162 Hy Kloc MISSING ADDRESS
IDL000099 Gayle L. Batt MISSING ADDRESS
IDL000077 Shannon McMillan MISSING ADDRESS
IDL000084 Donna Pence MISSING ADDRESS
IDL000152 John Gannon MISSING ADDRESS
IDL000052 Susan Chew MISSING ADDRESS
IDL000095 Elaine Smith MISSING ADDRESS
IDL000200 Ronald Nate MISSING ADDRESS
IDL000080 Pete Nielsen MISSING ADDRESS
IDL000160 Wendy Horman MISSING ADDRESS
IDL000114 Terry Gestrin MISSING ADDRESS
IDL000103 John Woude MISSING ADDRESS
IDL000206 Sage G. Dixon MISSING ADDRESS
IDL000198 Eric M. Redman MISSING ADDRESS
IDL000164 Luke Malek MISSING ADDRESS
IDL000042 Scott Bedke MISSING ADDRESS
IDL000054 Brent Crane MISSING ADDRESS
IDL000140 Neil A. Anderson MISSING ADDRESS
IDL000141 Robert Anderst MISSING ADDRESS
IDL000158 Brandon A. Hixon MISSING ADDRESS
IDL000091 Paul Shepherd MISSING ADDRESS
IDL000078 Mike Moyle MISSING ADDRESS
IDL000074 Lynn Luker MISSING ADDRESS
IDL000094 Kathleen Sims MISSING ADDRESS
IDL000073 Thomas Loertscher MISSING ADDRESS
IDL000086 Dell Raybould MISSING ADDRESS
IDL000104 Richard "Rich" Wills MISSING ADDRESS
IDL000196 Merrill Beyeler MISSING ADDRESS
IDL000043 Maxine Bell MISSING ADDRESS
IDL000048 Judy Boyle MISSING ADDRESS
IDL000105 Fred Wood MISSING ADDRESS
IDL000018 Patti Lodge MISSING ADDRESS
IDL000027 Jeff Siddoway MISSING ADDRESS
IDL000211 Mary Souza MISSING ADDRESS
IDL000050 Grant Burgoyne MISSING ADDRESS
IDL000071 Roy Lacey MISSING ADDRESS
IDL000049 Cherie Buckner-Webb MISSING ADDRESS
IDL000033 Steve Vick MISSING ADDRESS
IDL000010 Bart Davis MISSING ADDRESS
IDL000125 Todd M. Lakey MISSING ADDRESS
IDL000061 Jim Guthrie MISSING ADDRESS
IDL000015 Brent Hill MISSING ADDRESS
IDL000210 Lori Den Hartog MISSING ADDRESS
IDL000127 Fred S. Martin MISSING ADDRESS
IDL000081 Robert P. "Bob" Nonini MISSING ADDRESS
IDL000022 Curt McKenzie MISSING ADDRESS
IDL000110 Dan G. Johnson MISSING ADDRESS
IDL000217 Maryanne Jordan MISSING ADDRESS
IDL000030 Michelle Stennett MISSING ADDRESS
IDL000005 Bert Brackett MISSING ADDRESS
IDL000026 Dan Schmidt MISSING ADDRESS
IDL000023 Dean Mortimer MISSING ADDRESS
IDL000083 Jim Patrick MISSING ADDRESS
IDL000041 Clifford Bayer MISSING ADDRESS
IDL000016 Shawn Keough MISSING ADDRESS
IDL000002 Steve Bair MISSING ADDRESS
IDL000218 Kelly A. Anthon MISSING ADDRESS
IDL000219 Mark Harris MISSING ADDRESS
IDL000112 Jim Rice MISSING ADDRESS
IDL000062 Marv Hagedorn MISSING ADDRESS
IDL000024 Sheryl Nuxoll MISSING ADDRESS
IDL000100 Steven Thayn MISSING ADDRESS
IDL000035 Chuck Winder MISSING ADDRESS
IDL000014 Lee Heider MISSING ADDRESS
IDL000207 Abby Lee MISSING ADDRESS
IDL000186 Janie Ward-Engelking MISSING ADDRESS
LAL000802 Cedric Glover MISSING ADDRESS
LAL000822 Reid Falconer MISSING ADDRESS
LAL000841 Denise Marcelle MISSING ADDRESS
LAL000813 Stephanie Hilferty MISSING ADDRESS
LAL000809 Beryl Amedée MISSING ADDRESS
LAL000833 Jean-Paul Coussan MISSING ADDRESS
LAL000829 Ronnie Edwards MISSING ADDRESS
LAL000842 Jack McFarland MISSING ADDRESS
LAL000826 Jimmy Harris MISSING ADDRESS
NJL000194 Patricia Jones MISSING ADDRESS
NJL000202 Nicholas Chiaravalloti MISSING ADDRESS
NJL000203 R. Land MISSING ADDRESS
NJL000196 Arthur Barclay MISSING ADDRESS
NJL000197 Angela McKnight MISSING ADDRESS
NJL000201 Annette Chaparro MISSING ADDRESS
NJL000205 Joe Howarth MISSING ADDRESS
NJL000199 Joann Downey MISSING ADDRESS
NJL000198 Eric Houghtaling MISSING ADDRESS
NJL000200 James Kennedy MISSING ADDRESS
PRL000184 Angel Muñoz Suarez MISSING ADDRESS
PRL000070 Roberto Ruiz de Porras MISSING ADDRESS
PRL000189 Carlos Vargas Ferrer MISSING ADDRESS
PRL000195 Maria Charbonier Laureano MISSING ADDRESS
PRL000190 Luis Ortiz Lugo MISSING ADDRESS
PRL000188 Urayoan Hernandez Alvarado MISSING ADDRESS
PRL000214 Yashira Lebrón Rodríguez MISSING ADDRESS
PRL000186 Carlos Bianchi Anglero MISSING ADDRESS
PRL000067 Carlos Méndez Nuñez MISSING ADDRESS
PRL000194 Angel Matos Garcia MISSING ADDRESS
PRL000175 Sonia Pacheco Irigoyen MISSING ADDRESS
PRL000043 Héctor Torres Calderón MISSING ADDRESS
PRL000193 Javier Aponte Dalmau MISSING ADDRESS
PRL000176 Jose L Baez Rivera MISSING ADDRESS
PRL000182 Jose A. Rodriguez Quiles MISSING ADDRESS
PRL000075 Jose Melendez Ortiz MISSING ADDRESS
PRL000056 Víctor Vasallo Anadón MISSING ADDRESS
PRL000191 Jesus Santa Rodriguez MISSING ADDRESS
PRL000185 Efrain De Jesus Rodriguez MISSING ADDRESS
PRL000192 Ramón Cruz Burgos MISSING ADDRESS
PRL000187 Nelson Torres Yordan MISSING ADDRESS
PRL000181 Cesar Hernandez Alfonzo MISSING ADDRESS
PRL000155 Angel M. Rodriguez Otero MISSING ADDRESS
PRL000174 Pedro A. Rodriguez Gonzalez MISSING ADDRESS
PRL000021 Angel Martinez Santiago MISSING ADDRESS
PRL000164 Jose R. Nadal Power MISSING ADDRESS
PRL000165 Ramon L. Nieves Perez MISSING ADDRESS
PRL000160 Anibal J. Torres Torres MISSING ADDRESS
PRL000169 Ramon Ruiz Nieves MISSING ADDRESS
PRL000162 Angel R. Rosa Rodriguez MISSING ADDRESS
PRL000019 Carmelo Rios Santiago MISSING ADDRESS
PRL000173 Luis D. Rivera Filomeno MISSING ADDRESS
PRL000163 Maria De L. Santiago Negron MISSING ADDRESS
PRL000171 Miguel A. Pereira Castillo MISSING ADDRESS
PRL000025 Larry Seilhamer Rodriguez MISSING ADDRESS
PRL000007 Jose Dalmau Santiago MISSING ADDRESS
PRL000167 Maria T. Gonzalez Lopez MISSING ADDRESS
PRL000161 Rossana Lopez Leon MISSING ADDRESS
PRL000166 Jose O. Perez Rosa MISSING ADDRESS
PRL000168 Gilberto Rodriguez Valle MISSING ADDRESS
PRL000003 Itzamar Peña Ramirez MISSING ADDRESS
PRL000013 Jorge Suarez Caceres MISSING ADDRESS
PRL000011 Cirilo Tirado Rivera MISSING ADDRESS
PRL000002 Margarita Nolasco Santiago MISSING ADDRESS
PRL000009 Antonio Fas Alzamora MISSING ADDRESS
PRL000001 Thomas Rivera Schatz MISSING ADDRESS
PRL000008 Eduardo Bhatia Gautier MISSING ADDRESS
PRL000018 Migdalia Padilla Alvelo MISSING ADDRESS
PRL000170 Martin Vargas Morales MISSING ADDRESS
TXL000202 Dan Patrick MISSING ADDRESS

Also one is missing district.

TXL000202 Dan Patrick MISSING DISTRICT

Andy Lo

unread,
Jan 27, 2016, 11:18:52 AM1/27/16
to Open State Project
Thanks for letting us know Tomek.  We'll take a look at getting that resolved ASAP, probably after we've ensured that the scrapers are working for as many of the new legislative sessions as possible.  It looks like some of the scrapers need updates.  If you know how to look at the scraper code, I'm always willing to review pull requests on GitHub.  Otherwise, this report is very helpful, and thanks again for giving us a heads up.

In the future, you can report data quality issues on our JIRA issue tracker.

Tomek Pajor

unread,
Jan 31, 2016, 5:04:53 AM1/31/16
to Open State Project
No problem, I we have sometime this upcoming week so can try to squash some bugs.
Reply all
Reply to author
Forward
0 new messages