ND Scraper

[cha...@gmail.com]

Hi All,

Is the ND scraper running overnight?  I ask because my local scrape of ND is not correctly pulling in all Senators and Representatives.  In particular, it appears that the scraper is skipping all but the first person listed in a particular District (which always happens to be a Senator).   Additionally, billy-update is crashing when handling the bills in ND.

I see that the openstates.org site has correct data for all ND Senators and Representatives so I was not sure if the ND scraper was not updating the data or if there is some oddity in my local scraper (my local scraper for ND is unchanged from the git repository).

I have a notion of how to fix the scraper for the legislators, but don’t want to put too much time into it if the live scraper on the website is actually working.

thanks,
chadd

[James Turk]

Hi Chadd-

Yes the scrape is running nightly and hasn't had any issues.

There's a good chance something is misconfigured if your local scrapes aren't working but is there a reason you're running your own scrape of ND instead of pulling from the API?

-James

--
You received this message because you are subscribed to the Google Groups "Open State Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fifty-state-pro...@googlegroups.com.
To post to this group, send email to fifty-sta...@googlegroups.com.
Visit this group at http://groups.google.com/group/fifty-state-project.
For more options, visit https://groups.google.com/groups/opt_out.

[Chadd Williams]

James:

Thanks for the reply.  I'm just interested in contributing to this project and thought I'd poke around running various scrapers to see how they work.

I'll have to look into my setup more if ND is working correctly overnight.   Is there a specific version of Python needed to run these scripts? I'm using python 2.7.5 and all of the scrapers I have run except ND have worked correctly.  Python 3.3.2 gives a number of errors when running pip to pull down the requirements (even after commenting out the MySQL requirement).

thanks again,
chadd

--
You received this message because you are subscribed to a topic in the Google Groups "Open State Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fifty-state-project/-gd6qu7ase4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fifty-state-pro...@googlegroups.com.

[James Turk]

Ah yes, billy/openstates are both Python 2.7 only as many of the requirements aren't available yet.

If you have further trouble running ND feel free to follow up off-list with details about your environment/the specific errors.