running locally

[Joe Germuska]

Hey, gang:

Having just updated my repository after months (years?) away, this looks like it is out of date:

http://openstates.org/contributing/#running-your-scraper

I've got the full sunlight/master merged into my code, and I ran "pip install -U -r requirements.txt" (and dragged a TON of stuff in… overkill?)

Anyway, now when I run 

$ billy-update il

Traceback (most recent call last):

  File "/Users/germuska/.virtualenvs/openstates/bin/billy-update", line 8, in <module>

…

django.core.exceptions.ImproperlyConfigured: Requested setting DATABASES, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

It looks like that's from "pyes/djangoutils.py"

Is that a dependency of running the scraper locally? 

Joe

-- 
Joe Germuska
J...@Germuska.com * http://blog.germuska.com * http://twitter.com/JoeGermuska    

"I felt so good I told the leader how to follow."
-- Sly Stone 

[Alan Palazzolo]

Hey Joe,

I really know little about Django, but DJANGO_SETTINGS_MODULE is used
to point to your settings for Django to run. The following worked for
me on the command line:

export DJANGO_SETTINGS_MODULE=billy_settings.py

Though I could not get the billy_update to run after that.

--
Alan Palazzolo
Code for America
2011 Fellow
al...@codeforamerica.org

> --
> You received this message because you are subscribed to the Google Groups
> "Open State Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to fifty-state-pro...@googlegroups.com.
> To post to this group, send email to fifty-sta...@googlegroups.com.
> Visit this group at
> http://groups.google.com/group/fifty-state-project?hl=en.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Joe Germuska]

well, it looks like the Django settings file in the repo is

./site/openstates_site/settings.py

but simply setting that as the DJANGO_SETTINGS environment variable doesn't make it run either.

More to the point, the billy-update script probably shouldn't have a hard dependency on Django. It looks like the pyes lib brings that in, and I don't really know what that library does.

My guess is that everyone who routinely runs billy-update also works on the site, and so this dependency is easy to overlook.

It's also the case that I can actually use the bulk downloads for what I want most, but I figured I'd note it, and Github issues aren't enabled for openstates.

J

"Learn to fear any church that fears drums." --Regie Gibson

[James Turk]

This is definitely an oversight Joe, your guess was correct.  I'll take a look at getting this fixed up shortly.

[Alan Palazzolo]

So, Joe, where you able to get this running locally (just the
billy-update part)?

--
Alan Palazzolo
Code for America
2011 Fellow
al...@codeforamerica.org

[James Turk]

I've pushed an update that makes the pyes import conditional, it is still in requirements.txt but won't get imported and trigger this weird Django error anymore.  I just gave it a try with a clean checkout and it worked.

As to the larger problem here- we definitely need to rewrite that guide and possibly make an alternate requirements-light.txt for people that are just running scrapers.

[Kevin Schaul]

Not sure if this is worth opening a new thread or not, but I'm having issues getting billy to run locally, too.

Billy tells me it can't find the scrapers. I've tried multiple states, but I get an error with the following format:

(openstates)mb60953:openstates schauka$ billy-update mn --scrape

12:05:11 INFO billy: billy-update abbr=mn

    actions=scrape

    types=bills,legislators,votes,committees,alldata,events

    sessions=2013s1

    terms=2013-2014

12:05:11 INFO scrapelib: GET - https://www.revisor.mn.gov/revisor/pages/search_status/status_search.php?body=House

12:05:13 WARNING billy: metadata validation error: Required field 'lower_chamber_name' is missing

12:05:13 CRITICAL billy: Error: no legislators scraper found in module mn.legislators

Let me know if I can do anything to help debug. I'd really like to get this working (and then help update the docs).

Kevin

[James Turk]

This most likely means that the directory containing the states like "~/code/openstates/openstates/co"  isn't on the path for some reason.  Are you using the latest checkout of billy & openstates?

Might make sense to follow up off list as the specifics of your situation are likely unique, but in general this is controlled by the SCRAPER_PATHS setting in billy_settings.py (or your billy_local.py)

-James

--

You received this message because you are subscribed to the Google Groups "Open State Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fifty-state-pro...@googlegroups.com.
To post to this group, send email to fifty-sta...@googlegroups.com.

Visit this group at http://groups.google.com/group/fifty-state-project.