MN Bill scraper

[Alan Palazzolo]

Per Contribution guidelines, I am starting a thread here regarding my pull request which has some minor (but important) updates to the MN bill scraper:

https://github.com/sunlightlabs/openstates/pull/475

[James Turk]

Thanks!  All of those changes made a lot of sense and I actually just reviewed the PR on GitHub and merged it in.

On Tue, Apr 1, 2014 at 11:58 AM, Alan Palazzolo <al...@codeforamerica.org> wrote:

Per Contribution guidelines, I am starting a thread here regarding my pull request which has some minor (but important) updates to the MN bill scraper:

https://github.com/sunlightlabs/openstates/pull/475

--
You received this message because you are subscribed to the Google Groups "Open State Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fifty-state-pro...@googlegroups.com.
To post to this group, send email to fifty-sta...@googlegroups.com.
Visit this group at http://groups.google.com/group/fifty-state-project.
For more options, visit https://groups.google.com/d/optout.

[Alan Palazzolo]

Hey James,

Thanks for the quick reply and turnaround.  I am not seeing the changes in the API yet, but maybe I am missing something or just not being patient enough.

Best,

Alan

[James Turk]

We're actually getting a build error on it,

Traceback (most recent call last):
  File "/mnt/bamboo-ebs/openstates/virt/bin/billy-update", line 9, in <module>
    load_entry_point('billy==1.7.0', 'console_scripts', 'billy-update')()
  File "/mnt/bamboo-ebs/openstates/virt/src/billy/billy/bin/update.py", line 392, in main
    run_record += _run_scraper(stype, args, metadata)
  File "/mnt/bamboo-ebs/openstates/virt/src/billy/billy/bin/update.py", line 101, in _run_scraper
    scraper.scrape(chamber, time)
  File "./openstates/mn/bills.py", line 300, in scrape
    bill_version_url)
  File "./openstates/mn/bills.py", line 213, in get_bill_info
    bill_actions = self.extract_bill_actions(doc, chamber)
  File "./openstates/mn/bills.py", line 131, in extract_bill_actions
    if 'committee:referred' in action_type and committee[0]:
IndexError: list index out of range

I haven't had a chance to dive in yet, but probably just need a check that committee exists there.

--

[Alan Palazzolo]

Hey James,

I actually just saw that as I was going through some other changes. I
should have another pull request soon. Thanks.
--
Alan Palazzolo
Code for America
2011 Fellow
al...@codeforamerica.org
+1 770 596 1951

[Alan Palazzolo]

Fix has been merged. Many thanks, James!

I am not sure if the scraper is still updating and I am just not being
patient, but there seems to be some bills that are not showing the
committee in action fixes. It doesn’t seem to be consistent with
whether it is upper or lower. Locally, all these bills were able to
get parse the committee data. Some examples:

HF 826
http://openstates.org/mn/bills/2013-2014/HF826/
https://www.revisor.mn.gov/bills/bill.php?f=HF826&b=house&y=2014&ssn=0

SF 2726
http://openstates.org/mn/bills/2013-2014/SF2726/
https://www.revisor.mn.gov/bills/bill.php?f=SF2726&b=senate&y=2014&ssn=0

--
Alan Palazzolo
Code for America
2011 Fellow
al...@codeforamerica.org
+1 770 596 1951

[James Turk]

Hm.  From the looks of it the scrape has run successfully, anything being included in the scrape is showing up there.  

(Also maybe we should follow up on this off-list since it is probably noisy for most people)