can I work on any of these states?

[Dane]

Hi there,

I just noticed you all had a git repo and lots of code! Wow.
So...looking at the map I see that these three states aren't too far
along:

- Idaho
- South Carolina
- Missouri

For personal reasons I'd love to tackle some of these...I did notice
someone was poking around the Idaho state just the other day (in the
python project's openstates/id folder). I don't want to step on
anybody else that is working on stuff...can I work on any of these
states?

I'm guessing I just fork on github and commit that way?

Thanks!

Dane

[James Turk]

I know a there are some people are working on Idaho and South Carolina
right now. Missouri would be great though and you'd be free to work
without needing to worry about collisions.

Fork on github and let us know here when you have something that you'd
like us to test/merge.

Also please do ask for help if you run into any difficulties, and
thanks for contributing!

-James

> --
> You received this message because you are subscribed to the Google Groups "Open State Project" group.
> To post to this group, send email to fifty-sta...@googlegroups.com.
> To unsubscribe from this group, send email to fifty-state-pro...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/fifty-state-project?hl=en.
>
>

[Tami Crawford]

Hi,

I've done the bills and votes for South Carolina and submitted a pull request.  

It's ready to be reviewed so I can make whatever corrections/updates etc are needed.   

Tami

On Wed, Jun 22, 2011 at 11:25 PM, Dane <dsum...@gmail.com> wrote:

--
You received this message because you are subscribed to the Google Groups "Open State Project" group.
To post to this group, send email to fifty-sta...@googlegroups.com.
To unsubscribe from this group, send email to fifty-state-pro...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/fifty-state-project?hl=en.

--
tami.c...@gmail.com

[Dane]

Thanks,

I'll start on MO then.

[Dane]

Quick question...I don't see any unit-testing or testing of any kind
baked into this project. I've done some scraping work before and found
that pulling reference html/excel/pdf (whatever you're scraping) docs
and then testing against those is helpful...particularly when the
'live site' can change w/o anyone being aware. You can then at least
compare your scraping script against documents you know that it works
with vs the documents it no longer works with (compare the documents,
etc).

Is this something that I should not waste time on (ie, any testing
code I make wouldn't be pulled into the main project anyway)?
Personally I'd prefer to start by setting up my unit tests...but I
don't want to bother if its not something that would be utilized going
forward...

Thanks!

[James Turk]

We have tests on our core (billy/tests) but you're correct that on
individual scrapers we don't have tests. A big part of the reason for
this is that we try and design scrapers to break if the page changes
(rely on strict xpath/css rules over assuming that the 3rd div will
always be what you're after)

If you can come up with some tests that'd be useful to you that'd be
great, but in practice the real test is the nightly run. Once a state
is in production we (and the original author if they desire) will be
notified whenever something breaks:
http://openstates.sunlightlabs.com/jenkins/

(Also try running with --strict, this enforces our validation rules
which help avoid common mistakes)

-James

[Tami Crawford]

Hi,

I was also wondering about the tests for the scrapers and think it would be useful.

Even if the pages don't change completely, an edge case that wasn't handled properly or some new verbiage could be added that would cause things not to work.  In that case, the tests would not only catch it, but would also make it much easier to correct the problem, as well as help a new person come on board.

Tami

--
tami.c...@gmail.com

[Dane]

Quick update - I spoke too soon. There doesn't apper to be any
testing for the scrapers, but there is for the 'billy' project
does(I'm guessing this is the web front end and ultimate manager of
scraping results?).

Its using nose. I figure if we do some testing we should probably
follow suit. Make a tests/ folder in our State's scraper folder and
put nose tests there?

Thanks again...

On Jun 23, 9:57 am, Tami Crawford <tami.crawf...@gmail.com> wrote:
> Hi,
>
> I was also wondering about the tests for the scrapers and think it would be
> useful.
> Even if the pages don't change completely, an edge case that wasn't handled
> properly or some new verbiage could be added that would cause things not to
> work.  In that case, the tests would not only catch it, but would also make
> it much easier to correct the problem, as well as help a new person come on
> board.
>
> Tami
>
> On Thu, Jun 23, 2011 at 9:49 AM, James Turk <jt...@sunlightfoundation.com>wrote:
>
>
>
>
>
>
>
>
>
> > We have tests on our core (billy/tests) but you're correct that on
> > individual scrapers we don't have tests.  A big part of the reason for
> > this is that we try and design scrapers to break if the page changes
> > (rely on strict xpath/css rules over assuming that the 3rd div will
> > always be what you're after)
>
> > If you can come up with some tests that'd be useful to you that'd be
> > great, but in practice the real test is the nightly run.  Once a state
> > is in production we (and the original author if they desire) will be
> > notified whenever something breaks:
> >http://openstates.sunlightlabs.com/jenkins/
>
> > (Also try running with --strict, this enforces our validation rules
> > which help avoid common mistakes)
>
> > -James
>

> tami.crawf...@gmail.com

[James Turk]

Yes, we'd welcome tests for any state in the tests folder within that state.

Thanks Dane & Tami,

James