Test your scrapers (plus API wrapper)

[James McKinney]

Hello all,

I created a Ruby gem that wraps the ScraperWiki API. Just `gem install scraperwiki-api` and have a look at the readme for sample usage: https://github.com/opennorth/scraperwiki-api-ruby#the-scraperwiki-api-ruby-gem

I also wrote a bunch of RSpec matchers for operating on the API's output. We have tens of scrapers for my organization's database of Canadian elected officials (http://represent.opennorth.ca/), and I needed a way to validate the scrapers. Have a look at the examples in the readme: https://github.com/opennorth/scraperwiki-api-ruby#scraper-validations

Finally, I sometimes find it very annoying to debug a scraper on ScraperWiki, so I wrote this ScraperWiki mock in Ruby: https://gist.github.com/2759998 Unlike many mocks out there, it actually stores the results. Has anyone found a way to get ScraperWiki's datastore running locally? My mock is currently using Mongo.

I list a few more mocks in both Python and Ruby in this issue: https://bitbucket.org/ScraperWiki/scraperwiki/issue/141/document-the-various-fakerwiki-libraries

Let me know what you think!

Cheers,

--

James McKinney
Open North
+1.514.247.0223

http://opennorth.ca/

http://citizenbudget.com/ interactive budget consultations for municipalities

Twitter: @opennorth

[Chris Blower]

Hi James,

This is really cool - I love the RSpec matchers. Out of interest, what out of our API calls are useful to you?

Cheers,

Chris.

[James McKinney]

I most frequently use datastore/sqlite, and I use scraper/getinfo and scraper/getruninfo to generate my own reports.

I don't use the search API methods, though I do use the search on the website, and I don't use scraper/getuserinfo, as the info I want to know is also returned in scraper/getinfo under "userroles".