On languages and configuration

0 views
Skip to first unread message

bouvard

unread,
Aug 15, 2009, 9:12:57 PM8/15/09
to votersdaily
I've just committed a language-agnostic scheduler to the master
repository. I have also migrated per-scraper configuration to a
separate 'config' file that resides in each scrapers directory. This
should make it theoretically possible to support a variety of
languages for scrapers with a minimum of hassle. However, this raises
a quasi-philosophically point that I would like to address: in
documenting the schema for the database rather than the API for the
Python module I invited individuals to use whatever language they felt
comfortable with, however, I can not support every language and I
won't be able to adequately move forward with things if I have try
to. At the same time, I absolutely do not want to fragment the
codebase. So, my thinking regarding languages is that I will accept
pull requests for scrapers written in any language which meets the
following criteria:

1) Can be executed from the command line,
2) without compilation,
3) and with tools readily available on all platforms.
4) Someone must have done the work of writing an EventScraper-
analogue that has reached feature-parity with the Python version.

I think this will prevent me from getting into a situation where I
can't adequately support the project, but still provide plenty of
freedom to accept contributions in PHP, Ruby, Perl, Python, or
whatever else people write their best code in. I'm just glad people
want to help! :-)

One final caveat: I will, _always_, prefer scrapers be written in
Python. If there are two equally valid implementations of the same
scraper in two different languages, I will commit the Python one to
master, followed in order of preference by Ruby.

Thanks!
Chris
Reply all
Reply to author
Forward
0 new messages