Dan,
The only three OSS ILS's that I'm aware of are:
- Koha
- Evergreen
- Kuali OLE (still in dev)
Getting MARC records to export and index should (I suspect) be relatively straightforward from all three, so the areas of differentiation may be on how well they present API's for things like getting real time availability information, the ability to request an item or put it on hold, and the ability to present a "My Account" view of patron data via a RoR front end. Alas, I can't comment on any of the three's ability to do this, nor do I know anyone in the BL community whose done it (though perhaps this thread will unearth some).
Hope this helps. Good luck! Sounds like a great prospect,