If I can give you some advice on this (since actually I was trying to
do the same) just go with 2 different apps. Keep spree for what he is
doing well (handling payments, products, etc etc) and use your other
app by fetching and pulling from the same database. It's easy to share
sessions (actually I do not even need this since the logic for
accessing my 2nd app is way simpler than requiring another
authentication system).
I crushed into some weirdness trying to create an extension for my 2nd
app (it's actually a 1 controller app handling the way a registered
user, who payed via paypal standard can then upload some files for a
photocontest integrating swfupload - i will not go through here about
the massive headaches i had trying to integrate it into the app -).
You can read more on this here:
http://groups.google.com/group/spree-user/browse_thread/thread/b4fb06fbf0d2bb3e/65cd56435ca70a2d#65cd56435ca70a2d
Lately I was trying to include the 2nd app via engines as a plugin,
but I've actually gave up (at least for the moment) since it was
always complaining about
A copy of ApplicationController has been removed from the module tree
but is still active!