Yeah, this sucks. I've hit this issue myself, and I worked around it by wrapping the geo query, e.g.:
var geo = new SolrQueryByDistance(...)
var geoq = new SolrQuery(geo.Query) { Quoted = false }
IIRC there were issues with making it inherit AbstractSolrQuery because it's actually a localparam, but I can't remember what they were concretely :-(
It would be great if you could test a query on Solr like "{!geofilt ...} AND some_other_query", and "-{!geofilt ...}", if those work correctly I think it should be safe to make it inherit AbstractSolrQuery.
Cheers