there is no index plugin api afaik at this point.
start looking around src/mongo/db/storage/.
look at the README.md in that directory.
look at struct IndexKeyEntry and start finding what generates it. that should get you started.
sometimes with unfamiliar code i just do the operation of interest and trace in debugger and it takes me to all the relevant code...
keys are generated in different ways for "regular" values, and for geospatial, so if you find where that branches, that might be a logical place.
dwight