Well, I did personally encounter this issue more than a couple times on several projects. In most cases simply switching to raw id or read only where required works fine however it does add additional maintenance overhead for projects that started small and at some point database grows and some admin pages stop working. I even had to write some custom code once when a client wanted to keep the default select interface but there were too many of those selects with the same options for some objects (different number of inlines IIRC), I basically added some js code to make sure the generation and transmission of the list only takes place once. Ah, another part of the problem was that by default every single object is iterated and passed to force_text (I think) and that was costly and there wasn't an obvious way to use, like, a combination of model fields instead. Maybe some internals got changed recently and it got better though, I didn't review these parts of django code in a while.
That being said, I do see a value in using the default select sometimes (aforementioned small configuration tables are a good example) and I did see a lot of frustration when people see otherwise nice interface failing miserably just because some new user accounts or other data was added. I'm not sure forcing raw id is the right way to handle this but doing something that works by default (i.e. without you having to write a custom admin class or even form) regardless of table size would be a good idea. How about some field that intelligently guesses the table size and looks either like default select or like raw id field depending on that? A COUNT(*) query is usually much less costly than fetching the whole table and doing things with it. Additional (and simple enough) optimization could be done by making sure options for the same queryset on multiple fields or formset forms are only generated once, maybe we could cache them somewhere in the modelform/formset objects?
Also, although it might be obvious and even documented somewhere that __str__() method shouldn't perform additional db queries if you expect to use string coercion on a bunch of objects (actually I don't know if it is documented at all, I just hope so, correct me if I'm wrong), maybe some simple way to override string coercion method for the options and in similar places where a bunch of objects has to be converted to a list of strings could be a good idea (possibly on model/queryset level, some attribute or method that modelform would pick up and use to make lighter values() query instead of the default fetching, instantiation, coercion). I'm not sure what particular solution would be the best here though or I would've proposed one already, but it definitely shouldn't however be complicated or require much customization (if you need something complicated and customized you can always customize forms and fields and it stops being a problem), it should be something simple and effective for when you prefer to use the default classes wherever possible.
Ivan.