You haven't really overlooked anything. Currently, as you noted in the manual, you get the format in the request and then it's up to you how to handle that. I have thought about this a little and have various competing ideas on how to make this better. I'd love to hear any ideas you have on how you would like to see this expressed.
In the meantime, you could think about using a hook that reroutes all JSON requests to some other route, perhaps related to the original route. Something like that could separate the JSON stuff from the HTML stuff instead of mixing then in the same controller, as you mentioned. I'm not sure if this will work well for your case but something to think about.
Bret