I think that certainly depends on your needs.
In my experiences, at the first time I used to put Windsor config in app.config because I read that I can configure things without recompiling app, just like what you said. But then, after a period of experience and evaluation, I observed when I changed the config, I also had to modify the code. I haven't found a scenario where I can change to config only without recompile app. Also, configuring things in app.config is bit pain in the ass since I don't have compile-time checking, especially when it involves generic types configuration. Based on those reasons, I decided to move all config from file to code. And importantly, I feel better now :D.
--
Regards,
Maximilian Haru Raditya