Personally I look at it the other way around: things are clearer to me when I can see the point at which all configuration values are assembled and then passed down from parent to child, the same you would do with a purely functional program that can only operate on it's inputs. All I generally want to use config.exs for is global, static config (i.e. things which are only likely to change at compile-time, not runtime). Default values should be part of the application code, and expect to be overridden as necessary, rather than using config.exs/application env for that kind of thing. This keeps configuration closer to the code that needs it, and encourages a more functional approach to your design.
Because of that functional approach, it becomes considerably easier to test an Elixir program when you can start a supervision tree, a branch of it, or a leaf (a single process) by simply passing in the configuration it needs, which is going to be part of it's interface. Taking it a step further and allowing one to configure the names of things allows you to start many instances of what would normally be a singleton process, and test many scenarios in parallel - this is particularly handy with stateful things where you need to test different conditions which might conflict with other in-progress tests.
The more libraries that adopt this approach to configuration, the easier it gets to configure and test your whole system. The config providers framework I've made for Distillery is intended to be a way to deal with legacy applications, as well as providing a mechanism for applications which use other mechanisms for configuration (i.e. init-style configs, etcd/Consul, and so on) - but it is not intended to make it easy to put everything in config.exs/application env; I still strongly believe that we need to move away from that as much as possible, not out of ideological purity (which I hope is refuted by the config providers work), but out of knowing that if we can move away from it as a community, things will get much better for all of us as a whole.