Sub Dependency Resolvers can resolve lower level parameters that may not be registered in Windsor (like a connection string). Handler Selectors are used when you have a dependency that (usually) has more than one implementation registered and the IHandlerSelector can "help" Windsor decide which one to use. So:
ISubDependencyResolver - Resolves a dependency which is outside of Windsor
IHandlerSelector - Help Windsor which registered dependency to use
I wrote an article a few years ago for Visual Studio Magazine that gave examples of ISubDepedencyResolver and IHandlerSelector: