Traversal actually always happens. If a route is matched then traversal happens on the factory and path specified on the route. If unspecified, it will use the default root factory and the path "/". Thus under default conditions the context for a matched route is just the root of the default tree.
With this in mind you might have your root factory return a different tree if a route is matched (you could check for the presence of request.matched_route). This tree would not use your custom traverser.
You can always turn on pyramid.debug_routematch and other debugging to see what is going on as well.
Without seeing some code I can't help you diagnose "my routes don't work" but maybe now you have enough info to figure it out or paste something more useful for us.