It may be surprising but (if I understand you correctly) it's intended behavior. Switching tabs does just show and hide the tab's content, BUT, tabs that are not shown don't have their contents generated until they are shown. In other words, if a tab's output is invalidated while the tab is not shown, then that output will not be recalculated, not until the tab is shown. But once the tab is shown, then switching away from and back to that tab shouldn't cause any recalculation (unless, again, the output becomes invaliated in the meantime).
If you don't like this behavior, you can opt out of it on a per-output basis using the outputOptions function's suspendWhenHidden argument.