Ok, I think I have figured it out myself.
My confusion was around the way I thought the transitionBuilder was used. I thought it was only called when the item became the current item and you were then stuck with the animation that you wrapped it with then. But whenever the state changes transitionBuilder is called on the new item and on the previous item.
All you have to do is in the transitionBuilder is look to see if the child passed in represents the current item or not (most likely by looking at the key for the widget passed to the transition builder to see if it matches your current state) and then wrapping it with the animation you want.
For example taking the example code from the AnimatedSwitcher example code from the docs and changing the transitionBuilder parameter to this:
transitionBuilder: (Widget child, Animation<double> animation) {
if (child.key == ValueKey(_count)) {
return new ScaleTransition(child: child, scale: animation);
} else {
return new FadeTransition(child: child, opacity: animation);
}
},
would cause the previous value to fade away and the new value to scale up from 0 to full size.
To accomplish what I mentioned about only animating the new item, just have the else return child.