setState is the mechanism by which widgets are marked dirty, so once you call setState, it is assumed that it needs it. If you return a different widget from your `build` method, then that widget is also assumed to be dirty, since it is not the same widget as before.
One way you can make builds more efficient is by making sure to return the same widget from your build method. We use this a lot, for example, that's why `const` widgets are more efficient (they're always the same instance). It's also why AnimatedBuilder takes a `child` argument and a `builder` argument, and then passes the `child` to the builder: the builder can then make sure to always use the same child instance in its output each frame, thus limiting the parts that are dirtied (and rebuilt) to the parts that changed (the parts created by the builder itself).
HTH.
(I'm from the Flutter team.)