Yes, but not automatically. You either need to write a custom
interpolator (e.g., attrTween) or use a polygon with many intermediate
points. An example of the latter approach is here:
For a custom interpolator, you'd probably use SVG's elliptical arc
path commands, describing a rectangle with four arc segments. These
arc segments would have a very large radius, making them nearly
straight. Then, you'd interpolate to a circle consisting of the same
four arc segments, with the radius of each arc equal to the radius of
the circle. You could also use Bézier curves to approximate this
(which would allow the rectangle to be perfectly straight, but the
circle would only be approximate).
Mike