Yes, the issue is that the grid (or list) isn't an even multiple of the viewport size. The page physics will compute an animation for you that takes you to the page boundary, which ends up overscrolling in this case. When that animation ends, the "bounce" physics will create an animation to remove the overscroll.
The PageView widget avoids this problem by always having content that's an exact multiple of the viewport. It does this by using a SliverFillViewport widget, which fills the viewport.
Two approaches for fixing this behavior jump to mind, but both are fairly advanced:
1) You can create your own version of BouncingScrollPhysics (perhaps by subclassing the existing BouncingScrollPhysics) that bounces to even multiples of the viewport rather than to zero overscroll.
2) You can create a sliver that takes the SliverList or SliverGrid child and rounds its scroll extent up to a multiple of the viewport. You'd then use a CustomScrollView widget to use both this new sliver and the SliverList/SliverGrid.
My guess is that (2) will give you the best results (and would be generically re-usable), but it will involve learning a bit about how slivers work:
Adam