There has been one issue  related to privacy filed about this spec
so far. It mainly concerns the fact that visual viewport resize events
expose the height of the on-screen keyboard, though based on the
discussion around that, it looks like browsers currently resize the
layout viewport when showing an on-screen keyboard as well, so that
height is already exposed.
As far as tracking the movement of the visual viewport, in my mind
it's not materially different from tracking the movement of the layout
viewport (e.g. via window.scrollX/scrollY), and indeed prior to
Firefox 63 we would reflect the visual viewport offset in
window.scrollX/scrollY itself (though, granted, at the time
pinch-zooming was only possible on mobile). If you have concerns in
this area, it might make sense to reflect them in the linked spec
issue, or file a new spec issue about them.
One final note: being able to react to visual viewport movements and
size changes is an explicit design goal of this API. I am sensitive to
the fact that there is a tradeoff between user control and developer
control here, and that obnoxious uses of this API (e.g. always keeping
an ad inside the visual viewport) have the potential to degrade user
experience. I think there are also worthwhile uses of this API which
are not currently possible (for example, in some applications the user
may want a particular control to remain inside the visual viewport).
Perhaps this is an area where browser interventions (e.g. a mode where
visual viewport updates are not delivered to the website; compare
Firefox on Android's "Always enable zoom" checkbox, which overrides
user-scalable:no in the meta viewport tag) could come in useful,
depending on how the API ends up being used in the wild.