I'm assuming you got the x and y positions from some object on page number page_num.
The first thing to look at is the scroll position. x and y after converting is in screen space, while SetX/YScrollPos is in the coordinates of the scroll viewer. That is, the top left corner of the PDFViewCtrl is at position (0, 0) in screen space, but at position (GetHScrollPos and GetVScrollPos) in scroll coordinates. I would suggest trying
_viewer.SetHScrollPois(x + _view.GetHScrollPos());
_viewer.SetVScrollPois(y + _view.GetVScrollPos());
If that does not work, could you elaborate a little more on what you are doing?