Here's how my Logitech mouse behaves, for reference:
On Linux, the forward and back buttons map to Buttons 6 and 7 in Java
and Buttons 8 and 9 in X11, and the left and right wheel clicks map to
Buttons 4 and 5 in Java and Buttons 6 and 7 in X11. That makes sense,
because X11 uses Buttons 4 and 5 for scroll wheel up and down, but Java
uses a different type of event (MouseWheelEvent rather than MouseEvent)
for scroll wheel up and down, so it doesn't have to use Buttons 4 and 5
for that.
On macOS and Windows, the forward and back buttons map to Buttons 4 and
5 in Java, and the left and right wheel clicks produce a MouseWheelEvent
instead of a MouseEvent (i.e. they are indistinguishable, as far as Java
is concerned, from scroll wheel up and down.)
If you observe the same thing I observe with Java, then the button
mapping needs to be made contingent upon the client O/S:
X11 client
----------
Java Button 4 --> RFB Button 6
Java Button 5 --> RFB Button 7
Java Button 6 --> RFB Button 8
Java Button 7 --> RFB Button 9
macOS or Windows client
-----------------------
Java Button 4 --> RFB Button 8
Java Button 5 --> RFB Button 9
Double check which button numbers are used when you plug your mouse
directly into a Linux machine. If the forward and back buttons
correspond to X11 Buttons 8 and 9, then that is consistent with what I
observe, and those are the button numbers you should use for the forward
and back button RFB events. I would suggest testing the three major
client O/S's, as I did above, to see if your mouse behaves similarly
across those platforms. Also figure out what maps to Buttons 4 and 5 in
Java.
My viewer uses only Java, so I didn't test FLTK.