Decoding phase

[Nooshin Eghbal]

Hello,

I learned that for each rectangle of a frameBuffer update, we can use a different encoding method. 

I have two questions:

1) Do we need all rectangles of an update to be able to update the screen or we can update gradually as the rectangles arrive?

2) for each encoded rectangle, can we start decoding while receiving the parts or we much receive all parts to be able to start decoding?

Thanks,
Nooshin

[Pierre Ossman]

On 14/06/2024 13:26, Nooshin Eghbal wrote:
>
>
> Hello,
>
> I learned that for each rectangle of a frameBuffer update, we can use a
> different encoding method.
>
> I have two questions:
>
> 1) Do we need all rectangles of an update to be able to update the screen
> or we can update gradually as the rectangles arrive?
>

Yes and no. We try to wait for all rectangles to avoid giving a glitchy
experience for the user. Similar to how many applications wait for vsync
before drawing.

If it takes too long, though, we'll output things gradually.

> 2) for each encoded rectangle, can we start decoding while receiving the
> parts or we much receive all parts to be able to start decoding?
>

We decode them as they come in, so no waiting. We wait for a full
rectangle, though.

Regards
--
Pierre Ossman Software Development
Cendio AB https://cendio.com
Teknikringen 8 https://twitter.com/ThinLinc
583 30 Linköping https://facebook.com/ThinLinc
Phone: +46-13-214600

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[DRC]

On 6/14/24 7:26 AM, Nooshin Eghbal wrote:
> I learned that for each rectangle of a frameBuffer update, we can use
> a different encoding method.

Technically, yes.  The client-to-server SetEncodings message specifies
the encodings that the VNC viewer supports, with the preferred encoding
listed first.  Most VNC server implementations send all framebuffer
update (FBU) rectangles using the preferred encoding and do not change
the encoding until the VNC viewer sends a new SetEncodings message, but
the VNC server is not actually required to honor the preferred
encoding.  It is legal for it to use any encoding that the VNC viewer
supports.

https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#setencodings

> I have two questions:
>
> 1) Do we need all rectangles of an update to be able to update the
> screen or we can update gradually as the rectangles arrive?
>

You can update the screen incrementally as each rectangle arrives.  In
fact, that is how the TightVNC Viewer used to work. (TigerVNC, TurboVNC,
and LibVNCServer spun off, albeit in different ways, from TightVNC.) 
Back then, the TightVNC Viewer used raw X11 drawing commands on Un*x and
raw Win32 GDI commands on Windows, so blitting rectangles to the screen
was effectively single-buffered.  That was OK for raw X11 applications,
since X11 (in its raw form) is a single-buffered rendering system. 
However, most OpenGL applications use double buffering, so the TightVNC
Viewer implementation introduced unwanted flicker when using those
applications with VirtualGL.  To reduce the flicker, I implemented a
pseudo-double buffering mechanism whereby the TurboVNC Viewer decoded
all of the FBU rectangles into a linked list and, when the last FBU
rectangle had been received, blitted all of the rectangles from the
linked list to the screen at once.  These days, VNC viewers typically
use a true double buffering mechanism, decoding all of the rectangles
into the back buffer and swapping the buffers when the last rectangle in
the update has been received.  However, the RFB spec doesn't require
that.  You can go back to the original single-buffered approach if you want.

> 2) for each encoded rectangle, can we start decoding while receiving
> the parts or we much receive all parts to be able to start decoding?
>

You don't have to wait until the entire rectangle has been received. 
Generally speaking, as soon as you have enough data to reliably draw
something to the screen, you can draw it to the screen (bearing in mind
that it may create unwanted flicker, per above.)  The Tight encoding
splits FBU rectangles into "subrectangles" with different subencoding
types, such as JPEG or Raw or zlib-compressed indexed color.  When using
the Tight encoding, the original TightVNC Viewer blitted each individual
subrectangle as it was received and decoded, so it didn't even wait for
the entire FBU rectangle.  In fact, looking at the (ancient) code, it
also appears that it didn't even wait for an entire zlib-compressed
indexed color subrectangle.  It had a zlib buffer of a fixed size that
it filled from the network, then it decoded and blitted the pixels from
that buffer before going back to fill it again.  However, with JPEG
subrectangles, you would generally need to wait until the entire JPEG
image has been received.  That's because the Tight encoding typically
uses baseline (single-scan) JPEG images.  However, unless I am missing
something, nothing in the RFB spec requires JPEG images to be
single-scan.  The spec only requires them to be JFIF, so I guess you
could technically use progressive (multi-scan) JPEG images, which would
allow you to send and decode the lowest image frequencies first.  That
would effectively allow you to draw a pixelated version of the rectangle
or subrectangle and progressively refine it as higher-frequency scans
are received.  If the server used baseline JPEG images and compressed
them with restart markers, then technically you could also decompress
the image up to the next restart marker, draw what you have, then go
back to decompress up to the next restart marker.  Unless I am missing
something, the RFB spec doesn't forbid that.  However, it also doesn't
require it, so your implementation would have to retain compatibility
with other implementations that are more coarse-grained.

[Nooshin Eghbal]

Hello,

Thanks for your detailed response.

If the rectangles can be shown independently,  I am interested in capturing frameBuffer updates (sent from the server to client) with separate rectangles within each update to test some of my ideas.

I have looked into rfbproxy software to record updates but it is not clear the rectangles from the output file.

Any suggestion to record rectangles?

For example, should I write the encoded rectangles at the server side in a file before submitting them to the client?

Bests,

Nooshin

[DRC]

That would probably be the best approach.  rfbproxy is quite old and doesn't support any of the latest encodings.

--
You received this message because you are subscribed to the Google Groups "TigerVNC Developer Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tigervnc-deve...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tigervnc-devel/534a2b9a-1d5f-4f62-9a46-c0ffb7b64851n%40googlegroups.com.