vncHooksMoveCursor causes Xorg crash

[znshje]

Some random log out will happen after logging into the desktop environment recently. So I checked the journal, and find Xorg crashed:

arch systemd-coredump[18681]: [🡕] Process 870 (Xorg) of user 0 dumped core.

And I get the backtrace from the core dump file. The backtrace indicates an infinite recursive loop in the vncHooks.c. The detailed backtrace is attached in the end of this message.

Here's my environment:

TigerVNC: 1.15.0

OS: Arch Linux x86_64
Kernel: 6.12.13-1-lts
Resolution: 1920x1080, 2560x1440
DE: Plasma 6.3.0
WM: KWin
CPU: Intel i9-10850K (20) @ 5.200GHz
GPU: NVIDIA GeForce RTX 3090

Here's some debug information from the core dump file.

$ sudo coredumpctl -1 gdb Xorg  
          PID: 870 (Xorg)
          UID: 0 (root)
          GID: 0 (root)
       Signal: 11 (SEGV)
    Timestamp: Tue 2025-02-18 21:41:07 CST (46s ago)
 Command Line: /usr/lib/Xorg -nolisten tcp -background none -seat seat0 vt2 -auth /run/sddm/xauth_YbRQZY -noreset -displayfd 16
   Executable: /usr/lib/Xorg
Control Group: /system.slice/sddm.service
         Unit: sddm.service
        Slice: system.slice
      Boot ID: 8b430398b4334d93a9b8ce7f57718ee2
   Machine ID: 546e9c9b93f84da482914cbbf6aa1697
     Hostname: arch
      Storage: /var/lib/systemd/coredump/core.Xorg.0.8b430398b4334d93a9b8ce7f57718ee2.870.1739886067000000.zst (present)
 Size on Disk: 10.8M
      Message: Process 870 (Xorg) of user 0 dumped core.

GNU gdb (GDB) 16.2
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-pc-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
   <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/Xorg...
Reading symbols from /usr/lib/debug/usr/lib/Xorg.debug...

warning: Can't open file /memfd:xorg (deleted) during file-backed mapping note processing

warning: Can't open file /SYSV00000000 (deleted) during file-backed mapping note processing

warning: Can't open file /memfd:/.nvidia_drv.XXXXXX (deleted) during file-backed mapping note processing
[New LWP 870]
[New LWP 905]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
Core was generated by `/usr/lib/Xorg -nolisten tcp -background none -seat seat0 vt2 -auth /run/sddm/xauth_YbRQZY -noreset -displayfd 16'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  vncHooksMoveCursor (dev=0x5ac63f561dd0, screen=0x5ac63ed5e590, x=1367, y=992) at /usr/src/debug/tigervnc/tigervnc/unix/xserver/hw/vnc/vncHooks.c:1265
1265      (*miPointerPriv->spriteFuncs->MoveCursor)(dev, screen, x, y);
[Current thread is 1 (Thread 0x7b93247b2a00 (LWP 870))]
(gdb) bt

#0  vncHooksMoveCursor (dev=0x5ac63f561dd0, screen=0x5ac63ed5e590, x=1367, y=992) at /usr/src/debug/tigervnc/tigervnc/unix/xserver/hw/vnc/vncHooks.c:1265
#1  0x00007b93246d8949 in vncHooksMoveCursor (dev=<optimized out>, screen=<optimized out>, x=<optimized out>, y=<optimized out>) at /usr/src/debug/tigervnc/tigervnc/unix/xserver/hw/vnc/vncHooks.c:1265
#2  0x00007b93246d8949 in vncHooksMoveCursor (dev=<optimized out>, screen=<optimized out>, x=<optimized out>, y=<optimized out>) at /usr/src/debug/tigervnc/tigervnc/unix/xserver/hw/vnc/vncHooks.c:1265
#3  0x00007b93246d8949 in vncHooksMoveCursor (dev=<optimized out>, screen=<optimized out>, x=<optimized out>, y=<optimized out>) at /usr/src/debug/tigervnc/tigervnc/unix/xserver/hw/vnc/vncHooks.c:1265

...... (same lines)

#261899 0x00007b93246d8949 in vncHooksMoveCursor (dev=<optimized out>, screen=<optimized out>, x=<optimized out>, y=<optimized out>) at /usr/src/debug/tigervnc/tigervnc/unix/xserver/hw/vnc/vncHooks.c:1265
#261900 0x00005ac5feab94a9 in miPointerUpdateSprite (pDev=0x5ac63f561dd0) at ../xorg-server/mi/mipointer.c:431
#261901 mieqProcessInputEvents () at ../xorg-server/mi/mieq.c:571
#261902 0x00005ac5febe09b3 in ProcessInputEvents () at ../xorg-server/hw/xfree86/common/xf86Events.c:143
#261903 0x00005ac5feaa8058 in Dispatch () at ../xorg-server/dix/dispatch.c:508
#261904 dix_main (argc=13, argv=0x7ffdffceb048, envp=<optimized out>) at ../xorg-server/dix/main.c:272
#261905 main (argc=13, argv=0x7ffdffceb048, envp=<optimized out>) at ../xorg-server/dix/stubmain.c:34

[Pierre Ossman]

On 18/02/2025 14:58, znshje wrote:
> Some random log out will happen after logging into the desktop environment
> recently. So I checked the journal, and find Xorg crashed:
> arch systemd-coredump[18681]: [🡕] Process 870 (Xorg) of user 0 dumped core.
>
> And I get the backtrace from the core dump file. The backtrace indicates an
> infinite recursive loop in the vncHooks.c. The detailed backtrace is
> attached in the end of this message.
>

Thanks for the report. Any idea on what triggered this?

Regards,
--
Pierre Ossman Software Development
Cendio AB https://cendio.com
Teknikringen 8 https://twitter.com/ThinLinc
583 30 Linköping https://facebook.com/ThinLinc
Phone: +46-13-214600

A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?

[Pierre Ossman]

On 18/02/2025 17:03, znshje wrote:
>
> To avoid this issue, I have removed this config file to see if it still
> happens. This issue seems happening randomly at any time, and I didn't find
> an operation that could stably reproduce the issue.
>
> If you need more details, I'm willing to help to locate the problem.
>

I'm afraid I don't see anything in the code that can explain this. It
might have something to do with how it's used as an Xorg module. Most of
the testing is done in Xvnc.

It will be difficult for us to fix this without clear steps to reproduce it.

If you can do custom builds, then please test the following patch. It
will still crash, but the crash should tell us more where things go wrong.

diff --git a/unix/xserver/hw/vnc/vncHooks.c b/unix/xserver/hw/vnc/vncHooks.c
index 406452738..ad3f5f253 100644
--- a/unix/xserver/hw/vnc/vncHooks.c
+++ b/unix/xserver/hw/vnc/vncHooks.c
@@ -21,6 +21,7 @@
#include <dix-config.h>
#endif

+#include <assert.h>
#include <stdio.h>

#include "vncHooks.h"
@@ -87,6 +88,7 @@ typedef struct _vncHooksGCRec {
} vncHooksGCRec, *vncHooksGCPtr;

#define wrap(priv, real, mem, func) {\
+ assert(real->mem != func); \
priv->mem = real->mem; \
real->mem = func; \

[Noah Friedman]

This bug has been around for a couple of years now.  Automated regression testing of the changes the current developers make ought to be the one of their responsibilities, but they don't seem to test anything or have much regard for anyone with the constant user-visible and incompatible changes they keep introducing on top of that.  They just blink in surprise and expect you to outline for them everything they're doing that causes problems and then they tell you that your use cases aren't important to them. The only recourse is to fork the code, but frankly it would have been better if they'd just forked it themselves and left the implementation that all the distributions depend on alone.