Debugging font loading in X?

[Brian Cowan]

I have a real poser of a question, all because I'm stuck supporting a Motif application that seems to hide what it's doing deep in the bowels of viewkit...

My user is using a Red Hat Linux 6.10 host, and accessing the GUI via xrdp, which uses TigerVNC... On ONE and ONLY one host, the TigerVNC X server stalls 30 seconds on the first call to XLoadQUeryFont() and returns NULL (meaning that whatever it was looking for couldn't be loaded), a subsequent call then returns a non-null value (found and loaded font)...

I posted this on the user forum, and I'm not seeing a lot of traction... (Wrong audience?) What I'm looking for isn't a solution, but how to get visibility into what's being sent into X. If I can find THAT, maybe I can figure out why the X server gets stuck, and from there get the app and X to play nice together. It's either this or go spelunking within Viewkit, because the app I'm debugging literally never calls XLoadQueryFont() on it's own. But I have a sad feeling I may wind up deep in the Motif source code...

Is there something I can do to get X to tell me what's being asked of it that will actually work? I've seen other XWindows "trace" tools that interpose a display between the caller and an actual display, but those don't seem to want to build because my Linux is too new. (Seriously...).

[Brian Hinz]

> On ONE and ONLY one host,

I assume that you've already tried to determine what's different about this particular host than the others that work?

--
You received this message because you are subscribed to the Google Groups "TigerVNC Developer Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tigervnc-deve...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tigervnc-devel/a475ec5e-ef9e-40b4-8a79-78ee8b8fb5c4n%40googlegroups.com.

[Brian Cowan]

Yes, and a supposedly identical system I built doesn't do it. I'm desperate enough to ltrace the blasted X server. Which, of course, makes things sooooo much slower. I know that my test system does 2127 compares (strncmp) against x font strings, and a fun number of strrchr's against those strings as well. Need to know what the other one is doing since the user was trying to fix a DIFFERENT font issue before I got dragged in...

[Brian Cowan]

Not that the ltrace of the X server helped anything. I see it doing a whopping 2% of the string compares and opens that my system is doing. So, either the ltrace was done against the wrong X server (Possible since the X server is on a shared host) or I need to re-verify the version of TigerVNC the host is using. Joyful... Is there NOTHING I can do to make this easier to debug?

[Brian Cowan]

Actually, it turned out to be QUITE helpful, in finding yet another "curveball..." I found it parsing a string "tcp/myhost:7100"... 

"Oh, that's the X11 font server that runs on the local host to serve fonts for the application we develop."

Can anyone point me to a document telling me how to tell an X server to use a font server? It's no longer officially supported/allowed on RHEL 6+, of course...And my Google-Fu seems to be failing at the moment... I'm already having the customer point me to the package used to build said X Font Server...

I think, that once I get a font server installed, I can create a "FontPath.conf file in /etc/X11/xorg.conf.d and add this to it:

Section "Files" 

     FontPath "tcp/xxx.xxx.xxx.xxx:7100" 

EndSection

Does that sound/look about right? I'm stealing the instructions more or less from This bug report

Remote support is a pain some times... The host in question is 1000 miles away, in a "secure" (isolated) data center, operated by the (US Government) customer. I don't have any access, and the internet-capable machine the customer uses isn't allowed to connect to the network with the problem host. Then, with a non-standard (and unsupported by RH) font resolution/rendering configuration, we get weird results...

[Brian Hinz]

Have a look at the "-fp" option to Xvnc.  IIRC, you can pass it that string ("tcp/localhost:7100"), but you'll probably want to include the default font paths as well

--

You received this message because you are subscribed to the Google Groups "TigerVNC Developer Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tigervnc-deve...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tigervnc-devel/71c1f3ca-a75b-47a6-97ae-c78ca4119048n%40googlegroups.com.

[Brian Cowan]

And... There it is in the ps -elf collected earlier... "-fp /usr/share/X11/fonts/misc/,/usr/share/X11/fonts/Type1/,/usr/share/X11/fonts/75dpi/,/usr/share/X11/fonts/100dpi/,/usr/share/X11/fonts/TTF/,/opt/SharedData/fonts/,tcp/thishost:7100"  So I need to take a look @ the xrdp-sesman.ini file... And then need to install the CENTOS 5 xfs rpm that I pulled off archive.centos.org...

[Brian Cowan]

I got xfs installed, and edited the default font path to 

1. include all the directories in /etc/X11/fontpath.d

2. comment out no-listen=tcp

3. set the port.

I then killed iptables and ip6tables (to eliminate firewall issues)

I also added -fp ...,tcp/thishost:7100 (with some of the default font paths)...

Managed to hang my app. But unfortunately not in the same place, and it looked like it may not come back for hours.... I'll have to look at the setup later and see what I missed... 

[Brian Cowan]

And, the plot sickens... I managed to get the font server running, and listening to TCP connections, and all heck broke loose. My motif-based application went from taking < 1 second to start up to (I lost count after) over 10 minutes. This is with a more-or-less default configuration for everything. The only things I did were:

1. Made xrdp use rdp security (100% CPU usage issue)

2. Made the xfsd process listen for TCP connections and turned on lazy rendering.
3. added tcp/localhost:7100 to the -fp parameter.

4. Shut down the iptables and ip6tables firewalls to avoid that issue since I'm using a VirtualBox VM on a "Nat network" and only SSH and RDP ports are transferred.

I connect, start my application (the legacy ClearCase Unix diff/merge tool) and ... Nothing... For. 10. Minutes... No Font server, no problem, add font server, BIG problem.

What the heck am I missing?

TigerVNC version:

Xvnc TigerVNC 1.10.0 - built Dec 20 2019 07:12:07

Copyright (C) 1999-2019 TigerVNC Team and many others (see README.rst)

See https://www.tigervnc.org for information on TigerVNC.

Underlying X server release 12001000, The X.Org Foundation

xrdp.ini:

[globals]

bitmap_cache=yes

bitmap_compression=yes

port=3389

crypt_level=high

#channel_code=1

max_bpp=24

#black=000000

#grey=d6d3ce

#dark_grey=808080

#blue=08246b

#dark_blue=08246b

#white=ffffff

#red=ff0000

#green=00ff00

#background=626c72

[xrdp1]

name=sesman-Xvnc

lib=libvnc.so

username=ask

password=ask

ip=127.0.0.1

port=-1

security_layer=rdp

font server config:

#

# xfs font server configuration file

#

# allow a max of 10 clients to connect to this font server

client-limit = 10

# when a font server reaches its limit, start up a new one

clone-self = on

# alternate font servers for clients to use

#alternate-servers = foo:7101,bar:7102

# where to look for fonts

catalogue = /usr/share/X11/fonts/misc:unscaled,

            /usr/share/X11/fonts/75dpi:unscaled,

            /usr/share/X11/fonts/100dpi:unscaled,

            /usr/share/X11/fonts/Type1,

            /usr/share/fonts/default/Type1,

# in 12 points, decipoints

default-point-size = 120

# 75 x 75 and 100 x 100

default-resolutions = 75,75,100,100

# use lazy loading on 16 bit fonts

#deferglyphs = 16

deferglyphs = all

# Log errors via syslog.

use-syslog = on

# For security, don't listen to TCP ports by default.

#no-listen = tcp

sesman.ini

[Globals]

ListenAddress=127.0.0.1

ListenPort=3350

EnableUserWindowManager=1

UserWindowManager=startwm.sh

DefaultWindowManager=startwm.sh

[Security]

AllowRootLogin=1

MaxLoginRetry=4

TerminalServerUsers=tsusers

TerminalServerAdmins=tsadmins

[Sessions]

X11DisplayOffset=10

MaxSessions=10

KillDisconnected=0

IdleTimeLimit=0

DisconnectedTimeLimit=0

[Logging]

LogFile=/var/log/xrdp-sesman.log

LogLevel=DEBUG

EnableSyslog=1

SyslogLevel=DEBUG

[X11rdp]

param1=-bs

param2=-nolisten

param3=tcp

param4=-logverbose 10

[Xvnc]

param1=-bs

param2=-nolisten

param3=tcp

param4=-localhost

param5=-dpi

param6=96

param7=-Log

param8=*:syslog:100

param9=-fp

param10=/usr/share/X11/fonts/misc,/usr/share/X11/fonts/75dpi:unscaled,/usr/share/X11/fonts/100dpi:unscaled,tcp/localhost:7100

[Brian Hinz]

I still think you should go back to the original discrepancy - that there was only one problematic host - and figure out what is different about that one compared to the others.  For example, why did you have to install xfs on that one and not on any others?  To me, that strongly suggests a difference in the machine configurations.  Have you tried doing things like 'rpm -qa | sort > host_a.txt' and then diff that to the same report for a working machine to verify that they both have the same base set of packages installed?  You can google to find ways to locate and list all files that weren't installed via rpm/yum (in case you don't mandate all software be installed via yum).

--
You received this message because you are subscribed to the Google Groups "TigerVNC Developer Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tigervnc-deve...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tigervnc-devel/3cc28c9a-c6fc-4ad9-ba15-dcd1754832fcn%40googlegroups.com.

[Brian Cowan]

Well, the "problematic host" started life as a host that -- after an application version upgrade -- started having odd font-related behavior... Specifically it wasn't actually using the fonts the application setup files used. Yet the settings files were not changed. Wen I tried to reproduce using the X server that handled the console desktop, I couldn't reproduce it. After configuring xrdp, I didnt' get the mentioned problem, but instead got other font-related issues. When I compared the Xvnc server and the console desktop X server, I found that the Xvnc X server was based on a MUCH older X server than the console used. Better, the tigerVNC version in RHEL 6.10 is old enough to not be on the release history (1.1.0). 

So, we started by updating TigerVNC to use version 1.9.0+, which uses X.org 1.20. That resolved the "incorrect font" issue, but we now have this fun little issue where one of the tools takes 30 seconds in his environment and effectively forever on my test host. My test host has all the updates for RHEL 6.10, and the rpm -qa | sort outputs are pretty much the same, keeping in mind that we installed TigerVNC outside of RPM by downloading the bits from the web page...

At this point I wonder if current TigerVNC versions just hate the X font server -- which "isn't supposed to be there anymore..." Because a default configuration sent this into a loop. When I watch the xfs process with ltrace or strace, it's just sitting there waiting for work, Or at least thats all it seems to be doing.

Again, no X font server, no problem... Add the X font server and the problem occurs. And I'll be damned if I can understand why. 

[Brian Hinz]

So, we started by updating TigerVNC to use version 1.9.0+, which uses X.org 1.20. That resolved the "incorrect font" issue, but we now have this fun little issue where one of the tools takes 30 seconds in his environment and effectively forever on my test host. My test host has all the updates for RHEL 6.10, and the rpm -qa | sort outputs are pretty much the same, keeping in mind that we installed TigerVNC outside of RPM by downloading the bits from the web page...

We provided el6 RPMs for TigerVNC 1.10.1 (https://bintray.com/beta/#/tigervnc/stable/tigervnc/1.10.1?tab=files).  Those RPMs were built against the same Xorg source code as the native X server, have you tried those?

At this point I wonder if current TigerVNC versions just hate the X font server -- which "isn't supposed to be there anymore..." Because a default configuration sent this into a loop. When I watch the xfs process with ltrace or strace, it's just sitting there waiting for work, Or at least thats all it seems to be doing.

It's been forever since I used a font server, so I can't say. Do you have the fonts? I suppose you could try adding them to the system font path...

To view this discussion on the web visit https://groups.google.com/d/msgid/tigervnc-devel/52310c56-a411-4bd0-8d12-9bcba98f8957n%40googlegroups.com.

[Brian Cowan]

Brian, there aren't any RPM links there. Just .gz files. I'm currently using the 1.11 (latest) x86_64 gzip file. 

Now the "fun" part is that I can set the identical fontpath on the local console X server and none of the delays happen. The font server isn't even breathing hard... In MY hang situation, I'm stuck in a select call within  XListFontsWithInfo() waiting for the X server to come back... And it does -- eventually. The client host actually hangs in trying to load, then fails and goes on, but that fail takes 30 seconds.

I am getting an AMAZING amount of pushback on removing the XFS server from the picture, there are only 3-4 font directories not in the -fp parameter for Xvnc... But I think I have a way to get him to test -- on a per-session basis -- removing the x11 font server from the picture.... the ""xset fp" command. I'm going to have them cut out the font server by using xset fp= {fontpath without the darn font server}, test my app, then use xset fp+ to add the directories set as the catalogue for the x font server and test both my application and the one being developed by this customer. 

[Brian Hinz]

Try this link:

https://bintray.com/tigervnc/stable/tigervnc/1.10.1#files

They are there, bintray must have changed the way they display the files because I’m having a hard time navigating to it too

To view this discussion on the web visit https://groups.google.com/d/msgid/tigervnc-devel/b58c17d8-ba6a-4ce8-8b4f-bf184565bdadn%40googlegroups.com.

--

Sent from Gmail Mobile

[Brian Cowan]

Thanks!

I have learned to love xset.

Using that to non persistently remove the font server proved to my increasingly recalcitrant customer that it was the font server's "fault." And then using it to add the "extra" fonts from the font server configuration proved that they didn't need that 14-year old security vulnerability masquerading as a redundant system service. I really appreciate being able to use the list as a sounding board, and will chime in to help where I can. I may not know the internals of the VNC viewer/server, but I do know troubleshooting and problem definition -- even if my rambling in this thread may infer otherwise...

I'm going to try that el6 build now that I can see it. (Next time, I will use a DESKTOP browser ...) And, 1.10.1 from the RPM is still not happy with the xfs font server... I don't know why, but this isn't what I would call a significant problem since the most current font server is from 2007....

Brian