Problem with VGL Setup for remote rendering
94 views
Skip to first unread message
ulsch
Feb 20, 2019, 7:43:37 AM2/20/19
to VirtualGL User Discussion/Support
Hello VGL-team,
currently i am setting up a environment where virtualgl remotely on a workstation renders and sends the data to a client. So the typical VGL application without any fancy constellation or hardware. For this I use the latest VGL version 2.6.1.
Client1: Linux Ubuntu 18.04
Client2: Windows 10 with Linux Ubuntu 18.04 wsl and vcxsrv X server
Workstation: Linux Ubuntu 18.04, nvidia quadro
The connection can be established via vglconnect but a VGLrun only works via vglrun -c proxy <executable> which leads to quite low frame rates.
vglrun +pr +v -c proxy /opt/VirtualGL/bin/glxspheres64
[VGL] NOTICE: Automatically setting VGL_CLIENT environment variable to
[VGL] xxxxxxxx, the IP address of your SSH client.
Polygons in scene: 62464 (61 spheres * 1024 polys/spheres)
[VGL] Shared memory segment ID for vglconfig: 48562541
[VGL] VirtualGL v2.6.1 64-bit (Build 20190101)
[VGL] Opening connection to 3D X server :0
Visual ID of window: 0x21
Context is Direct
[VGL] Using Pbuffers for rendering
OpenGL Renderer: llvmpipe (LLVM 6.0, 256 bits)
[FBX] WARNING: MIT-SHM extension not available. Will use X pixmap
[FBX] drawing instead.
[VGL] Using pixel buffer objects for readback (BGR --> BGRA)
[VGL] NOTICE: PBO readback is not behaving asynchronously. Disabling PBOs.
[VGL] This could be due to a mismatch between the readback pixel format
[VGL] (BGRA) and the Pbuffer pixel format (BGR).
[VGL] Try setting VGL_FORCEALPHA=1.
Readback - 176.59 Mpixels/sec- 158.23 fps
13.330503 frames/sec - 14.876841 Mpixels/sec
Blit - 15.38 Mpixels/sec- 13.78 fps
Total - 14.66 Mpixels/sec- 13.14 fps
Readback - 184.28 Mpixels/sec- 165.13 fps
15.410686 frames/sec - 17.198325 Mpixels/sec
Blit - 15.00 Mpixels/sec- 13.44 fps
Total - 14.96 Mpixels/sec- 13.41 fps
Readback - 184.09 Mpixels/sec- 164.95 fps
15.084290 frames/sec - 16.834067 Mpixels/sec
Blit - 16.52 Mpixels/sec- 14.80 fps
Total - 15.89 Mpixels/sec- 14.24 fps
Readback - 187.55 Mpixels/sec- 168.06 fps
14.632749 frames/sec - 16.330148 Mpixels/sec
Blit - 15.11 Mpixels/sec- 13.54 fps
Total - 14.91 Mpixels/sec- 13.36 fps
Readback - 187.72 Mpixels/sec- 168.21 fps
using vglrun with compression (-c jpeg) it gets stuck and aborts with a timeout
vglrun +pr +v +tr -c jpeg /opt/VirtualGL/bin/glxspheres64
[VGL] NOTICE: Automatically setting VGL_CLIENT environment variable to
[VGL] xxxxxxxxxxx , the IP address of your SSH client.
Polygons in scene: 62464 (61 spheres * 1024 polys/spheres)
VGL] Shared memory segment ID for cfconfig: 48529773
[VGL] VirtualGL v2.6.1 64-bit (Build 20190101)
[VGL] Opening connection to 3D X server :0
Visual ID of window: 0x21
Context is Direct
[VGL] Using Pbuffers for rendering
OpenGL Renderer: llvmpipe (LLVM 6.0, 256 bits)
...
VGL 0xf512f740] glXSwapBuffers (dpy=0x01f70930(localhost:21.0) drawable=0x00280002 [VGL]
ERROR: Could not connect to VGL client. Make sure that cfclient is
[VGL] running and that either the DISPLAY or VGL_CLIENT environment
[VGL] variable points to the machine on which cfclient is running.
[VGL] ERROR: in connect--
[VGL] 283: Connection timed out
VGL_CLIENT variable is not set
DISPLAY is automatically set to localhost:21.0
In another comparable environment this runs the virtualGL transmission with compression error free with frame rates in the range of 50-60fps which allows a smooth working on the client side. Unfortunately I have no clue what caused this error. i'd be happy to get some hints and help about that
Best regards
DRC
Feb 20, 2019, 1:13:16 PM2/20/19
to virtual...@googlegroups.com
-c proxy is designed for use with an X proxy (such as TurboVNC, the X
proxy we produce), so it's not surprising that it performs poorly over a
remote connection. X proxies act as virtual X servers, so they are
designed to be run on the same machine as VirtualGL. As such, -c proxy
sends the rendered frames from VirtualGL to the X server using
XShmPutImage(), so the frames are not compressed in any way.
Check the vglconnect log (in {home directory}/.vgl/ on the client
machine) and see whether vglclient actually received a connection from
the server. If it didn't, then I would suspect that you need to open
port 4242 in the client's firewall. This is one of many reasons why X
proxies are the most popular way of running VirtualGL these days, but I
acknowledge that the VGL Transport still has its uses and is the most
straightforward way of running VirtualGL if you are already using a
remote X environment with a client-side X server.
DRC
On 2/20/19 6:43 AM, ulsch wrote:
> Hello VGL-team,
>
> currently i am setting up a environment where virtualgl remotely on a
> workstation renders and sends the data to a client. So the typical VGL
> application without any fancy constellation or hardware. For this I use
> the latest VGL version 2.6.1.
> Client1: Linux Ubuntu 18.04
> Client2: Windows 10 with Linux Ubuntu 18.04 wsl and vcxsrv X server
> Workstation: Linux Ubuntu 18.04, nvidia quadro
>
> The connection can be established via vglconnect but a VGLrun only works
> via vglrun -c proxy <executable> which leads to quite low frame rates.
>
> * vglrun +pr +v -c proxy /opt/VirtualGL/bin/glxspheres64*
> /[VGL] NOTICE: Automatically setting VGL_CLIENT environment variable to/
> /[VGL] xxxxxxxx, the IP address of your SSH client./
> /Polygons in scene: 62464 (61 spheres * 1024 polys/spheres)/
> /[VGL] Shared memory segment ID for vglconfig: 48562541/
> /[VGL] VirtualGL v2.6.1 64-bit (Build 20190101)/
> /[VGL] Opening connection to 3D X server :0/
> /Visual ID of window: 0x21/
> /Context is Direct/
> /[VGL] Using Pbuffers for rendering/
> /OpenGL Renderer: llvmpipe (LLVM 6.0, 256 bits)/
> /[FBX] WARNING: MIT-SHM extension not available. Will use X pixmap/
> /[FBX] drawing instead./
> /[VGL] Using pixel buffer objects for readback (BGR --> BGRA)/
> /[VGL] NOTICE: PBO readback is not behaving asynchronously. Disabling
> PBOs./
> /[VGL] This could be due to a mismatch between the readback pixel format/
> /[VGL] (BGRA) and the Pbuffer pixel format (BGR)./
> /[VGL] Try setting VGL_FORCEALPHA=1./
> /Readback - 176.59 Mpixels/sec- 158.23 fps/
> /13.330503 frames/sec - 14.876841 Mpixels/sec/
> /Blit - 15.38 Mpixels/sec- 13.78 fps/
> /Total - 14.66 Mpixels/sec- 13.14 fps/
> /Readback - 184.28 Mpixels/sec- 165.13 fps/
> /15.410686 frames/sec - 17.198325 Mpixels/sec/
> /Blit - 15.00 Mpixels/sec- 13.44 fps/
> /Total - 14.96 Mpixels/sec- 13.41 fps/
> /Readback - 184.09 Mpixels/sec- 164.95 fps/
> /15.084290 frames/sec - 16.834067 Mpixels/sec/
> /Blit - 16.52 Mpixels/sec- 14.80 fps/
> /Total - 15.89 Mpixels/sec- 14.24 fps/
> /Readback - 187.55 Mpixels/sec- 168.06 fps/
> /14.632749 frames/sec - 16.330148 Mpixels/sec/
> /Blit - 15.11 Mpixels/sec- 13.54 fps/
> /Total - 14.91 Mpixels/sec- 13.36 fps/
> /Readback - 187.72 Mpixels/sec- 168.21 fps/
> /[VGL] NOTICE: Automatically setting VGL_CLIENT environment variable to/
> /[VGL] xxxxxxxxxxx , the IP address of your SSH client./
> /Polygons in scene: 62464 (61 spheres * 1024 polys/spheres)/
> /VGL] Shared memory segment ID for cfconfig: 48529773/
> /[VGL] VirtualGL v2.6.1 64-bit (Build 20190101)/
> /[VGL] Opening connection to 3D X server :0/
> /Visual ID of window: 0x21/
> /Context is Direct/
> /[VGL] Using Pbuffers for rendering/
> /OpenGL Renderer: llvmpipe (LLVM 6.0, 256 bits)/
> /.../
> /VGL 0xf512f740] glXSwapBuffers (dpy=0x01f70930(localhost:21.0)
> drawable=0x00280002 [VGL] /
> /ERROR: Could not connect to VGL client. Make sure that cfclient is/
> /[VGL] running and that either the DISPLAY or VGL_CLIENT environment/
> /[VGL] variable points to the machine on which cfclient is running./
> /[VGL] ERROR: in connect--/
> /[VGL] 283: Connection timed out/
> /
> /
proxy we produce), so it's not surprising that it performs poorly over a
remote connection. X proxies act as virtual X servers, so they are
designed to be run on the same machine as VirtualGL. As such, -c proxy
sends the rendered frames from VirtualGL to the X server using
XShmPutImage(), so the frames are not compressed in any way.
Check the vglconnect log (in {home directory}/.vgl/ on the client
machine) and see whether vglclient actually received a connection from
the server. If it didn't, then I would suspect that you need to open
port 4242 in the client's firewall. This is one of many reasons why X
proxies are the most popular way of running VirtualGL these days, but I
acknowledge that the VGL Transport still has its uses and is the most
straightforward way of running VirtualGL if you are already using a
remote X environment with a client-side X server.
DRC
On 2/20/19 6:43 AM, ulsch wrote:
> Hello VGL-team,
>
> currently i am setting up a environment where virtualgl remotely on a
> workstation renders and sends the data to a client. So the typical VGL
> application without any fancy constellation or hardware. For this I use
> the latest VGL version 2.6.1.
> Client1: Linux Ubuntu 18.04
> Client2: Windows 10 with Linux Ubuntu 18.04 wsl and vcxsrv X server
> Workstation: Linux Ubuntu 18.04, nvidia quadro
>
> The connection can be established via vglconnect but a VGLrun only works
> via vglrun -c proxy <executable> which leads to quite low frame rates.
>
> /[VGL] NOTICE: Automatically setting VGL_CLIENT environment variable to/
> /[VGL] xxxxxxxx, the IP address of your SSH client./
> /Polygons in scene: 62464 (61 spheres * 1024 polys/spheres)/
> /[VGL] Shared memory segment ID for vglconfig: 48562541/
> /[VGL] VirtualGL v2.6.1 64-bit (Build 20190101)/
> /[VGL] Opening connection to 3D X server :0/
> /Visual ID of window: 0x21/
> /Context is Direct/
> /[VGL] Using Pbuffers for rendering/
> /OpenGL Renderer: llvmpipe (LLVM 6.0, 256 bits)/
> /[FBX] WARNING: MIT-SHM extension not available. Will use X pixmap/
> /[FBX] drawing instead./
> /[VGL] Using pixel buffer objects for readback (BGR --> BGRA)/
> /[VGL] NOTICE: PBO readback is not behaving asynchronously. Disabling
> PBOs./
> /[VGL] This could be due to a mismatch between the readback pixel format/
> /[VGL] (BGRA) and the Pbuffer pixel format (BGR)./
> /[VGL] Try setting VGL_FORCEALPHA=1./
> /Readback - 176.59 Mpixels/sec- 158.23 fps/
> /13.330503 frames/sec - 14.876841 Mpixels/sec/
> /Blit - 15.38 Mpixels/sec- 13.78 fps/
> /Total - 14.66 Mpixels/sec- 13.14 fps/
> /Readback - 184.28 Mpixels/sec- 165.13 fps/
> /15.410686 frames/sec - 17.198325 Mpixels/sec/
> /Blit - 15.00 Mpixels/sec- 13.44 fps/
> /Total - 14.96 Mpixels/sec- 13.41 fps/
> /Readback - 184.09 Mpixels/sec- 164.95 fps/
> /15.084290 frames/sec - 16.834067 Mpixels/sec/
> /Blit - 16.52 Mpixels/sec- 14.80 fps/
> /Total - 15.89 Mpixels/sec- 14.24 fps/
> /Readback - 187.55 Mpixels/sec- 168.06 fps/
> /14.632749 frames/sec - 16.330148 Mpixels/sec/
> /Blit - 15.11 Mpixels/sec- 13.54 fps/
> /Total - 14.91 Mpixels/sec- 13.36 fps/
> /Readback - 187.72 Mpixels/sec- 168.21 fps/
>
> using vglrun with compression (-c jpeg) it gets stuck and aborts with a
> timeout
>
> *vglrun +pr +v +tr -c jpeg /opt/VirtualGL/bin/glxspheres64*
> using vglrun with compression (-c jpeg) it gets stuck and aborts with a
> timeout
>
> /[VGL] NOTICE: Automatically setting VGL_CLIENT environment variable to/
> /[VGL] xxxxxxxxxxx , the IP address of your SSH client./
> /Polygons in scene: 62464 (61 spheres * 1024 polys/spheres)/
> /VGL] Shared memory segment ID for cfconfig: 48529773/
> /[VGL] VirtualGL v2.6.1 64-bit (Build 20190101)/
> /[VGL] Opening connection to 3D X server :0/
> /Visual ID of window: 0x21/
> /Context is Direct/
> /[VGL] Using Pbuffers for rendering/
> /OpenGL Renderer: llvmpipe (LLVM 6.0, 256 bits)/
> /.../
> /VGL 0xf512f740] glXSwapBuffers (dpy=0x01f70930(localhost:21.0)
> drawable=0x00280002 [VGL] /
> /ERROR: Could not connect to VGL client. Make sure that cfclient is/
> /[VGL] running and that either the DISPLAY or VGL_CLIENT environment/
> /[VGL] variable points to the machine on which cfclient is running./
> /[VGL] ERROR: in connect--/
> /[VGL] 283: Connection timed out/
>
>
> VGL_CLIENT variable is not set
> DISPLAY is automatically set to /localhost:21.0/
>
> VGL_CLIENT variable is not set
> /
> /
ulsch
Feb 21, 2019, 6:45:28 AM2/21/19
to VirtualGL User Discussion/Support
Think simple instead of complicated :-)
Thank you very much, the port was blocked!
Message has been deleted
ulsch
Apr 1, 2019, 5:36:00 AM4/1/19
to VirtualGL User Discussion/Support
Picking up this issue once again.
I observe strong different performance between two comparable systems (in different facilities), which seems not plausible to me:
Visualization Enviroment 1:
VGL Server:
Hardware:
CPU: 2*Intel Xeon E5-2687W v4, 12x3.0Ghz (3.5 Ghz max)
MB : Supermicro Motherboard X10DAi
RAM: 256GB DDR4 ECC 2400Hz (CT32G4RFD424A)
GPU: Nvidia Quadro M6000 12GB
SSD: 2TB Samsung PM863
HDD: 24TB RAID 6 (5xHUH728080ALE600)
LAN: 10Gbit Supermicro AOC-STG-I2T
Software:
Ubuntu 12.04 LTS
VirtualGL 2.6.1
VGLClient (Desktop PC):
CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
LAN: 1Gbit
Software: Ubuntu 16.04LTS
vglrun +pr /opt/VirtualGL/bin/glxspheres64
Readback - 1279.52 Mpixels/sec- 936.68 fps
625.544299 frames/sec - 854.508525 Mpixels/sec
Compress 0 - 91.66 Mpixels/sec- 67.10 fps
Total - 112.60 Mpixels/sec- 82.43 fps- 140.59 Mbits/sec (19.2:1)
Readback - 1276.81 Mpixels/sec- 934.69 fps
620.622386 frames/sec - 847.785074 Mpixels/sec
Compress 0 - 84.02 Mpixels/sec- 61.51 fps
Total - 107.23 Mpixels/sec- 78.50 fps- 134.42 Mbits/sec (19.1:1)
CPU Loading:
Server: 170% average
Client: 270% average
Visualization Enviroment 2:
VGL Server:
Hardware:
CPU: 2*Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz (3.7 Ghz max)
RAM: 256GB DDR4 ECC 2666MHz
GPU: Nvidia Quadro P5000 16GB [GP104GL]
LAN: 1Gbit
Software:
Ubuntu 18.04 LTS
VirtualGL 2.6.1
VGLClient (laptop):
CPU: Intel(R) Core(TM) i7-6600 CPU @ 2.60GHz
LAN: 1Gbit
Software: Windows 10 with Linux Ubuntu 18.04 WSL, VcXsrv X-Server
vglrun +pr /opt/VirtualGL/bin/glxspheres64
Readback - 336.98 Mpixels/sec- 301.95 fps
44.329628 frames/sec - 49.471864 Mpixels/sec
Compress 0 - 122.81 Mpixels/sec- 110.04 fps
Total - 32.88 Mpixels/sec- 29.46 fps- 55.18 Mbits/sec (14.3:1)
Readback - 333.44 Mpixels/sec- 298.78 fps
44.390165 frames/sec - 49.539424 Mpixels/sec
Compress 0 - 122.34 Mpixels/sec- 109.62 fps
Total - 27.66 Mpixels/sec- 24.78 fps- 47.42 Mbits/sec (14.0:1)
CPU Loading:
Server: 388% average
Client: 77% average
I don't understand why the CPU loading of the VGL server in environment 2 is so high compared to environment 1 and a significant lower framerate of 25fps compared to 80fps is achieved.
There is no firewall active in the environments which may cause an overhead.Do you have any hints where to look at?
Looking forward to your reply.
DRC
Apr 1, 2019, 12:53:29 PM4/1/19
to virtual...@googlegroups.com
Here's what I think is happening:
- You are using an unsupported 2D X server configuration on the Windows
client. I haven't had a chance to play with WSL yet, so I don't know
how the VirtualGL Client interacts with a 2D X server in that
environment, but it's a safe bet that the MIT-SHM extension is not going
to work if vglclient is running under WSL and the 2D X server isn't.
(MIT-SHM allows an X11 application to send images to the X server
through shared memory, so it requires that both the X11 application and
the X server share the same underlying SysV shared memory implementation.)
The only officially supported Windows X server for the VirtualGL Client
is Cygwin/X, using the Cygwin build of vglclient. Otherwise, I
recommend using TurboVNC, since it eliminates the need for a client-side
X server on Windows clients. The VGL Transport is really more of a
legacy feature, and it's mainly useful for Linux-->Linux on a high-speed
LAN.
References:
https://cdn.rawgit.com/VirtualGL/virtualgl/2.6.1/doc/index.html#hd004003
https://cdn.rawgit.com/VirtualGL/virtualgl/2.6.1/doc/index.html#hd005003
https://cdn.rawgit.com/VirtualGL/virtualgl/2.6.1/doc/index.html#hd007
- The slow 2D X server is slowing down the end-to-end performance of the
VGL Transport when using your Windows client.
- You should always disable frame spoiling ('vglrun -sp') when
benchmarking. Otherwise, if the end-to-end performance (which, in the
case of the VGL Transport, is usually limited by the client/2D X server)
is slow, then the benchmark (which is trying to render frames as quickly
as it can, because it's a benchmark) will cause a lot of frames to be
spoiled on the server, and that will increase the server CPU load
unnecessarily. Frame spoiling is only meant to be used with interactive
applications.
Reference:
https://cdn.rawgit.com/VirtualGL/virtualgl/2.6.1/doc/index.html#hd0017
> --
> You received this message because you are subscribed to the Google
> Groups "VirtualGL User Discussion/Support" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to virtualgl-use...@googlegroups.com
> <mailto:virtualgl-use...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/virtualgl-users/2f635957-50fa-4149-8903-8324d5ef972a%40googlegroups.com
> <https://groups.google.com/d/msgid/virtualgl-users/2f635957-50fa-4149-8903-8324d5ef972a%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.
- You are using an unsupported 2D X server configuration on the Windows
client. I haven't had a chance to play with WSL yet, so I don't know
how the VirtualGL Client interacts with a 2D X server in that
environment, but it's a safe bet that the MIT-SHM extension is not going
to work if vglclient is running under WSL and the 2D X server isn't.
(MIT-SHM allows an X11 application to send images to the X server
through shared memory, so it requires that both the X11 application and
the X server share the same underlying SysV shared memory implementation.)
The only officially supported Windows X server for the VirtualGL Client
is Cygwin/X, using the Cygwin build of vglclient. Otherwise, I
recommend using TurboVNC, since it eliminates the need for a client-side
X server on Windows clients. The VGL Transport is really more of a
legacy feature, and it's mainly useful for Linux-->Linux on a high-speed
LAN.
References:
https://cdn.rawgit.com/VirtualGL/virtualgl/2.6.1/doc/index.html#hd004003
https://cdn.rawgit.com/VirtualGL/virtualgl/2.6.1/doc/index.html#hd005003
https://cdn.rawgit.com/VirtualGL/virtualgl/2.6.1/doc/index.html#hd007
- The slow 2D X server is slowing down the end-to-end performance of the
VGL Transport when using your Windows client.
- You should always disable frame spoiling ('vglrun -sp') when
benchmarking. Otherwise, if the end-to-end performance (which, in the
case of the VGL Transport, is usually limited by the client/2D X server)
is slow, then the benchmark (which is trying to render frames as quickly
as it can, because it's a benchmark) will cause a lot of frames to be
spoiled on the server, and that will increase the server CPU load
unnecessarily. Frame spoiling is only meant to be used with interactive
applications.
Reference:
https://cdn.rawgit.com/VirtualGL/virtualgl/2.6.1/doc/index.html#hd0017
> You received this message because you are subscribed to the Google
> Groups "VirtualGL User Discussion/Support" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to virtualgl-use...@googlegroups.com
> <mailto:virtualgl-use...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/virtualgl-users/2f635957-50fa-4149-8903-8324d5ef972a%40googlegroups.com
> <https://groups.google.com/d/msgid/virtualgl-users/2f635957-50fa-4149-8903-8324d5ef972a%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.
Reply all
Reply to author
Forward
0 new messages