VirtualGL with a remote GPU cluster

[Jason Edgecombe]

Hello,

I'm trying to set up VirtualGL in the following configuration with three machines:

• application server (A) -  runs the X application and is the X proxy using StarNet FastX. one user per app server.
  
• GPU cluster (G) - hosts the 3D X server and multiple GPUs (one per application server)
• client machine (C) - a user's laptop that runs the FastX client.

I currently have things working where machines A and G are on the same machine and VirtualGL works, but I can't seem to get things to work when the 3D graphics card and X server are on a different machine. I've tried setting VGL_DISPLAY to the X display for machine G, but I'm getting the error "No protocol specified" when running something like "DISPLAY=machine_G:1 glxinfo" even without VirtualGL.

Is this is a model that can work with VirtualGL?

My end goal is to set up a virtual computer lab where each user has a dedicated virtual machine with server-side GPU acceleration (in the server room). Each VM guest is hosted by a machine with 4 Nvidia GPUs. I've been unsuccessful using PCI pass-through to make the video cards work in the guest, so I'm trying to see if I can use 3D acceleration with a GPU that is over the network from the X client (application server). End users will use FastX to connect to the VMs that function as the application server.

Any help is appreciated.

Thanks.

Jason

---------------------------------------------------------------------------
Jason Edgecombe | Linux Administrator
UNC Charlotte | The William States Lee College of Engineering
9201 University City Blvd. | Charlotte, NC 28223-0001
Phone: 704-687-1943
jwed...@uncc.edu | http://engr.uncc.edu |  Facebook
---------------------------------------------------------------------------
If you are not the intended recipient of this transmission or a person responsible for delivering it to the intended recipient, any disclosure, copying, distribution, or other use of any of the information in this transmission is strictly prohibited. If you have received this transmission in error, please notify me immediately by reply e-mail or by telephone at
704-687-1943.  Thank you.

[DRC]

Not as such, because VirtualGL must run on the same server as the
application. You can, however, run the X proxy on a different server
than the VirtualGL/Application server:
https://cdn.rawgit.com/VirtualGL/virtualgl/master/doc/index.html#hd009002

I'm not sure I fully understand how virtual machines fit into your
environment. Can you elaborate? I'm also not sure why VMs are even
necessary, unless you're particularly paranoid about security. If FastX
is truly an X proxy, then it virtualizes the X server, and VirtualGL
virtualizes the GPU. If you need to run a VM, then you should be able
to run VirtualBox or VMWare directly on G using VirtualGL:
https://cdn.rawgit.com/VirtualGL/virtualgl/master/doc/index.html#hd0013
https://cdn.rawgit.com/VirtualGL/virtualgl/master/doc/index.html#hd0014

On 2/14/19 2:31 PM, Jason Edgecombe wrote:
> Hello,
>
> I'm trying to set up VirtualGL in the following configuration with three
> machines:
>

> * application server (A) -  runs the X application and is the X proxy

> using StarNet FastX. one user per app server.

> * GPU cluster (G) - hosts the 3D X server and multiple GPUs (one per
> application server)
> * client machine (C) - a user's laptop that runs the FastX client.

[Jason Edgecombe]

Hi DRC,

I'm running VMs because my use case is for university students in engineering classes. These students need some isolation between their sessions in order to give some performance and stability guarantees. One of the main applications needs CAD-level graphics and a good amount of CPU.

The application server (A) is a VM that is running on the GPU server (G). My initial attempts have been to use PCI-passthrough to expose the GPU to the VM, but that has not worked so far. Currently, the VM host (G) is Windows Hyper-V and the guest (A) is Ubuntu 16.04.  My tests so far with using a RHEL7 VM host with KVM RHEL7 guest have not worked either. I was hoping that GPU's over the network was an option.

The FastX server process on the application server (A) hosts the 2D X server. I can't really separate it as the X proxy. I can't do any fun ssh tricks behind the scenes, because we don't have password-less ssh because of Kerberized NFS home directories. The root user cannot access any files hosted on Kerberized NFS shares unless it has credentials.

I'm not committed to any particular solution, but I need a solution that works with Kerberized NFS4 home directories and doesn't assume that root can read user files. I also need some reasonable resource isolation for performance.  I have 10+ Dell Precision Rack 7910 servers with 4 GPUs each that are supposed to serve as a virtual computer lab with 3D support for Linux applications. I already have a different time-sharing cluster that hosts large CPU and memory jobs for long-running jobs. The CPU cluster also just got some Nvidia M2000 GPUs for basic GPU support.

I'm confused about why GPU's over the network won't work. If all communication to the 3D X server uses the X11 protocol and isn't out-of-band, then I would expect it to work.

I hope that this clarifies things.

Thanks,

Jason

---------------------------------------------------------------------------
Jason Edgecombe | Linux Administrator
UNC Charlotte | The William States Lee College of Engineering
9201 University City Blvd. | Charlotte, NC 28223-0001
Phone: 704-687-1943
jwed...@uncc.edu | http://engr.uncc.edu |  Facebook
---------------------------------------------------------------------------
If you are not the intended recipient of this transmission or a person responsible for delivering it to the intended recipient, any disclosure, copying, distribution, or other use of any of the information in this transmission is strictly prohibited. If you have received this transmission in error, please notify me immediately by reply e-mail or by telephone at
704-687-1943.  Thank you.

--
You received this message because you are subscribed to the Google Groups "VirtualGL User Discussion/Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to virtualgl-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/283712a9-1118-eadc-f89f-7f6b789900fb%40virtualgl.org.
For more options, visit https://groups.google.com/d/optout.

[DRC]

VirtualGL currently has to access the GPU through an X server (the "3D X
server"), and since your GPU-attached host is running Windows, you can't
run a 3D X server on the host (Cygwin and other readily-available X
server solutions for Windows don't support hardware-accelerated OpenGL.)
Even if your GPU-attached host were running Linux, the approach you
suggest would be:

- insecure, since you'd have to open the 3D X server to TCP communication

- incompatible, since indirect OpenGL rendering would have to be used
between VirtualGL and the 3D X server. Many newer OpenGL features
wouldn't work.

- slow, since indirect OpenGL rendering would have to be used and since
VirtualGL would have to read back rendered frames through a socket.

PCI pass-through is the only way I can think of to do what you're trying
to do.

DRC

> Phone: 704-687-1943 <tel:704-687-1943>
> jwed...@uncc.edu <mailto:jwed...@uncc.edu> | http://engr.uncc.edu |

>  Facebook
> ---------------------------------------------------------------------------
> If you are not the intended recipient of this transmission or a person
> responsible for delivering it to the intended recipient, any disclosure,
> copying, distribution, or other use of any of the information in this
> transmission is strictly prohibited. If you have received this
> transmission in error, please notify me immediately by reply e-mail or
> by telephone at

> 704-687-1943 <tel:704-687-1943>.  Thank you.

> <mailto:virtualgl-users%2Bunsu...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/virtualgl-users/283712a9-1118-eadc-f89f-7f6b789900fb%40virtualgl.org.
> For more options, visit https://groups.google.com/d/optout.
>

> --
> You received this message because you are subscribed to the Google
> Groups "VirtualGL User Discussion/Support" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to virtualgl-use...@googlegroups.com

> <mailto:virtualgl-use...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/virtualgl-users/CAAR6MGCUYbJz3qh%2BZr%3Df2QiHw7JbgD%3D%2BBGZsT7S9KOdui31%2BnA%40mail.gmail.com
> <https://groups.google.com/d/msgid/virtualgl-users/CAAR6MGCUYbJz3qh%2BZr%3Df2QiHw7JbgD%3D%2BBGZsT7S9KOdui31%2BnA%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[Jason Edgecombe]

OK, thanks for clarifying. I'll continue to explore PCI pass-through.

Sincerely,

Jason

---------------------------------------------------------------------------
Jason Edgecombe | Linux Administrator
UNC Charlotte | The William States Lee College of Engineering
9201 University City Blvd. | Charlotte, NC 28223-0001
Phone: 704-687-1943

jwed...@uncc.edu | http://engr.uncc.edu |  Facebook
---------------------------------------------------------------------------
If you are not the intended recipient of this transmission or a person responsible for delivering it to the intended recipient, any disclosure, copying, distribution, or other use of any of the information in this transmission is strictly prohibited. If you have received this transmission in error, please notify me immediately by reply e-mail or by telephone at

704-687-1943.  Thank you.

To unsubscribe from this group and stop receiving emails from it, send an email to virtualgl-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/498aa64d-4d4c-ba1a-b5d0-f866d33d05d9%40virtualgl.org.