rootless 3D X server (Xorg)?
Youssef Ghorbal
Jason Edgecombe
UNC Charlotte | The William States Lee College of Engineering
9201 University City Blvd. | Charlotte, NC 28223-0001
Phone: 704-687-1943
jwed...@uncc.edu | http://engr.uncc.edu | Facebook
---------------------------------------------------------------------------
If you are not the intended recipient of this transmission or a person responsible for delivering it to the intended recipient, any disclosure, copying, distribution, or other use of any of the information in this transmission is strictly prohibited. If you have received this transmission in error, please notify me immediately by reply e-mail or by telephone at
704-687-1943. Thank you.
--
You received this message because you are subscribed to the Google Groups "VirtualGL User Discussion/Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to virtualgl-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/7fc96bfe-4c8a-476f-8fc8-275b1e73c67a%40googlegroups.com.
DRC
In the long term, this will be addressed in VirtualGL 3.0 by the
introduction of an EGL back end, which will allow VirtualGL to be
used without a 3D X server (with some limitations.) That feature
is still under development, however. Googling led me to this
page, which suggests that what you want is possible (perhaps even
using GDM), but I haven't tried it personally.
Doug O'Neal
Dr.
Douglas O’Neal (contractor)
Manager, HPC Systems Administration Group, ITOG
Frederick National Laboratory for Cancer Research
Leidos Biomedical Research, Inc.
Email: Douglas.O’Ne...@nih.gov
DRC
The answer depends heavily on how you
define "securely" and "over-allocating resources". Please
clarify.
DRC
Doug O'Neal
- Only the user submitting the batch job has access to the gpu and the user has access only to the gpu(s) allocated through the batch system.
- Ideal situation is for Xorg or an equivalent daemon to be started when the batch job starts and is killed when the job exits. Daemon should run as the user, possibly with /dev/nvidia? owned by the user. A chown can be included in the slurm prolog script.
- If Xorg has to be running continuously, it should not take resources (gpu system time or memory) away from the non-display jobs when they have the gpu allocated. Do we need one daemon per gpu and how to we restrict access based on slurm resource requests?
- More minor but still a problem. Running Xorg headless still blocks access to the virtual consoles using HPE servers and iLO to connect
DRC
Follow-up questions:
- Are your 3D applications purely non-interactive? That is, is it important for users to be able to interact with or see the output of the applications in real time? If the answer is "never", then that simplifies the solution.
- Do these applications use GLX? EGL? Vulkan? Do they have X11 GUIs or need to use X11 for anything other than accessing the GPU?
If the applications are sometimes or always interactive, have X11
GUIs, and use GLX or EGL to access a GPU, then ideally you would
use TurboVNC with VirtualGL and the VirtualGL EGL back end. The
way that would work is:
2. The job scheduler picks an execution node and a GPU on that node.
3. The job scheduler starts a new TurboVNC session on the execution node. (Note that some job schedulers require the -fg switch to be passed to /opt/TurboVNC/bin/vncserver in order to prevent TurboVNC from immediately backgrounding itself.)
4. The job scheduler temporarily changes the permissions and ownership for the devices (/dev/dri/card*, /dev/dri/render*, /dev/nvidia*) corresponding to the chosen GPU so that only the submitting user can access the GPU.
5. The job scheduler executes the 3D application with DISPLAY pointed to the newly-created TurboVNC session and VGL_DISPLAY pointed to the chosen GPU's DRI device.
There are multiple ways in which TurboVNC sessions can be managed:
- Some sites use custom web portals that create a new TurboVNC session through a job scheduler; populate a VNC "connection info" file with the TurboVNC session's hostname, display number, and one-time password; and download the connection info file to the user's browser, where it can be opened with the TurboVNC Viewer. Re-connecting to the TurboVNC session involves much the same process, except that the job scheduler simply generates a new OTP for the existing session rather than starting a new session.
- Some sites do basically the same thing without the web portal. In that case, the job scheduler prints the hostname and display number of the newly-created TurboVNC session, and users are required to enter that information into the TurboVNC Viewer manually and authenticate with the TurboVNC session using an authentication mechanism of the SysAdmin's choosing. (TurboVNC supports static VNC passwords, Unix login credentials or any other PAM-based authentication mechanism, one-time passwords, time-based one-time passwords, X.509 certificates, and SSH, and SysAdmins can force a particular authentication and encryption mechanism to be used on a system-wide basis.)
- If the users have direct SSH access to the execution nodes, then they could also use the TurboVNC Session Manager, which handles authentication, encryption, and session management through SSH. (In that case, a user would only need to know the hostname of the execution node on which their session is running.)
Potential wrinkles:
- The VirtualGL EGL back end generally works fine with straightforward OpenGL applications, but there are a couple of esoteric application behaviors (generally related to complex X11/OpenGL interactions) that still trip it up. You would need to test your applications and make sure that they all work properly with the EGL back end before declaring that a 3D X server will never be necessary.
-If you are dealing with multi-GPU applications that expect to be
able to directly connect to separate GPU-attached X
servers/screens in order to access GPUs for the secondary
rendering processes (e.g. ParaView "back in the days" before it
supported EGL), then that complicates things. It should still be
possible to use VirtualGL as a GLX-to-EGL translator in that
case. It would just require special values of the VGL_DISPLAY and
VGL_READBACK environment variables to be set for each rendering
process.
If the 3D applications are purely non-interactive, then you wouldn't necessarily need VirtualGL. VirtualGL is basically only useful for displaying to a remote system, because the two most common Un*x remote display use cases are: (1) remote X11 (client-side physical X display), in which case you need VirtualGL in order to avoid sending OpenGL primitives and data over the network, and (2) an X proxy (server-side virtual X display), in which case you need VirtualGL because X proxies lack GPU acceleration. You generally only need VGL if a 3D application is displaying something that a user needs to see or interact with in real time. However, you could still use VirtualGL as a GLX-to-EGL translator if your non-interactive 3D applications use GLX to access a GPU. If the non-interactive 3D application needs an X server for some purpose, such as creating a dummy window or a Pixmap, then you could start an Xvfb instance instead of TurboVNC, since the user would never need to see or interact with the application's output in real time.
tl;dr: I don't actually know how to start independent 3D X
servers using a job scheduler, and I'm not sure if starting
GPU-attached Xorg instances under non-root accounts is even
possible. (Someone else please correct me if I'm wrong.) Sites
that use job schedulers and need to use the VirtualGL GLX back end
will typically run a full-time 3D X server with a dedicated screen
for each GPU. In that case, everything I said above applies,
except that you would point VGL_DISPLAY to the GPU's screen rather
than its DRI device. The full-time 3D X server shouldn't use any
GPU compute resources when it is idle, but it will use some GPU
memory (not a lot, though-- like 32-64 MB if the GPU is configured
for headless operation.) However, the security situation is less
palatable, since nothing would technically prevent another user
from pointing VGL_DISPLAY to a screen attached to a GPU that has
been allocated for another user. I really think that the EGL back
end is your best bet, if you can make it work.
Faldegast
--
You received this message because you are subscribed to the Google Groups "VirtualGL User Discussion/Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to virtualgl-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/fbd6771f-df33-7960-47ab-468086fcf144%40virtualgl.org.
Jason Edgecombe
UNC Charlotte | Office of OneIT
---------------------------------------------------------------------------
If you are not the intended recipient of this transmission or a person responsible for delivering it to the intended recipient, any disclosure, copying, distribution, or other use of any of the information in this transmission is strictly prohibited. If you have received this transmission in error, please notify me immediately by reply e-mail or by telephone at
704-687-1943. Thank you.
[Caution: Email from External Sender. Do not click or open links or attachments unless you know this sender.]
--
You received this message because you are subscribed to the Google Groups "VirtualGL User Discussion/Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to virtualgl-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/fbd6771f-df33-7960-47ab-468086fcf144%40virtualgl.org.