Tecplot / tec360 issues
Richard Ems
turbovnc-2.2.90-20211222.x86_64
NVIDIA Corporation GM204GL [Tesla M60]
/net/c1m1/data_1/opt/Tecplot360/360ex_2021r2/bin/../bin/libtpsdkintegrationmanager.so(_ZN7tecplot3sdk11integration15ManagerAbstract5startEv+0x19) [0x7f93fe92b1b9]
/net/c1m1/data_1/opt/Tecplot360/360ex_2021r2/bin/tec360-bin() [0x487ef5]
/lib64/libc.so.6(__libc_start_main+0xf3) [0x7f93fb4c0493]
/net/c1m1/data_1/opt/Tecplot360/360ex_2021r2/bin/tec360-bin() [0x489d09]
Tecplot received shutdown signal.
Signal #11
Please send the output above to Tecplot support: sup...@tecplot.com
/net/c1m1/data_1/opt/Tecplot360/360ex_2021r2/bin/tec360: line 349: 276303 Aborted (core dumped) "$0-bin" "$@"
Bypass the OpenGL library and use the LLVM OSMesa library for off-screen
image rendering. This option does not require an X server connection.
DRC
No idea. I'll have to try to reproduce it on my local machine.
DRC
--
You received this message because you are subscribed to the Google Groups "VirtualGL User Discussion/Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to virtualgl-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/c52cd569-4ff3-4bef-ad59-dc83a2938b97n%40googlegroups.com.
DRC
I have installed Tecplot 360 with an evaluation license. Please specify the exact steps needed to reproduce the issue using one of the Tecplot examples.
DRC
Richard Ems
To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/d9d33270-0115-79e9-2694-5dc358d4f3e5%40virtualgl.org.
DRC
Please clarify how this is supposed to
work. 'module load' is not a valid Bash command.
==========
> vglrun ./Tec360_FlowViz_batch.sh
./Tec360_FlowViz_batch.sh: line 3: module: command not found
Tecplot 2021.2.1.9698 - 21:07:35 Feb 3 2022
[linux64-centos6.10]
Color maps directory :
/usr/local/tecplot/360ex_2021r2/colormaps
Configuration File :
/usr/local/tecplot/360ex_2021r2/tecplot.cfg
Process Temp Dir : /usr/tmp/tecplot_drc/tpaqmYUE0
Tecplot internal font file:
/usr/local/tecplot/360ex_2021r2/tecplot.fnt
LaTeX Configuration File:
/usr/local/tecplot/360ex_2021r2/tecplot_latex.mcr
Your license expires at the end of 26-Mar-2022.
Update your license, or contact sup...@tecplot.com for help.
Tecplot executable :
/usr/local/tecplot/360ex_2021r2/bin/tec360-bin
Tecplot Home Directory : /usr/local/tecplot/360ex_2021r2
QuickMacro File :
/usr/local/tecplot/360ex_2021r2/tecplot.mcr
Addon Startup File :
/usr/local/tecplot/360ex_2021r2/tecplot.add
AddOn Loaded : Tecplot Subzone Data Loader
AddOn Loaded : SZL Remote Loader
AddOn Loaded : CFD Analyzer
AddOn Loaded : CGNS Loader
AddOn Loaded : DEM Loader
AddOn Loaded : DXF Loader
AddOn Loaded : EnSight Loader
AddOn Loaded : FLOW-3D Loader
AddOn Loaded : Fluent Common Fluid Files Loader
AddOn Loaded : Fluent Data Loader
AddOn Loaded : General Text Loader
AddOn Loaded : HDF Loader
AddOn Loaded : PLOT3D Loader
AddOn Loaded : PLY Polygon File Loader
AddOn Loaded : Text Spreadsheet Loader
AddOn Loaded : Kiva Loader
AddOn Loaded : Finite Element Analysis Loader
AddOn Loaded : Write Data as Text
AddOn Loaded : VTK Data Loader
AddOn Loaded : FVCOM netCDF Loader
AddOn Loaded : CONVERGE HDF5 File Loader
AddOn Loaded : CONVERGE Output File Loader
AddOn Loaded : Exodus File Loader
AddOn Loaded : Telemac Data Loader
AddOn Loaded : HDF5 Loader
AddOn Loaded : Akima Spline
AddOn Loaded : General Curve Fit
AddOn Loaded : Stineman Interpolation
AddOn Loaded : extendmcr
AddOn Loaded : Advanced Quick Edit Tool
AddOn Loaded : Multi Frame Manager
AddOn Loaded : Create Multiple Frames
AddOn Loaded : Tensor Eigensystem
AddOn Loaded : Key Frame Animation
AddOn Loaded : Aux Data Editor
AddOn Loaded : Time Series Plot
AddOn Loaded : Extract Blanked Zone
AddOn Loaded : Extract Precise Line
AddOn Loaded : Extract Over Time
AddOn Loaded : Link Solution Time
AddOn Loaded : Extend Time MCR
AddOn Loaded : Strand Editor
AddOn Loaded : Measure Distance
==========
DRC
To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/CAEqczzz-YhSLWHB2sSHHruJva_D_LvmOL6EPCVV%3DCwkcmrU6pQ%40mail.gmail.com.
Richard Ems
To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/e3b796a1-f680-4c3e-26d9-b1f7fc410664%40virtualgl.org.
DRC
I tried it with '$useVGL = 1' as well and was still unable to reproduce the issue. (Note that, from the point of view of the application, setting '$useVGL = 1' in turbovncserver.conf or starting the TurboVNC session with '-vgl' is the same as launching the application with 'vglrun +wm'.)
I also tried with both CentOS 7.9 and CentOS Stream 8. My CentOS 7.9 machine has a Quadro K5000 with driver v470.xx. My CentOS 8 machine has a Quadro P620 with driver v510.xx.
Is there any way that you could try
running the Tecplot batch job on the 3D X server without
VirtualGL? If it fails in the same way, then it isn't our bug.
DRC
To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/CAEqczzw1c3MNUZT%2BV%2Be%2Bd3RzuJuXHYSrW1bCWCyBqKsEwxaQUA%40mail.gmail.com.
Richard Ems
Is there any way that you could try running the Tecplot batch job on the 3D X server without VirtualGL? If it fails in the same way, then it isn't our bug.
To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/0aa7f795-a393-0eea-c045-3c0ad5af4995%40virtualgl.org.
DRC
Is there any way that you could try running the Tecplot batch job on the 3D X server without VirtualGL? If it fails in the same way, then it isn't our bug.Do you mean on a local attached display? That won't be easy, since this graphics workstation is part of a cluster located far away from me or the user getting this bug.
I have just contacted Tecplot Support and I'm waiting for their response and a trial license to test myself using my own environment.
The 3D X server is on the VirtualGL server, which should be the same machine on which Tecplot 360 is installed. Unless the machine has been configured otherwise, the 3D X server should be listening on Display :0.0, so try:
xauth merge
/etc/opt/VirtualGL/vgl_xauth_key
LD_PRELOAD= DISPLAY=:0.0 ./Tec360_FlowViz_batch.sh
I remember to have to preload libvglfaker.so for starccm+ to work properly. Anything similar perhaps needed here for tec360? Just some more guessing. Worth a try?
Preloading libvglfaker.so is
literally what vglrun does. I am not trying to make Tecplot 360
work properly. I am trying to make it fail.
To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/CAEqczzzf5QZ%3Dq-tp3QyxMurbh5RV_JGbkuWfo1LF1zD19%3DVwSg%40mail.gmail.com.
Richard Ems
To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/d96c534d-a8d0-21bd-7b40-5baea12de838%40virtualgl.org.
DRC
OK, that's an important data point. Now I just need to be able to reproduce the failure. :|
Richard Ems
--
You received this message because you are subscribed to the Google Groups "VirtualGL User Discussion/Support" group.
To unsubscribe from this group and stop receiving emails from it, send an email to virtualgl-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/14b5014b-6277-0440-e9c5-11f7e517419a%40virtualgl.org.
DRC
I assume that you are using the GLX back end when the error occurs, but I couldn't make the error occur with the EGL back end either. Which nVidia driver version are you using? I can try to match your driver version as closely as possible in case that has something to do with the failure.
Otherwise, I think the only way forward is
to get temporary access to a machine on which the failure can be
reproduced.
DRC
To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/CAEqczzzNxRQN5N5sFQr08coZV67O5WkNCTBpGa9kE_d5x%2BfmbA%40mail.gmail.com.
DRC
Also, can you send me a trace log from running the batch job with 'vglrun +tr +v'? That may reveal something about what the application is doing when the failure occurs.
DRC
To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/CAEqczzzNxRQN5N5sFQr08coZV67O5WkNCTBpGa9kE_d5x%2BfmbA%40mail.gmail.com.
Richard Ems
To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/0a7a6eb3-5afc-3a8c-9b4e-a1212ecb17ab%40virtualgl.org.
DRC
I tested 510.xx, so apparently that isn't
the issue. The GLX back end is the default, but the EGL back
end can be selected with 'vglrun -d egl' or 'vglrun -d
{DRI-device}' (where {DRI-device} is a device under /dev/dri,
such as /dev/dri/card0. I should be able to diagnose the issue
using a regular (non-administrative) account, as long as I have
SSH access. I will e-mail you an SSH public key.
DRC
To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/CAEqczzw%2B3Na122FXLwxCtyYYRYK_NOJ67X7AAGsjniLABpJuUg%40mail.gmail.com.
Richard Ems
To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/1a86354e-6214-97fc-c643-d4d325323085%40virtualgl.org.
DRC
to see if it has any stereo visuals. Because of libglvnd, this causes
the system's Mesa GLX vendor library to be loaded into the 3D
application process. This apparently interacts badly with Tecplot 360,
perhaps because Tecplot 360 has its own Mesa implementation. I have no
idea why the issue only occurs in batch mode, but I do know that the
issue is triggered by a specific sequence of GLX calls, so that probably
has something to do with it.
The quick workaround is to set VGL_PROBEGLX=0 in the environment. I
will modify the vncserver script in TurboVNC to do that automatically,
since TurboVNC will never have any stereo visuals to probe. Thus, this
won't be a problem again for you after you upgrade to TurboVNC 3.0. But
I also need to see if I can find a more robust solution for VirtualGL,
since this will affect other X proxies. VGL probably needs to learn how
to play more nicely with libglvnd.
I have asked for an extension of my Tecplot eval license, so hopefully
I'll be able to play with this more next week.
DRC
On 3/29/22 8:15 AM, Richard Ems wrote:
> Hi DRC,
>
> 2021 at
> https://kb.tecplot.com/2021/01/08/troubleshooting-linux-crashes/
> <https://kb.tecplot.com/2021/01/08/troubleshooting-linux-crashes/>
> that you probably already know, but I thought I would send the link here
> just in case. It may be helpful for others finding this thread.
>
> Thanks again
> Richard
>
>
>
> On Fri, 25 Mar 2022 at 17:47, 'DRC' via VirtualGL User
> Discussion/Support <virtual...@googlegroups.com
>
> I tested 510.xx, so apparently that isn't the issue. The GLX back
> end is the default, but the EGL back end can be selected with
> 'vglrun -d egl' or 'vglrun -d {DRI-device}' (where {DRI-device} is a
> device under /dev/dri, such as /dev/dri/card0. I should be able to
> diagnose the issue using a regular (non-administrative) account, as
> long as I have SSH access. I will e-mail you an SSH public key.
>
> DRC
>
> On 3/25/22 3:38 PM, Richard Ems wrote:
>> NVIDIA-SMI 510.47.03 Driver Version: 510.47.03 CUDA Version:
>> 11.6
>> How would I select one or the other back end?
>> And yes, giving you access to the workstation should be fine.
>> Let me run some more tests myself, but the workstation is on
>> production, so I have limited timeframes to test.
>>
>> Richard
>>
>>
>> On Fri, 25 Mar 2022 at 12:53, 'DRC' via VirtualGL User
>> Discussion/Support <virtual...@googlegroups.com
>>
>> I assume that you are using the GLX back end when the error
>> occurs, but I couldn't make the error occur with the EGL back
>> end either. Which nVidia driver version are you using? I can
>> try to match your driver version as closely as possible in
>> case that has something to do with the failure.
>>
>> Otherwise, I think the only way forward is to get temporary
>> access to a machine on which the failure can be reproduced.
>>
>> DRC
>>
>> On 3/25/22 7:14 AM, Richard Ems wrote:
>>> Can I provide more setup info or do some more testing?
>>> How can I be helpful?
>>> Richard
>>>
>>> On Tue, 22 Mar 2022 at 17:30, 'DRC' via VirtualGL User
>>> Discussion/Support <virtual...@googlegroups.com
>>>
>>> OK, that's an important data point. Now I just need to
>>> be able to reproduce the failure. :|
>>>
>>>
>>> On 3/22/22 2:56 PM, Richard Ems wrote:
>>>> Hi DRC,
>>>>
>>>> Just tested and it does not fail just using the local X
>>>> server. See attached output.
>>>>
>>>> Thanks,
>>>> Richard
>>>>
>>>>
>>>> On Tue, 22 Mar 2022 at 15:19, 'DRC' via VirtualGL User
>>>> Discussion/Support <virtual...@googlegroups.com
>>>>>>
>>>>>> Please clarify how this is supposed to
>>>>>> work. 'module load' is not a valid Bash
>>>>>> command.
>>>>>>
>>>>>> ==========
>>>>>>
>>>>>> > vglrun ./Tec360_FlowViz_batch.sh
>>>>>> ./Tec360_FlowViz_batch.sh: line 3: module:
>>>>>> command not found
>>>>>> Tecplot 2021.2.1.9698 - 21:07:35 Feb 3
>>>>>> 2022 [linux64-centos6.10]
>>>>>> Color maps directory :
>>>>>> /usr/local/tecplot/360ex_2021r2/colormaps
>>>>>> Configuration File :
>>>>>> /usr/local/tecplot/360ex_2021r2/tecplot.cfg
>>>>>> Process Temp Dir :
>>>>>> /usr/tmp/tecplot_drc/tpaqmYUE0
>>>>>> Tecplot internal font file:
>>>>>> /usr/local/tecplot/360ex_2021r2/tecplot.fnt
>>>>>> LaTeX Configuration File:
>>>>>> /usr/local/tecplot/360ex_2021r2/tecplot_latex.mcr
>>>>>> Your license expires at the end of
>>>>>> 26-Mar-2022.
>>>>>> Update your license, or contact
>>>>>> sup...@tecplot.com
>>>>>>>> line 349: 276303 Aborted (core
>>>>>>>> dumped) "$0-bin" "$@"
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Any ideas what is going wrong?
>>>>>>>> When running with tec360 option
>>>>>>>> "--osmesa" all runs fine, but it
>>>>>>>> takes 17 min to run.
>>>>>>>>
>>>>>>>> --osmesa
>>>>>>>> Bypass the OpenGL library and
>>>>>>>> use the LLVM OSMesa library for
>>>>>>>> off-screen
>>>>>>>> image rendering. This option
>>>>>>>> does not require an X server connection.
>>>>>>>>
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Richard
>>>>>>>
>>> --
>>> You received this message because you are subscribed to
>>> the Google Groups "VirtualGL User Discussion/Support" group.
>>> To unsubscribe from this group and stop receiving emails
>>> from it, send an email to
>>> virtualgl-use...@googlegroups.com
>>> https://groups.google.com/d/msgid/virtualgl-users/14b5014b-6277-0440-e9c5-11f7e517419a%40virtualgl.org
>>> --
>>> You received this message because you are subscribed to the
>>> Google Groups "VirtualGL User Discussion/Support" group.
>>> To unsubscribe from this group and stop receiving emails from
>>> it, send an email to
>>> virtualgl-use...@googlegroups.com
>>> <https://groups.google.com/d/msgid/virtualgl-users/CAEqczzzNxRQN5N5sFQr08coZV67O5WkNCTBpGa9kE_d5x%2BfmbA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
>> You received this message because you are subscribed to the
>> Google Groups "VirtualGL User Discussion/Support" group.
>> To unsubscribe from this group and stop receiving emails from
>> it, send an email to
>> virtualgl-use...@googlegroups.com
>> <https://groups.google.com/d/msgid/virtualgl-users/0a7a6eb3-5afc-3a8c-9b4e-a1212ecb17ab%40virtualgl.org?utm_medium=email&utm_source=footer>.
>> --
>> You received this message because you are subscribed to the Google
>> Groups "VirtualGL User Discussion/Support" group.
>> To unsubscribe from this group and stop receiving emails from it,
>> send an email to virtualgl-use...@googlegroups.com
>> <https://groups.google.com/d/msgid/virtualgl-users/CAEqczzw%2B3Na122FXLwxCtyYYRYK_NOJ67X7AAGsjniLABpJuUg%40mail.gmail.com?utm_medium=email&utm_source=footer>.
> --
> You received this message because you are subscribed to the Google
> Groups "VirtualGL User Discussion/Support" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to virtualgl-use...@googlegroups.com
> <https://groups.google.com/d/msgid/virtualgl-users/1a86354e-6214-97fc-c643-d4d325323085%40virtualgl.org?utm_medium=email&utm_source=footer>.
> --
> You received this message because you are subscribed to the Google
> Groups "VirtualGL User Discussion/Support" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to virtualgl-use...@googlegroups.com
> <https://groups.google.com/d/msgid/virtualgl-users/CAEqczzwyD1DQkdSKvF-gDUSo%2B4xvZDaVvwZ2A2XQR5sVrr0AoA%40mail.gmail.com?utm_medium=email&utm_source=footer>.
Richard Ems
I will add VGL_PROBEGLX=0 to our environment.
Kind regards
Richard
> El 1 abr. 2022, a la(s) 19:53, 'DRC' via VirtualGL User Discussion/Support <virtual...@googlegroups.com> escribió:
>
> The issue is related to the fact that VirtualGL probes the 2D X server to see if it has any stereo visuals. Because of libglvnd, this causes the system's Mesa GLX vendor library to be loaded into the 3D application process. This apparently interacts badly with Tecplot 360, perhaps because Tecplot 360 has its own Mesa implementation. I have no idea why the issue only occurs in batch mode, but I do know that the issue is triggered by a specific sequence of GLX calls, so that probably has something to do with it.
> To view this discussion on the web visit https://groups.google.com/d/msgid/virtualgl-users/29357079-1148-4d2d-d299-16ff80719f62%40virtualgl.org.
DRC
VGL_PROBEGLX=0 automatically, so you can get the fix simply by upgrading
your TurboVNC Server RPM.
https://turbovnc.org/DeveloperInfo/PreReleases
Ultimately, I was unable to reproduce the issue outside of Tecplot 360,
so I hypothesize that the issue is related to the fact that Tecplot 360
links with its own version of Mesa. If my hypothesis is correct, then
it doesn't make sense to change VirtualGL's visual probing behavior in
general, because technically, VirtualGL isn't doing anything wrong.
However, it does make sense to disable that behavior in environments for
which it will never be useful, including TurboVNC.
Notes:
-- At the time VirtualGL builds the visual attribute table for the 2D X
server, VGL doesn't actually know which image transport will be used.
(In fact, the image transport can be changed dynamically using the
VirtualGL Configuration dialog.) Thus, unfortunately VGL cannot
intelligently set a default value for VGL_PROBEGLX.
-- The issue can also be worked around, without disabling GLX probing,
by setting __GLX_VENDOR_LIBRARY_NAME=nvidia. This simulates what
VirtualGL would do if it interposed the GLX_EXT_libglvnd extension.
That doesn't seem to be the right approach in general, however, because
nVidia's GLX client library doesn't fully work with an X server running
Mesa. This seems to support my hypothesis that VGL is doing the right
thing and that the issue is application-specific.
DRC