[eq-dev] Weird performance drop with multi-pipe configs
m0bl0
Hi,
sorry if this is not strictly Equalizer-related, but maybe someone here can
help...
In my Equalizer application, I'm seeing very weird performance drops when I
use more than one pipe per node (on a multi-GPU machine), e.g.:
In a simple one-window config:
520 fps
In a one-pipe, 2-window config (where both windows render exactly the same
scene):
220 fps (about as expected)
The "interesting" case:
In a 2-pipe, 2-window config, with exactly the same output as the previous
one:
60 fps (here I would expect 500 fps)
In particular, it makes almost no difference if both pipes map to the same
physical GPU or to two different ones (with device 0 / device 1).
With a 2-node config (still on the same one machine) I'm getting > 500 fps
again, so only the case with multiple pipes per node is broken. eqPly works
fine with the same (2-pipe) config, so the problem should be in my
application. However, some profiling shows that in the 2-pipe case, all the
gl* calls just take an unreasonable amount of time (regular stuff like
glBindTexture, glBindBuffer, glUniformxx etc).
Does anybody have any idea what could cause this?
Thanks,
Marc
--
View this message in context: http://software.1713.n2.nabble.com/Weird-performance-drop-with-multi-pipe-configs-tp6052917p6052917.html
Sent from the Equalizer - Parallel Rendering mailing list archive at Nabble.com.
_______________________________________________
eq-dev mailing list
eq-...@equalizergraphics.com
http://www.equalizergraphics.com/cgi-bin/mailman/listinfo/eq-dev
http://www.equalizergraphics.com
Stefan Eilemann
On Tue, Feb 22, 2011 at 5:08 PM, m0bl0 [via Software]
<[hidden email]> wrote:
> In my Equalizer application, I'm seeing very weird performance drops when I
> use more than one pipe per node (on a multi-GPU machine), e.g.:
> In a 2-pipe, 2-window config, with exactly the same output as the previous
> one:
> 60 fps (here I would expect 500 fps)
>
> In particular, it makes almost no difference if both pipes map to the same
> physical GPU or to two different ones (with device 0 / device 1).
>
> With a 2-node config (still on the same one machine) I'm getting > 500 fps
> again, so only the case with multiple pipes per node is broken. eqPly works
> fine with the same (2-pipe) config, so the problem should be in my
> application.
>
A couple of ideas - I assume you are using nVidia cards (Quadro on
Windows or any on Linux):
- Are you creating any auxiliary context in your code?
- Try setting EQ_CHANNEL_IATTR_HINT_STATISTICS NICEST, this causes
regular glFinish()'s which, contrary to popular belief, improve
performance on certain drivers with multi-threaded rendering.
- Try using the last 186 series driver.
Can you outline what your rendering code does and which GL features it uses?
If all fails, do you have a nVidia contract to talk to? This looks
very much like a driver-related thingie.
HTH,
Stefan.
View this message in context: Re: Weird performance drop with multi-pipe configs
m0bl0
Hi,
thanks for the reply.
Stefan Eilemann wrote:
>
> A couple of ideas - I assume you are using nVidia cards (Quadro on
> Windows or any on Linux):
>
Yes, this is with two Quadro FX 5800s. The numbers are from Win7, but the
behavior on Linux is very much the same.
Stefan Eilemann wrote:
>
> - Are you creating any auxiliary context in your code?
>
No.
Stefan Eilemann wrote:
>
> - Try setting EQ_CHANNEL_IATTR_HINT_STATISTICS NICEST, this causes
> regular glFinish()'s which, contrary to popular belief, improve
> performance on certain drivers with multi-threaded rendering.
>
This seems to improve the performance slightly (65 fps instead of 60).
Stefan Eilemann wrote:
>
> - Try using the last 186 series driver.
>
I'm using the 266 drivers. Do you mean I should go back to the older ones?
This wouldn't work on Win7 because the GPU affinity was still broken then
(iirc), but I could try it on Linux.
Stefan Eilemann wrote:
>
> Can you outline what your rendering code does and which GL features it
> uses?
>
The rendering is pretty straightforward, it renders triangle lists from a
VBO with a simple vertex and fragment shader (just coordinate transforms and
texturing). It uses around 100 draw calls per frame (glDrawElements), and
binds a VBO, texture and shader for each one.
Stefan Eilemann wrote:
>
> If all fails, do you have a nVidia contract to talk to? This looks
> very much like a driver-related thingie.
>
To me too, but experience has taught me not to question others' code before
I'm certain that mine isn't broken ;)
But no, I don't have a direct nVidia contact.
Thanks again,
Marc
--
View this message in context: http://software.1713.n2.nabble.com/Weird-performance-drop-with-multi-pipe-configs-tp6052917p6060496.html
Sent from the Equalizer - Parallel Rendering mailing list archive at Nabble.com.
_______________________________________________
Stefan Eilemann
On Thu, Feb 24, 2011 at 3:29 PM, m0bl0 [via Software]
<[hidden email]> wrote:
>> - Try using the last 186 series driver.
>
> I'm using the 266 drivers. Do you mean I should go back to the older ones?
> This wouldn't work on Win7 because the GPU affinity was still broken then
> (iirc), but I could try it on Linux.
bet yet. Maybe there are newer drivers?
Unfortunately ever since the 190 drivers the affinity situation is
somewhat of a mess.
>> If all fails, do you have a nVidia contract to talk to? This looks
>> very much like a driver-related thingie.
>
> To me too, but experience has taught me not to question others' code before
> I'm certain that mine isn't broken ;)
you, try 186 on xp64, which afaik will give you proper performance. It
might be that you can do something in your code, but without getting
some hints on why this behavior happens with the new multi-GPU driver
code we will get nowhere.
> But no, I don't have a direct nVidia contact.
here). If I find the time, I'll follow this up on my end and keep you
posted.
Cheers,
m0bl0
Stefan Eilemann wrote:
>
> Correct, I didn't know you're on Windows 7. 266 or later is your best
> bet yet. Maybe there are newer drivers?
>
There are 267 drivers now, but they don't change anything here.
Stefan Eilemann wrote:
>
> I'm pretty confident this is a driver-related issue. If feasible for
> you, try 186 on xp64, which afaik will give you proper performance. It
> might be that you can do something in your code, but without getting
> some hints on why this behavior happens with the new multi-GPU driver
> code we will get nowhere.
>
I did some testing on an XP64 machine, but my application uses some features
from GL 3.2, which is supported only in the 191 and later drivers. I guess I
could rewrite it to use the extensions instead...
With the 191 or later drivers, my application (and also eqPly) crashes the
driver when I use an on-screen window on the second GPU (pbuffer or FBO
works). Is this a known bug? (eqPly with 186 works OK on the same machine).
Stefan Eilemann wrote:
>
> Try posting your problem on forums.nvidia.com (please post a link
> here). If I find the time, I'll follow this up on my end and keep you
> posted.
>
OK, I'll do that and post the link.
Cheers,
Marc
--
View this message in context: http://software.1713.n2.nabble.com/Weird-performance-drop-with-multi-pipe-configs-tp6052917p6072671.html
Sent from the Equalizer - Parallel Rendering mailing list archive at Nabble.com.
_______________________________________________
Stefan Eilemann
<ml-node+6072671-...@n2.nabble.com> wrote:
> With the 191 or later drivers, my application (and also eqPly) crashes the
> driver when I use an on-screen window on the second GPU (pbuffer or FBO
> works). Is this a known bug? (eqPly with 186 works OK on the same machine).
Yes. The multi-GPU implementation in the driver changed with the 190
series to support the Win7 driver model. This broke GPU-affinity
windows until the 265 series. On XP, you can activate the old
implementation using the "enablemmb.exe" from nVidia, if you can get
hold of it [1].
HTH,
Stefan.
[1] http://www.derivative.ca/wiki/index.php?title=Using_Multiple_Graphic_Cards
m0bl0
Stefan Eilemann wrote:
>
> Yes. The multi-GPU implementation in the driver changed with the 190
> series to support the Win7 driver model. This broke GPU-affinity
> windows until the 265 series. On XP, you can activate the old
> implementation using the "enablemmb.exe" from nVidia, if you can get
> hold of it [1].
>
OK, using the disableMMB.exe, it works on XP, but with exactly the same
performance problems as on Win7/Linux.
Also, I finally posted on the nVidia forums:
http://forums.nvidia.com/index.php?showtopic=194330
Cheers,
Marc
--
View this message in context: http://software.1713.n2.nabble.com/Weird-performance-drop-with-multi-pipe-configs-tp6052917p6077844.html
Sent from the Equalizer - Parallel Rendering mailing list archive at Nabble.com.
_______________________________________________
Stefan Eilemann
below one answer from nVidia:
-----
There are two issues affinity on Win7 - basic use of affinity with no Window and use of affinity with associated with window.
The use of affinity with no associated window should now be fixed in latest R265 drivers (Steve or Shalini will correct me if I'm wrong). So latest Quadro drivers on the web should have it.
The use of affinity with an associated window is still sub-optimal and will be fixed in a R270 driver which will be available in a month or so. Don't know the exact date at this stage.
-----
One workaround may be to use an affinity FBO for rendering, and a non-affinity window receiving an output frame from the FBO.
m0bl0
Stefan Eilemann wrote:
>
> The use of affinity with no associated window should now be fixed in
> latest R265 drivers (Steve or Shalini will correct me if I'm wrong). So
> latest Quadro drivers on the web should have it.
>
> The use of affinity with an associated window is still sub-optimal and
> will be fixed in a R270 driver which will be available in a month or so.
> Don't know the exact date at this stage.
> -----
>
> One workaround may be to use an affinity FBO for rendering, and a
> non-affinity window receiving an output frame from the FBO.
>
Hi Stefan,
thanks for the followup!
Unfortunately, using FBOs for rendering does not improve the performance in
my case (even without the output/input frame), on both XP and 7.
Oh well, at least there's some hope it might get fixed at some point...
Cheers,
Marc
--
View this message in context: http://software.1713.n2.nabble.com/Weird-performance-drop-with-multi-pipe-configs-tp6052917p6087761.html
Sent from the Equalizer - Parallel Rendering mailing list archive at Nabble.com.
_______________________________________________