[QLab] Performance issues

[Jonathan Wald]

Hi folks

I'm running a theatre production with Qlab, and I'm having some performance issues I'm hoping you can help with.  

The production uses two LCD monitors and a video projector, all three of which I'm hoping to operate through Qlab; and there are sound cues which I'd like to operate through Qlab as well.

I am running the program on a Mac Pro 2.66 1 X Quad Xeon with a 2nd graphics card so there are 4 DVI outputs; the monitors and the projector, as well as an LCD screen for the desktop, are all connected using VGA cabling, via DVI to VGA adapters.  I can't use DVI cabling because the distances are too long - up to 30 meters.  The system is 10.5.

Currently, the media is on an external hard drive.  The video files were originally H264 mp4s and 1080i50 mov files, but I have since tried using photo jpeg as my codec, which seems to work a bit better.  The original resolution for the footage is 1920 x 1080; but the computer seems unable to process multiple signals at that resolution through VGA cables, so I'm experimenting with shrinking the footage down to 1280 x 768; the projector is 1024x768.

The big problem I'm having - aside from the inability of the computer to deal with the full HD resolution - is that my videos jump a fair bit, and often the picture will freeze for seconds at a time while the audio continues.

I discovered one thing that was causing jumps - firing a group with a video cue and a stop cue in it (to stop a previous video cue).  When the stop cue fires, the current video cue skips.  So I've taken my stop cues out of groups, and that helps a bit, though it is unfortunate not to be able to group them.

Another thing that causes jumps or freezes is running too many video cues at once - even three at a time seems to gum up the system - almost like overhead is accruing, even though it's not supposed to.

But I'm still getting some jumps, and definitely still getting freezes.  I was thinking I could try putting the media on the computer's hard drive, which is otherwise empty - it's got a couple of hundred gigs available.  Or I could try upgrading the system to 10.6.

Based on what I've described, does anyone have any sense of whether those fixes would help, or suggestions for possible others?

Thanks,

Jonathan 

[Dominic.bilkey]

Johnathon, 

     Can I ask which graphics cards you are using?

Dom

________________________________________________________
WHEN REPLYING, PLEASE QUOTE ONLY WHAT YOU NEED. Thanks!
Change your preferences or unsubscribe here:
http://lists.figure53.com/listinfo.cgi/qlab-figure53.com
Follow Figure 53 on Twitter here: http://twitter.com/Figure53

[Jonathan Wald]

Hi Dom,

It's got 2 x ATI Radeon HD 2600 XT.

And I looked at the specs - it actually turns out it's a 2x Quad Core 2.8 ghz and not 2.66.  The rental house must have upgraded me!

Made another interesting discovery about the stop cues - even when I separate them out of a group - ie, I don't run a video cue and a stop cue (referencing a previous video cue) at the same time, the stop cue can still interfere with the current playing video cue, causing the image to freeze for a few seconds while the sound continues on.  I wonder if it could be solved by changing the number/frequency of the keyframes?  

Thanks,

Jonathan

[Stephen Pruitt]

Hey Jonathan, 

I've run multiple video outs on the exact system you're using many times, so I can tell you that it can be done. The 2600 cards are not super-great, but they are capable of keeping up if you've got the system set up right. I'm almost certain that getting your video to the internal drives and making sure you have enough RAM will solve 90% of your problems, if not all, but here are some suggestions, in order of importance, from my experience. 

First thing - definitely put the video on your internal hard drives. An external just isn't going to be able to keep up with that data stream - it won't even be close for running multiples. Ideally, if you can dig up another hard drive or two, take advantage of the 4 internal bays and run off two or three hard drives. If you're pulling your data from multiple places, the computer will be able to keep up much better. The ideal set-up would be to fill up those other three internal bays and put the data for each output on a different drive, keeping the system drive separate. 

Second - How much RAM does the system have? You're going to need a minimum of 8GB, and things will probably work better with 16. 

Third - I'd definitely drop down to 1024 x 768, since that's the native resolution of your projector. Even though the LCD panels are capable of showing higher, no one is ever going to see that level of detail from the audience. 

Fourth - This one was counter-intuitive for me. If you're running the same video simultaneously on more than one output (say, one LCD and the projector), create a different cue for each one, rather than making it one cue and using two screens within the cue. Even though it seems like it would be easier on the system to do it once and send it to multiple outputs, Qlab prefers having one cue per send, even if they are doing the same thing. 

Fifth - Make sure your video cards are in the bottom two PCI slots. Occasionally someone will bump one up to the third, to get more airflow between them or whatever, but those 3rd and 4th slots receive much less bandwidth than the first two (4x vs 16x). Ideally, you might call your supplier of the computer and ask them for a third card, then run one send off each one, plus your system screen doubled up on the first one. If each card only has to handle one signal, then that bandwidth issue becomes less important. 

Most likely, if you make those tweaks, you'll be able to move back to H.264 rather than photoJPG, if you want to. 

Stephen Pruitt

Fluxion Scenic and Light

www.fluxiondesigns.com

[Sean Dougall]

I'd absolutely second all of Stephen's suggestions. One quick
clarification, though:

On 2/26/11 at 5:42 AM, stephen....@gmail.com (Stephen
Pruitt) wrote:

>Second - How much RAM does the system have? You're going to
>need a minimum of 8GB, and things will probably work better
>with 16.

Because QLab v2 relies on QuickTime 7, it's stuck in 32-bit
land, which means it can only access the first 4 GB of RAM.
Having 8 GB rather than 4 or less will definitely make a
difference, because most of the OS X 10.5 processes are 64-bit
and can use the higher memory space, out of QLab's way. But
unless you're running other memory-intensive applications at the
same time, you may not actually see much of a difference between
8 GB and 16.

We're hoping this can change in the future, and that QLab v3 can
be 64-bit. No promises yet, though -- Apple has been letting
QuickTime X languish half-finished for a while now. But finding
a way to make it work is high priority.

Cheers,
Sean

[Jonathan Wald]

Hi Stephen and Sean,

Thanks so much for your suggestions and advice.  It's good to know that what I want to do can be done!

I will first of all move the media to the internal drive and see if I can get the rental house to up the RAM.  Best I can tell, the video cards are already in the two bottom slots - I've never used a Mac Pro before, but that's how it looks from the outside - the DVI outputs are in the lower half of the card area.

In terms of the resolution, is it important that all the screens are running at the same resolution, or can they run at different resolutions?  I've got a lot of clips, and it'd be a big job to redo the resolution of all of them - I'm assuming that it's better to change the resolution of the media clip itself, rather than making QLab or the computer adjust the resolution on the fly?

Thanks again, I really appreciate the help.

Cheers

Jonathan

[Christopher Ashworth]

On Feb 26, 2011, at 2:35 PM, Jonathan Wald wrote:
>
> In terms of the resolution, is it important that all the screens are running at the same resolution, or can they run at different resolutions? I've got a lot of clips, and it'd be a big job to redo the resolution of all of them - I'm assuming that it's better to change the resolution of the media clip itself, rather than making QLab or the computer adjust the resolution on the fly?

It's not important for all screens to be the same resolution.

It's best if the media can match the resolution of the screen it will be displayed on. That minimizes the amount of work required, since no scaling is needed.

-C

[Jonathan Wald]

Thanks - I realise my original post wasn't clear - while the projector is 1024x768 native, the two monitors are actually full HD monitors - it's just the computer won't run them at HD, so I was shrinking that footage down to 1280x768 and shrinking the projector footage to 1024x768.

Thanks for your help,

Jonathan

[luckydave]

> Thanks - I realise my original post wasn't clear - while the projector is 1024x768 native, the two monitors are actually full HD monitors - it's just the computer won't run them at HD, so I was shrinking that footage down to 1280x768 and shrinking the projector footage to 1024x768.

If you're connecting to the monitors with VGA or DVI, they'll likely represent themselves to the computer as 1366x768 - the 16:9 partner to 1024x768. If you connected with an HDMI cable, you'd probably be able to do at least 1080i, if the monitors support that. Oddly, for a 720p tv, 1366x768 is its native pixel depth most of the time. Often, 1080p tv's that are less than 50" or so actually have the same number of pixels, or so I gather. Remember, HD is a marketing term more than a standard.

Anyway, the point is, if your movies are scaled before they're played back, that's less computation in the moment of playback, and so likely better performance. I think the biggest issue you'll have with this one is hard disk access. Three movie files at the same time is a lot to ask. Add a hard drive or three, as Stephen suggested.

luckydave
luck...@figure53.com

[sam kusnetz]

since there is unfortunately no way to quantitatively measure GPU performance, from the perspective of a user, there is no real way to be sure which part of the machine is getting overwhelmed and causing your dropouts.

i bet a whole dollar that it's either the hard disk or the graphics card, and lucky for us all there is a super easy way to figure it out: make several versions of your videos, some of which are highly compressed to have small file sizes, and others which are gently compressed to have large files sizes. if the large-file-size videos play back smoothly, then your problem was overtaxing the video cards. if your small-file-size videos play back smoothly, then your problem was overtaxing the hard disk.

i recently did a show with lots of video that required transparency and so all my cues were encoded with the animation codec which uses a lot of disk space, but apparently is fairly light on the GPU. i ran that show off a pre-unibody macbook pro with the cues on an external 7200 RPM hard disk connected via firewire 800. no problem.

another trick i've learned is to encode a bunch of versions of videos with progressively lower data rates... it's amazing how low the data rate can get for some videos before you see artifacts. the lower the data rate, the happier the hard drive.

cheerio
sam

[Jonathan Wald]

Hi Sam,

Thanks for this info.

I'm doing my editing in Final Cut Studio. As far as compression goes, do you notice any difference if you use a codec like animation or photo jpeg in the sequence settings themselves, versus using a higher quality codec in the sequence, and then compressing the file via compressor or another program?

Thanks,

Jonathan

[Jonathan Wald]

HI folks,

I wrote a week ago about problems I was encountering with Qlab and multiple videos, and you all gave me some very helpful advice.  Unfortunately, I tried all the fixes you suggested which I was able to, and I'm still having the issues, so I'm wondering if you have any more thoughts.

To recap, I'm running a theatre production with Qlab which uses two LCD monitors and a video projector.  The monitors are set at 1280x768; the projector is native 1024x768.  The monitors and the projector, as well as an LCD screen for the desktop, are all connected using 20 - 40meters of VGA cabling, via DVI to VGA adapters.

I have a Mac Pro 2.88 2 X Quad Xeon with a 2nd graphics card so there are 4 DVI outputs.  I've now upgraded to 12GB of ram; and there are now 3 hard drives on the computer, all 7200rpm, and they're nowhere near full.  The system is 10.5.  The second graphics card is in the slot with the highest possible bandwidth.  They are 2 x ATI Radeon HD 2600 XT.

The big problem I'm having - aside from the inability of the computer to deal with the full HD resolution - is that my videos jump a fair bit, and often the picture will freeze for seconds or more at a time while the audio continues.

I've compressed and output them all in photo JPEG, and even reduced the quality so that the files are very small.  And I've moved the videos to different hard drives. That helps, particularly getting small file sizes, but still, when I try and run three videos at once, the videos freeze, and often synch is never recovered - I have to restart the program in order to stop the lag.  (Two videos is usually okay.)

I discovered one thing that was causing jumps - when a stop cue fires, the current playing video often skips.  So I've positioned stop cues at places where that matters less.

I wasn't able to upgrade the graphics cards unfortunately - but my understanding from a previous poster was that because shrinking the file size helped, it was likely not a graphics card issue.

I tried booting in safe mode; but when I did, Qlab wouldn't recognise my projector, and almost immediately crashed!

Any more ideas?  My show opens tomorrow, and I'm now at a point where I am going to have to bring in a second computer to operate one of the video screens.

Thanks,

Jonathan

[Christopher Ashworth]

Hi Jonathan,

On Mar 6, 2011, at 3:02 PM, Jonathan Wald wrote:
>
> Any more ideas?

Did you try assigning each cue to only a single screen/projector?

-C

[Jonathan Wald]

Yes - mostly the monitors and projectors are running different videos, which start and stop at different times - so even at the start, I was rarely running video to more than one screen/projector at the same time. But the few times they do run the same thing, I made different files, each of which is the correct resolution for the screen/projector, so that Qlab doesn't have to do any scaling.

J.

[Christopher Ashworth]

Hm...gonna have to ponder this.

One note on the following:

On Mar 6, 2011, at 3:02 PM, Jonathan Wald wrote:
>

> I wasn't able to upgrade the graphics cards unfortunately - but my understanding from a previous poster was that because shrinking the file size helped, it was likely not a graphics card issue.

I don't recall what the cards were, and they're probably fine, but FWIW I'd hesitate to say that any component of the rendering chain can be easily counted out as an issue unless it is explicitly tested.

I haven't ever worked with long cable runs using video, and I don't know why it would matter, but out of curiosity have you tried connecting the machine up to screens with "normal" length cabling to see if there was any difference?

Also, I forget, did you send us a copy of the workspace to look at? If not, try sending a copy to sup...@figure53.com and I'd like to take a look at it.

Best,
Chris

[Christopher Ashworth]

On Mar 6, 2011, at 3:22 PM, Christopher Ashworth wrote:
>
> I don't recall what the cards were,

Er, duh. Sorry! You told me what they were in your last post. Anyway, point still stands.

[sam kusnetz]

>> I wasn't able to upgrade the graphics cards unfortunately - but my understanding from a previous poster was that because shrinking the file size helped, it was likely not a graphics card issue.
>
> I don't recall what the cards were, and they're probably fine, but FWIW I'd hesitate to say that any component of the rendering chain can be easily counted out as an issue unless it is explicitly tested.

it was i who suggested this logic, my thinking being that a version of the video with reduced file size with equal picture quality necessarily makes the graphics card work harder while making the hard disk work less. ergo, if that scenario results in smoother playback, it can be a good indicator that the hard disk was a bottleneck and the video card was not.

not proof, mind you.

i would spend good money on a video card performance monitor. sigh...

regarding the OP, this is seriously vexing to me because your description of your setup sounds quite reasonable to me and i feel like i've had success with very similar setups. i'll let you know if i think of anything...

cheerio
sam

[Ted Pallas]

Long cable runs - I drop a distribution amp into anything over 50ish feet.  If you're ONLY running VGA in your cable path you're generally ok up to 150' or so, but after 75' interference becomes more and more dangerous.

Ted Pallas
Live Media Designer
Sandwich Construction Consultant
ted dot pallas -at- gmail dot com
516.286.9661

[Christopher Ashworth]

On Mar 6, 2011, at 5:13 PM, sam kusnetz wrote:

> i would spend good money on a video card performance monitor. sigh...

http://www.atpurpose.com/atMonitor/

Includes some stats about the graphics cards, such as available vram, current FPS for screens, etc.

On Mar 6, 2011, at 5:13 PM, sam kusnetz wrote:
>
> regarding the OP, this is seriously vexing to me because your description of your setup sounds quite reasonable to me

My thought as well. :/

-C

[Jonathan Wald]

Hey thanks everyone for your continuing feedback.

I am only running VGA in my cable path, so I would have thought I'd be okay.  

I don't know if the video lengths are relevant, but some are quite short; and I have a number of cues that I put on infinite loop, though never many at once.  

I thought about using the "reset" command instead of the "stop" command, but haven't tried that yet - the problem occurs immediately in the show run in any case, so I wouldn't have thought anything was being held over.....

Jonathan

[Keith Smith]

On 7 Mar 2011, at 01:47, Jonathan Wald wrote:

> I am only running VGA in my cable path, so I would have thought I'd be okay.

Your lengths sound fine to me.

> [...] and I have a number of cues that I put on infinite loop, though never many at once.

>
> I thought about using the "reset" command instead of the "stop" command, but haven't tried that yet - the problem occurs immediately in the show run in any case, so I wouldn't have thought anything was being held over.....

Two things here:

First:
In my experience (well, I'm newish to QLab, but have been doing video for ages) Quicktime doesn't handle infinite video loops very well (there is always a bit of a lag at the loop point) and QLab seems to handle them even less well (very laggy at the loop point). In a recent production (last weekend in fact) I had a 10 second clip that loops seamlessly but never played back correctly at the loop point (no matter what player I used, QLab included). To fix it, I created a 15 minute clip (which more than covered the period the talent needed) in Final Cut which just contained the 10 second clip over and over again. That way I didn't need to rely on the looping feature and all worked well in QLab.

Second:
When you say "occurs immediately in the show run" can I just make sure that you are pre-loading the cues before hitting go? If your problem is right at the start then perhaps QLab is just busy collecting and buffering all the media. If you haven't already done so, you may want to put all these cues in a group so that the entire group is pre-loaded before you execute the go.

Regards,
Keith.

[Jonathan Wald]

Hi Kevin,

I found the same thing with the loops - and have been using the same solution, creating longer video files so that the break in the loop happens less often. And trying to balance that with not making the file size too long!

With pre-loading, I have noticed - sometimes, not consistently - that it does help avoid lag at the very start of a cue. However, this problem is bigger than that. This is a case of, I load cue one on the first monitor, which is a 7-minute infinitely looping clip of approximately 300mb - getting much smaller seriously compromised quality. Then I try and load (with or without pre-load) another much smaller infinitely looping clip on monitor 2, and sometimes it plays fine; but more often both of them start seriously freezing, or jumping, or both. And it's not just those two cues, or I'd think it was a problem with the files....

J.

>
> First:
> In my experience (well, I'm newish to QLab, but have been doing video for ages) Quicktime doesn't handle infinite video loops very well (there is always a bit of a lag at the loop point) and QLab seems to handle them even less well (very laggy at the loop point). In a recent production (last weekend in fact) I had a 10 second clip that loops seamlessly but never played back correctly at the loop point (no matter what player I used, QLab included). To fix it, I created a 15 minute clip (which more than covered the period the talent needed) in Final Cut which just contained the 10 second clip over and over again. That way I didn't need to rely on the looping feature and all worked well in QLab.
>
> Second:
> When you say "occurs immediately in the show run" can I just make sure that you are pre-loading the cues before hitting go? If your problem is right at the start then perhaps QLab is just busy collecting and buffering all the media. If you haven't already done so, you may want to put all these cues in a group so that the entire group is pre-loaded before you execute the go.
>
>
> Regards,

> Keith.________________________________________________________

[Geoff Hollingshead]

I'm not sure the file sizes would be much of an issue.

Back in the fall I was running a show with 2 panasonic dw6300 projectors image blended and my qlab monitor. I was running pretty much raw video (approx 1gb of data per min of video) outputting at the projectors native resolution of 1280x800 (I believe).

I don't remember the exact system specs but it was about a dual core 2.8ghz processor, 6gb of ram, 2 hard drives (one for the system drive running apps, the other with all the show files)

I had 2 graphics cards, I believe they are the geforce cards, each card would have 1 projector and 1 display screen for the operator. So a setup similar to what you are running.

As pervious people have stated I think there could be an issue using the infinite loop. You might be better off creating your loops in final cut and then exporting.

Not sure if this helps but just letting you know what I've had success with in the past.

Cheers

Geoff Hollingshead
Head of Sound
Arts Club Theatre

Sent from my iPhone
>
------------------------------ IMPORTANT NOTICE ------------------------------
This email transmission and any accompanying attachments contain confidential
information intended only for the use of the individual or entity named above.
Any dissemination, distribution, copying or action taken in reliance on the
contents of this email by anyone other than the intended recipient is strictly
prohibited. If you have received this email in error please immediately delete
it and notify sender at the above email address.
------------------------------ IMPORTANT NOTICE ------------------------------

[sam kusnetz]

> http://www.atpurpose.com/atMonitor/
>
> Includes some stats about the graphics cards, such as available vram, current FPS for screens, etc.

oh my god! atmonitor! where have you been all my life?

thank you, this is excellent.

regarding your issues with stops causing stutter in playing cues, do you mostly have several stops firing at once? if you haven't already, try spacing them out.

cheerio
sam

[Jonathan Wald]

Hi

I do still have some media files on the drive running the apps - as I have three different screens to go to, and only three hard drives, I left some on the main drive so that the other drives never had to share. Could that be an issue?

I'll have a look at getting rid of the infinite loops entirely - a huge amount of work though, may not be able to do it for this show! Would be interesting to know if that's the issue....

Thanks,

Jonathan

On 07/03/2011, at 1:12 PM, Geoff Hollingshead wrote:

> I don't remember the exact system specs but it was about a dual core 2.8ghz processor, 6gb of ram, 2 hard drives (one for the system drive running apps, the other with all the show files)
>

> As pervious people have stated I think there could be an issue using the infinite loop. You might be better off creating your loops in final cut and then exporting.
>
> Not sure if this helps but just letting you know what I've had success with in the past.
>
> Cheers
>
> Geoff Hollingshead
> Head of Sound
> Arts Club Theatre

________________________________________________________

[Jonathan Wald]

Yes, I've split the stops so that wherever possible I don't have too many at one time - definitely helped, but didn't solve, the problem.

J.

[Christopher Ashworth]

On Mar 6, 2011, at 10:17 PM, Jonathan Wald wrote:
>
> I'll have a look at getting rid of the infinite loops entirely - a huge amount of work though, may not be able to do it for this show! Would be interesting to know if that's the issue....

I do not know a reason that setting a video on an infinite loop would cause any issues. (Unless there is a memory leak in that circumstance that I do not know about -- which of course I definitely want to know about if people are seeing it.)

The loop point of a video is indeed currently susceptible to a delay, because QLab does not currently double-buffer video (i.e. have a second video ready and waiting at the start time). At the loop point it "rewinds" as quickly as possible to the start time and then proceeds to continue playing the file.

To my knowledge, this would not account for the kind of drops you are seeing.

Would love to see a copy of the workspace file, if possible. May not lead anywhere, but could possibly provide more clues.

Best,
Chris

[Jonathan Wald]

Hey Chris,

What kind of copy? The file itself, or a screen cap?

J.

[Christopher Ashworth]

On Mar 6, 2011, at 10:42 PM, Jonathan Wald wrote:

> Hey Chris,
>
> What kind of copy? The file itself, or a screen cap?

The QLab ".cues" file itself. You can send a copy to sup...@figure53.com. (Not the media files, just the QLab workspace file.)

[Mark Valenzuela]

A total shot in the dark, perhaps, but do you have the last released version of 10.5? (was it 10.5.8?)

Is it possible the culprit could be related to the OS or to whatever version of QuickTime is on the machine?

Mark V.

Sent from my iPod. Pardon the typos.

[Stephen Pruitt]

I'm another one stumped here. As I said before, I've done multiple videos on this same system, and once I made some basic changes, that I put out before in this thread, I didn't have any issues. 

From the description of the issues, this really sounds like a processing or graphics problem. If it were VGA related, the video signal would be iffy, but I can't see it making these sorts of problems. If it were the a hard drive/data flow issue, I think it would be occurring when videos start, rather than stop. 

I do think that altering the loops to be full length movies would be helpful. It was good to hear Chris's explanation of why video loops have never really worked for me in Qlab, and I've had to do the same thing a few times, after going through a period where I thought the problem was my 13" MacBooks's lack of a discrete graphics card. Finally after running one particular sequence on a tricked out Mac Pro, I decided that it was just the looping process itself and started building my loops in Final Cut or iMovie. 

Out of curiosity, since the problem is the Stop commands, would it be possible to just let those videos run in the background until there is a more opportune time to stop them? Fade them out and let them go until other video has also completed, perhaps? For that matter, have you tried stopping them via Fades (stop when complete) instead of a straight Stop Cue? Have you tried Pausing them first, then stopping them? I've sometimes found that even though doing something a slightly different way *shouldn't* make a difference, or is a messier bit of programming, it just works. Not always, obviously, but if I were having these sorts of issues the day before the show opened, it would be worth a try. 

Stephen Pruitt

Fluxion Scenic and Light

www.fluxiondesigns.com

To: "Discussion and support for QLab users." <ql...@lists.figure53.com>
Subject: Re: [QLab] Performance issues Redux
Message-ID: <856870D1-F256-4596...@yahoo.com>
Content-Type: text/plain; charset="us-ascii"

Hey thanks everyone for your continuing feedback.

I am only running VGA in my cable path, so I would have thought I'd be okay.  

I don't know if the video lengths are relevant, but some are quite short; and I have a number of cues that I put on infinite loop, though never many at once.  

I thought about using the "reset" command instead of the "stop" command, but haven't tried that yet - the problem occurs immediately in the show run in any case, so I wouldn't have thought anything was being held over.....

Jonathan