Broken Video Cue File Paths

[Michael Heath]

I'm having an occasional,  recurring issue with video cue target's being erroneously marked as broken. 

Randomly (at least, I haven't found any pattern), every cue that references a particular image filewill be marked as broken. If I go to the cue's settings, the file box shows the path in yellow text rather than the expected stylized file hierarchy. As soon as I click on the path, and choose the exact same file in the same location, that cue is fixed, as well as all others that reference that file. I've ran tests before I fix the 'broken' file reference, and in every way I can think of the file exists and always had.

It may be worth noting that the files in question are on Google Drive, synchronized to the machine with Google Drive file stream. I am unsure if this is part of the issue, though, because there are cues that reference other files in the same folder that have no issues. In fact, I've had Group cues that reference two files in the same folder simultaneously, and one cue will break while the other is fine. I've also ran a bash script to constantly "test -f" the file in question, and the bash script never loses access to the file even when QLab does. 

Has anyone seen anything like this? Any suggestions?

[Sam Kusnetz]

Hello Michael

This is almost certainly due to the file being in Google Drive.

We strongly recommend against linking to media in shared folders, because the sharing mechanism (Google Drive in this case) may at any time remove, replace, or rename the file.

Google Drive is particularly opaque when it comes to figuring out what it does and when and why.

If you move this same file outside of Google Drive, does the issue persist?

Best

Sam

--
Contact support anytime: sup...@figure53.com
Follow Figure 53 on Twitter: https://twitter.com/Figure53
User Group Code of Conduct: https://figure53.com/help/code-of-conduct/
---
You received this message because you are subscribed to the Google Groups "QLab" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qlab+uns...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/qlab/CAHat%3D1QL%3D89hfEdLXHxmFsavt8oNbHP64-61xJMvjU-c0y62%2BA%40mail.gmail.com.

—

Sam Kusnetz | Figure 53

s...@figure53.com

[Michael Heath]

Thanks for the quick response!

I agree that it's not ideal, but the way our organization distributes and updates certain content, it's worked well. We'ved used (and use) this setup with many other production applications and never had any issues. 

You're right that with the opaqueness of Google Drive File Stream, it makes it almost impossible to prove (either way) if it's contributing to this issue. It's worth noting that the files in question are set to persistent in Google Drive File Stream, so they are locally cached and available offline. Certainly my next step is to move the files fully local, but with how intermittent this issue has been it will be a while before I have any definitive idea on if that 'fixes' the isuse.

I guess my bigger question is, I'm confused about how and when QLab 'tests' files. Even if there was a drop in access or other problem with Google Drive, it seems odd that Qlab would automatically test that file and mark the cue as broken, but have no automatic mechanism for restoring the cue once access is restored. If the access issue is temporary and transient, why isn't qlab's marking of the target as broken also transient? For that matter, why does it get marked as broken (and require manual intervention to fix) for cues that aren't loaded, pre-loaded, etc or in any way active? I'm guessing that qlab is doing some non-standard I/O stuff for performance reasons, but in that case it is very confusing for the qlab interface to display the 'broken' target as a (perfectly valid) file path. 

[Sean Dougall]

Hi Michael,

QLab isn’t doing any non-standard file I/O stuff, but Google Drive certainly is—and there are file system details that are generally hidden to the user that may be coming into play.

As a bit of background: QLab uses macOS’s built-in file system events to monitor changes to file targets of Video cues. (It does this on Video cues specifically, so that you can make changes to still images in Photoshop, etc., and see them immediately reflected in your projections.) If a file gets replaced or updated, one of two things can happen—sometimes the original file itself is modified, but more often a new file is created and swapped in at the same path (but with a new “inode”, in Unix speak). When this happens, QLab looks at the file path to reload from the new version.

Normally, if you’re saving from Photoshop or something, that replacement process is atomic, so by the time QLab gets notified that something has changed, the new file is in place and ready to be reloaded (so the cue never breaks). Shared file systems don’t make that guarantee; they’re designed to put a placeholder at the file path so that you can see that the file exists while they download its contents. If QLab gets a notification that the old file has disappeared, and it looks at the path for a new file but doesn’t find any valid data, it will consider the cue broken. Since it no longer has a valid inode to track, the OS’s file system event notifications stop coming in regardless of what you do at the path, so QLab never knows to look for a new file when the download is complete.

It’s true that we can’t say with 100% certainty that that’s what is happening with Google Drive. However, this isn’t a problem that has been reported outside of shared file systems, so I would second Sam’s recommendation to start by making truly local (not just cached) copies of the media files for QLab to use. There may be tweaks we could make on the QLab side that would work around this specific problem, but I would still be extremely wary of anything that can arbitrarily make a file disappear in the middle of a performance.

Cheers,

Sean

[Michael Heath]

Thanks Sean. 

I assume when you say "macOS's built-in file system events" you're talking about the FSEvent API? I had done a monitor app using kernel queues that didn't register anything on the file(s) in question, but I'm going to try the same thing with the FSEvent API. I've done a fair amount of programming using Google Drive File Stream file descriptors (in other contexts, not for QLab), so I want to dig into this a bit and submit a bug report to Google if applicable.

[Sean Dougall]

Correct, it uses FSEventStream to watch files for changes. The mechanism that cues use to store their file targets is a combination of bookmarks, absolute file paths, and relative file paths, because it has to deal with a lot of different contingencies around media files getting replaced, moved, bundled with the workspace, etc. It’s possible that another application that works, say, only with relative paths would be better able to handle a file disappearing and being replaced non-atomically, because it would only have one place to look to resolve the link—maybe that’s why the problem is coming up only with QLab?

In any case, I’m both curious to hear what you learn, and feel that making local copies will be both easier and more robust.

Cheers,
Sean