C++ MovieReader - updating inputs and managing external resources

[Tommy Hinks]

Hi,

I'm implementing a Gaffer node for reading frames from movie files using libav (ffmpeg) as the backend. My approach is based on the OpenImageIOReader in Gaffer. This means using a dual-node approach, where the ImageReader/OpenImageIOReader pair is replaced by a MovieReader/AvReader pair. 

As for the ImageReader, the MovieReader mostly just delegates work to child nodes. The two setups are very similar, except that some input plugs are slightly different.

The AvReader, however, is quite different from the OpenImageIOReader. Rather than generating filenames based on sequences (e.g. my_image.####.exr) we want to point the AvReader to one file and from there let the user select which video stream (by index) to read from. As the frame changes in Gaffer we seek and decode suitable image data from the video stream and present it as channel data. This means that we want to keep the file open until the input filename changes. We use a similar frame caching strategy as the OpenImageIOReader (having ripped out and borrowed the IECorePreview::LRUCache class), but we don't want to reopen the video file each time a frame is requested.

The first stumbling point is that we have an input plug (videoStreamIndex) that depends on another input plug (fileName). As I understand things, we are supposed to update the values of output plugs in the overridden compute-method, but when are we supposed to update input plugs? When a new file name is selected we would like to regenerate the video stream index presets and set the value to what we guess is the best stream (which obviously means that we have to open the file first and read the header).

The second questions is related to managing the decoder. In our case the decoder stores low-level resources to streams and other information that libav is able to read. The AvReader node contains a std::unique_ptr<Decoder> that needs to be reset when the file name changes. I guess we could check in compute() if the filename has changed, but I am wondering if there is a better (standard) way of managing external resources in Gaffer nodes? 

Any help is much appreciated!

Best,

T

[John Haddon]

Hi Tommy,

This sounds like a great project!

The first stumbling point is that we have an input plug (videoStreamIndex) that depends on another input plug (fileName). As I understand things, we are supposed to update the values of output plugs in the overridden compute-method, but when are we supposed to update input plugs? When a new file name is selected we would like to regenerate the video stream index presets and set the value to what we guess is the best stream (which obviously means that we have to open the file first and read the header).

The difficulty here is that the `fileName` plug doesn't necessarily have a single static value. If it contains a `${contextVariable}` expansion, or is driven by an expression, then the filename van vary dynamically based on Gaffer's context.

 

For presets, our usual approach is to compute them dynamically on demand, rather than to have static registrations. There's a pretty simple example of that in the ImageReaderUI, where we make presets for the `start.frame` plug based on querying the available frames dynamically. We compute the available frames as an output plug on the node, so that it gets cached using Gaffer's usual mechanisms and we don't repeat the query every time the presets are requested. You can see the preset registrations here : https://github.com/GafferHQ/gaffer/blob/main/python/GafferImageUI/ImageReaderUI.py#L159-L160.

Updating the `videoStreamIndex` value is trickier, and is the sort of thing we tend to avoid, because in Gaffer the notion of one "current" value often doesn't hold. But the way to do it is to connect to `plugSetSignal()`, which will be emitted when the value of the `fileName` plug changes, and from which you are permitted to change the value of the `videoStreamIndex` plug. There's an example of that in the Attributes node here : https://github.com/GafferHQ/gaffer/blob/main/src/GafferScene/Attributes.cpp#L62.

An alternative to messing about in `plugSetSignal` might be to allow the user to provide `-1` (or some other sentinel) as a value, meaning "pick the best one for me on the fly". This would have the benefit of being something you could do dynamically for any filename in any context. That's probably the more "Gaffery" approach, as a lot of Gaffer's power comes from using a small number of nodes but making them context-sensitive using expressions and spreadsheets.

The second questions is related to managing the decoder. In our case the decoder stores low-level resources to streams and other information that libav is able to read. The AvReader node contains a std::unique_ptr<Decoder> that needs to be reset when the file name changes. I guess we could check in compute() if the filename has changed, but I am wondering if there is a better (standard) way of managing external resources in Gaffer nodes?

We generally avoid any kind of stored state (in this case `unique_ptr<Decoder>`) on nodes, because as described above, there isn't "one current state", and in fact a node can be queried in any context from any thread at any time (and the `fileName` can depend on the context). Even with a static `fileName`, things like TimeWarps make life quite interesting :

 

In that example, the Merge will be interleaving calls to the MovieReader with different frame times, and different threads will be interleaving differently as tiles are computed in parallel. If the "current time" is part of the Decoder state, then you have a problem even if the filename is static.

One option would be to say "we're not going to support any of that", use `plugSetSignal()` to update `m_decoder`, and throw if you are called in an unsuitable context. That will preclude using a lot of Gaffer's better features with the node, but might be fine if you only need to use it in constrained circumstances. And might also be a reasonably strategy to get something simpler working first.

The alternative would be to avoid all state on the AvReader node, and instead use a cache mapping filenames to decoders, and look up into that cache dynamically. That's the approach we take in the SceneReader node, where a single USD or ABC file will contain an entire animation, and we call `SceneReader::scene()` in every compute rather than store state on the node : https://github.com/GafferHQ/gaffer/blob/main/src/GafferScene/SceneReader.cpp#L711-L752. In that example the SceneInterface is analogous to your Decoder class and SharedSceneInterfaces is the cache. But where things might get trickier for you is with time handling. A single SceneInterface supports concurrent queries at any point in time, but my naive assumption would be that your Decoder might have it's own internal time state? How you deal with that would depend on the interface `libav` provides, which unfortunately falls within my vast pool of ignorance. Happy to do a bit of thinking about that if you can provide some `libav` education though.

Hopefully this helps keep you moving, although there are obviously some open questions still. Please do get back in touch if I can help with anything further...

Cheers...

John

Any help is much appreciated!

Best,

T

--
You received this message because you are subscribed to the Google Groups "gaffer-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gaffer-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gaffer-dev/5f600316-e64d-4d50-b91f-068a507f1c07n%40googlegroups.com.

[Tommy Hinks]

Hi John,

Thanks for your detailed reply! I appreciate the explanations about the fundamental Gaffer design choices, this will surely help us make more informed decisions going forward.

As for `libav`, you are correct in assuming that decoders store some internal state for retrieving images. Also, there are some additional parameters beyond `video_stream_index`, e.g. "filter graphs" and such. I'm thinking that a reasonable approach is to create a `DecoderCache` that stores opened decoders, which are essentially then a file handle together with metadata about the streams in the file.

However, I think it will also be necessary to have a `FrameCache` (similar to `OpenImageIOReader`) since decoding frames can be quite expensive, especially given that we are not necessarily decoding them in consecutive order, but rather support random access via "seeking". We will need to be careful when populating the `FrameCache` so that we don't end up with multiple threads requesting frames from the same decoder concurrently. As a first attempt we could always lock while decoding a frame, although this might not be the most efficient solution. A better solution might be to associate each decoder with its own mutex, so that we can lock each decoder separately.

I have been studying `ImageReader`/`OpenImageIOReader` in some detail to try to understand the inner workings, I've got a few questions related to that. I've created a [Miro board](https://miro.com/app/board/uXjVNKEP_90=/) (can you access this?) to map things out visually. This is still a work in progress.

[ImageReader::affects](https://github.com/GafferHQ/gaffer/blob/main/src/GafferImage/ImageReader.cpp#L390-L393)
Here I'm not sure what "parent" means, is it the node that owns the plug or is it the output plug that the input is connected to (or something else entirely)?

[ImageReader::compute](https://github.com/GafferHQ/gaffer/blob/main/src/GafferImage/ImageReader.cpp#L449-L453)
For `availableFrames` the corresponding output plug on `OpenImageIOReader` is simply [forwarded](https://github.com/GafferHQ/gaffer/blob/main/src/GafferImage/ImageReader.cpp#L177), which makes sense since the low-level reader must be able to determine the valid frame range(s). I'm guessing that `fileValid` is a bit different since it must also take the "frame masking" into account, which is a concept that the low-level reader doesn't know about.

I'll be doing some implementing and will post more when I run into more interesting cases!

Best,

T

[John Haddon]

On Fri, Dec 1, 2023 at 1:57 PM 'Tommy Hinks' via gaffer-dev <gaffe...@googlegroups.com> wrote:

I have been studying `ImageReader`/`OpenImageIOReader` in some detail to try to understand the inner workings, I've got a few questions related to that. I've created a [Miro board](https://miro.com/app/board/uXjVNKEP_90=/) (can you access this?) to map things out visually.

I'm afraid I just get a "We are bringing the systems back, please bear with us" message on that link.

[ImageReader::affects](https://github.com/GafferHQ/gaffer/blob/main/src/GafferImage/ImageReader.cpp#L390-L393)
Here I'm not sure what "parent" means, is it the node that owns the plug or is it the output plug that the input is connected to (or something else entirely)?

Everything in Gaffer is parented into a hierarchy. The ScriptNode is the root node of a `.gfr` file, with the nodes you see in the GraphEditor being children of that root. Nodes may also have their own child nodes in an internal network. Plug are also part of this hierarchy, and are either parented directly to a node, or to another plug.

The `parent<Type>()` method in your link always returns the immediate parent of the object you call it on, cast to the type in the template argument. The case will fail and return `nullptr` if the parent isn't of the requested type.

An ImagePlug is the type of plug that Gaffer uses for passing image data around. It has several child plugs, each corresponding to some property of the image - see https://www.gafferhq.org/documentation/1.3.8.0/WorkingWithImages/AnatomyOfAnImage/index.html.

So, returning to the linked code, it is saying "if `input` is a child of the `__intermediateImage` plug, then dirty the corresponding child of the `out` plug". Which is another way of saying "every property of the `out` plug depends on the corresponding property of the `__intermediateImage` plug.

 

[ImageReader::compute](https://github.com/GafferHQ/gaffer/blob/main/src/GafferImage/ImageReader.cpp#L449-L453)
For `availableFrames` the corresponding output plug on `OpenImageIOReader` is simply [forwarded](https://github.com/GafferHQ/gaffer/blob/main/src/GafferImage/ImageReader.cpp#L177), which makes sense since the low-level reader must be able to determine the valid frame range(s). I'm guessing that `fileValid` is a bit different since it must also take the "frame masking" into account, which is a concept that the low-level reader doesn't know about.

Yes, that's exactly right. A node is responsible for providing values for its output plug. In the case of `availableFrames`, the ImageReader is using a direct connection to do that, which means it doesn't need to do anything else. But for other output plugs, it is responsible for computing their values in `compute()`, implementing `hash()` to provide a lightweight signature for use as a cache key, and implementing `affects()` to declare dependencies.

 

I'll be doing some implementing and will post more when I run into more interesting cases! 

Good luck! Do let us know how you get on, and let us know if we can help with anything else...

Cheers...

John

[Tommy Hinks]

Apologies for my failed markdown attempts, it seems to be rendering as text. Not sure why it's not working properly.

Please try the link to the Miro board again (https://miro.com/app/board/uXjVNKEP_90=/), it seems there was a brief outage but it should be back online now. This is just me trying to map out how things work, it is still not complete.

We have decided to open-source our code (https://github.com/ilpvfx/ilp_gaffer_movie). This is still very much a work in progress, but our efforts can be tracked here.

Pretty soon I will be delving into the color space handling. This will be interesting since libav has its own way of handling color spaces. I think we will somehow need to consolidate that with how OCIO does things, I'll keep you posted.

Best,

Tommy

[John Haddon]

On Wed, Dec 6, 2023 at 8:55 AM 'Tommy Hinks' via gaffer-dev <gaffe...@googlegroups.com> wrote:

Apologies for my failed markdown attempts, it seems to be rendering as text. Not sure why it's not working properly.

I don't think that's you - Google Groups just doesn't support Markdown. Feel free to use the GitHub discussions board over at https://github.com/GafferHQ/gaffer/discussions if you want those niceties - I kindof wouldn't mind it if everyone drifted over there really.

We have decided to open-source our code (https://github.com/ilpvfx/ilp_gaffer_movie). This is still very much a work in progress, but our efforts can be tracked here.

Nice one! Thank you!

 

Pretty soon I will be delving into the color space handling. This will be interesting since libav has its own way of handling color spaces. I think we will somehow need to consolidate that with how OCIO does things, I'll keep you posted.

Hmm, yeah, that sounds interesting. Colour isn't really my strong point, but let us know if we can help with anything...

Cheers...

John