GafferML and Non ONNX Inference?

[Sachin Shrestha]

Hey gaffer devs,

I have been exploring the GafferML inference pipeline and from what I understand, it currently focuses primarily on onnx models which works well for many use cases.

However, for more complex generative workflows like diffusion based image generation, there seems to be limited onnx support out there. I have experimented by simply using python nodes and it works but I'm curious if there is a more native integration for non-onnx pipelines in the works or discussions.

Cheers,

Sachin 

[lucien...@gmail.com]

Hey Sachin,

It's true that the current support for ONNX is mostly for feed forward models. Any kind of recurrent networks or loop like diffusion is hard to implement with only native node.

I've done some work to experiment with diffusion using GafferML but some of it had to be done with Expression nodes running some python for the loop mechanics in particular but still using ONNX for the heavy lifting with VAE, text encoder and UNet.

We discussed with John and Eric some ideas to allow for this kind of architectures in full native ONNX by exposing the graph within Gaffer so please let us know about your use case.

Note that there exists full ONNX diffusion runtime like this already https://github.com/axodox/axodox-machinelearning

Hope this helps!

Cheers,

Lucien

[Sachin Shrestha]

Hi Lucien,

Thank you for sharing your insights. I have also experimented with python nodes running external models and while it works (sort of), it is not a native experience within gaffer. The onnx eco-system out there for diffusion models is very small and I have seen some older SD1.5 models ported over. I could be wrong here but I think some of the newer models may have different architecture that may not allow a direct porting to onnx framework. The general pytorch and hugging face's diffusers/safetensors eco-system is huge and seems to have become the de-facto framework of choice.

My primary use case is rnd with image and video diffusion models. ComfyUI is great but does have a lot of deployment overhead and has had security risks in the past. It seems like GafferML could be a nice alternate platform for vfx given its already excellent support for all industry standard frameworks. I already use it sometimes to test segmentation models which are simpler to port to onnx. But a full diffusion pipeline may be more headache than its worth currently in my smaller experiments.

Cheers,

Sachin