GafferML and Non ONNX Inference?

[Sachin Shrestha]

Hey gaffer devs,

I have been exploring the GafferML inference pipeline and from what I understand, it currently focuses primarily on onnx models which works well for many use cases.

However, for more complex generative workflows like diffusion based image generation, there seems to be limited onnx support out there. I have experimented by simply using python nodes and it works but I'm curious if there is a more native integration for non-onnx pipelines in the works or discussions.

Cheers,

Sachin 

[lucien...@gmail.com]

Hey Sachin,

It's true that the current support for ONNX is mostly for feed forward models. Any kind of recurrent networks or loop like diffusion is hard to implement with only native node.

I've done some work to experiment with diffusion using GafferML but some of it had to be done with Expression nodes running some python for the loop mechanics in particular but still using ONNX for the heavy lifting with VAE, text encoder and UNet.

We discussed with John and Eric some ideas to allow for this kind of architectures in full native ONNX by exposing the graph within Gaffer so please let us know about your use case.

Note that there exists full ONNX diffusion runtime like this already https://github.com/axodox/axodox-machinelearning

Hope this helps!

Cheers,

Lucien