SkImageFilter vs SkRuntimeShader effects concatenation questions

[franz]

Hello!
For my project, I would like to concatenate a bunch of custom effects to an input image in an efficient way. In a nutshell, I would like to achieve something similar to Apple's CIImage, with custom CIKernels. Skia seems the best option to do that.

I see two main APIs that can achieve this:
- SkRuntimeEffect, which becomes eventually an SkShader (via SKRuntimeShaderBuilder.makeShader). The effects can be concatenated here using SkRuntimeShaderBuilder.child.
- SkImageFilter, effects can be contacted here using SkImageFilters.runtimeShader.

From what I understood, SkRuntimeEffect (and more in general SKShader) are just ways to build fragment shaders. SkImageFilter instead does more as it interacts with the context and can create intermediate textures when applied.

Let's say that I have a custom multi-sampling effect (e.g. a custom sharpen filter that samples the input shader N times, with N big), which I want to apply 2 times (e.g. input image -> filtered once -> filtered twice)...
I can create a SkRuntimeEffect for the effect, then create my SkShader for the first pass using SkRuntimeShaderBuilder.makeShader, then create the shader for the second pass by using another SkRuntimeShaderBuilder for the same effect, to which I pass the first shader as a child. IIUC, this will create a single fragment shader that will apply both of the effect passes in a single shader, thus sampling the input image N*N times (which is inefficient in this case).

However, if I use SkImageFilter and for each effect pass I create a SkImageFilter from the SkRuntimeShaderBuilder, then the two passes are going to be rendered in two rendering passes, using intermediate textures, right?

So I can decide whether to use a separate rendering pass (and intermediate texture) or not by concatenating SkRuntimeEffects using SkImageFilter (separate rendering passes) or SkRuntimeShaderBuilder (everything in a single pass), right?

Skia is not performing any optimization on whether to use a separate pass or not by default, right? If, for example, I concatenate two SkImageFilters that are generated from two SkRuntimeEffect.makeForColorFilter (which should be concatenated in a single rendering pass, in a single fragment shader), skia is still going to use two rendering passes if I use SkImageFilter to concatenate them, right?
So if I want to have everything optimized, I have to perform this optimization myself by manually deciding when to concatenate them as SkShaders vs when to concatenate them as SkImageFilters, right?

If I also want control over the region of interest (ROI) of my custom effects, I was thinking of using SkRuntimeShaderBuilder.makeShader to concatenate the effects I want to be in a single rendering pass and obtain the corresponding SkShader, then use SkImageFilters.Shader, since I can specify a crop rect (which would be my ROI).

In general, is there any detailed documentation on how image filters and runtime shaders work, and what is the optimal way of using them?

Thanks a lot for the help!

[Brian Osman]

First: Yes. Everything you wrote is correct (great job figuring all that out, given how poorly it's documented!). There are other differences in image filters and shaders (mostly things like the crop rect, as you mentioned, but also which kinds of operations support using one or the other -- for example, an image filter can be used in a saveLayer operation to filter the layer contents, but a simple shader can't).

Regardless, your observations about the performance tradeoffs are 100%. Your specific question about cases where it would make sense to combine image filters into a single pass is something that's been on our backlog for a long time, but there are quite a few architectural things getting in the way -- we would like to get there eventually though. (Essentially, make it possible to figure out if we should combine some passes of an image filter tree, rather than use a new intermediate surface for each one).

You can use a crop rect if you want, but if you're using a pure shader solution (eg, you've built a runtime-shader from your effect), you can also control where it happens by simply changing the geometry you draw. (For example, just draw a rectangle with your shader. This may have different behavior though, in terms of sampling the input content outside of the geometry for something like a blur). Unfortunately, there isn't really any documentation on how this all works together. But we're happy to answer additional specific questions.

--
You received this message because you are subscribed to the Google Groups "skia-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to skia-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/f834f18e-5329-4a87-802e-e9308c1790a1n%40googlegroups.com.

[franz]

Thanks a lot for the prompt answer.

I will investigate more the internals to understand how to proceed. Currently, this optimization would prevent me from using the SkImageFilter API, as I don't see an easy way to mix the concatenation of SkImageFilter with SkRuntimeEffects that are concatenated using SkRuntimeShaderBuilder. (The way I mentioned in the previous message doesn't seem feasible).
For example, if I have 3 SkRuntimeEffects, A, B, and C, where B and C are children of A and I want them to be concatenated in a single SkShader (and thus A+B+C is a single SkShader), there is no way for me to specify the inputs of B and C as SkImageFilters AFAIS: SkImageFilters.runtimeShader allows me to set the direct children of the SkRuntimeEffect referenced by the SkRuntimeShaderBuilder (A), but not the children of the children effects (B, C).

If I want to do this optimization I have to opt out of SkImageFilter and keep references to the SkRuntimeShaderBuilder all the way to the rendering phase, when I would finally specify the inputs by taking an image snapshot of the last rendering pass(es), getting the SkShaders for those intermediate images and specify them as inputs to the leaf nodes of my SkRuntimeShaderBuilder tree, and then recursively build the SkShaders all the way up to the last SkRuntimeEffect I want to concatenate in the next rendering pass. 

This is quite painful and it would be much smoother to just create the description of the tree using SkImageFilters and let Skia do the magic :D. I would also lose all the other optimizations of SkImageFilter, such as the intermediate caches and some ROI (or crop rect) analysis that is embedded in the SkImageFilters implementation, right? 

Is there a rough estimate of when could this optimization be prioritized (months, years, never)? I see that the SkImageFilters and SkRuntimeEffects APIs are already available to the Android developers in the latest Android releases.
Also, what are the other optimizations included in the SkImageFilter implementation that I'd lose by implementing the render passes manually?

Thanks a lot!