Some questions about SkMesh

[A. Chi]

Hello.

I have a use case where I need to draw some 3d content on a Skia canvas. So far I've been using other libraries to create an api dependent texture and transferring it to Skia as a SkImage, but this has proven to be very problematic because of the different ways to do so on the different platforms and gpu apis.

I'm very interested in using SkMesh because it offers most of what I need to render the 3d content, while not having to worry about any context switching or synchronization.

However there is a key missing feature that's stopping me from migrating everything to pure skia and that is : texture support in shader code.

I've tried to do something like "uniform shader image", as one would for a runtime effect (shader, blender, colorfilter ...), but unfortunately it's not supported. I've also tried to build skia without this check https://github.com/google/skia/blob/main/src/sksl/ir/SkSLVarDeclarations.cpp#L192 but it failed a couple of asserts later.

Would it possible to add image support to vertex/fragment programs supplied to a SkMeshSpecification ? I'd say something to achieve the same functionality as SkImage::makeRawShader is all that's needed for my use case.

Of course it would be great to also have full effect support, colorspace conversion, other sampler types like samplerCube, level of details... but none of this is really as important  since it can be implemented on top of basic texture support.

Also I remember SkMesh being added a while back under the name SkVertices3d, only to be removed, so I'm glad it's back now. Just wish we could get image support, because otherwise it's really such a great feature :)

Thanks in advance.

[Brian Osman]

Hi,

It's true that the mesh fragment shader doesn't support uniform shaders (we've been talking about adding that feature, but have no concrete plans yet). However, if you only need a single texture applied to your mesh, you can do that: You put the image shader with your texture on the SkPaint's shader field, and the mesh fragment shader just returns the coordinates that are used to apply that shader. Does that make sense?

--
You received this message because you are subscribed to the Google Groups "skia-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to skia-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/9dd9e386-6074-4d76-bef2-54d896d771e8n%40googlegroups.com.

[A. Chi]

Come to think of it, I don't know why I ever thought SkMesh would be doing any depth testing. The relevant documentation states that the position in the varyings should be of type float2 meaning that Skia has no way of telling what my Z coordinates are. Also the output of the vertext shader is in device coordinates.

So I guess SkMesh is not going to work, and I'll have to go back to writing a lot of code to get my own opengl/metal code to behave nicely with Skia. But I'll keep watching SkMesh development since it's an interesting future nonetheless.

Thanks for the work you're doing on Skia guys !

[A. Chi]

Thanks for the reply Brian.

Your suggestion gave me an idea where I might be able to get away with the lack of shader support by getting the interpolated varyings in the fragment shader as the output color and doing the actual computations that need to also access the textures in a blender or a shader that has access to the SkMesh drawing result.

But I've noticed something weird that I didn't consider at first because I was drawing the whole mesh in one color while being under the assumption that depth testing was handled by SkMesh. Both of these single draw renders are done with the same geometry, with the exception of the supplied face order :

Here the 2 tris for the red face are last in the vertex buffer.

Whereas here, yellow is last.

I thought at first that it was because of Skia's use of CCW front faces, but that has no effect, as it seems that the tris are drawn in the order they're supplied in without any depth testing. Can you confirm that that's the case ?

I'm still on chrome110 if that matters.

Thanks.

On Wednesday, June 7, 2023 at 9:05:56 PM UTC+1 brian...@google.com wrote:

[Brian Osman]

Yes, that's correct. There is no depth testing, so the vertex data is submitted in the order it's received, and the normal GPU rasterization rules apply. (Effectively, it will draw triangular meshes as if each triangle was rendered in order ... later triangles will paint over earlier ones).

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/b30a8a1a-28d8-45c3-8d86-bfe300a0a739n%40googlegroups.com.

[A. Chi]

Just noticed that M120 allows using child shaders with SkMesh.

Any chance we could also get some kind of depth testing in a future release ? I understand that that would introduce further complexity by having 2 ways to handle the position (currently only float2).

Ideally we'd return a float4 and choose a comparison function in the spec. But failing that, maybe just some pointers as to how to implement something like this manually in shader code.

[John Stiles]

What did you have in mind? Skia wants to control the depth value (to allow more batching while respecting painter's-order) so it's unlikely that we'd allow you to change it. 

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/6c517492-aa21-46c6-91da-846d6ee71185n%40googlegroups.com.

[A. Chi]

My usecase would be to draw 3d on a Skia canvas without calling any other apis directly. So far the only thing missing from SkMesh is the depth support to draw something like a textured Cube.

But it seems that SkMesh had a different design goal in mind, and is more like a improved Canvas.drawVertices rather than what I had thought initially.

[John Stiles]

Sorry, drawing in 3D with depth support is not something Skia supports. You are better served to render 3D using native APIs and then copy the Skia rendering into your 3D view (or vice versa).

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/8dc9c383-4ee2-43db-9de2-fd109e1d154dn%40googlegroups.com.

[Greg Daniel]

Unfortunately Skia currently has no concept of depth on the public API. We've talked about it many times but it would not be a small change to add. So as of now we have no plans to add depth support to any API including SkMesh.

I'm not sure of any current way for you to mimic it in a custom shader in Skia unless we exposed some sort of public storage object for you to be able to read and write in the shader. Though something like that may be more possible to add to skia in the future than depth.

Greg

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/8dc9c383-4ee2-43db-9de2-fd109e1d154dn%40googlegroups.com.

[William Candillon]

We've been experimenting with some 3d transformations with Skia, and right now, this involves two things:
1. Sort the triangles on the CPU (which is somewhat fair since we only want to do simple 3d things).

2.  Create groups of triangles together (since we can only test the depth of a vertex and the pixel depth, we make groups of triangles and calculate the average z value. Seems to work well enough for simple examples ).

Of course, some depth testing or the ability to create custom buffers to do such an operation feels extremely appealing.

In terms of scope, I feel like p5.js really got it right: you can build simple 3d primitives, but it's definitely not a 3d engine (https://p5js.org/reference/#group-3D).

And looking at how digital artists are using p5.js, it seems to make sense: They build mostly 2D art, but sometimes that nice 3D primitive comes in extremely handy (this has been our experience as well).

Kind regards,

William