Issue with Skia text rendering

[Pavel Dey]

Hi team,

I am trying to apply a text using canvas->drawString() function. It is part of an android application where the surface is created using the java code. A media decoder is using that surface to render a video. After rendering each frame I am retrieving that surface in CPP. From there I am getting the canvas and calling draw operation on that. So basically the text is expected to appear on top of the video.

But when I call drawString() function like below, it does not render anything on the screen. 

canvas->drawString(text, textPosX, textPosY, font, p);

But if somehow if I use the canvas for some other draw operation before text rendering, then it renders the text. For example, if I use the following code to draw a rectangle (which will hardly be visible in the screen), it renders the text.

SkPaint pp;
pp.setImageFilter(SkImageFilters::Blur(0.0, 0.0, nullptr));
canvas->drawRect(SkRect::MakeWH(1, 1), pp);

Please note that if I do not set the image filter, I can see few filled rectangles, one for each character in the text, which is probably the boundary of those characters.

Somehow, probably this issue is linked with the way I am retrieving the surface. I am doing the following things.
1. Create GL interface using GrGLMakeNativeInterface function.

2. Create GrDirectContext using GrDirectContext::MakeGL from the GL interface.

3. Create GrBackendRenderTarget using GrDirectContext.

4. Create SkSurface using the output of step 3.

These steps are supposed to be executed for every frame of input video. But to optimise the performance I am running step 1 and 2 only once and using their values in the subsequently in loop because these two are costly operations. 3 and 4 are anyway run in loop. With this set up the issue mentioned above is observed. But if I execute step 2 also every time and only optimise step 1, it works fine.

What could be the reason for it? 

Thanks

Pavel

[Brian Salomon]

Hi Pavel,

I'm curious how you're creating a GrBackendRenderTarget every frame. We don't have a GrDirectContext API for that. Are you flushing and submitting the work every frame?

Brian

--
You received this message because you are subscribed to the Google Groups "skia-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to skia-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/897e85bb-e352-4ae2-bb1c-c12d55ab19a1n%40googlegroups.com.

[Pavel Dey]

Hi Brian,

The code looks like this.

//----GL Setup begins
if(isFirstTime) {
glInterface = GrGLMakeNativeInterface();
if (!glInterface.get()) {
return;
}

GrContextOptions options;
options.fDisableDistanceFieldPaths = true;
dContext = GrDirectContext::MakeGL(std::move(glInterface), options);

if (!dContext.get()) {
return;
}


SkColorType colorType;
// setup surface for fbo0
GrGLFramebufferInfo fboInfo;
fboInfo.fFBOID = 0;

GrGLTextureInfo texInfo;
texInfo.fID = textureId;
texInfo.fTarget = GL_TEXTURE_2D;

auto wideColorGamut = false;
if (wideColorGamut) {
fboInfo.fFormat = GL_RGBA16F;
texInfo.fFormat = GL_RGBA16F;
colorType = kRGBA_F16_SkColorType;
} else {
fboInfo.fFormat = GL_RGBA8;
texInfo.fFormat = GL_RGBA8;
colorType = kN32_SkColorType;
}
backendRT = GrBackendRenderTarget(width, height, 0, STENCIL_BUFFER_SIZE, fboInfo);

SkSurfaceProps props(0, kUnknown_SkPixelGeometry);
renderTarget = SkSurface::MakeFromBackendRenderTarget(
dContext.get(), backendRT, kBottomLeft_GrSurfaceOrigin, colorType,
nullptr, &props);

canvas = renderTarget->getCanvas();

I am using this canvas object subsequently for drawing operations.

I am actually not using flush and submit on surface object. But for measuring the execution time I was using that command once before taking cpu time stamp. But it wont be there in the final code. Can that play any role?

Variables 

-Pavel

[Brian Salomon]

Hi Pavel,

Yes, if you don't call flush and submit then Skia may buffer up work and never send it to the 3D API (GL in this case).  You'll definitely want to reuse the GrDirectContext across frames. You only need to recreate that if you make a new GL context.

Brian

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/22c7c487-0346-4d40-bf29-7d5de9566e0bn%40googlegroups.com.

[Pavel Dey]

Hi Brian,

Ok. So I should probably use flush and submit when I am creating GrDirectContext once and reusing it every time. But I have couple of concerns.

1. Will repetitive call to flush and submit (eg- after every frame) not make the program slow? From some of my previous experiments it occurred to. However, my understanding could be wrong.

2. In my current application, I am calling flush and submit along with cpu sync (for the purpose of measuring execution time stamp). Then I don't understand why am I still facing this issue.

Pardon my inadequate knowledge on this topic. I am quite fuzzy about what is going on in the backend when 'GrGLMakeNativeInterface' and 'GrDirectContext::MakeGL' functions are getting called.

[Pavel Dey]

Hi Brian,

One more thing I observed. If before rendering the text on the canvas if I 

1) take a snapshot of the canvas and store it in SkImage

2) Clear the canvas

3) draw the image back on the canvas using drawImage

and then render the text, it appears correctly on top of the video surface. And yes, I have used flush and submit function and the end of draw operation. taking image snapshot is clearly an inefficient way of making things work. But even after debugging the code I could not understand what is missing in my original code to render text without doing any additional operation.

Thanks

Pavel

[Brian Osman]

Hi Pavel,

Are you doing other GL rendering with your GL context outside of Skia? (To get or render the video frames?). If so, it might just be a state problem. In that case, you can tell Skia that the GL state may have changed, by calling resetContext. That will force Skia to forget any cached GL state, and re-configure anything it's going to use (like texture wrap modes, etc...).

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/9d09d01e-14ef-4991-93b2-fc538205fc46n%40googlegroups.com.

[Brian Salomon]

This sounds like the most likely issue to me as well.

GrGLMakeNativeInteface just fills out a struct of function pointers to OpenGL commands. What "native" means depends on how Skia is compiled but typically GL functions obtained by wglGetProcAddress on Windows, eglGetProcAddress on Android, ...

GrDirectContext is Skia's abstraction over an OpenGL context, a Vulkan device, a D3D device, etc. It's lifetime should be related to the underlying 3D API device/context's lifetime. We cache texture/buffer resource, shader programs, etc there. It's relatively expensive to recreate.

Flush-and-submit is the only way to guarantee that Skia's work for a frame is submitted to the GPU before a swap buffers occurs. On GL the submit is pretty much a no-op but on other APIs flush corresponds to preparing command buffers and submit to submitting the command buffers to a queue.

Brian

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/CAMcBjo4vYwTEdNHV%2Bx-Rxhez2Yss8TXJKFBtF7t-Vfonwq44kg%40mail.gmail.com.

[Pavel Dey]

Thanks to both of you for your answers. Brian guessed correctly. I am rendering video frames outside Skia. Since this is an android application, I am decoding a media and rendering the output on the GL Surface before entering into the Skia world where the rest of the text rendering process occurs. This happens in a loop. Once for each video frame. After using resetContext I got the correct output. So certainly it was an issue with the state. Now I am calling resetContext and flushandSubmit for every frame.

But as Brian Salomon mentioned, clearing the cache frequently is relatively expensive what is the recommended approach for my use case? Certainly I cannot render the video frames inside Skia. Then what should be the correct way to handle this without compromising the performance?

Thanks

Pavel 

[Brian Salomon]

Calling resetContext() once per frame shouldn't be a significant performance problem. I was referring to recreating the GrDirectContext every frame as an expensive operation. Also, you may be able to draw the video using Skia. If it is in an AHardwareBuffer you could use SkImage::MakeFromAHardwareBuffer.

Brian

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/07d092a8-b2ff-4326-9673-aa548023150cn%40googlegroups.com.

[Pavel Dey]

I see. The current solution sounds good then. Also, I will explore and see if drawing the video frames can be brought into Skia without impacting the rest of the code. Thanks a lot again for helping me out.

Thanks

Pavel