Render GL_TEXTURE_EXTERNAL_OES to 3 different Surfaces

[mrousavy]

Hey all!

I'm trying to render a GL_TEXTURE_EXTERNAL_OES texture (created with glGenTextures on a 1x1 pbuffer OpenGL context) to Skia, draw stuff onto it ONCE, then render the resulting Frame to 3 different Surfaces (Preview View, Video Recording, and Frame Analysis).

This is my current code:

sk_sp<SkImage> SkiaRenderer::renderFrame(OpenGLContext& glContext, OpenGLTexture& texture) {

// 1. Activate the OpenGL context w/ 1x1 pbuffer surface (eglMakeCurrent)

glContext.use();

// 2. Initialize Skia

if (_skiaContext == nullptr) {

_skiaContext = GrDirectContext::MakeGL();

}

// TODO: Do I need that?

_skiaContext->resetContext();

_skiaContext->resetGLTextureBindings();

// 3. Create the offscreen Skia Surface

if (_offscreenSurface == nullptr) {

GrBackendTexture skiaTex = _skiaContext->createBackendTexture(texture.width,

texture.height,

SkColorType::kN32_SkColorType,

GrMipMapped::kNo,

GrRenderable::kYes);

GrGLTextureInfo info;

skiaTex.getGLTextureInfo(&info);

_offscreenSurfaceTextureId = info.fID;

__android_log_print(ANDROID_LOG_INFO, TAG, "Created Texture %i!", info.fID);

_offscreenSurface = getSkiaSurface(info.fID, texture.width, texture.height);

}

sk_sp<SkImage> frame = getSkiaTexture(texture);

SkCanvas* canvas = _offscreenSurface->getCanvas();

//canvas->clear(SkColors::kCyan);

auto duration = std::chrono::system_clock::now().time_since_epoch();

auto millis = std::chrono::duration_cast<std::chrono::milliseconds>(duration).count();

canvas->drawImage(frame, 0, 0);

// TODO: Run Skia Frame Processor

SkRect rect = SkRect::MakeXYWH(150, 250, millis % 2000 / 10, millis % 2000 / 10);

SkPaint paint;

paint.setColor(SkColors::kRed);

canvas->drawRect(rect, paint);

_offscreenSurface->flushAndSubmit();

// TODO: Do I need eglSwapBuffer for the 1x1 pbuffer?

glContext.flush();

return _offscreenSurface->makeImageSnapshot();

}

void SkiaRenderer::renderTextureToOutputSurface(OpenGLContext& glContext, sk_sp<SkImage> image, EGLSurface outputSurface) {

// 1. Activate the OpenGL context for the given output surface (eglMakeCurrent)

glContext.use(outputSurface);

// 2. Initialize Skia

if (_skiaContext == nullptr) {

_skiaContext = GrDirectContext::MakeGL();

}

// TODO: use this later kRenderTarget_GrGLBackendState | kTextureBinding_GrGLBackendState

_skiaContext->resetContext();

_skiaContext->resetGLTextureBindings();

// 3. Wrap the target output surface (FBO0 on this glContext)

sk_sp<SkSurface> surface = getSkiaSurface(0, image->width(), image->height());

SkCanvas* canvas = surface->getCanvas();

//canvas->clear(SkColors::kCyan);

canvas->drawImage(image, 0, 0);

// TODO: Remove this

auto duration = std::chrono::system_clock::now().time_since_epoch();

auto millis = std::chrono::duration_cast<std::chrono::milliseconds>(duration).count();

SkRect rect = SkRect::MakeXYWH(150, 250, millis % 3000 / 10, millis % 3000 / 10);

SkPaint paint;

paint.setColor(SkColors::kGreen);

canvas->drawRect(rect, paint);

// This does eglSwapBuffers()

_skiaContext->flushAndSubmit();

// does eglSwapBuffers

glContext.flush();

}

And getSkiaSurface/getSkiaTexture looks like this:

sk_sp<SkSurface> SkiaRenderer::getSkiaSurface(GLuint frameBufferId, int width, int height) {

GrGLFramebufferInfo frameBufferInfo {

.fFBOID = frameBufferId,

.fFormat = GR_GL_RGBA8,

};

GLint sampleCnt;

glGetIntegerv(GL_SAMPLES, &sampleCnt);

GLint stencilBits;

glGetIntegerv(GL_STENCIL_BITS, &stencilBits);

GrBackendRenderTarget renderTarget(width,

height,

sampleCnt,

stencilBits,

frameBufferInfo);

SkSurfaceProps props(0, kUnknown_SkPixelGeometry);

sk_sp<SkSurface> surface = SkSurfaces::WrapBackendRenderTarget(_skiaContext.get(),

renderTarget,

kBottomLeft_GrSurfaceOrigin,

SkColorType::kN32_SkColorType,

nullptr,

&props,

nullptr);

return surface;

}

sk_sp<SkImage> SkiaRenderer::getSkiaTexture(OpenGLTexture& texture) {

GrGLTextureInfo textureInfo {

// OpenGL will automatically convert YUV -> RGB if it's an EXTERNAL texture

.fTarget = texture.target,

.fID = texture.id,

.fFormat = GR_GL_RGBA8,

.fProtected = skgpu::Protected::kNo,

};

GrBackendTexture skiaTexture(texture.width,

texture.height,

GrMipMapped::kNo,

textureInfo);

sk_sp<SkImage> image = SkImages::BorrowTextureFrom(_skiaContext.get(),

skiaTexture,

kBottomLeft_GrSurfaceOrigin,

kN32_SkColorType,

kOpaque_SkAlphaType,

nullptr,

nullptr);

return image;

}

..my custom OpenGLTexture wrapper just holds the ID and the target (GL_TEXTURE_EXTERNAL_OES or GL_TEXTURE_2D)

The problem here is that with the current code, the Camera Frame does not appear on the screen. I am 100% certain that the EXTERNAL texture contains the Frame, but Skia just doesn't render it to the screen later on.

Any thoughts?

Related; https://groups.google.com/u/1/g/skia-discuss/c/LoH9k2_FGPM

[Brian Salomon]

Hi, I noticed at least one problem inspecting the code, see note below.

 ^^^ Here you have a texture ID but the function getSkiaSurface() treats it as a framebuffer object ID, which is a different kind of OpenGL object. Hooray for no type safety in OpenGL!

Moreover, if you just need Skia to make an offscreen surface use SkSurfaces::RenderTarget(). It will create and manage the lifecycle of an OpenGL texture ID, FBO, etc under the hood. So this would look something like:

if (_offscreenSurface == nullptr) {
    _offscreenSurface = SkSurfaces::RenderTarget(_skiaContext,
                                                 skgpu::Budgeted::kNo,

                                                 SkImageInfo::MakeN32Premul(texture.width, texture.height));
}

--
You received this message because you are subscribed to the Google Groups "skia-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to skia-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/ec07c4f1-74a2-4e3e-bf18-967e0dbcabe9n%40googlegroups.com.

[mrousavy]

Thanks Brian, this was it!!!!

Amazing stuff.

Now the transform matrix isn't applied properly, but I guess this is just a bit of fiddling around..

The Android docs state:

Retrieve the 4x4 texture coordinate transform matrix associated with the texture image set by the most recent call to updateTexImage().
This transform matrix maps 2D homogeneous texture coordinates of the form (s, t, 0, 1) with s and t in the inclusive range [0, 1] to the texture coordinate that should be used to sample that location from the texture. Sampling the texture outside of the range of this transform is undefined.
The matrix is stored in column-major order so that it may be passed directly to OpenGL ES via the glLoadMatrixf or glUniformMatrix4fv functions.
If the underlying buffer has a crop associated with it, the transformation will also include a slight scale to cut off a 1-texel border around the edge of the crop. This ensures that when the texture is bilinear sampled that no texels outside of the buffer's valid region are accessed by the GPU, avoiding any sampling artifacts when scaling.

My Transform Matrix for this particular image is:

0.000000 -1.000000 0.000000 0.000000 -1.000000 0.000000 0.000000 0.000000 0.000000 0.000000 1.000000 0.000000 1.000000 1.000000 0.000000 1.000000

(printed from m[0] to m[15])

And in Skia it renders the image wrong by -90 degrees.

I guess Skia uses different origins than OpenGL here?

This is the OpenGL shader that works and rotates properly with the transform matrix:

attribute vec4 aPosition;

attribute vec2 aTexCoord;

uniform mat4 uTransformMatrix;

varying vec2 vTexCoord;

void main() {

gl_Position = aPosition;

vTexCoord = (uTransformMatrix * vec4(aTexCoord, 0.0, 1.0)).xy;

}

[Greg Daniel]

Skia's logical origin from the SkCanvas POV is in the top left corner. When you wrap a GLTexture or Framebuffer into an SkSurface you can tell us what the origin for that backend object is and we'll make sure draws and/or reads from it happen correctly. But regardless of the origin you pass into the wrapped SkSurface, the SkCanvas also has a TopLeft origin.

To view this discussion on the web visit https://groups.google.com/d/msgid/skia-discuss/56403f9e-839d-46d8-abc5-5c8cbc9ea65bn%40googlegroups.com.