Rendering GL_TEXTURE_EXTERNAL_OES to offscreen GL_TEXTURE_2D using Skia just freezes

[mrousavy]

Hey all!

I got an Android Camera2 setup where my Camera streams frames into an OpenGL pipeline.

I successfully built the OpenGL pipeline like this:

1. I create one OpenGL context with 3 surfaces:
   
   1. 1x1 pbuffer EGLSurface (for initializing textures and stuff)
   2. 1920x1080 window EGLSurface (preview)
   3. 4000x2000 window EGLSurface (video recording)
2. I create a GL_TEXTURE_EXTERNAL_OES texture and pass that to the Camera

Then, on every Frame the Camera streams into my EXTERNAL_OES texture:

1. Camera streams frame into EXTERNAL_OES texture
2. I use a pass-through shader to render the EXTERNAL_OES texture to each of the 3 surfaces. This also applies a transformation matrix

This code works perfectly fine and I can show a 1080p preview and record 4k videos all in the same pipeline.

Now I want to integrate Skia into the rendering aspect here, so the rendering part should look like this:

1. Camera streams frame into EXTERNAL_OES texture
2. I render frame from EXTERNAL_OES texture to an offscreen TEXTURE_2D texture and apply some Skia drawing ontop of that (e.g. a red box). I found GrDirectContext::createBackendTexture helpful, which basically just creates an offscreen Frame Buffer as far as I know.
3. Then, I want to render that TEXTURE_2D to each of the 3 surfaces now using the pass-through shader.

I've been debugging this for hours now but I just can't get this Skia part to work. The first frame draws successfully, but then it freezes. I have weird artefacts on the screen, and it just doesn't work appropriately.

This is the codebase: https://github.com/mrousavy/react-native-vision-camera/tree/feat/android-skia/android/src/main/cpp

And here's the relevant code pieces (marked them as snippets for easier viewing):

1. onFrame(..): Called once the Camera put a new Frame into _inputTexture (EXTERNAL_OES).
2. SkiaRenderer::renderFrame(..): Called to render the Frame onto an offscreen texture (TEXTURE_2D) and add some additional Skia commands (draw a red rectangle).
3. OpenGLRenderer::renderTextureToSurface(..): Called with the newly rendered to offscreen texture (TEXTURE_2D) that contains the frame and the red rectangle. This will render the Frame to the output EGLSurface.
4. PassThroughShader::draw(..): Actually doing the pass-through rendering of the 2D texture that contains my Skia drawing. This is the shader that it uses. (sampler2D instead of samplerExternalOES)

Without Skia it worked, but with the Skia middleman integration now, the first Camera frame renders and from then on it just freezes and shows weird artefacts the entire time. Here's a video demo of what's going wrong: twitter post

I understand that using Skia for everything would maybe simplify this a bit, but Skia is an optional plugin in my project. The passing through (texture -> EGLSurface) should still happen in pure OpenGL.

What am I doing wrong here? Is the 1x1 pbuffer approach wrong?

Not directly related to the freezing, but can I use that 4x4 GL transformation matrix in Skia?

I'd appreciate any help, also happy to pay $150/h for consulting hours/fixing if anyone wants to do that.

Thanks!

[mrousavy]

If you don't want to click the links, here's the code snippets:

1. onFrame:

void VideoPipeline::onFrame(float* transformMatrixParam) {

// 1. Activate the offscreen context

_context->use();

// 2. Prepare the texture we are going to render

OpenGLTexture& texture = _inputTexture.value();

// 3. (Optional) If we have Skia, render to a separate offscreen framebuffer which the outputs will then read from

if (_skiaRenderer != nullptr) {

auto newTexture = _skiaRenderer->renderFrame(*_context, texture);

__android_log_print(ANDROID_LOG_INFO, TAG, "Rendered from Texture #%i -> Texture #%i!", texture.id, newTexture.id);

texture = newTexture;

}

// not sure if i need to do that

glBindTexture(texture.target, texture.id);

glBindFramebuffer(GL_FRAMEBUFFER, DEFAULT_FRAMEBUFFER);

// 4. Render to all outputs

if (_previewOutput) {

__android_log_print(ANDROID_LOG_INFO, TAG, "Rendering to Preview..");

_previewOutput->renderTextureToSurface(texture, transformMatrix);

}

if (_recordingSessionOutput) {

__android_log_print(ANDROID_LOG_INFO, TAG, "Rendering to RecordingSession..");

_recordingSessionOutput->renderTextureToSurface(texture, transformMatrix);

}

}

2. SkiaRenderer::renderFrame():

OpenGLTexture SkiaRenderer::renderFrame(OpenGLContext& glContext, OpenGLTexture& texture) {

// 1. Activate the OpenGL context (eglMakeCurrent)

glContext.use();

// 2. Initialize Skia

if (_skiaContext == nullptr) {

GrContextOptions options;

// TODO: Set this to true or not? idk

options.fDisableGpuYUVConversion = false;

_skiaContext = GrDirectContext::MakeGL(options);

}

// TODO: use this later kRenderTarget_GrGLBackendState | kTextureBinding_GrGLBackendState

_skiaContext->resetContext();

// 3. Create the offscreen Skia Surface

if (_offscreenSurface == nullptr) {

GrBackendTexture skiaTex = _skiaContext->createBackendTexture(texture.width,

texture.height,

SkColorType::kN32_SkColorType,

GrMipMapped::kNo,

GrRenderable::kYes);

GrGLTextureInfo info;

skiaTex.getGLTextureInfo(&info);

_offscreenSurfaceTextureId = info.fID;

SkSurfaceProps props(0, kUnknown_SkPixelGeometry);

_offscreenSurface = SkSurfaces::WrapBackendTexture(_skiaContext.get(),

skiaTex,

kBottomLeft_GrSurfaceOrigin,

0,

SkColorType::kN32_SkColorType,

nullptr,

&props,

// TODO: Delete texture!

nullptr);

}

GrGLTextureInfo textureInfo {

// OpenGL will automatically convert YUV -> RGB because it's an EXTERNAL texture

.fTarget = texture.target,

.fID = texture.id,

.fFormat = GR_GL_RGBA8,

.fProtected = skgpu::Protected::kNo,

};

GrBackendTexture skiaTexture(texture.width,

texture.height,

GrMipMapped::kNo,

textureInfo);

sk_sp<SkImage> frame = SkImages::BorrowTextureFrom(_skiaContext.get(),

skiaTexture,

kBottomLeft_GrSurfaceOrigin,

kN32_SkColorType,

kOpaque_SkAlphaType,

nullptr,

nullptr);

SkCanvas* canvas = _offscreenSurface->getCanvas();

canvas->clear(SkColors::kCyan);

canvas->drawImage(frame, 0, 0);

auto duration = std::chrono::system_clock::now().time_since_epoch();

auto millis = std::chrono::duration_cast<std::chrono::milliseconds>(duration).count();

SkRect rect = SkRect::MakeXYWH(150, 250, millis % 3000 / 10, millis % 3000 / 10);

SkPaint paint;

paint.setColor(SkColors::kGreen);

canvas->drawRect(rect, paint);

_offscreenSurface->flushAndSubmit();

OpenGLTexture result {

.id = _offscreenSurfaceTextureId,

.target = GL_TEXTURE_2D,

.width = texture.width,

.height = texture.height,

};

return result;

}

..and then to render that offscreen TEXTURE_2D to the output EGLSurfaces:

3. OpenGLRenderer::renderTextureToSurface(..):

void OpenGLRenderer::renderTextureToSurface(const OpenGLTexture& texture, float* transformMatrix) {

if (_surface == EGL_NO_SURFACE) {

__android_log_print(ANDROID_LOG_INFO, TAG, "Creating Window Surface...");

_context->use();

_surface = eglCreateWindowSurface(_context->display, _context->config, _outputSurface, nullptr);

}

// 1. Activate the OpenGL context for this surface

_context->use(_surface);

OpenGLError::checkIfError("Failed to use context!");

// 2. Set the viewport for rendering

glViewport(0, 0, _width, _height);

glDisable(GL_BLEND);

glClearColor(1.0f, 0.0f, 0.0f, 1.0f);

glClear(GL_COLOR_BUFFER_BIT);

// 3. Bind the input texture

//glBindTexture(texture.type, texture.id);

// 4. Draw it using the pass-through shader which also applies transforms

_passThroughShader.draw(texture, transformMatrix);

// 5. Swap buffers to pass it to the window surface

_context->flush();

OpenGLError::checkIfError("Failed to render Frame to Surface!");

}

4. PassThroughShader::draw(..):

void PassThroughShader::draw(const OpenGLTexture& texture, float* transformMatrix) {

// 1. Set up Shader Program

if (_programId == NO_SHADER || _shaderTarget != texture.target) {

if (_programId != NO_SHADER) {

glDeleteProgram(_programId);

}

_programId = createProgram(texture.target);

glUseProgram(_programId);

_vertexParameters = {

.aPosition = glGetAttribLocation(_programId, "aPosition"),

.aTexCoord = glGetAttribLocation(_programId, "aTexCoord"),

.uTransformMatrix = glGetUniformLocation(_programId, "uTransformMatrix"),

};

_fragmentParameters = {

.uTexture = glGetUniformLocation(_programId, "uTexture"),

};

_shaderTarget = texture.target;

glGenBuffers(1, &_vertexBuffer);

}

glUseProgram(_programId);

// 3. Set up the vertices

glBindBuffer(GL_ARRAY_BUFFER, _vertexBuffer);

glBufferData(GL_ARRAY_BUFFER, sizeof(VERTICES), VERTICES, GL_STATIC_DRAW);

// 3. Pass all uniforms/attributes for vertex shader

glEnableVertexAttribArray(_vertexParameters.aPosition);

glVertexAttribPointer(_vertexParameters.aPosition,

2,

GL_FLOAT,

GL_FALSE,

sizeof(Vertex),

reinterpret_cast<void*>(offsetof(Vertex, position)));

glEnableVertexAttribArray(_vertexParameters.aTexCoord);

glVertexAttribPointer(_vertexParameters.aTexCoord,

2,

GL_FLOAT,

GL_FALSE,

sizeof(Vertex),

reinterpret_cast<void*>(offsetof(Vertex, texCoord)));

glUniformMatrix4fv(_vertexParameters.uTransformMatrix, 1, GL_FALSE, transformMatrix);

// NOTE: Do I need to use a different texture than GL_TEXTURE0 here?

// 4. Pass texture to fragment shader

glActiveTexture(GL_TEXTURE0);

glBindTexture(texture.target, texture.id);

glUniform1i(_fragmentParameters.uTexture, 0);

// 5. Draw!

glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);

}

This is the shaders:

static constexpr Vertex VERTICES[] = {

{{-1.0f, -1.0f}, {0.0f, 0.0f}}, // bottom-left

{{1.0f, -1.0f}, {1.0f, 0.0f}}, // bottom-right

{{-1.0f, 1.0f}, {0.0f, 1.0f}}, // top-left

{{1.0f, 1.0f}, {1.0f, 1.0f}} // top-right

};

static constexpr char VERTEX_SHADER[] = R"(

attribute vec4 aPosition;

attribute vec2 aTexCoord;

uniform mat4 uTransformMatrix;

varying vec2 vTexCoord;

void main() {

gl_Position = aPosition;

vTexCoord = (uTransformMatrix * vec4(aTexCoord, 0.0, 1.0)).xy;

}

)";

// This is NOT because we are rendering from an offscreen TEXTURE_2D (skia)

static constexpr char FRAGMENT_SHADER[] = R"(

precision mediump float;

varying vec2 vTexCoord;

uniform sampler2D uTexture;

void main() {

gl_FragColor = texture2D(uTexture, vTexCoord);

}

)";

[mrousavy]

I'm thinking that Skia might just do a ton of stuff with the OpenGL context that's out of my hands - so it would theoretically be possible for me to just do all the rendering using Skia - then I don't do any custom OpenGL commands myself.

I played around with that but couldn't get it to work - Skia seems to not properly switch to the EXTERNAL texture to prepare it for rendering.

I created a separate topic for this since this might be out of scope for this one here: https://groups.google.com/u/1/g/skia-discuss/c/DBDDy83nKW8