VideoCapture with custom surface

[Javier Gerardo Martinez Salomon]

Hi CameraX folks,
I'm currently using `camerax 1.3.4` and I'm having a hard time to make work the VideoCapture usecase appropriately with a custom surface and display the preview correctly.

I'm using the following snippet to produce a `SurfaceTexture` from a custom view that extends from `SurfaceView`:

private fun initGlExecutor(anySurface: Any?, adapter: CallbackAdapter) {
eglExecutor.submit {
/*
* Initializing a thread for EGL requires the following steps:
*
* 1) Get the EGL Display 'eglGetDisplay()'
* 2) Initialize the EGL Display 'eglInitialize()'
* 3) Create the EGL Config for the display 'eglChooseConfig()'
* 4) Create the EGL Context for the display 'eglCreateContext()'
* 5) Bind the EGL Display to a native window surface 'eglCreateWindowSurface()'
* 6) Make the bound EGL Surface active 'eglMakeCurrent()'
*
* after that, any writes to the EGL Surface need to be propagated to the native surface
* with 'eglSwapBuffers()'
*/

anySurface?.let { displaySurface ->
// Internally gets and initializes the EGL Display and Config, creates the EGL Context
val eglContext = getEglContext()

eglWrapper = EglWrapper(eglContext).also {
// Create EGLSurface for writing to display preview
originalWindowSurface = when (displaySurface) {
is Surface -> EglWindowSurface(it, displaySurface)
is SurfaceTexture -> EglWindowSurface(it, displaySurface)
else -> throw RuntimeException("Invalid surface: $displaySurface")
}
// Create EGLSurface for writing to video output
//outputWindowSurface = EglWindowSurface(it, videoWriter.inputSurface)
}

// Set all writes to go to display preview surface
originalWindowSurface?.makeCurrent()

// Create drawing utility
glesDrawer = GLESDrawer().also {
// Create Texture for Camera to write to
surfaceTexture = SurfaceTexture(it.textureId)
}

// Set listener to input texture so all writes can be passed on to display preview and video output if enabled
surfaceTexture?.setOnFrameAvailableListener(adapter)
}

surfaceTexture?.let {
LiveDataUtil.updateValue(_surfaceLiveData, it)
}
}
}

I keep track of the device orientation by getting the rotation out of the default display using the WindowManager and mapping the values to the appropriate degrees:

fun getDegreeRotationFromNaturalOrientation(activityContext: Context) =
when (getRotationFromNaturalOrientation(activityContext)) {
Surface.ROTATION_90 -> 90
Surface.ROTATION_180 -> 180
Surface.ROTATION_270 -> 270
else -> 0
}

Then when a frame is made available with `onFrameAvailable` I draw the frame in the "correct" orientation with:

private fun onDrawFrame() {
surfaceTexture?.updateTexImage()
surfaceTexture?.getTransformMatrix(tMatrix)

originalWindowSurface?.let {
glesDrawer?.draw(
tMatrix,
it.getWidth(),
it.getHeight(),
orientation
)

it.swapBuffers()
}
}

This has been working great so far, and handles orientations gracefully when using the Preview usecase alone, but once I added the VideoCapture usecase I started to have difficulties with handling the orientation.

For a Pixel 6, when binding both usecases and in portrait mode the preview looks fine and the video recording is successful, however, when I use landscape mode the preview orientation is not quite right despite the video being correctly oriented. It seems to me that this is a drawing issue, however, I don't understand how to correctly display the preview when using both Preview and VideoCapture, if I only use Preview the preview displays correctly in any orientation.

1. Is there any additional consideration that I'm not handling when using both the Preview and VideoCapture usecases?
2. Do you have any examples to share on how to correctly handle and display the preview in the correct orientation?

Any help is welcome, unfortunately, I can't use the PreviewView since it poses bigger architectural challenges and I want a simple Camera View to display things correctly.

Thanks in advance.