OverlayEffect does not work correctly

[Tomáš Válek]

There are some problems with OverlayEffect.

1. canvas.drawRect(0f,0f,200f,200f, paint) produces different result on different devices. See screenshot. Zero coords are wrong. Blue rectangle is from CameraX OverlayEffect, the red rectangle is correct rectange drawn by customView with onDraw() method. You can see the blue rectangle on tablet (Lenovo TAB M10 plus Android 13) is cut. The same problem is on the left smartphone (Galaxy S20+ Android 13).

2. OverlayEffect on the right smartphone (Pixel 2 Android 13) produces some preview errors and unexpected overlay repeats.

CameraX v.: 1.4.0-alpha04

private fun getOverlayEffect(): CameraEffect {
handlerThreadBackground = HandlerThread("handlerThreadBackground").apply { start() }
handlerBackgroundThread = Handler(handlerThreadBackground.looper)

// https://developer.android.com/reference/androidx/camera/core/UseCaseGroup.Builder#addEffect(androidx.camera.core.CameraEffect)
val effect = OverlayEffect(OverlayEffect.PREVIEW, 0, handlerBackgroundThread) { // uncomment for development
it.printStackTrace()
}

effect.setOnDrawListener { frame ->
Static.log(TAG, "FRAME: size: " + frame.size + ", rotation: " + frame.rotationDegrees)
val canvas = frame.overlayCanvas
Static.log(TAG, "CANVAS: size: " + canvas.width+"x" + canvas.height + ", saveCount: " + canvas.saveCount + ", isHardwareAccelerated: " + canvas.isHardwareAccelerated)

val paint = Paint()
paint.color = Color.BLUE // Nastavení barvy čtverce
paint.style = Paint.Style.FILL // Vyplnění stylu
canvas.drawRect(0f,0f,200f,200f, paint)

return@setOnDrawListener true
}

return effect
}

[Xi Zhang]

Thanks for reporting the issue with the picture. It helps us to understand the issue. Basically, the issue is that the coordinate system of the Canvas is not what you expect. Your red rect is drawn in the UI coordinate system, but the Canvas is NOT in the UI coordinate system. You can check out the javadoc here: 

• https://developer.android.com/reference/androidx/camera/effects/OverlayEffect#setOnDrawListener(androidx.arch.core.util.Function%3Candroidx.camera.effects.Frame,java.lang.Boolean%3E)
• https://developer.android.com/reference/androidx/camera/effects/Frame#getSensorToBufferTransform()

To simplify things, a camera frame goes through 3 stages until it reaches the UI: Camera sensor -> effect -> PreviewView (assuming that you are using PreviewView). And the coordinate system is the "effects". To transform it to the PreviewView coordinate system, you need to do something like the following:

val sensorToUi = previewView.sensorToViewTransform
if (sensorToUi != null) {
  val sensorToEffect = frame.sensorToBufferTransform
  val uiToSensor = Matrix()
  sensorToUi.invert(uiToSensor)
  uiToSensor.postConcat(sensorToEffect)
  val canvas = frame.overlayCanvas
  canvas.setMatrix(uiToSensor)
  canvas.drawRect(0f, 0f, 100f, 100f, Paint())
}

For the full code sample, see: https://android-review.googlesource.com/c/platform/frameworks/support/+/3075863/1/camera/integration-tests/viewtestapp/src/main/java/androidx/camera/integration/view/OverlayEffectsFragment.kt

[Tomáš Válek]

Ad 1.: I do not have to set: canvas.setMatrix() or canvas.rotate() due to devices in the screenshot are rotated to the same orientation as camera sensor (frame.rotationDegrees is 0) so "Your red rect is drawn in the UI coordinate system, but the Canvas is NOT in the UI coordinate system." Yes it is. But OK, I tried your example but the result is the same like in the screenshot.

Ad 2.: Please answer to this point also.

Thank you.

Dne pondělí 6. května 2024 v 18:33:52 UTC+2 uživatel xi...@google.com napsal:

[Xi Zhang (张熹)]

Even if the UI is in the same orientation as the camera sensor, we still need transformation because there could be scaling/cropping. But you are right that this is probably not the root cause. The canvas transformation should not affect the aspect ratio, which is also wrong in your picture.  I wonder what the rest of your setup looks like. If you don't mind uploading a minimal reproducible code sample to GitHub, I can take a look at it.

For your question 2, I can reproduce a similar issue when drawing the rect out of the boundary of the frame. The overlay is backed by a SurfaceTexutre, so I think this should be an Android framework behavior, intended or not. Let's get the rect drawn within the boundary first and see if the issue remains.

[Tomáš Válek]

Ad 1. OK, you are right about scaling/cropping. When I used your advice, it helps to show rectangle exactly to the same place like in Ui. Thank you. Rest setup is basic without any extra setup.

previewUseCase = Preview.Builder().Build()

try {

// Unbind use cases before rebinding
cameraProvider.unbindAll()

// Order by priority
val useCaseGroup: UseCaseGroup.Builder = UseCaseGroup.Builder()
previewUseCase?.let { useCaseGroup.addUseCase(it) }

val overlayEffect = getOverlayEffect()
useCaseGroup.addEffect(overlayEffect)

cameraProvider.bindToLifecycle(lifecycleOwner, cameraSelector, useCaseGroup.build())

} catch (e: Exception) {
e.printStackTrace()
}

Ad 2.:
Code: Draw overlay Text + one blue rectangle

val paintText = TextPaint().apply {
color = Color.YELLOW // Set the text color
textSize = titleTextSize.toFloat() // Set the text size
isAntiAlias = true // Smooth out the edges of what is being drawn
}

val paintRect = Paint().apply {
color = Color.BLUE
style = Paint.Style.FILL
alpha = 25 // 0..255
}

canvas.translate(pivotX, pivotY)
val staticLayout = StaticLayout.Builder.obtain(text, 0, text.length, paintText, staticLayoutWidth).apply {
setAlignment(Layout.Alignment.ALIGN_NORMAL)
setLineSpacing(1.0f, 1.0f)
setIncludePad(false)
}.build()
staticLayout.draw(canvas)

canvas.drawRect(0f, 0f, 200f, 200f, paintRect)

Produces different results:

2A.: Pixel 2 Android 12. WRONG: White flashing + overlay repeats text (in place) + blue rectangles problem (I draw only one rectangle).

 

2B.: Pixel 2 Android 13. WRONG: Overlay repeats text (in place, see seconds) + blue rectangles problem (I draw only one rectangle with alpha 22, so the rectangle is overlaped several times).

 

Samsung Galaxy S20+ Android 13: WORKS WELL. Text is changed every second, text is not overlaped, blue rectangle is not overlaped, there is only one blue rectangle with correct alpha.

Thank you for your answer.

Dne pondělí 6. května 2024 v 20:48:23 UTC+2 uživatel xi...@google.com napsal:

[Xi Zhang (张熹)]

Thanks for the feedback. I can reproduce the issue on Pixel 2. This seems to be a device compatibility issue where the Android API misbehaves on an older device. Unfortunately, there isn't an easy fix for it. There are a few options but they require time to investigate and implement. If they are fixed, it will be added in a future version. Filed https://issuetracker.google.com/issues/339451836 to track the issue.

[Xi Zhang]

If you are using PreviewView, could you try if it helps to set the mode to COMPATIBLE?

previewView.implementationMode = PreviewView.ImplementationMode.COMPATIBLE

[Xi Zhang (张熹)]

I added a code sample to our test app, for reproducing the issue. Somehow I can no longer reproduce it. This is the change I made:

https://android-review.googlesource.com/c/platform/frameworks/support/+/3075863

If you are still experiencing this issue, you can check out if this code works for you.

[Tomáš Válek]

I try to set previewView to PreviewView.ImplementationMode.COMPATIBLE mode before or after bindToLifecycle(), but does not help. Still blinking like in the screenshot in the first post.

Dne čtvrtek 23. května 2024 v 20:19:04 UTC+2 uživatel xi...@google.com napsal:

[Nick Song]

you can try this code:

val overlayEffect = OverlayEffect(CameraEffect.PREVIEW or CameraEffect.VIDEO_CAPTURE,
0, Handler(Looper.getMainLooper())
)