Clarification on Supported Targets for OverlayEffect

[Siddhant]

Hello,

We are currently using the OverlayEffect in CameraX (Version 1.4.1) to overlay timestamp and geo-coordinates on the image. We have 2 different overlay effects. First effect only has CameraEffect.PREVIEW as target and second effect has  CameraEffect.IMAGE_CAPTURE only. According to the addEffect documentation, it states that the IMAGE_CAPTURE as the only target is supported.

However, when we specify CameraEffect.IMAGE_CAPTURE as the only target for the OverlayEffect, we encounter an IllegalArgumentException, which mentions that only three specific target combinations are supported by the OverlayEffect(show in the image below).

Could you please clarify if this is bug or are we doing something wrong here?

Thank you.

[Xi Zhang]

Hi this is working as intended. The preview/recording uses GPU buffer and the image capture uses CPU buffer, to optimize the performance, we have different APIs for it. To target IMAGE_CAPTURE, you will need to create a CameraEffect with a ImageProcessor. Please see this API: 

https://developer.android.com/reference/androidx/camera/core/CameraEffect#CameraEffect(int,java.util.concurrent.Executor,androidx.camera.core.ImageProcessor,androidx.core.util.Consumer%3Cjava.lang.Throwable%3E)

Please let me know if you have more questions.

[Siddhant]

Thank you for the clarification. We implemented the CameraEffect as you suggested, and it works well overall. However, we’ve run into an issue where the text is being drawn vertically instead of horizontally. From what I understand, this is because the canvas is created using the dimensions of the Bitmap, which inherits the camera’s orientation.

I believe we may need to apply the same sensor-to-UI coordinate conversion that we use for the overlay effect. We tried passing the PreviewView for this conversion, but it results in an error when attempting to capture an image.

Could you suggest the best way to draw the text in the correct orientation? I’ve attached the code and a screenshot for your reference.

class WatermarkImageEffect(
val timeStampDate: String,
val timeStampGeoCord:String,
val context: Context,
) : CameraEffect(
IMAGE_CAPTURE,
Runnable::run,
WatermarkProcessor(timeStampDate, timeStampGeoCord, context),
{ throwable -> }) {
internal class WatermarkProcessor( val timeStampDate: String,
val timeStampGeoCord:String,
val context: Context,) : ImageProcessor {
override fun process(request: ImageProcessor.Request): ImageProcessor.Response {
val inputImage = request.inputImage
val bitmapIn = inputImage.toBitmap()
inputImage.imageInfo.sensorToBufferTransformMatrix
val bitmapOut = createProcessedBitmap(bitmapIn, timeStampDate, timeStampGeoCord,
context )
return ImageProcessor.Response {
createOutputImage(bitmapOut, inputImage)
}
}
/**
* Creates output image
*/
@SuppressLint("RestrictedApi")
private fun createOutputImage(newBitmap: Bitmap, imageIn: ImageProxy): ImageProxy {
return RgbaImageProxy(
newBitmap,
imageIn.cropRect,
imageIn.imageInfo.rotationDegrees,
imageIn.imageInfo.sensorToBufferTransformMatrix,
imageIn.imageInfo.timestamp
)
}
/**
* Apply text on the [Bitmap]
*/
private fun createProcessedBitmap(
bitmapIn: Bitmap,
timeStampDate: String,
timeStampGeoCord:String,
context: Context,
): Bitmap {
val paint = Paint()
val textSizeInPx = TypedValue.applyDimension(
TypedValue.COMPLEX_UNIT_DIP,
13f,
context.resources.displayMetrics
)
// Create paint for timestamp
val timestampPaint = Paint().apply {
color = android.graphics.Color.WHITE
textSize = textSizeInPx
typeface = Typeface.DEFAULT_BOLD
setShadowLayer(3f, 1f, 1f, BLACK)
isAntiAlias = true
}
val textHeight = timestampPaint.fontMetrics.descent - timestampPaint.fontMetrics.ascent
val textWidthTimestamp = timestampPaint.measureText(LocalDate.now().toString())
val textWidthGeoCord = timestampPaint.measureText(timeStampGeoCord)
val textWidth = max(textWidthTimestamp, textWidthGeoCord)
val canvas = Canvas(bitmapIn)
val x = (canvas.width - textWidth)
val y = (canvas.height - textHeight - textHeight)
canvas.drawText(LocalDate.now().toString(), x, y, timestampPaint)
canvas.drawText(timeStampGeoCord, x, y + textHeight, timestampPaint)
canvas.drawBitmap(bitmapIn, 0f, 0f, paint)
return bitmapIn
}
}
}

Expected

Actual

[Scott Nien]

OverlayEffect doesn't implement the ImageProcessor so it can't be used with CameraEffect.IMAGE_CAPTURE only as Xi said. 

But alternatively, you can have the OverlayEffect with CameraEffect.PREVIEW | CameraEffect.IMAGE_CAPTURE | CameraEffect.VIDEO_CAPTURE
This way the effect will also be applied on the captured image.

Is this what you want ? 

--
You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camerax-develop...@android.com.
To view this discussion visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/81717953-49f5-466d-9bba-a00de8e6b735n%40android.com.