New developer, trying to display captured ImageProxy and let user decide to keep or re-take

[Omri Nachmani]

I am developing an app that intends to use the ML Kit face recognition on still images rather than live camera feed. I've used the CameraX tutorial to set everything up but can't figure out how to display the image after capture, and can't find a single good answer online. I am also a new developer on Kotlin so if you have an answer, please spell it out. My code:

package com.example.faceanalysis

import android.Manifest
import android.content.pm.PackageManager
import android.net.Uri
import android.os.Bundle
import android.util.Log
import android.widget.Toast
import androidx.appcompat.app.AppCompatActivity
import androidx.camera.core.*
import androidx.camera.lifecycle.ProcessCameraProvider
import androidx.core.app.ActivityCompat
import androidx.core.content.ContextCompat
import com.google.mlkit.vision.common.InputImage
import com.google.mlkit.vision.face.FaceDetection
import com.google.mlkit.vision.face.FaceLandmark
import kotlinx.android.synthetic.main.activity_main.*
import java.io.File
import java.text.SimpleDateFormat
import java.util.*
import java.util.concurrent.ExecutorService
import java.util.concurrent.Executors
import com.example.faceanalysis.MainActivity.FaceAnalysis as FaceAnalysis1

typealias LumaListener = (luma: Double) -> Unit

class MainActivity : AppCompatActivity() {

private var imageCapture: ImageCapture? = null
private lateinit var outputDirectory: File
private lateinit var cameraExecutor: ExecutorService

override fun onCreate(savedInstanceState: Bundle?) {
super.onCreate(savedInstanceState)
setContentView(R.layout.activity_main)

// Request camera permissions
if (allPermissionsGranted()) {
startCamera()
} else {
ActivityCompat.requestPermissions(
this, REQUIRED_PERMISSIONS, REQUEST_CODE_PERMISSIONS)
}
// CameraX tutorial code:

// Set up the listener for take photo button
camera_capture_button.setOnClickListener { takePhoto() }

outputDirectory = getOutputDirectory()

cameraExecutor = Executors.newSingleThreadExecutor()
}

private fun takePhoto() {
// Get a stable reference of the modifiable image capture use case
val imageCapture = imageCapture ?: return

// Create time-stamped output file to hold the image
val photoFile = File(
outputDirectory,
SimpleDateFormat(FILENAME_FORMAT, Locale.US
).format(System.currentTimeMillis()) + ".jpg")

// Create output options object which contains file + metadata
val outputOptions = ImageCapture.OutputFileOptions.Builder(photoFile).build()

// Set up image capture listener, which is triggered after photo has
// been taken
imageCapture.takePicture(ContextCompat.getMainExecutor(this), object : ImageCapture.OnImageCapturedCallback() {
override fun onError(exc: ImageCaptureException) {
Log.e(TAG, "Face capture failed: ${exc.message}", exc)
}

override fun onCaptureSuccess(image: ImageProxy) {
// Code to display image from image proxy
// code to ask user if they would like to use this image
//call to close camera if they say yes, call to keep camera if they say no
val savedUri = Uri.fromFile(photoFile)
val msg = "Face capture succeeded: $savedUri"
Toast.makeText(baseContext, msg, Toast.LENGTH_SHORT).show()
Log.d(TAG, msg)
}
})
}

private fun startCamera() {
val cameraProviderFuture = ProcessCameraProvider.getInstance(this)

cameraProviderFuture.addListener(Runnable {
// Used to bind the lifecycle of cameras to the lifecycle owner
val cameraProvider: ProcessCameraProvider = cameraProviderFuture.get()

// Preview
val preview = Preview.Builder()
.build()
.also {
it.setSurfaceProvider(viewFinder.surfaceProvider)
}

imageCapture = ImageCapture.Builder()
.build()

//val imageAnalyzer = ImageAnalysis.Builder()
// .build()
// .also {
// it.setAnalyzer(cameraExecutor, FaceAnalysis)


// }

// Select back camera as a default
val cameraSelector = CameraSelector.DEFAULT_BACK_CAMERA

try {
// Unbind use cases before rebinding
cameraProvider.unbindAll()

// Bind use cases to camera
cameraProvider.bindToLifecycle(
this, cameraSelector, preview, imageCapture)

} catch(exc: Exception) {
Log.e(TAG, "Use case binding failed", exc)
}

}, ContextCompat.getMainExecutor(this))
}

private fun allPermissionsGranted() = REQUIRED_PERMISSIONS.all {
ContextCompat.checkSelfPermission(
baseContext, it) == PackageManager.PERMISSION_GRANTED
}

private fun getOutputDirectory(): File {
val mediaDir = externalMediaDirs.firstOrNull()?.let {
File(it, resources.getString(R.string.app_name)).apply { mkdirs() } }
return if (mediaDir != null && mediaDir.exists())
mediaDir else filesDir
}

override fun onDestroy() {
super.onDestroy()
cameraExecutor.shutdown()
}

override fun onRequestPermissionsResult(
requestCode: Int, permissions: Array<String>, grantResults:
IntArray) {
if (requestCode == REQUEST_CODE_PERMISSIONS) {
if (allPermissionsGranted()) {
startCamera()
} else {
Toast.makeText(this,
"Permissions not granted by the user.",
Toast.LENGTH_SHORT).show()
finish()
}
}
}

companion object {
private const val TAG = "CameraXBasic"
private const val FILENAME_FORMAT = "yyyy-MM-dd-HH-mm-ss-SSS"
private const val REQUEST_CODE_PERMISSIONS = 10
private val REQUIRED_PERMISSIONS = arrayOf(Manifest.permission.CAMERA)
}

[Patrick Liao]

Hi,

There are two approaches (without ImageAnalysis!) to this as far as I concern:

1. Try using ``ImageCapture.OnImageCapturedCallback()`` when you call ``ImageCapture.takePicture()`` since it would not save the picture to the storage to you but its ``onCaptureSuccess(ImageProxy)`` give you an ImageProxy instance, which could be converted into ``Image`` for you to use (+It is in YUV format, which should be beneficial in your case).

2. ImageCapture.OnImageSavedCallback() could also make this happen. With its ``onImageSaved(ImageCapture.OutputFileResults)`` method, you can obtain the URI of the saved image with ``outputFileResults.getSavedUri()`` and you can go from there. The disadvantage over the first method is that it is now in JPEG and a delay would occur this method is called by the system ONLY AFTER CameraX saves the file to storage.

Hope this helps.

Cheers,

Patrick Liao 

[Omri Nachmani]

Thanks Patrick,

The first approach sounds like the ideal solution. Now since I am only starting out with android dev overall (I come from python dev, just learned kotlin), what code / classes would i need to display this image proxy? Do i need to great a new activity all together (if so how does the new activity receive imageProxy) or can I somehow stop camera feed and display the image, restart camera feed if user discards photo?

Would appreciate advice on the correct direction ! 

[Patrick Liao]

Hi,

You are welcome.

ImageProxy is passed to you with the callback method. And for displaying it, I would not bother creating a new activity but use something like an ImageView or a Fragment that goes right on top of the current screen. For camera hibernation, you can try to wrap your camera activity inside a fragment and the ImageView thing as another, then you create a "master" activity with only one fullscreen element -- a fragment placeholder. This way, you can hide the camera fragment and display ImageView fragment when you need to, and CameraX will do the hibernation stuff for you since CameraX is lifecycle-aware (i.e. when you "hide" the camera fragment, everything will act as if onPause() is called). 

Cheers,

Patrick Liao

[Scott Nien]

Thanks Patrick for sharing the tips. Yes please use ImageCapture.OnImageCapturedCallback if you would like to show the captured image without saving to the disk. This would be faster. 

Some caveats for the OnImageCapturedCallback though: 
  1) It is actually JPEG format  (see javadoc ) but MLKit InputImage.fromMediaImage also accepts Image with JPEG format so that's probably fine. 

  2) The image is provided without rotation applied. Developers should apply the image.getImageInfo().getRotationDegrees() clockwisely. If MLkit is used, you should pass the image.getImageInfo().getRotationDegrees() as the rotationDegrees parameters in the InputImage.fromMediaImage call. 

--
You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camerax-develop...@android.com.
To view this discussion on the web visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/f3208ed8-36c1-4f62-a486-5e069b55ef2an%40android.com.

[Omri Nachmani]

Thank you Scott, i will make sure to apply the rotation. Such a helpful communtiy!

[Omri Nachmani]

So i have managed to set up camerax in a fragment and have an imageview waiting for the image in another fragment. My current struggle is to send the image proxy to the other fragment. I am using NavDirections that uses Bundle to pass Safe Args from one fragment to another. In the design tab I can add arguments but the type must be declared and I don't know how to ensure it can interpret the image type. Is there a better way? 

Code below:

CameraXFragment: takePhoto function

private fun takePhoto() {
// Get a stable reference of the modifiable image capture use case
val imageCapture = imageCapture ?: return

// Setup image capture listener which is triggered after photo has
// been taken
imageCapture.takePicture(ContextCompat.getMainExecutor(safeContext), object : ImageCapture.OnImageCapturedCallback() {
override fun onError(exc: ImageCaptureException) {
Log.e(TAG, "Photo capture failed: ${exc.message}", exc)
}

@SuppressLint("UnsafeExperimentalUsageError")
override fun onCaptureSuccess(image: ImageProxy) {
val faceImage = image.image
findNavController().navigate(CameraXViewDirections.actionCameraXViewToReviewImageFragment(faceImage))
super.onCaptureSuccess(image)

}

})
}

ReviewImageFragment

override fun onCreateView(
inflater: LayoutInflater, container: ViewGroup?,
savedInstanceState: Bundle?
): View? {
val binding = DataBindingUtil.inflate<FragmentReviewImageBinding>(
inflater, R.layout.fragment_review_image, container, false)
// Inflate the layout for this fragment
val args = reviewImageFragmentArgs.fromBundle(requireArguments())
binding.imageView.setImageResource(args.faceImage)
return binding.root
}

navigation.xml:

<?xml version="1.0" encoding="utf-8"?>
<navigation xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:app="http://schemas.android.com/apk/res-auto"
xmlns:tools="http://schemas.android.com/tools"
android:id="@+id/navigation"
app:startDestination="@id/cameraXViewFragment">

<fragment
android:id="@+id/cameraXViewFragment"
android:name="com.example.faceanalysis.CameraXView"
android:label="CameraXView" >
<action
android:id="@+id/action_cameraXView_to_reviewImageFragment"
app:destination="@id/reviewImageFragment" />
</fragment>
<fragment
android:id="@+id/reviewImageFragment"
android:name="com.example.faceanalysis.reviewImageFragment"
android:label="fragment_review_image"
tools:layout="@layout/fragment_review_image" >
<argument android:name="faceImage" />
</fragment>
</navigation>

[Scott Nien]

// Resent it to the all group. 

Hi Omri, 

I don't think imageView.setImageResources will work since it is used to set the drawable resources. 

Please try the following code snippet to draw the ImageProxy (JPEG format) into the ImageView 

    ByteBuffer buffer = imageProxy.getPlanes()[0].getBuffer();
    byte[] bytes = new byte[buffer.capacity()];
    buffer.get(bytes);
    Bitmap bitmapImage = BitmapFactory.decodeByteArray(bytes, 0, bytes.length, null);
    imageProxy.close();   

    // Must run in UI thread
    imageView.setRotation((float)imageProxy.getImageInfo().getRotationDegrees()); 
    imageView.setImageBitmap(bitmapImage);

[Omri Nachmani]

This worked! I can now display the recently captured image!!! You are all amazing and this community is great. Thank you !!