ConcurrentCamera use case group doubts

[Siddhant]

Hello. I am looking into the Concurrent Camera and have some questions. I am using camerax 1.5.0 rc-01. I want the same layout as a video call i.e. main camera view with a small camera view on corner. 

While the out of option provided by using CompositionSettings is great, I would like to have secondary view with rounded corner radius and border which does seem not possible with CompositionSettings.

I am not able to find any concrete documentation and sample regarding UseCaseGroup to be used in SingleCameraConfig. According to to code snippet in this docs section, it seems possible to have multiple previews now but isn't camerax restricted to 1 preview?

[Scott Nien]

Hi Siddhant, 

Yes,  you can actually have multiple Preview instances (and multiple VideoCapture instances) when using ConcurrentCamera. 

So currently there are two options 

1) Two Previews (and optionally two VideoCaptures)
     each camera streams to different Preview And VideoCapture 

2) One Preview (and optionally one VideoCapture),  you pass the same UseCaseGroup with CompositionSetting:
     CameraX composes the front and back camera output into one Preview and one VideoCapture. 

Scott

--
You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camerax-develop...@android.com.
To view this discussion visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/b4b7f9b7-85ce-4c2f-a9d5-40a2316c5d2fn%40android.com.

[Siddhant]

Alright. Thank you. So if I am understanding correctly, if I choose the 1st option i.e. 2 different previews, there will be 2 separate video capture and won't be combined into one. For combined capture, I have to use the second option, correct?

[Scott Nien]

Yes, Exactly 

[Siddhant]

Thank you for clarifying this. So I guess I will have to drop the customization of the PIP view. One last question, can we take picture while we are recording video in dual concurrent mode?

[Scott Nien]

>One last question, can we take picture while we are recording video in dual concurrent mode?

Unfortunately, you can't take pictures in this case.  See the guaranteed configurations for concurrent stream below

https://developer.android.com/reference/android/hardware/camera2/CameraDevice#concurrent-stream-guaranteed-configurations

As you can see,  only two streams are required. So if you are showing preview and recording video,  then there are two streams configured already. 

But there is a workaround, which you can use PreviewView.getBitmap to take a snapshot of preview. 

[Siddhant]

Sorry for late reply. I have some queries with regards to your reply: 

• Will there be any performance overhead If I use the workaround you are suggesting? 
• With the workaround I believe I won't be getting the screen blink effect we get when we capture image and have to implement manually?
• Is there anything else I should watch out while using getBitmap while recording?

[Scott Nien]

There is not much performance overhead but the quality is worse than regular still capture as regular still capture (ImageCapture) will have the Jpeg image optimized. 

It is basically a snapshot of the PreviewView and the size is exactly the same as PreviewView.   And all the other ImageCapture features are not available including screen flash. 

May I know what's the use case of your app ?  Why do your apps need to record and take pictures on concurrent cameras ?

This might be something we can support in the future. 

[Siddhant]

So due to a task requirement, our team is looking into feasibility of feature implementation having concurrent camera view which supports recording and having ability to snap picture while recording so that user can point to specifically any important item in recording. Samsung camera supports this so we thought it would be great if we can do so. It would be awesome if camerax can support this in future.

I actually found a sample app called Socialite made by your team I presume. https://github.com/android/socialite. A green screen effect is implemented in this with concurrent cam. I was able to modify the code to remove green screen effect and add imageCapture use case while it still has previewUseCase and videoCaptureUseCase. It does not use CompositionSettings. I am attaching screen recording and image that I took while doing so.

[Scott Nien]

The SocialLite app has two modes: Photo and Video.  Preview and ImageCapture are bound in Photo mode.  Preview And VideoCapture are bound in Video mode.  This is supported. 

[Siddhant]

Yes, It has 2 modes but I am in video mode in the recording I sent you. I added that white image shutter button on the start of the row component in video mode myself. It wasn't out of box. I believe the front cam image is being provided using the greenscreen effect? I am trying to understand why we do not get illegal exception since we have attached more than 2 use cases to useCaseGroupBuilder. Attaching code for more clarity.

if (effectMode == EffectMode.GREEN_SCREEN) {
// Concurrent camera setup for green screen effect
var primaryCameraSelector: CameraSelector? = null
var secondaryCameraSelector: CameraSelector? = null

// Iterate through available concurrent camera infos to find suitable primary
// (front-facing) and secondary (back-facing) cameras.
for (cameraInfos in cameraProvider.availableConcurrentCameraInfos) {
primaryCameraSelector = cameraInfos.first {
it.lensFacing == CameraSelector.LENS_FACING_FRONT
}.cameraSelector
secondaryCameraSelector = cameraInfos.first {
it.lensFacing == CameraSelector.LENS_FACING_BACK
}.cameraSelector

if (primaryCameraSelector == null || secondaryCameraSelector == null) {
// If either a primary or secondary selector wasn't found, reset both
// to move on to the next list of CameraInfos.
primaryCameraSelector = null
secondaryCameraSelector = null
} else {
// If both primary and secondary camera selectors were found, we can
// conclude the search.
break
}
}

if (primaryCameraSelector != null && secondaryCameraSelector != null) {
//preview and video use cases have already been added before. Adding effect and image capture use case
useCaseGroupBuilder.addEffect(greenScreenEffect)
useCaseGroupBuilder.addUseCase(imageCaptureUseCase)

val segmentedSelfieUseCaseGroupBuilder = UseCaseGroup.Builder()
.addUseCase(imageAnalysisUseCase)

val primary = ConcurrentCamera.SingleCameraConfig(
primaryCameraSelector,
segmentedSelfieUseCaseGroupBuilder.build(),
lifecycleOwner,
)

val secondary = ConcurrentCamera.SingleCameraConfig(
secondaryCameraSelector,
useCaseGroupBuilder.build(),
lifecycleOwner,
)

cameraProvider.unbindAll()
val concurrentCamera = cameraProvider.bindToLifecycle(
listOf(primary, secondary),
)

return@launch
}
}

[Scott Nien]

That's a great finding! Thanks!  

Turns out the effect will enable the "StreamSharing" feature for it so it can have one PRIV stream for both the preview and VideoCapture.  Hence it is possible to support the ImageCapture. 

To workaround it,  you can set an effect to the UseCaseGroup like this. 

private class DefaultEffect extends CameraEffect {
    protected DefaultEffect(int targets, @NonNull Executor executor,
            @NonNull Consumer<Throwable> errorListener) {
        super(targets, executor, DefaultSurfaceProcessor.Factory.newInstance(DynamicRange.SDR),
                errorListener);
    }
}

[Siddhant]

We’ve made a lot of progress since our last update, and everything is working well overall. Thanks for your help. However, we recently discovered an issue. This only happens for concurrent mode.

We’re providing the front image through an OverlayEffect, and we’re only setting the targets to PREVIEW or VIDEO_CAPTURE (Doing this cause of feature requirement where only back camera should appear in image, not both). For some reason, this causes the camera quality to drop to 640x360, and the resulting video is recorded at the same resolution. The use cases provided are image capture and video capture.

The quality selector for the VideoCapture use case is set to HD (720p). Interestingly, when we add the IMAGE_CAPTURE target, we get the expected 1280x720 resolution.