Lifecycle managing with multiple Preview UseCases

[Zoran Smilevski]

Hi!

I have a special type of suggestion/request regarding managing lifecycle with multiple UseCases. Now it's happening that when one of UseCases is stopped, whole CameraX stop. It would be nice if API could be more flexible regarding that.

Let me go into more details why I need this. 

We have an app which is streaming camera and user also see preview of what is streamed. CameraX is created in foreground service with two Preview UseCases, one for PreviewView and another for encoder and streamer. First Preview is shared with UI over LiveData and when fragment is visible it shows preview. Everything fine here until we minimize fragment (collapse/minimize it like in audio player, so you can browse around an app while still streaming), which is done with MotionLayout. In that minimized view we don't have a PreviewView so it's lifecycle is stopped which also unbinds all UseCases. Here comes a problem for us. Because all UseCases are stopped, also encoder and streamer is stopped. And whole concept of minimized view is broken. Hope it's clear enough where our problem is :)

So, we tackle that with a custom surface provider based on SurfaceTextureProvider. That solves a problem with lifecycle, but problem arrives with correctly transforming preview. We tried with using PreviewTransformation, but we couldn't make it work correctly, guess some required plumbing was missing. So we went with some older implementations of preview transformation, but it's not always perfect. Stil looking into that how to make it reliable, because sometimes transformstions are correct, sometimes are stretched.

If there is some other solution to that, please let me know, would be glad if you could help me.

TLDR; 

Would it be possible to create a PreviewView with an option to not being bound to lifecycle and so not unbind UseCase when removed? Does this make sense or is there any other solution for that?

[Charcoal Chen]

Hi,

Thanks for using CameraX.

You can refer to the FakeLifecycleOwner.java to implement your own customized LifecycleOwner to fulfill your requirement and use it to bind the use cases. Then, you can control the camera streams to not be stopped when the fragment is minimized. 

[Zoran Smilevski]

Thanks for answering so quickly! I did few test yesterday based on your answer and that sadly doesn't solve my problem :(

I didn't had a lifecycle problem because Service lives longer than PreviewView and so it survives it. But here are my two obsevrations based on PreviewView implementations I didn't know before:

1. TextureView (implementationMode = PreviewView.ImplementationMode.COMPATIBLE): it works fine when camera is started while PreviewView is visible. It even works when I go to homescreen and play with other apps. But if I start camera when PreviewView is not visible (from minimized view) it won't produce frames. PreviewView must be visible when starting camera or it doesn't work, it's just black rectangle. 

 

2. SurfaceView: it starts fine when PreviewView is not visible. It also works when in minimized mode and I'm navigating around in app. But it stops producing frames a second or two when I go to homescreen (out of app). When I come back I need to restart camera to start producing frames again.

So what I'm looking for is a combination of both. That could be started when PreviewView is not visible (SurfaceView) and that it works when app is in background (TextureView).

It's hard to describe everything with words :) I can do a liitle demo and put it here so you can see what I mean if necessary.

[Xi Zhang (张熹)]

If I understand correctly, your goal is to:

1. have a always-on streaming in the background, and
2. display or hide preview in foreground based on user actions.

We plan to support video streaming in the future, but currently there is no ETA. For now, there are a couple of things you can try.

First of all, CameraX is not designed to support 2 use cases of the same type. e.g. the 2 Preview use cases in your scenario. It might work, but there is no guarantee. If you still want to go this route, could you try unbinding Preview when the UI is hidden, and rebinding when the UI is displayed? For example, when the preview is active, do:

cameraProvider.bindToLifecycle(lifecycleOwner, cameraSelector, previewForPreview, previewForStreaming)

And when the preview is hidden, do:

cameraProvider.bindToLifecycle(lifecycleOwner, cameraSelector, previewForStreaming)

Alternatively, you could replace the previewForStreaming with an ImageAnalysis use case and stream its output. e.g. with an ImageWriter. It's less efficient but more stable.

Also, creating a custom SurfaceProvider should work too, as you suggested in the first post. You are right about the transformation being tricky. Both the SurfaceView and the TextureView rely on the info from the camera to correct the preview. I assume that info is what gets lost in the redirection. To plumb it through, your custom SurfaceProvider should compensate for the info in your custom SurfaceRequest. It's complicated and requires a good understanding on how the stack works.

--
You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camerax-develop...@android.com.
To view this discussion on the web visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/53fd1e0a-d467-4513-acf4-bb63f1fb1903n%40android.com.

[Zoran Smilevski]

Thanks for your answer! Yes it's that exactly.

I tried with ImageAnalysis first, but couldn't make it work because image from front camera was 270 degress rotated, and with YUV library I was not able to rotate it correctly and to mantain colors (it was always green or purple in some places). Also I think this would be too slow or too inefficient. Because of our streaming protocol which is raw h264 baseline frames I need to rotate them correctly. So I solved it like I answered in this stackoverflow post https://stackoverflow.com/a/63528509/1004550 with introduction of OpenGLRenderer in between preview and encoder. This works perfectly.

For rebinding do I need to unbind all first and then bind again and so restart encoder? Or can I just bind multiple times?

With custom PreviewView implementation, a lot of API is also restricted to library or private so it's hard to copy stuff and make it work.

Anyway, it's still great to use camerax, to help with dealing with camera. Stable image is more important than unperfect preview in some cases. Have an idea to also try with picture-in-picture, so PreviewView will always be visible. But still looking for solution when this is not possible.

[Zoran Smilevski]

If anyone else will face similar problem. I solved that problem by using only one preview use case as suggested above, but modifying OpenGLRenderer, where I added additional native context, on which I attach another surface output surface (same as done in 'attachOutputSurface' function). Important is to create an additional native context with display, config, context, pbufferSurface, program and textureId from preview native contex. And then when rendering in 'renderLatest' you draw to both surfaces by switching context with makeCurrent(additionalContext). Transformations are done the same way for both surfaces and stream and preview looks great now.

[Zoran Smilevski]

Here's a my take on that what I described above. If someone can check and comment I will be very thankful.