Simultaneous mic usage - CameraX Video Capture & 3rd Party SDK

[Aubrey Haskett]

Hi CameraX folks - wondering if anyone has come across this or knows a way to make this work:

I am using CameraX's VideoCapture use case to record video. At the same time, I am streaming to a 3rd party via an SDK. I have the view working and sending info to both places, but not the audio. My audio is only going to VideoCapture, not my 3rd party SDK.

How I do the view:

1. I create a custom surfaceProvider. The surface given to that surfaceProvider is generated by the 3rd party SDK.
2. I then use my surfaceProvider to build a preview, which I give to CameraX as the preview use case: 

val surfaceProvider = cameraState.customSurfaceProvider
val preview = Preview.Builder().build().also { it.setSurfaceProvider(surfaceProvider) }

That's all great and works perfectly. But the audio isn't working for the stream, only the recording.

How I do the audio:

• The 3rd party SDK has an area where you can attach items like a camera or mic. I'm attaching the device's mic.
• This works and provides the streamed video with audio when I am NOT using the VideoCapture use case at the same time. As soon as a start using VideoCapture alongside the stream, the audio only goes to the recording, not the stream.
• I queried the SDK and it says that it has properly attached the mic and all should be well. So I believe VideoCapture is consuming the audio and the stream is failing silently. 

Has anyone faced this before? I see a convo in here from last year that might help me but want to get more recent opinions. I really want to continue using CameraX instead of Camera2. 

Thanks. 

[Aubrey Haskett]

I have a suspicion that the stream via the SDK is receiving silence because two threads can't access the same mic. I am not 100% sure which mic(s) the SDK is trying to use versus CameraX's videoCapture, because both obscure that code. CameraX source code shows a whole lotta of checks to find the right mic(s). So I haven't confirmed this theory, but I think it's the case. 

I am thinking I may have to not use CameraX for VideoCapture. I'll have to set up MediaRecorder with a custom surface (image input I'll send to the SDK's stream, the recording mp4, and the user's device preview), along with an audio stream (goes to recording and SDK's stream). :( Sad because there are still a ton of freebies I want to use from CameraX but have to keep customizing it. 

[tonyt...@google.com]

Yes, it depends on the device implementation, but on most devices, the same audio source can only be accessed by one client at a time. CameraX uses CAMCORDER by default. If the SDK tries to access the same source simultaneously, it could receive silence (see the behavior). With that, there's a chance that the audio only goes to one of the clients even if you switch the implementation to MediaRecorder for video recording.

[Leo Huang]

One thing to add. I did some tests and found it behaves on different SDK levels.

Is your test device pre-Android 10? If yes, you could run into the behavior as mentioned in behavior "Pre-Android 10 behavior". Then you can try on newer SDK devices.

```

One more change was added in Android 9: only apps running in the foreground (or a foreground service) could capture the audio input. When an app without a foreground service or foreground UI component started to capture, the app continued running but received silence, even if it was the only app capturing audio at the time.

```

Although the doc of behavior doesn't mention the behavior "within the same application on Android 10+", my test result shows that two AudioRecords instances can receive audio data simultaneously. (Ps. CameraX VideoCapture and MediaRecorder both use AudioRecord under the hook.) Even if two AudioRecords have different AudioSource, sample rate, encoding settings and running on different processes.

'tonyt...@google.com' via Android CameraX Discussion Group <camerax-d...@android.com> 於 2024年7月12日 週五 上午11:52寫道：

--
You received this message because you are subscribed to the Google Groups "Android CameraX Discussion Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to camerax-develop...@android.com.
To view this discussion on the web visit https://groups.google.com/a/android.com/d/msgid/camerax-developers/ef26531d-ce72-483c-aa9f-02b7451f9d3fn%40android.com.

[Aubrey Haskett]

Ahh, I bet that's what's happening. I can't know which mic is being used by the SDK - they have everything shrouded from me. But I bet it's the CAMCORDER, because when I independently run that SDK stream alongside plain old AudioRecord (so, take CameraX VideoCapture out of the equation entirely), I do get audio if I manually set the AudioRecord mic to MIC rather than CAMCORDER. Huh.

I guess I could still use CameraX VideoCapture and: 

1. Save the video only (no audio) internally
2. Record audio at the same time using a byteBuffer to go to both the SDK and an mp3
3. Then mux the VideoCapture video and recorded mp3 at the end for a resulting mp4 with audio

That's a little crazy and just going back to Camera2 might be more proper. I'll have to think about it. 

Thanks for the help! 

[Aubrey Haskett]

Thanks for the reply. 

No - I wasn't test pre-Android 10. I am on a Google Pixel 8 Pro with Android 14. 

Thanks for letting me know CameraX VideoCapture uses AudioRecord under the hook. I just noted in another comment that I tried AudioRecord on it's own, but directed it to use the MIC...that worked and I was able to save an mp3 to device while the SDK presumably used the CAMCORDER (SDK doesn't tell me what they're using, unfortunately. I've asked the third-party company so we'll see if they tell me). 

[Leo Huang]

Btw, if you would like to change the audio source of VideoCapture, there is currently a way as mentioned here via Java reflection. 

aubrey....@gc.com 在 2024年7月18日 星期四凌晨2:22:33 [UTC+8] 的信中寫道：