PSA: Stopping source and firing ‘ended’ event on audio track if missing input audio is detected in Chrome

[Henrik Grunell]

A mechanism to detect missing input (mic) audio data has been implemented in Chrome[1]. After around 15 seconds of no input audio data, the audio track and its source will be stopped and the ‘ended’ event is fired[2].

The reason for this is that the stream typically doesn’t recover from this state. The application can choose what to do, for example try to restart the input stream or inform the user that there is a problem.

This change will be in M60.

Regards,

Henrik

[1] https://chromium.googlesource.com/chromium/src/+/021d9e05dc8384ba3d5f7c1c1834ee0bf9fd3e5c

[2] https://w3c.github.io/mediacapture-main/#event-mediastreamtrack-ended

[Warren McDonald]

I have tried to test this behaviour on current 60 beta, but cannot trigger the track end behaviour.

Complete mute of mic input device results in no change in track status even after 30 seconds.

Has this change been withdrawn?

Withdrawing would actually be preferable as this behaviour would have unpredictable impact on many applications where user muting preference are unknown.

If this is implemented, then we will have to instruct users to only use our application mute mic function and never their system or audio peripheral mute.

[Iñaki Baz Castillo]

2017-06-30 0:40 GMT+02:00 Warren McDonald <warren....@gmail.com>:
> I have tried to test this behaviour on current 60 beta, but cannot trigger the track end behaviour.
>
> Complete mute of mic input device results in no change in track status even after 30 seconds.

I'm a bit lost here. Muting an audio track should not trigger "ended"
event at all.

Definition of the MediaStreamTrack "ended" event:

> The MediaStreamTrack object's source will no longer provide any data, either because the user revoked the permissions, or because the source device has been ejected, or because the remote peer permanently stopped sending data.

Muting a mic is not that because obviously, you still want to be able
to unmute it.


--
Iñaki Baz Castillo
<i...@aliax.net>

[Warren McDonald]

Agree, it would be good to understand the motivation for the idea of assuming the track can be ended, just because it has had no signal for an arbitrary period of time.

[Dag-Inge Aas]

Not to answer on behalf of Google here, but I'm 99% sure that this is related to the infamous Chrome Audio bug that has been plaguing OSX users for some years now. That is that Chrome will for some reason fail to obtain the microphone but still resolve with the microphone track, so that you believe you have a microphone input, but it is in fact silent.

More details here: https://bugs.chromium.org/p/webrtc/issues/detail?id=4799

On Fri, Jun 30, 2017 at 10:37 AM, Warren McDonald <warren....@gmail.com> wrote:

Agree, it would be good to understand the motivation for the idea of assuming the track can be ended, just because it has had no signal for an arbitrary period of time.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/fd482671-e0c0-473e-861a-9aa70042e8a3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Dag-Inge Aas
Tech Lead @ appear.in

[Iñaki Baz Castillo]

2017-06-30 10:37 GMT+02:00 Warren McDonald <warren....@gmail.com>:
> Agree, it would be good to understand the motivation for the idea of assuming the track can be ended, just because it has had no signal for an arbitrary period of time.

The track "ended" event fires when the remote sends us a SDP in which
a sending audio/video track has been removed. That's AFAIK the only
way to signal that a remote track has definitely ended.

AFAIR "ended" event is not fired when calling stop() on a local track
(not sure about remote tracks).

[Taylor Brandstetter]

The track "ended" event fires when the remote sends us a SDP in which
a sending audio/video track has been removed. That's AFAIK the only
way to signal that a remote track has definitely ended.

Note that this behavior may be going away in the standard... See https://github.com/w3c/webrtc-pc/issues/1181

The current direction of the working group is to do something where "removeTrack" causes the track to be removed from the stream on the other side, but not ended: https://github.com/w3c/webrtc-pc/pull/1402

This is because after you call "removeTrack", you can still do "sender.replaceTrack" to start sending again. So it's not possible for the track on the remote side to go to "ended", because that's irreversible.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.
To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CALiegfmwpyOkZsh8g-R6_uaDGXVeCFMd0QFnDfJ1RDZmbQWMww%40mail.gmail.com.

[Silvia Pfeiffer]

Yeah, I think the commit message confirms that:

* Detection is done in AudioInputDevice as this corresponds to a physical device which is what we want to detect missing callbacks for.
* When detected we call CaptureError() on the capturer, which stops the source and puts the track in ended state and fires the ended event.

Best Regards,
Silvia.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/CAOKXErqyzhX7RN-5-GiUy2%3DPciwxoS%3DJyrzi4GURApV%3DFraYkg%40mail.gmail.com.

[Warren McDonald]

OK. So the critical description here is "missing callbacks" 

Will a working audio device that has been muted in its audio circuit, outside of the audio interface/device, cause missing callbacks. If the answer to this is true, then this is the wrong way to handle this, as that will be a common situation with table speaker/mics and headphones with inline switches. 

If only misbehaving currently instantiated AudioInputDevices, that have never, or have stopped, providing callbacks, will cause this to trigger, then that sounds quite reasonable, as the device state cannot be determined. 

If it is the latter, could the PSA be reworded to explain that this is targeted at failed devices, rather than missing audio signal.  

Could we please get a more detailed description of the targeted device scenario. 

Thanks

    
Warren

On Saturday, 1 July 2017 08:33:30 UTC+10, Silvia Pfeiffer wrote:

Yeah, I think the commit message confirms that:

* Detection is done in AudioInputDevice as this corresponds to a physical device which is what we want to detect missing callbacks for.
* When detected we call CaptureError() on the capturer, which stops the source and puts the track in ended state and fires the ended event.

Best Regards,
Silvia.

On 30 Jun. 2017 8:29 pm, "Dag-Inge Aas" <dagi...@appear.in> wrote:

Not to answer on behalf of Google here, but I'm 99% sure that this is related to the infamous Chrome Audio bug that has been plaguing OSX users for some years now. That is that Chrome will for some reason fail to obtain the microphone but still resolve with the microphone track, so that you believe you have a microphone input, but it is in fact silent.

More details here: https://bugs.chromium.org/p/webrtc/issues/detail?id=4799

On Fri, Jun 30, 2017 at 10:37 AM, Warren McDonald <warren....@gmail.com> wrote:

Agree, it would be good to understand the motivation for the idea of assuming the track can be ended, just because it has had no signal for an arbitrary period of time.

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/fd482671-e0c0-473e-861a-9aa70042e8a3%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Dag-Inge Aas
Tech Lead @ appear.in

--

---
You received this message because you are subscribed to the Google Groups "discuss-webrtc" group.

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrt...@googlegroups.com.

[Henrik Grunell]

Muting a device does not stop callbacks with audio data from being delivered, thus does not trigger this detection.

The motivation for the change is to detect device or audio system failures, which is normally not possible to recover from other than by restarting the application.

Regards,

Henrik

To unsubscribe from this group and stop receiving emails from it, send an email to discuss-webrtc+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/discuss-webrtc/d1212d49-df6e-459b-b7f3-9df7ce891f4b%40googlegroups.com.

[Warren McDonald]

We have just seen this in action on a live system and the impact is not so nice.

In our current configuration the Ended event is causing a recapture which results in a new stream, which results in a renegotiation, which then gives us a stable camera track for another 15 seconds (no audio of course), and then repeats the whole sequence every 15 seconds. 

So it looks like we will have to change logic to do something like count the number of unexpected track ended events and then exit with user information if it continues to happen. Assuming of course that a recapture may result in an active audio track. If we know that the condition is not recoverable, then we may as well abort on first instance.   

Happy days.