WiFi stack toxic to extreme low audio latency on Flo
101 views
Skip to first unread message
Brad Justice
Sep 17, 2015, 4:24:02 PM9/17/15
to android-ndk
I am running Jack Audio Connection Kit under Android 5.0.2 R1 on a
Nexus 7 2013. Jack is running directly on top of ALSA and is
essentially a parallel stack to the standard Android audio stack.
Currently the implementation is stable with three 256 sample buffers
at 96K sampling rate, which is 8.1 ms of buffer latency. Unfortunately
achieving stability with a buffer configuration this low requires
disabling WiFi.
There are two distinct failure patterns. I can toggle between them by
enabling or disabling verbose WiFi debugging. With verbose debugging
disabled, I see numerous streaming underrun events, each terminating
with a watchdog reset. These begin shortly after the Jack server is
started. With verbose debugging enabled, the streaming underrun events
are replaced by single underrun events for roughly one hour. At that
point I do get a streaming underrun/watchdog reset event.
I am running Jack with real time priority of 56. I have tried
priorities of up to 70 without positive result. Examining the logs, I
find nothing that connects the failure events to a change in WiFi
state, just the periodic scans that occur repeatedly without an
accompanying failure. If there is no network connection or if WiFi is
fully disabled I see no underruns.
I am posting here for two reasons. First to notify Google that the
Nexus WiFi stack appears to be toxic to extreme low latency audio and
that enabling verbose debugging has a significant positive effect on
problem behavior on 5.0.2 R1.
Second, I am attempting to eliminate the failures while maintaining a
network connection, at least for debugging purposes. Suppressing WiFi
scanning while Jack is running does improve matters at the cost of
roaming support. This would be an acceptable solution, but I still see
an underrun/reset event after 30 to 40 minutes of operation. Perhaps
someone with a better understanding of the WiFi stack could recommend
other lines of investigation.
Bradley Justice
Nexus 7 2013. Jack is running directly on top of ALSA and is
essentially a parallel stack to the standard Android audio stack.
Currently the implementation is stable with three 256 sample buffers
at 96K sampling rate, which is 8.1 ms of buffer latency. Unfortunately
achieving stability with a buffer configuration this low requires
disabling WiFi.
There are two distinct failure patterns. I can toggle between them by
enabling or disabling verbose WiFi debugging. With verbose debugging
disabled, I see numerous streaming underrun events, each terminating
with a watchdog reset. These begin shortly after the Jack server is
started. With verbose debugging enabled, the streaming underrun events
are replaced by single underrun events for roughly one hour. At that
point I do get a streaming underrun/watchdog reset event.
I am running Jack with real time priority of 56. I have tried
priorities of up to 70 without positive result. Examining the logs, I
find nothing that connects the failure events to a change in WiFi
state, just the periodic scans that occur repeatedly without an
accompanying failure. If there is no network connection or if WiFi is
fully disabled I see no underruns.
I am posting here for two reasons. First to notify Google that the
Nexus WiFi stack appears to be toxic to extreme low latency audio and
that enabling verbose debugging has a significant positive effect on
problem behavior on 5.0.2 R1.
Second, I am attempting to eliminate the failures while maintaining a
network connection, at least for debugging purposes. Suppressing WiFi
scanning while Jack is running does improve matters at the cost of
roaming support. This would be an acceptable solution, but I still see
an underrun/reset event after 30 to 40 minutes of operation. Perhaps
someone with a better understanding of the WiFi stack could recommend
other lines of investigation.
Bradley Justice
Glenn Kasten
Sep 17, 2015, 4:36:10 PM9/17/15
to android-ndk
Brad,
My theory is that this is most likely due to excessive interrupt disabling or preemption disabling in the kernel, power management, or a security (DRM) kernel.
Any of those could keep a realtime task from being woken up at the right time.
However, there are many other possible root causes.
To confirm which, I recommend using the systrace tool to see what is happening in the system at the time of the underrun. See
I also recommend watching this video (6 minutes)
which discusses the relationships between late task wakeup, audio buffer sizes, and audio latency:
and this article on contributors to audio latency:
If you narrow down the cause and would like to file a bug report, please do so here:
Thanks,
Glenn
Felix Homann
Sep 19, 2015, 4:57:32 AM9/19/15
to android-ndk
Am Donnerstag, 17. September 2015 22:24:02 UTC+2 schrieb Brad Justice:
I am running Jack Audio Connection Kit under Android 5.0.2 R1 on a
Nexus 7 2013.
May I ask _how_ exactly you are running Jack on Android?
Glenn Kasten
Sep 28, 2015, 1:25:32 PM9/28/15
to android-ndk
Brad Justice
Nov 6, 2015, 4:59:21 PM11/6/15
to android-ndk
Glenn
I wanted to comment on some of your suggestions regarding my low audio latency investigation.
I wanted to comment on some of your suggestions regarding my low audio latency investigation.
>My theory is that this is most likely due to excessive interrupt disabling or preemption disabling in the kernel,
>power management, or a security (DRM) kernel.
Interrupt or preemption disabling sounds right, and I believe that I have definitively established the cause as somewhere in the WiFi stack. Test runs go from very few - often zero - xruns with WiFi disabled to many thousands of xruns and numerous Jack watchdog resets with WiFi enabled.
>To confirm which, I recommend using the systrace tool
I am familiar with systrace and it has been a good tool for me in the past. Unfortunately I no longer consider it useful at this point in my development. I am tracking down faults unrelated to the activity of my application; faults that occur many minutes, even hours, into my test runs at seemingly random moments. My best solution has been to invoke systrace repeatedly from a loop in a bash script ensure it is running at the moment of failure. Unfortunately systrace itself generates a flood of xruns and watchdog resets.
These points are discussed in greater detail, along with test results, in my blog at http://proaudioandroid.blogspot.com/ .
At this point I have statistical evidence that the WiFi stack is the cause of my current xrun problem. Verbose WiFi debug log offers no clues. As systrace is problematic, I am continuing the investigation by making speculative code changes. Additional suggestions are welcome.
With my audio buffer at 2.7 ms I think I am the canary in the coal mine.
Thanks for your assistance on this an so many other issues,
Brad Justice
Glenn Kasten
Nov 6, 2015, 5:39:35 PM11/6/15
to android-ndk
Brad,
Thanks for your updates on your testing, and the limitations you have found in the systrace tool.
I appreciate your suggestions and will share them with the systrace team.
As mentioned in the "chicken video" (link at September 17 reply),
I'm interested in finding and fixing such performance problems as early
as possible, and your feedback is quite helpful towards that.
Glenn
Reply all
Reply to author
Forward
0 new messages