Add IEM (In Ear Monitoring) capability for traditional live performance

[Mats Andersson]

Background
At the time I write this, vaccines for the current pandemic are starting to roll out, i.e. we can start to think about life going back to a more normal situation. Both Sonobus and Jamulus are fantastic tools for online rehearsals, but the end result of rehearsals is in many cases a live performance. IEM systems are great for ensuring a tight and synchronized live performance, but traditional solutions are relatively costly and not so flexible

• 800 USD/mono channel is typical for the IEM system itself
• To provide individual mixes for each member, then quite expensive soundboards/soundcards are also needed (you normally want yourself to be a bit louder than "front of house" mix, and you normally also want to add a click track),

Very recent developments
Recently, a very cost effective approach has arrived:

• A computer with a multi-channel sound-card does the A/D conversion (i.e. acts as a traditional transmitter)
• A good quality Wi-Fi router is connected to the computer over Gigabit LAN
• Normal mobile phones are used as portable D/A converters (i.e. acts as a traditional receiver, with the added bonus of working in stereo)

Two commercial alternatives are now on the market providing this solution, both charging ~100 USD/connection:

• Audiofusion (www.audiofusionsystems.com). Latency measurements places it in the 22~24 ms region, https://www.youtube.com/watch?v=iQ2doYpHmEg&t=643s
• Stagewave (www.stagewave.io). They officially state down to 17ms latency, and I did indeed arrive at this figure in my own simple testing (using Oblique Audio RTL Utility).
• Note that Myxyr seems to have been first with this idea, but from what I can tell that project/product looks dead? https://www.myxyr.com/en/

My initial proposal

• Sonobus adds multiple input channel support. Up to 8 mono channels probably covers more than 99% of the potential end users?
• Sonobus adds some sort of IP multicast support. In an IEM scenario (only 1 sender, many receivers), then all clients can receive the same multichannel stream. This is beneficial for both bandwidth/latency, and synchronicity.

On the IEM client (think cellphone), then the following would be needed

• Receive many multichannel mono signals, and mix them down to a stereo signal
• Simplified GUI, just click an "IEM client" button and it will automatically enter a mode where: Buffer size is set to "Auto", microphone is turned off (with no ability to accidentally turn it on), and an overall layout where focus is on mixing/panning the channels

I have already done some successful basic testing. Below 10ms latency seems perfectly doable with a good Wi-Fi router in the 5G band (so far only tested with 802.11 AC, I have just bought an AX router).

What do you think?

[Mats Andersson]

I did some more detailed testing now, see below screenshot

• Using a direct cable between input and output on my Presonus Firestudio Mobile operating at 48kHz and 64 samples buffer, I consistently measure 274 samples or ~5.7ms round-trip delay
• Adding a complete Sonobus system, I measure around 1000 samples or ~20.9 ms, meaning that the Sonobus system has a total latency of 20.9-5.7~15.2ms, i.e. it is already better than the 2 previously listed commercial IEM alternatives

In my testing I was using

• Windows 10 computer with Sonobus 1.30, operating at 16-bit PCM
• An iPhone 7 with Sonobus 1.0.3 (TestFlight version number), operating at 64 buffer size. It settled on 7ms buffer
• A six antenna TP-Link WDR-7500 (I know, not the best but certainly not the worst) WiFi router. It supports and operates at 802.11 AC speeds, which my iPhone 7 also does
• A fairly standard XMOS USB sound card with Thesycon drivers. Standalone round trip latency at 48k/64 buffer is ~5.6 ms, i.e. roughly 2.8ms delay comes from the A/D conversion in this sound card alone

I also had a further look at the high level requirements of multichannel operation

• Adding multiple PCM channels is of course possible. There are some standards, but basically they are in most applications added as stereo pairs
• The Opus codec supports up to 255 channels in a single stream, i.e. it is very well suited already from this point of view. In addition, using Opus instead of PCM keeps bandwidth down which is beneficial from latency point of view, especially considering when many users are connected at the same time

I have not looked further into potential ways to add multicast.

[Jesse Chappell]

Interesting ideas! I was planning on adding multiple input support in
a future version (it was going to be next, but I got carried away
doing the public group feature instead).

Jesse

> --
> You received this message because you are subscribed to the Google Groups "SonoBus Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sonobus-user...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sonobus-users/5802033d-8e88-4544-a221-51572610dcdfn%40googlegroups.com.

[Mats Andersson]

Great! As you can probably imagine by now, Sonobus can relatively easy become the global standard IEM system with very widespread use (in addition to what it already does so well) :-)

How do we proceed here? My proposal is that I create GitHub issues like "IEM - Short feature request description", that can be worked on and ticked off individually?

• I wish that I could help with the actual coding, but I am more of an advanced user/system builder than a coder

[Daniel Zimmermann]

Hi Mats,

thanks for your ideas and measurements!

IEM is a cool additional use case that might not divert too much from the main goal of sonobus.

I just wanted to make sure every reader understands the difference between latency and bandwidth, because you made a connection between them in your last sentence. 

I compare latency with the length of the road, and bandwidth with the number of lanes - given that all packets drive with ~speed of light, there is no way more lanes will make your road shorter. 

But since even the best opus settings costs 2.5ms of latency, you gain a significant amount of time by not compressing.

If you have 10+ channel, your upload bandwidth gets clogged since Internet providers tend to still provide last-Millennium-broadband. But if you need to opus decode 10+ channels in ultra low latency with a realtime scheduler, you raspi will be insufficient.

 

The race to zero latency means that if you have all endpoints in the same LAN, you can consider Layer 2 alternatives like AVB (now TSN) that can give you guaranteed submillisecond bounded latency, or a layer 3 AoIP standard as AES67 or other proprietary ones. Those are mostly used in pro audio and broadcast and unfortunately not too approachable by the open source community.

But TSN as the only IEEE standard (the guys that define the Ethernet) is free and open: https://github.com/Avnu/OpenAvnu.

I tested soundjack.eu and https://github.com/gisogrimm/ovbox since May on raspi and PCs, with a total audio e2e latency <10ms.

Most musicians will whine about 10ms, but it is better than the latency that some mipro wireless IEM systems achieve ;-)

I like to measure air pressure wave in mic to air pressure wave out headphones of the other device(the only relevant latency), but parts of that chain are easier to compare a single aspect. Internet latency is normally the major part and never very stable.

 

My sound card measurements made with Jack-audio are in the forum https://forum.digital-stage.org/t/measure-device-latency-in-linux/263 if you're interested, and I'd be very happy if this list would grow! :-)

Jesse, you created a great application, and I like the user interface better than all the others so far. 

Happy to be part of the community!

Daniel 

[Mats Andersson]

Hi Daniel,

Your road analogy is perfectly valid assuming that there are no other vehicles on the road. However, if there are, then bottlenecks will be created which ultimately will increase the time it takes to get from A to B. Especially when we are talking consumer Wifi, this is easy to demonstrate:

• First run a looping ping test with no load on the Wifi network. Should give you 1-2ms ping time.
• Now start a file transfer that saturates the Wifi network (single large file, SSD drive on both sender and receiver). What happens to your ping time?

Thanks for your interesting links, there are indeed some positive things coming out of the current situation! In particular I had a look at some of the latency measurements. You have probably also read the information available at DAWBench and Gearslutz. I understand that you have focused on Raspberry Pi. All my testing is done on a highly latency optimized Win10 desktop PC, so for me I do not need to consider a lack of CPU processing power. With that said, multicast of multi-channel PCM16 coded sound should anyhow be the best option for IEM. The reason I still mention Opus is:

• The functionality is principally already there, since Opus specifies up to 255 channels of audio. This means that the time for the main developers to put out an initial version that can be tested by the community is most likely smaller in this case. You can think about this like agile software development cycles, for example: Sprint 1=Opus ; Sprint 2=Opus with Multicast ; Sprint 3=PCM16 ; Sprint 4=PCM16 with Multicast
• Consumer Wifi often has performance issues at high loads (low power SoC CPUs, traffic queuing algorithms that favour high throughput rather than low latency etc.). Opus compression could easily be what alleviates this, especially with many sound channels and receivers
  

Yes, there is a lot of work going on for TSN. Professionally I work with the largest electrical power transmission systems on the planet, and for instance inclusion of IEEE 802.1AS (PTP) made the control system design considerably easier. In one decade, then all current and voltage measurement data will probably go over wired Ethernet. Principally everything is possible including TSN/AVB/AES67, but:

• How much extra work for the developers of Sonobus would this add? Is that extra work re-usable for the main goals of this project?
• How many (affordable) consumer units support any of these newer standards, especially when we are talking Wifi?

Once Wifi 7 including 802.11be support is released and has widespread adoption then this might be worth re-visiting, but for now I personally see no reason to. For an IEM system, you anyhow just have one-way communication, so microsecond synchronism between the receivers is not critical.

Yes, I also read that the Mipro IEM system latency is roughly in the region that I measured! In my experience, vocalists and keyboard players are the most latency sensitive, while drum and bass (the typical rhythm section) handles static latency just fine. Click track also helps to synchronize the overall performance.

[Lars Kjebekk]

Hi! I also came across sonobus while searching for IEM solition to our local church. We will be doing some testing in the coming days also, but there could be a huge potential here. I second Mats suggestion about multiple channel input. 

Thanks for a fantastic product and happy to be a member of the community!

br

Lars

[Mats Andersson]

Some smaller updates: I have now checked the same thing again, but with two important variations
--WiFi 6 capable router (Huawei AX3 Pro) compared with WiFi 5 capable router (TP-Link Archer C7 / WDR7500)
--WiFi 6 capable phone (iPhone SE2) compared with WiFi 5 capable phone (iPhone 7)

Interestingly enough, I get very similar performance in all 4 scenarios. I.e. I can draw some important but somewhat preliminary conclusion with regards to Sonobus IEM usage:
--WiFi 6 offers no tangible benefit over WiFi 5 as long as the amount of users is relatively small (not surprisingly)
--iPhone 7 is already "fast enough" for this use case (64 audio buffer size)

Should the Sonobus phone app evolve into "something so easy that even the drummer can use it", then I will propose that my band members buy used iPhones for IEM purposes.