Playing in time

220 views
Skip to first unread message

Kevin Doren

unread,
Dec 31, 2020, 8:24:00 PM12/31/20
to SonoBus Users
One of the major challenges of playing together online is timing. Actually, it's THE major challenge.  I'm trying to understand how musicians can play in time in a peer-to-peer system like SonoBus, or does it only work when the delay is really small?  How well does it work in practice, and what are the limitations?

Server-based systems such as Jamulus deal with this by making the server the time reference.  All audio goes to the server, which mixes it and sends the mix back to each player.  Everyone hears the same thing, although some hear it later than others.  It's each musician's responsibility to listen to the (delayed) mix coming back from the server, and to play with that.  This works because your brain can adapt, up to a point (that point seems to be around 50 ms).

Peer-to-peer solutions such as SonoBus eliminate the round trip to the server, which lowers latency while scaling up the bandwidth requirements as the number of musicians increases.

But how do you play in time without a common point of reference?  If the delays are really small it might not matter much.  But as delay increases, I am listening to me in the present and you in the past, while you listen to you in the present and me in the past.

So how can we play in time?  Maybe I am missing something here, but it doesn't look to me like SonoBus currently has a way to introduce a common time reference.  Without that, I think it would only work well if the delay is really low.

Couple of thoughts.

1) SonoBus has a metronome function that you can send to others.  However this appears to just send the audio, so others hear it with delay, which guarantees they will be out of time.  I think it might be more useful if one player's metronome could command the other players' metronomes to click at the same (real) time.  Then then players would have a common reference.  They would still hear the other players' audio with delay, so I'm not sure how workable that would be in practice, but at least each musician would have a common time reference.

2) Another possibility would be to have each player's SonoBus client time-synchronize the audio streams, including the local user, when it mixes them.  To do this it would need to intentionally add different delay amounts to each stream, including the user's own, in order to  produce a time-synchronized mix that people could play to.  Like playing on a Jamulus server, it would mean each user would need to play to a delayed mix, but the upside is that everyone would hear the same thing.  The advantage would be that, by eliminating the server hops, and buffer/decompress/mix/compress cycle at the server, SonoBus would have lower latency and be workable at greater distances.

-Kevin

Carlos Aguayo

unread,
Dec 31, 2020, 10:05:58 PM12/31/20
to Kevin Doren, SonoBus Users
Option #1 can be attained by syncing clocks to a central NTP time source, and then starting an identically-configured metronome on each  client at a prearranged time.  I'd started checking this out, but never really figured out how to make it inject the signal; jack should make that somewhat easier.  I don't really think it would take much coding, but the "at" command is useless on the mac; it is much more feasible to use "at" on linux.  There's already a metronome for jack injection, and if you google this topic, you'll even find some academic proof-of-concept projects out there.

I hope someone makes this happen soon, I could even imagine a Jamulus client benefitting from such a sync'ed local signal.

Carlos


--
You received this message because you are subscribed to the Google Groups "SonoBus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonobus-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonobus-users/716d02a9-09e8-4d5a-a69f-1f4ad9e34e9cn%40googlegroups.com.

Ben Bogart

unread,
Jan 1, 2021, 12:21:36 PM1/1/21
to SonoBus Users
Or PTP might be even better than NTP.

Right now we loop the mixed audio back to each player with jacktrip (we haven't been able to use sonobus for several reasons yet but hoping to when it has full multitrack output not through plugins :) ). With our terrible internet the latency in this system is starting to wear on the musicians.  I can sat that it is totally possible to play reasonably well hearing the final output late and it is a better experience that trying to play not knowing when you are sounding with the other musicians.

I've been thinking about something like a clock (SMPTE or other) synchronized with PTP that buffers the local stream the length of the longest latency incoming stream plus a jitter buffer.  Then all the streams would be slightly delayed but synchronized and there would be no need for roundtrip for each musician to hear the same audio. The delay in many cases might be tolerable.

This is somewhat like your option 2, but without the metronome which I feel adds unnecessary musical restrictions and reduces musicality in performance.

Tom Holden

unread,
Jan 1, 2021, 4:29:26 PM1/1/21
to Kevin Doren, SonoBus Users
On Thu, Dec 31, 2020 at 8:24 PM Kevin Doren <ke...@doren.org> wrote:

Server-based systems such as Jamulus deal with this by making the server the time reference.  All audio goes to the server, which mixes it and sends the mix back to each player.  Everyone hears the same thing, although some hear it later than others.  It's each musician's responsibility to listen to the (delayed) mix coming back from the server, and to play with that.  This works because your brain can adapt, up to a point (that point seems to be around 50 ms).

I'm not sure that you can say that the Jamulus server is any better as a time reference. Yes, everyone hears the same scattered sync whereas on Sonobus, each is going to hear a differently scattered sync. But, in both cases, do you not have to appoint a designated leader with whom everyone tries to sync? Maybe the advantage of Jamulus is that it is your own audio return from the server that you attempt to sync with the leader, which assures that you are in sync at the server and on the recordings it makes and apparently so to all other clients who attempt to do the same thing.Without a return audio from some common point, it does seem that a Sonobus group is more vulnerable to falling apart for a given latency. 

Peer-to-peer solutions such as SonoBus eliminate the round trip to the server, which lowers latency while scaling up the bandwidth requirements as the number of musicians increases.
Sonobus cannot assure lower latency than Jamulus if the route between peers is essentially as long as the route between clients. That seems to be the case in my area where even local peer connections seem to be routed through Internet backbone routers in the metropolitan area 100km away, in which there are also Jamulus servers on various Virtual Machine farms. 

But how do you play in time without a common point of reference?  If the delays are really small it might not matter much.  But as delay increases, I am listening to me in the present and you in the past, while you listen to you in the present and me in the past.
I forget the name but there is a service that plays a reference track locally at each client against which the musician plays. The server collects all the recordings (concurrently, I think) and automatically assembles them into a mix for playback to all the clients very shortly after the end of the reference. Musicians are performing live but not with each other. However, one could build up a mix sequentially so that increasingly one is performing with a mix of the other clients.

Tom

Ben Bogart

unread,
Jan 1, 2021, 4:35:03 PM1/1/21
to SonoBus Users
> do you not have to appoint a designated leader with whom everyone tries to sync?
No!  We don't.  We play as an ensemble with everyone listening to everyone.  We don't have bass or drums. We adjust to eachother as needed.  We also are working with significant (about 80ms) delay right now and its a way better situation than trying to play separately and hoping it syncs up later.

howdood

unread,
Jan 1, 2021, 6:55:34 PM1/1/21
to Ben Bogart, SonoBus Users
Hi all, first time post here, but I've been on the Jacktrip users for a while. I guess I've got a positive and a negative here. Bad stuff out of the way first: if everyone is playing along to a reference track, there's no point trying to do it 'live', as to stay in time you'll have to not listen to the other 'live' players anyway - better to record separately until you get a take you like then layer it. Loads of community choirs in London have been doing that recently, and the end result has been awesome.

But plus point: don't take it for granted that unworkable latency is a fact of life. In most circumstances I've seen discussed the main factor that pushes that over the threshold is endpoint soundcards not being set up properly (ie high period size leading to high buffer length between soundcard and PC itself). If you're all in the same city, you can get end-to-end pure network time of sub-20ms for delivery of UDP packets, plus 5ms jitter buffer at each end: so as long as you have soundcard latency of 2.6ms-ish each end you're able to play as well as if you were stood 40ft away from each other in a field. My first advice for anyone struggling with drag in online playing would be to calculate their soundcard latency and reduce it if possible. Especially with jacktrip, I've seen plenty of 'how to' guides that leave the end user soundcards contributing more latency than the network itself - up to 10ms at each end if not tweaked well. A second tip would be to set things up so you hear your own part only with the network latency added via loopback so you can compensate for it - that's second nature for players of instruments like tubas and tambourines where the sound comes much later than the muscle movement anyway.

As proof of concept - here's a 100% live no reference track online gig we did a while back (using jacktrip via an AWS server hub) where we were running 24-27ms end to end latency between participants, including soundcard latency. https://www.pscp.tv/w/1rmGPYlZBDyJN Sorry about all the mistakes, but we were feeling our way with the tech as well! I've had issues getting sonobus to pick up jackd on linux so not migrated fully yet but am pretty confident that it will work as well if not better.
Hope that's useful!
h

--
You received this message because you are subscribed to the Google Groups "SonoBus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonobus-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonobus-users/1e0e6f98-1882-4b9c-bf9e-46f34b6474f9n%40googlegroups.com.

howdood

unread,
Jan 1, 2021, 6:58:39 PM1/1/21
to SonoBus Users
ps - lag on the video is/was a loola-to-periscope encoding issue. The real thing went out synced perfectly! Don't get me started on big name companies enforcing their own in-house resampling...

Kenneth Fields

unread,
Jan 2, 2021, 9:36:55 PM1/2/21
to howdood, Ben Bogart, SonoBus Users
hi,
there is such a reluctance to move on to the next phase music practice.
forget syncing to a military 4/4 march rhythm on the downbeat - which you can only do while in the same city.
with net music you will sync on a pulse based on the delay between cities.
both cities will be playing in a unique time/space frame particularly their own.
They don’t hear the same thing. two (or more) versions for the price of one jam.
the blending of time/space frames (poly-chronotopic) is the name of the game for net music.
when other cities join, you will need to free yourself into moving in a polyrhythmic/polytemporal universe.
It is still a shared sonic space - decentralized, no authoritative privileged POV.

The way Artsmesh works, is with multiple metronomes. we calculate the ping time of each peer
divided into 60k ms, which gives you the beat per second. if it is too fast, we pin the pulse to the
1/8, 1/16th or 1/32nd note. We haven’t automated that to the point of a WAN ableton link thing yet though (re recent email topic).
I think average jitter is stable enough on decent connections to make this a feasible practice.

Ken









Jesse Chappell

unread,
Jan 2, 2021, 11:30:09 PM1/2/21
to Kenneth Fields, Ben Bogart, SonoBus Users, howdood
Providing the round trip latencies displayed in terms of 8ths or 16ths at a certain tempo is on the TODO list. Then automating the jitter buffer times to get them all lined up to divisible values is a pretty easy next step. Could be a very cool thing to experience :)

Jesse 

Paul Simon Limbrick

unread,
Jan 3, 2021, 5:35:56 AM1/3/21
to Jesse Chappell, Kenneth Fields, Ben Bogart, SonoBus Users, howdood
Hi
This would be interesting and feeds into my experiences of the creative processes coming out of net-based activities. One of the things I do with online ensembles is to first establish an understanding of the shared latency, then use that in the work. It varies from day to day, often as a result of the network activity beyond our control, and follows the capacity affected by the time of day, social networking, peak times for gaming/streaming, and business activity. It is interesting to have to connect to and integrate into this milieu. Finding ways to consolidate and refine a creative connectivity is a way forward. Instead of a separate and often frustrated space for musical creativity, the activity is in a space where is actions and processes can be observed and participated in. Helping us understand and be able to access common points of consistency and reference can only lead to expressing new and exciting new things.
Shared latency, conversion to musical language; yes please

Best Simon

Sent from my iPad

On 3 Jan 2021, at 04:30, Jesse Chappell <je...@sonosaurus.com> wrote:



Mike O'Connor

unread,
Jan 3, 2021, 7:21:16 AM1/3/21
to Paul Simon Limbrick, SonoBus Users, Jesse Chappell, Kenneth Fields, Ben Bogart, howdood
i'm thinking this concept needs a name.  so we can put it in the names of public sessions that want to work on it.  so we can read and write and search for it.  etc.

does is already have a name, and i just don't know it?  

Reply all
Reply to author
Forward
0 new messages