Puzzler: why does audio traffic that's not on 4464 <--> 61002 vary with audio volume?

[Mike O'Connor]

hi all,

here's a little series of puzzlers, followed by an explanation of how i came to them.

why...

- only 2/3 of the audio traffic between two Jacktrip-connected computers flows on the 4464 - 6100N pair of ports

- the remaining 500k of bandwidth is spread over a large number of other port-pairs

- those additional port pairs appear to be random when audio volume is high, but start to use known IP ports like 993 and 995 when the gain is reduced to zero

- the traffic-per-port-pair changes with audio volume

background/discussion

i was trying to diagnose a crackly connection a few days ago and decided to take a look at traffic by IP port just to see if a script-kiddie was interfering with our stream.  that turned out to be a dead end, but i noticed this really interesting behavior that appears to be normal.  i'm wondering whether Jacktrip or Jack is doing this, and whether it's normal/intended.

here's a little 30-second silent movie to show what i mean.  

https://haven2.com/video/non4464traffic.mp4

there are two computers.  i'm running the IFTOP utility on both.  the one on the left is the hub server, the one on the right is a hub client.  there's a ton of traffic on port 5900 that you can ignore -- they're screen sharing.  these are local Macs, but i've also run this test to a remote Linode with the same behavior.  the three columns of the display are average traffic over 2, 10 and 40 second intervals

towards the top of the display you'll see the port 4464-61002 pipes.  that port-pair stays really steady at 1 mbps.  

at the beginning of the video, the volume on the audio-sending computer (the one on the right) is full on, with a fairly compressed signal that's peaking about -6 db.

as your eye goes down, you see a zillion other port pairs that look ephemeral.  and they also look like random, non-assigned ports.  i haven't explored how many of them there are, but since the overall bandwidth is 1.5 mbps, and they each have tiny traffic there must be a LOT of them.  with a LOT of churn.

when i zero the volume on the audio, a couple interesting things happen.  

- traffic starts to load up on a the non 4464 pairs

- the pairs are no longer anchored way up in the random range -- they're anchored in known ports like 256, 993 and 995

- some of the pairs aren't anchored in a port at all

can anybody point me in the right direction?  i can't frame a good search query for this one.

thanks,

mike o'connor

[Scott Miller]

This would definitely begin to get at behavior I’ve noticed wrt sustained tones and especially low frequencies, which consume higher bandwidth relative to perceived loudness.

I wonder, Mike, have you tried filtering the audio streams at constant gain? I would imagine a LPF would have significant impact.

Scott

[Scott Miller]

Oops, meant High Pass Filter (HPF). 

[IOhannes m zmölnig]

On 1/15/22 15:18, Mike O'Connor wrote:
> can anybody point me in the right direction? i can't frame a good search query for this one.

no idea, but 'iftop' is most certainly the wrong tool to analyze this.
your probably capture the traffic with something like wireshark, and
then analyse the weirdo packages, to see what their payload is (e.g.
whether this is jacktrip traffic)

a quick glance on your video (although i'm really really bad at parsing
such videos (including youtube tutorials that teach the use of
keystrokes)) shows that (at least some of ) your weirdo connections use
the same port for incoming and outgoing traffic.

my first guess is that this hints that the weirdo traffic is indeed
coming from jacktrip - in which case i think that you might have hit
some bug :-)


VDCx
IOhannes

[Mike O'Connor]

wow!  wireshark has come quite a way since the last time i tried it.  i've never really understood how to use it, but there are more built-in clues these days.

here's the approach i've sketched out.  

- recreate that test scenario

- capture separate 1-second logs of UDP packets for:

- no jacktrip connection

- jacktrip running high-volume

- jacktrip running low-volume

- dump those into Excel to do analysis

is that how you'd approach it?

--
You received this message because you are subscribed to the Google Groups "jacktrip-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jacktrip-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jacktrip-users/279a13b1-fb26-3b7f-6748-cea7b44a68be%40iem.at.

[Dave Adams]

I don't think Excel will be very useful for what you describe.

Wireshark has been (even in its "Ethereal" days) very good at capturing network traffic and truly outstanding at protocol analysis.  It will, for example, show where a "conversation" is in a stream of traffic.

I used to have quite a few cheat-sheets.  I trimmed those way back, since I usually ended up do web searches for the features I needed.

In this case, feature #1 would be capture filters.
    1) filter for just the addresses of the jacktrip conversation ends
    2) filter OUT things NOT in regular jacktrip, usually by port
          so if things like email and bonjour are running on your macs you  don't get that traffic

Feature #2 would be processing filters.
    - try different ports to see how often they appear, and when
    - filter to highlight the  conversation stream to see who starts the stream with the unexpected port numbers

Once you really see the packet pattern(s) it might be worth exporting that view to excel, to match merge against your log of what you did.

I think that, once you get capture and post processing filters figured out, you'll find wireshark very useful for what you are researching!

[Mike O'Connor]

hi Dave,

thanks!  that's a big clue right there.  the trick to this puzzler is that it appears that there are LOTs of port-pairs in use at any given time.  the number and distribution of these is what i'm trying to understand.  so i can filter on the known JT ports (4464 and 6100n) but the ephemeral ports on the other ends of those weird connections are all over the place.  that's the behavior i was trying to highlight in that 30-second video at the top of this thread.  here's the link again

https://haven2.com/video/non4464traffic.mp4

the tool i'm using in that video (IFTOP) is great because it displays the ports along with the connections, but it is only showing a few (ranked by traffic).  i want to see them all -- so my thought is to dump those logs into Excel (or mySQL if there are too many for Excel) to get a sense of how many there are and their distribution in those three scenarios.  can i do something like that directly in wireshark?  a good "how to build wireshark filters" tutorial may be my next step, eh?

To view this discussion on the web visit https://groups.google.com/d/msgid/jacktrip-users/3befa988-1e3f-48cc-8b60-f3bb0cfbe082n%40googlegroups.com.

[Mike O'Connor]

oh!  i think i've found a toehold in Wireshark.  i think Statistics/Conversations is going to push me miles forward.  i've got to get on with my day, but that looks very promising.

i also found a bunch of great "how to write filters" tutorials -- thanks for the nudge.

mike

To view this discussion on the web visit https://groups.google.com/d/msgid/jacktrip-users/DDBD3576-F62C-4389-A285-A941959E682B%40gmail.com.

[Dave Adams]

when you get time to get back to it, will your filtering leave conversations intact?

One of the reasons "stateful firewalls" can provide extra protection is their ability to pass traffic based on prior related traffic.

This is particularly important for things like voice over IP, where the first exchange is the "tip of the iceberg."  A simple, easy to identify starting exchange, sometimes even from a TCP port, leads to scads of harder to identify UDP exchanges.  And the key to it is the sequence in which the conversations originate.

That's one of the things Wireshark will really help on. 

BTW: another big advantage of Wireshark is that, once you have your sequence clearly and completely represented, you can send as filtered file to another Wireshark user.  That, in turn, really simplifies getting another set of eyes on the question at hand.

[Aaron Wyatt]

Is there any additional information on this? This sort of behaviour seems so far outside of the scope of how the networking code operates that I'd be incredibly surprised if the traffic was coming from JackTrip. There are two objects that handle the networking that run on separate threads. They're only created once per JackTrip related JACK client, and they share a common socket so that we can use the same port to send and receive. The code to bind the socket is only run once, and throws an error if the specified port isn't available. So I don't know how there could be any binding of multiple random ports. (And we're not using any sort of high level library for networking - we're making the bind call directly, so it's not hidden somewhere under layers of abstraction.) The sending thread chugs away calling the same blocking function in the ring buffer - it gets audio packets at regular intervals as soon as there's data there. And it's the same amount of data whether there's silence or cacophony.

You can see the expected results below. The only capture filter I'm running is to block non-UDP packets, just to filter out any other noise. (I'm running it on a laptop that doesn't have much else running, so there's not much other UDP to contend with.) I'm not limiting by port at all. But what you see is a stream of equally sized packets flowing between client and server on the expected ports. The packets from the client are roughly twice as big because it's running at double the frames/period that the server is (which became possible thanks to the Anton Runov's work on the jitter buffer). Looking at the server, we're running at 128 frames/period. Each audio sample is 16bits, so 2 bytes. This gives as 256 bytes per period per channel for a total of 512 bytes per audio packet in a stereo send. The JackTrip header is 16 bytes, giving us the number we see here of 528 bytes as the payload of each UDP packet.

Not doubting what you're seeing at all, but if it's coming from JackTrip then it's operating so far outside of what the code should allow for that we have major problems.

[Mike O'Connor]

hi Aaron,

i haven't circled back to this, sorry.  this is really outside my experience/understanding so it's been hard for me to get a toehold.  

if you'd like to recreate what i did, head back to the original post.  i was using IFTOP between two computers that weren't doing much except Jacktrip.  what fascinated me was the way that the pattern of the traffic changed as i changed the gain on a computer -- both the apparent distribution of the ports and the size of the packets.  the whole thing was really weird, well beyond my expertise.  that's what the little MP4 movie shows.

i think your best approach would be to see if you could recreate that scenario and the work from there.  

thanks a lot for looking into this.

mike

To view this discussion on the web visit https://groups.google.com/d/msgid/jacktrip-users/12a85dee-f6a5-45ba-96d1-f4f2e97a271dn%40googlegroups.com.
<wireshark.png>

[Aaron Wyatt]

The problem is that I can't recreate it, which is what that wireshark capture shows. (And that was with a mix of both pure silence and music being routed over the link.) Obviously that's only a fraction of the capture, but the rest is like that as well, with traffic being dominated by JackTrip between those two ports. If you wanted to do a wireshark capture and send it through, I could take a look at it (although there are potential privacy concerns with that), but iftop doesn't really show much of what's going on. I also don't see anything like it when I run iftop myself. What command line arguments are you using with it? When i run it with 'iftop -P -f "udp" -i <interface>' (the filter argument means I'm only looking at udp traffic) all I see is traffic across those two ports with a minimal amount of other traffic thrown in (things like DNS and mDNS queries and ntp traffic).

[Mike O'Connor]

hi Aaron,

ah!  that's good!  i'd much rather be wrong than identify a Jacktrip issue.

i came up with a hypothesis overnight that i can test (soon...  because... deadlines).  i'm thinking that what i'm seeing might be AVB data between audio interface on the testing machine and a rendering machine on the same LAN segment.  it'll be easy to check by rerunning the IFTOP setup and disabling/disconnecting the AVB connection.  

your test supports that hypothesis fersure.  i'll try really hard to find a gap in the action and run that disconnect-AVB test as soon as i can.

thanks!

mike

To view this discussion on the web visit https://groups.google.com/d/msgid/jacktrip-users/5e64972b-0cf6-4798-aa88-6d1ec495d126n%40googlegroups.com.

[Aaron Wyatt]

Awesome. No rush, and it's greatly appreciated.

A

[Mike O'Connor]

wow.  epic day.  two fiber cuts at my house in the same day.  beware siding-installers bearing sharp tools.  anyway, a little unexpected time due to meetings being...  unreachable.

i killed the AVB network dead.  no joy.  so i thought i would do a narrated version of that IFTOP setup just to give you a feel for what's going on.  here's a link to a little 5 minute video where i walk through the setup and run a couple sessions with Jacktrip 1.5.1 on each end.  it ends with a plea for help configuring a Wireshark session.  i'm happy to run Wireshark while this is going on, but it's beyond me how to set it up.

here's the most exciting media you're gonna see tonight!  it's called....

Weird Packets!

https://haven2.com/video/WeirdPackets.mp4

To view this discussion on the web visit https://groups.google.com/d/msgid/jacktrip-users/6ccb92d9-99e2-4e44-9a66-12f6077b7b58n%40googlegroups.com.

[Aaron Wyatt]

Very weird. I tried to reproduce again and just can't. Was testing on linux before but moved to mac and same thing. Everything flows through the expected ports. For wireshark, you can just run a capture on the interface without any filters. So just double click on the appropriate interface on the screen that shows at launch. Just don't do anything that might send unencrypted private or sensitive information across the network while you're doing the capture. (You'll just need to run it for 5-10 seconds, and you can then save the capture by going to file > save.) In the meantime, you could also try running "lsof -i | grep jacktrip" on the command line while seeing this traffic and see what it outputs. It will list the ports that are being used by the app.

A