Does your Zoom app have session admin priviledges?
With the PC and fiber capacity you have, I can't see how you'd have problems on those two fronts.
Before I'd try hard-wire patching I'd double check the Zoom muting.
Then I'd check the sound on each participant in full voice. All it takes is ONE member with sub-par upload bandwidth and the whole mix can get quite ugly.
I hate to suggest the next item, because it's kinda like "did you plug it in?" But I HAVE spent debug time looking in arcane places, THEN asked the obvious question. So, in this case, have you double checked the routing of the "patch panel?" I learned working with Ardour to record some tests that it was TOO easy to overlook a troublesome link in the Linux Jack "Connect" screen. I eventually removed ANY connection that was not ESSENTIAL to capturing, then playing back, with Ardour. Capture and Playback required different connections. Any time I forgot that I got nasty results in the jacktrip mix.
THEN I might check out the hard-wire. THAT has challenges too. In particular, just getting around line-level/mic-level/headphone-level volume AND impedence mismatches can be quite a headache. I now have a little box of assorted resistors for soldering different pads to 1/8" TRS jacks for jack_delay and/or amp output/audio input adapters.
I think the non-mac equivalents of Blackhole will become more available without installation headaches. I intend to go that route at some point.