TL; DR: You will need some intermediary DSP between quisk and the RS-HFIQ
So the problem with WSJT-X talking to the RS-HFIQ over Hamlib is it requests a mode setting where the RS-HFIQ doesn't have a mode, ignoring the philosophical argument that I/Q is a mode:
`Hamlib error: Feature not available while getting current mode`
... unfortunately, WSJT-X doesn't handle this issue gracefully, and I wish it would. Maybe it's maintainer(s) could fix that. One may also propose to the Hamlib devs to include IQ as a 'mode.' Then someone (might end up being me) would modify Hamlib's RS-HFIQ source, as well as other IQ-related radios like FUNCube Dongle to support the mode-setting request to return `IQ`. There's still a lingering issue of whether or not WSJT-X would handle the new IQ mode reply in a constructive way so it seems to all come back to WSJT-X.
Apparently WSJT-X doesn't really support IQ audio anyway, though it can sort of fake it:
... this means there will still need to be an intermediary means of converting the baseband audio to I/Q and back- such as GNU Radio Companion or Quisk. It's a bit of software to run, like having multiple boxes on a desk forming a signal chain.
I found that Grig will control the RS-HFIQ. In that case WSJT-X would run with rig set to 'none' while tuning and keyup would be handled with grig. Grig is strictly a control application- it does no signal processing.
The raw-IQ nature of the RS-HFIQ makes it very powerful at the cost of moving a lot of control and responsibility to the radio operator instead of inside the equipment. I learned to like Quisk. I had to write a bit of C and Python code, as well as file bugs and pull requests with a few project maintainers - months of effort- but it ended up being very rewarding: I might be able to use the RS-HFIQ and the HARDROCK-500 as my 'forever radio' w/ HF. On the other hand, many don't have the time for all of the technical snags that inevitably pop up -and there is no shortage of them - and would be happier with something more monolithic.