.print ac format=raw {(VR(out)*VR(in)+VI(out)*VI(in))/VM(in)**2} {(VI(out)*VR(in)-VR(out)*VI(in))/VM(in)**2}
.END
This won't even give you exactly what you want, as the real and imaginary parts of K will be in separate variables in the rawfile. But it does get the values in there.
It is unfortunate that it is this difficult. The Xyce expression library was developed long before Xyce had AC analysis or Harmonic Balance, and extending the expression library in its current design is very difficult. We have a path forward for replacing the expression package with something more maintainable and extensible, but it will not be in the upcoming 6.3 release. Until then, the expression package can only produce real numbers, and all variables in expressions must be real.
It is possible to define user functions for complex division to simplify the .print line somewhat, but not much:
*test of complex division output with user functions
* real and imaginary part of K=vo/vi
.func kr(vir,vii, vor,voi) {(vor*vir+voi*vii)/(vir*vir+vii*vii)}
.func ki(vir,vii, vor,voi) {(voi*vir-vor*vii)/(vir*vir+vii*vii)}