OK, so this is at least mostly: "It's not just any X server, you have to be running (apparently) X on 64-bit Linux."
Like, it generally works on an amd64 laptop running Debian Bookworm. It fails on an i386 laptop running ... some earlier Debian or Ubuntu, not with the swrast stuff but (with ssh -X or -Y something about screwed up auth, if I just xhost+ I just get blank window frames). MacOS/XQuartz fails as described above.
I'm about to try compiling tvcon on x64 and on Mac and seeing if anything changes if I connect directly to the terminal port rather than sshing into the pidp10 with DISPLAY set correctly.
Adam