That's pretty much how it goes on ARM SBCs, the vendor has to apply a varying amount of patches to the Linux kernel for it to have decent functionality because the SOCs are generally designed to be run on Android with all sorts of proprietaryness. I went on an ARM board kick and bought up all sorts of different SBCs, though I've never been a huge fan of the rpi but the rpi4 is starting to seem like a decent board. I have a Rockpro64, and a Rock96, they both use the RK3399 chip. The PCIE slot on the RockPro64 is a neat gimmick but it's a silly form factor to have a PCIE cart plugged into a board that is half it's size. There are plenty of SBCs that have an M.2 slot for an NVME drive and if it's implemented right it should be just as useable as a straight PCIE slot with a M.2 to PCIE adaptor/extender.
I don't much care about the PCIE slot, its sure faster for getting data from the host PC to the FPGA card but if you're using a Mesa card, you can't beat the hardware generated step/encoder interfaces no matter what you do with a modern PC. New processors are fast but latencies just keep getting worse, ARM doesn't seem to be hugely better than x86 in this regard. So the Ethernet cards have been great for me but Mesa seems to be increasingly supporting SPI devices, so that might be pretty interesting since almost every SBC has SPI on the GPIO and therefore the FPGA wouldn't be tying up the ethernet port. HM2 has some sort of support for SPI, I'm not sure if it's part of smart serial or what, maybe Michael can answer that. I think SPI is even in some way compatible with RS422/485 so there might be alot of possibilities there.
By far, the best ARM SBC I've used is the Odroid N2, and that's the board I'd like to see make some traction. Think I'll make a new post for it, see if any of the MK kernel gurus want to take a stab at it.