I've had absolutely dismal luck with the M1 Mac Mini and multiple displays, and I've tried everything. Have one sitting here that is basically a boat anchor at the moment. Just when you think it's working you reboot and it forgets displays, changes resolutions, ugh.
Now, what DOES work really well is a base model M1 Macbook air and a 4k 2x2 video wall controller. That will give you four 1080P outputs, you just map four surfaces in QLab to the corners, and it works flawlessly. I had one running a submarine attraction at our local aquarium for about 2 months, 7 days a week, 10 hours a day. Reboots were no problem, it remembered the output.