Hi Remco,
I am trying to help Yang do this analysis via CIPRES, and I appreciate your suggestion. However, there is no message regarding running out of memory under Help/Message, and increasing the Java heap size from 8g to 12g does not make any difference. I think that there may be a 64 MB limit on the size of the alignment file, since an alignment with 4,200 taxa that is 62 MB in size loads, while the preferred alignment with 4,623 taxa that is 68 MB in size fails.
As regards the practicality of analyzing such a large data set, that depends upon having access to fast GPUs. I found that analysis of the data set with 4,200 taxa for 10M generations should be possible in 24 hours using four NVIDIA V100 GPUs but would take weeks to analyze on 24 Intel Haswell cores. Of course, it might be that more generations will be needed to get acceptable ESS values.
In any case, do let Yang and me know whether you are able to modify BEAUti 2.x.x to accommodate larger alignments.
Thanks, Wayne