Efficiently parallelise

[Buz Wilson]

Andres mentioned poy plugins recently and I was digging around for more information when I hit this page: http://www.csc.fi/english/research/sciences/bioscience/programs/poy/index_html

Their opening line caught my attention:
" The current installation may be used at most with 8 processors. According to our preliminary tests, the program does not efficiently parallelize to more processors."

Is this a general issue, or an issue related to the Murska on-line system? How does the latest poy4 scale to the large parallel systems available to you? I only have an 8 processor system now, but I hope to expand it as time & funding becomes available. Knowing what the issues are ahead of time would be helpful.

I would also be interested to know which parallel processing framework you are using to run poy, as I would like to do a bench test of two network connected PCs. So far I have been using mpich-shmem, but for separate systems, another version will be needed.

Cheers

Buz Wilson
Australian Museum

[Andres Varon]

Hello Buz,

On Aug 15, 2009, at 6:18 PM, Buz Wilson wrote:

Andres mentioned poy plugins recently and I was digging around for more information when I hit this page: http://www.csc.fi/english/research/sciences/bioscience/programs/poy/index_html

Their opening line caught my attention:
" The current installation may be used at most with 8 processors. According to our preliminary tests, the program does not efficiently parallelize to more processors."

It is a false generalization. Suppose you do:

build (1)

will the program speedup with more than one processors? No.

build (8)

will the program speedup with more than 8 processors? No

build (10000)

who has 10000 processors in this mailing list? ;-) But it would scale linearly with 10000 processors. OK, OK, in _theory_ it would scale. I have only tested with up to 256 ... 

So, how efficiently and scalable is the program depends on how the script is written. That's why report (script_analysis:"script_file") exists.

Is this a general issue, or an issue related to the Murska on-line system? How does the latest poy4 scale to the large parallel systems available to you? I only have an 8 processor system now, but I hope to expand it as time & funding becomes available. Knowing what the issues are ahead of time would be helpful. 

I would also be interested to know which parallel processing framework you are using to run poy, as I would like to do a bench test of two network connected PCs. So far I have been using mpich-shmem, but for separate systems, another version will be needed.

Again, it all depends on the script. I will use simple examples. Suppose you run the following script in parallel:

build (1000)

swap ()

select ()

The bottleneck is not in the communications layer. Very little messages will be sent between processors to run that script. But if you run the following script in parallel:

build ()

swap (visited:"file")

select ()

then A LOT of info will be moved around between processors, because only one of them is allowed to output to a file (that is a constraint of MPI). Therefore, every process has to send the trees to be printed to one process, who takes care of putting them in a file. That is a lot of data flying around. And then the communications layer will be a bottleneck.

Overall, I don't think that the communications are a bottleneck with modern Gigabit and above networks. And scripts can be written to avoid introducing  bottlenecks that would hurt performance. If you are concerned, check what report (script_analysis) says about the script that you want to run. 

best,

Andres

[Buz Wilson]

Thanks much, Andres. We see that build() is parallelised, and I assume that search() is, too. If this is the case, then poy systems should scale well. I wonder why they had that comment on their website; probably more to do with their web interface, than the underlying software.

Now the second query (I suppose I should restrict myself one query per message) was what is the recommended MPI implementation for multiple PCs over the network (I'll upgrade my personal router to a Gigabit model Real Soon Now*).

Cheers

Buz

* I suppose no one remembers Jerry Pournell's Chaos Manor column in Byte Magazine.