Does HDF backend slows down with MPI multiprocessing using schwimmbad?

[NAVIN CHAURASIYA]

Hi everyone,

I tried to use the HDF backend with parallelised walkers (of the order of about 100) using MPIPool from schwimmbad on a cluster.

I noticed that when I don't parallelise it, the full mcmc run finishes about 10 times faster. However when I parallelise it, it becomes slow. I have stuggled to find out this.

The schwimmbad version : 0.3.2.

emcee version : 3.1.1

I'm not sure if I have provided useful information to guess the issue.

I'd be willing to pass any relevant information on this from my side, if anyone wants to help.

Is there a known issue of compatibility of emcee with schwimmbad to cause this?

Thank you.

Regards,

Navin Chaurasiya.