how many walkers are really needed?

[immu...@gmail.com]

I have a function that is fairly expensive to calculate and I did some tests recently with 64, 128 and 256 walkers and adding more walkers did not make much of an impact. The corner plots looked nearly identical and when I calculate 5,10, 50, 90, 95 and maximum likelihood and these differed by less than .1% usually.

I use 50*tf as a stopping criteria and I have tried running that longer and it doesn't make any real difference in the interval I care about.

How can I figure out the minimum number of walkers I really need?

Thanks

[Dan Foreman-Mackey]

It sounds like you've done the right experiments and there's no general rule that I know. I generally find that after a certain point the number of walkers doesn't make a huge difference (depending on the number of CPUs that you're running with). I normally use 64 or fewer.

--
You received this message because you are subscribed to the Google Groups "emcee users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to emcee-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/emcee-users/0ae420ff-f9eb-4aff-9767-221e0ad5ea7cn%40googlegroups.com.

--

Dan Foreman-Mackey

Associate Research Scientist

Flatiron Institute

http://dfm.io

[immu...@gmail.com]

Okay that makes me feel better. I just wanted to make sure I was not missing something that has not come up in my reading. In my case it doesn't make sense to drop below 64 just because of the hardware I am running on so I will just stay with that.

Thanks for the help

[JP]

> I normally use 64 or fewer.

I thought that the recommended was to use hundreds of walkers, as increasing them, at least in my machine, gives a significant impact in terms of performance.

From the paper emcee: The MCMC Hammer:

> With emcee you want (in general) to run with a large number of walkers, like hundreds.

Also, does parallelization change the recommended number of walkers or does it have nothing to do with it?

[Dustin Lang]

It depends on the problem and your computing resources.  If you have N cores available to you, you can use at least N*2 walkers and keep basically the same time per step.  High dimensional problems will generally require more walkers, just to span the basis.

To view this discussion on the web visit https://groups.google.com/d/msgid/emcee-users/d316f8d9-384d-4685-adc9-e063fbd686adn%40googlegroups.com.