Question about Celery

[ilaria_mosca]

Dear OQ-Developers,

I'm writing to have some explanations on Celery. In order to reduce the computation time I would like to use more than one processor for running OpenQuake. I think I installed and run celery properly after having installed and run RabbitMQ. But I haven't understood how I  can control the number of processors used in a job. How do I select the number of processors? and is there a suggested criterium to define how many processors to select with respect to the time window or/and the number of stochastic event sets for a event based PSHA? 

I selected see_per_logic_tree_path=10000 (this parameter indicate the number of stochastic event set in a event based PSHA, isn't it?) and so the computational time is very long using only one processor. This explains why I would like to use the parallel computation.

Thank you in advance for any help.

Kind regards,

Ilaria

[Luigi Panzeri]

Hi Ilaria,

By default Celery runs with a concurrency level equal to number of CPU/cores available on your machine. In general you can select the number of threads used by celery with the -c command line argument, e.g.

celeryd -c 8  

This will start celery with 8 threads, that means that it can use up to 8 cores in the same time).

As far I know, there is no systematic study of the performance of oq-engine with the concurrency level varying.

However, in this case the rule of thumb is that you get a performance gain by increasing the concurrency only if your calculation is CPU bound (so the time spent in computation is bigger than the time spent in input/output).

Moreover, the event based calculations, I have seen so far, perform better by using the default number of parallel threads. So, my suggestion is to keep the default.

Cheers,

Luigi

2013/7/3 ilaria_mosca <ilaria....@gmail.com>

--
You received this message because you are subscribed to the Google Groups "OpenQuake Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openquake-de...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Sriram]

Hey Ilaria,

One suggestion I would like to add to Luigi's response. I have noticed that you are requesting 10000 SES per logic tree branch. I am not sure what is your requirement but you can control the size of SES by controlling the length of SES ( through investigation_time) and number of ses per branch. Theoretically, an investigation time of 10000 years gives an SES equivalent to 10000 SESs 1 year long each concatenated. But in Openquake implementation, generating 10000 SESs per branch takes significantly more time than 1 SES 10000 years long. 

Hope this suggestion is of some use, let me know if I can help

Peace,

Sriram 

--
Peace,
Sriram

[ilaria_mosca]

Dear Luigi and Sriram,

thank you very much for your answers. They both helped me to understand better the parallel computation of OpenQuake.

Cheers,

Ilaria