cluster or parallel computing

[Venkatessh]

Hi,

I would like to know how to run the Fermi Science Tools using enrico in either a cluster or a multicore computer.
Is it sufficient to mention yes in the configuration file to submit a job to a cluster?


Thank you,
R Venkatessh

[Christoph Deil]

Hi!

Thanks for trying out enrico!

we use enrico to submit jobs for parallel processing of each energy bin in a spectrum or time bin in a light curve using the Submit = yes option:

[Spectrum]
   #Submit the job to a cluster?
   Submit = yes

[LightCurve]
   #Submit the job to a cluster?
   Submit = yes

See https://enrico.readthedocs.org/en/latest/configfile.html

This will only work for the grid engine job submission system (http://en.wikipedia.org/wiki/Oracle_Grid_Engine), i.e. if you use qsub ( http://gridscheduler.sourceforge.net/htmlman/htmlman1/qsub.html ) to submit jobs.

If you use some other job scheduling system it will currently not work, but should be quite easy to add.

See https://github.com/gammapy/enrico/tree/master/enrico/submit if you can do it yourself or make a github issue and we will try to do it.

To use multiple cores on one machine it should be possible to simply call `bash` instead of `qsub` when running each tool, but I haven't tried this.

I also wanted to point out an announcement from the fermi-soft mailing list to you from December 2012, where they added parallel processing capabilities to gtapps_mp and LATAnalysisScripts. I haven't tried these tools yet. Unfortunately this mailing list is private, so you have to sign up at http://fermi.gsfc.nasa.gov/ssc/library/newsletter/ to see the archive, so I've pasted that post below.

Please let us know what worked for you in the end or if you run into any problems making it work on your cluster. Getting batch submission to work on a new cluster typically needs some experimentation (use very small datasets so that you see if it works or not immediately), but it shouldn't be hard to make it work.

Hope that helps.

Christoph

------------------

*** Posting from fermi-soft on 2012-12-06

Dear Fermi User,

The FSSC is pleased to announce a new set of User Contributed Tools, gtapps_mp.
These tools are python modules that divide up some of the more computationally
intensive tasks and submit the resulting smaller jobs as separate processes on a user's
computer. This can drastically decrease the time needed to perform an analysis of
LAT data.

The current tools supported are gtltcube, gtexpmap and gtdiffrsp. The gtapps_mp
package can be downloaded from the FSSC User Contributed Tools website:

http://fermi.gsfc.nasa.gov/ssc/data/analysis/user/

For more details on how to use gtapps_mp, see the README file associated with the
package. Support for more tools may be available in the future.


Additionally, the LATAnalysisScripts have been updated to version 0.1.11.
See the change log associated with the release for more details.

Regards,
Fermi Science Support Center

-- 

[Venkatessh]

Hi,

Thank you for the quick response. Your tool is well documented in terms of the usage and setup (at least I didn't have any problems with that).
Having some experience with the Fermi Science Tools, I like the way the package is structured. However, I would be glad if you could clarify my doubt on running it on multicore machines. I am trying to get access to a cluster and I am sure there will be no problem running in that using "qsub".
But for the time being, I have a 12 core machine at my disposal and would be glad to run the data reduction using "enrico".

You mentioned the use of cmd "bash" in place of "qsub". Does this mean the replacement of the cmd in the "__init__.py" script?

Also, thanks for mentioning the fermi-soft mailing list.


R Venkatessh

[Christoph Deil]

Yes, that's what I meant. I'll try to add it tomorrow:
https://github.com/gammapy/enrico/issues/16

Christoph

[Venkatessh]

Hi,

I tried fiddling with some of the commands to make parallel computing work in my 12-core machine. Still no success.

R Venkatessh

[Venkatessh]

Hi,

I tried to use GNU Parallel library to utilize the parallel programming capability. Haven't had any great success until now. I begin to doubt if it is because of the hard-coded binaries that are part of the Fermi Science Tools that causes this problem.


R Venkatessh