Problem parallelizing calls?

[Marnin Wolfe]

Hi there,

I am having a problem parallelizing jobs (via R). Basically, I want to run a lot of jobs at the same time on a server. I typically do this using the "foreach" package and/or "plyr" functions (which uses foreach). I'm using the "doParallel" library as my backend to do this.

When I do this, after the first analysis completes on a given thread, the next one terminates prematurely. Likelihoods, AIC, etc. are listed as NA in the output (I do get an output returned) and I get this message:

forrtl: No such file or directory

forrtl: severe (28): CLOSE error, unit 49, file "Unknown"

Image              PC                Routine            Line        Source             

airemlf90          0000000100B9E475  Unknown               Unknown  Unknown

airemlf90          0000000100B9A41F  Unknown               Unknown  Unknown

airemlf90          000000010003A800  Unknown               Unknown  Unknown

airemlf90          0000000100006B53  Unknown               Unknown  Unknown

airemlf90          000000010000108E  Unknown               Unknown  Unknown

I have a hunch that this has to do with the naming / storage of temporary files (parameter files for airemlf90). Could there be a problem when many R sessions try to access airemlf90 and/or write parameter files? Or some kind of a memory leak?

Any help will be appreciated.

Thanks,

Marnin

[Facundo Muñoz]

Marnin,

I have noticed the behaviour you report when using parallelisation based on forking (e.g. mclapply). And indeed it is due to a fragile set-up of temporary files.

Lacking the time for fixing it properly, my workaround for this has been precisely the use of foreach + doParallel.

Take at look at this issue [1] to see whether it helps you setting up things properly, and let me know.

Alternatively, you could also leverage the "remote computing" feature (with option breedR.bin = "submit") in order to parallelise jobs. See ?remote for details. Note that you can define your "remote" server as "localhost".

Hope it helps   

        ƒacu.-

[1] https://github.com/famuvie/breedR/issues/92

--
Report issues at https://github.com/famuvie/breedR/issues
---
You received this message because you are subscribed to the Google Groups "breedR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to breedr+un...@googlegroups.com.
To post to this group, send email to bre...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/breedr/6a61d271-23d1-41d0-8d8f-686e7b9be1f1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marnin Wolfe]

Thanks so much, Facundo! That did fix the problem and I can easily and successfully parallelize now.

Just a note: discovered the R package "doFuture", which is like a more flexible backend than doParallel. By using a "multisession" instead of a forking-based method, I had no problems.

Cheers,

Marnin

[Facundo Muñoz]

Cool!

Out of curiosity. Was `doFuture` necessary to fix your issue, or could you handle it with `doParallel`?

Thanks for your report!

        ƒacu.-

To view this discussion on the web visit https://groups.google.com/d/msgid/breedr/049f7187-726c-4bd5-8a7f-2638e07b2577%40googlegroups.com.

[Marnin]

I believe I could have done it with doParallel, but doFuture handles the passing of libraries and env. objects by default AND has more options for the method of parallel.

~Marnin

You received this message because you are subscribed to a topic in the Google Groups "breedR" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/breedr/JhZH9ExJeTE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to breedr+un...@googlegroups.com.

To post to this group, send email to bre...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/breedr/8aaf5aea-d70b-0316-656c-ed2d4e69c7a5%40cirad.fr.