multicore usage of Rmosek

[rogerk...@gmail.com]

Having tried to use Rmosek in a foreach loop and gotten a rather mysterious error message,

I'm wondering whether this is likely to be something unsupported -- a brief passage in the

userguide suggests  that this might be the case -- or whether it is likely to be my "pilot error."

I don't want to do anything fancy, just to solve independent problems in a foreach loop.

I wouldn't have thought that this violated any license agreement, but I suppose that this is

also a possibility.

RK

[edadk]

Hi Roger

What was the error message?

[Henrik Alsing Pedersen]

Dear Roger,

The Rmosek package is not currently thread safe. Solving multiple
problems in parallel is on the "Missing Feature" wishlist in the
userguide. For development purposes: did you use the "foreach"
package? if yes, with what parallel backend (e.g. registerDoMC)?

Regards,
Henrik Alsing Friberg

[rogerk...@gmail.com]

Henrik,

Yes,  I was using the foreach package, and registerDoMC.  This is about

the extent of my prior experience with multi-core methods in R;  it would be

great if it were possible to run Rmosek in this mode, my (statistical) applications

involve simulations with the need to solve many (relatively small) problems.

Roger

[Henrik Alsing Friberg]

Hi Roger,

The next release of Rmosek (v.1.2.3) fixes a platform-specific bug which you may have been unlucky enough to provoke. I have attached a multicore test script which may guide you to a safe parallel implementation. The thing is that the multicore package copies the entire address space to new processes. This work okay most of the time, but if a MOSEK environment have been cached in the package before %dopar%, all spawned processes will try to use the same environment in a way that is not thread-safe. All spawned processes will also try to release the cached MOSEK environment which will cause something like
*** glibc detected *** /.../R: double free or corruption (!prev): 0x0000000002844590 ***

Since the parallelization happens outside my code, I cannot control it. I might be able to detect the fork so each spawned process can fetch their own MOSEK environment, but unless it is a major problem it is not something I will look into as the solution would have to look outside of the processes address space (e.g. the harddrive) and this will definitely cause a performance overhead. Instead, I suggest calling mosek_clean before every %dopar%, to release the cached MOSEK environment.

So take a look at the attached script and try it with the next Rmosek v1.2.3 release when it reaches CRAN. Hope it works for you ;-)

Kind regards,
Henrik Alsing Friberg

[rogerk...@gmail.com]

Hi Henrik,

Thanks, this sounds like an excellent development.  Can you make any guesses about when 1.2.3 will arrive on CRAN, it

seems 1.2.2 is still there.

Roger

[Henrik Alsing Friberg]

Dear Roger

Usually the Rmosek package have been tested using the "--no-install"
flag, because the CRAN servers did not have MOSEK installed. Now they
do, and because of their "-pedantic" gcc flag and the typedef of "long
long" in mosek.h, warnings were produced (no 64bit integer support in
C++98 standard). The only way to get rid of these warnings was to
modify mosek.h which will happen in MOSEK 6.0.0.136 planned to be
released after the easter holiday next week. The package will then
overwrite "long long" with "int64_t" which is a warning-free
compiler-specific version of "long long". Until this version reaches
the CRAN servers, and everything checks out without warning, I am not
allowed to release.

If you have an SVN client you can checkout the project before that:
svn checkout svn://scm.r-forge.r-project.org/svnroot/rmosek/

You can install directly from this directory, or perform a "R CMD
build" to make the tar.gz..

Kind regards,
Henrik Alsing Friberg

[rogerk...@gmail.com]

Hi Hendrik,

I've now installed Rmosek 1.2.3 via svn as suggested.  When running your test_multicore.R, I get:

Optimization interrupted.

ERROR: MSK_RES_ERR_LICENSE_NO_SERVER_SUPPORT: The license server does not support the requested feature.

Optimization interrupted.

ERROR: MSK_RES_ERR_LICENSE_MAX: Maximum number of licenses is reached.

it continues, and begins to report iterations, but then again dies with the messages about Max number of licenses reached.

Is this is an inherent problem with a single licenses, or is there something wrong with my installation?

Roger

[Henrik Alsing Friberg]

Hi Roger,

The multicore package works by spawning new processes each having
their own MOSEK environment. Since each MOSEK environment requests a
separate license, you will need one license per process spawned by the
parallelization operator %dopar%. As you guessed, the error you see is
caused by the fact that you only have one license installed.

Since you come from the academical world, you may find interest in our
Academic Initiative where a free and unlimited license can be
obtained:
http://mosek.com/sales/Academic%20initiative/

This will allow you to solve optimization problems in parallel with as
many processes as your system allows.

Kind regards,
Henrik Alsing Friberg

[Henrik Alsing Friberg]

Hi Roger,

I have added a new section of parallel optimization in the Advanced Topics chapter, of the newest userguide found on
  http://rmosek.r-forge.r-project.org/

Please let me know if you find my description helpful, and if something important is missing.

[rogerk...@gmail.com]

Hi Henrik,

I've now installed a 64 bit mosek and verified that with the new license I can use foreach

which is really an excellent development.  Thanks very much for all your efforts.  The new

section of the userguide is also good. For my simulations I'm always operating in batch

mode from the console, so your dire warnings about  environmental  cleanliness won't

be a problem, I hope.  Certainly, they haven't been thus far.  I do have another rather

bizarre installation question, but I think that I better open a new thread for this tomorrow.

Thanks again,

Roger