multiprocessing and cvxopt on windows

489 views
Skip to first unread message

Martin Cousineau

unread,
Mar 24, 2017, 12:02:59 PM3/24/17
to CVXOPT
I am having a problem using cvxopt and multiprocessing on Windows. It does work fine on MacOs however.


Any help is appreciated.

Martin Cousineau

unread,
Mar 27, 2017, 2:40:51 PM3/27/17
to CVXOPT
Let me describe in more details my problem. On Windows, multiprocessing.Pool() gives an error when I just import cvxopt without using it. I do not have the same problem on MacOs. Could it be due to my configuration of cvxopt on windows or the libraries it is using. On both machines, cvxopt has been installed using conda and use python 3.4. 

Dima Pasechnik

unread,
Mar 27, 2017, 7:49:15 PM3/27/17
to CVXOPT


On Monday, March 27, 2017 at 7:40:51 PM UTC+1, Martin Cousineau wrote:
Let me describe in more details my problem. On Windows, multiprocessing.Pool() gives an error when I just import cvxopt without using it. I do not have the same problem on MacOs. Could it be due to my configuration of cvxopt on windows or the libraries it is using. On both machines, cvxopt has been installed using conda and use python 3.4. 
 
you talk about "an error" you get by just importing, but never explain what kind of error this is.
Could you post the log?
 

Martin Cousineau

unread,
Mar 27, 2017, 10:11:46 PM3/27/17
to CVXOPT
The output is the following:

Traceback (most recent call last):
  File "test.py", line 49, in <module>
    main()
  File "test.py", line 41, in main
    pool = Pool()
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\c
ontext.py", line 118, in Pool
    context=self.get_context())
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\p
ool.py", line 168, in __init__
    self._repopulate_pool()
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\p
ool.py", line 233, in _repopulate_pool
    w.start()
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\p
rocess.py", line 105, in start
    self._popen = self._Popen(self)
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\c
ontext.py", line 313, in _Popen
    return Popen(process_obj)
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\p
open_spawn_win32.py", line 49, in __init__
Traceback (most recent call last):
Traceback (most recent call last):
    None, None, False, 0, None, None, None)
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>
OSError: [WinError 8] Not enough storage is available to process this command
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\s
pawn.py", line 106, in spawn_main
The code that generates this error is the following:
import numpy as np
from sklearn.metrics.pairwise import rbf_kernel
from sklearn.utils import resample
import cvxopt
from functools import partial
from multiprocessing import Pool

def fun_full(idx, X_ini, T_ini, gamma):
    X = X_ini[idx,:]
    T = T_ini[idx]
    x0 = X[T==0]
    x1 = X[T==1]
    n0 = x0.shape[0]
    n1 = x1.shape[0]
    K = rbf_kernel(x0,x0,gamma=gamma)
    K = cvxopt.matrix(K)
    kappa = np.sum(rbf_kernel(x0,x1,gamma=gamma),axis=1)*n0/n1
    kappa = cvxopt.matrix(kappa)
    G = cvxopt.spmatrix(-1.,range(n0),range(n0),(n0,n0),tc='d')
    h = cvxopt.matrix(np.zeros((n0,)),tc='d')
    A = cvxopt.matrix(np.ones((1,n0)),tc='d')
    b = cvxopt.matrix(n0*np.ones((1,)),tc='d')
    cvxopt.solvers.options['show_progress'] = False
    sol = cvxopt.solvers.qp(K, -kappa, G, h, A, b)
    obj = sol['primal objective']
    return obj

def main():
    np.random.seed(0)
    n = 200
    p = 5
    B = 2
    X_ini = np.random.rand(n,p)
    T_ini = np.round(np.random.rand(n))
    idx_ini = np.arange(len(T_ini))
    idx_boot = []
    for i in range(B):
        idx_boot.append(resample(idx_ini))
Everything is also available on the previous stack overflow link.

Dima Pasechnik

unread,
Mar 28, 2017, 3:58:21 AM3/28/17
to CVXOPT
Thanks. As you talked about "an error" it was not clear whether you meant the error already shown, or something else.
Did you try importing cvxopt after Pool rather than before?

Martin

unread,
Mar 28, 2017, 4:28:48 AM3/28/17
to CVXOPT
I am guessing that you are using CVXOPT 1.1.8 from the Omnia channel? If yes, could you try CVXOPT 1.1.9 to see that solves your problem? (You should be able to install this from pypi with pip.)  

Martin Cousineau

unread,
Mar 28, 2017, 5:58:50 AM3/28/17
to CVXOPT
Using CVXOPT 1.1.9 and importing cvxopt after Pool did not solve the problem when using cvxopt within the fun_full function.

I get the following error:

Traceback (most recent call last):
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "<string>", line 1, in <module>
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\s
pawn.py", line 106, in spawn_main
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\s
pawn.py", line 106, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\s
pawn.py", line 115, in _main
    exitcode = _main(fd)
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\s
pawn.py", line 115, in _main
    prepare(preparation_data)
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\s
pawn.py", line 226, in prepare
    prepare(preparation_data)
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\s
pawn.py", line 226, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\s
pawn.py", line 278, in _fixup_main_from_path
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\s
pawn.py", line 278, in _fixup_main_from_path
    run_name="__mp_main__")
    run_name="__mp_main__")
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\runpy.py", line 2
40, in run_path
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\runpy.py", line 2
40, in run_path
    pkg_name=pkg_name, script_name=fname)
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\runpy.py", line 9
6, in _run_module_code
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\runpy.py", line 9
6, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\runpy.py", line 8
5, in _run_code
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\runpy.py", line 8
5, in _run_code
    exec(code, run_globals)
  File "C:\Users\mcousi7\Dropbox\Code\Workspace\DepOpt\src\test.py", line 6, in
<module>
    exec(code, run_globals)
  File "C:\Users\mcousi7\Dropbox\Code\Workspace\DepOpt\src\test.py", line 6, in
<module>
    import cvxopt
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\site-packages\cvx
opt\__init__.py", line 32, in <module>
    import cvxopt
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\site-packages\cvx
opt\__init__.py", line 32, in <module>
    import cvxopt.base
    import cvxopt.base
ImportError: DLL load failed: The paging file is too small for this operation to
 complete.
ImportError: DLL load failed: The paging file is too small for this operation to
 complete.
Fatal Python error: Failed to initialize Windows random API (CryptoGen)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\s
pawn.py", line 106, in spawn_main
    exitcode = _main(fd)
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\s
pawn.py", line 115, in _main
    prepare(preparation_data)
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\s
pawn.py", line 226, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\multiprocessing\s
pawn.py", line 278, in _fixup_main_from_path
    run_name="__mp_main__")
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\runpy.py", line 2
40, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\runpy.py", line 9
6, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\runpy.py", line 8
5, in _run_code
    exec(code, run_globals)
  File "C:\Users\mcousi7\Dropbox\Code\Workspace\DepOpt\src\test.py", line 6, in
<module>
    import cvxopt
  File "C:\Users\mcousi7\AppData\Local\Continuum\Anaconda3\lib\site-packages\cvx
opt\__init__.py", line 32, in <module>
    import cvxopt.base
ImportError: DLL load failed: The paging file is too small for this operation to
 complete.

Martin

unread,
Mar 29, 2017, 2:09:51 PM3/29/17
to CVXOPT
I am afraid that it is not an error that we have seen before. From the error message, it sounds like a memory issue ("The paging file is too small for this operation to complete.")? 

Martin Cousineau

unread,
Mar 29, 2017, 2:58:08 PM3/29/17
to CVXOPT
Maybe but this code runs fine on my macbook that got way less memory. Also, the same code works on windows if I replace the multiprocessing lines with a loop that does the same thing. Finally, on windows, the multiprocessing lines work as long as I replace this function with another one that do not use cvxopt.
Reply all
Reply to author
Forward
0 new messages