multiprocessing compatibility issue with Gurobi

[Jason]

The following simple multiprocessing of a square function works fine:

    

    from multiprocessing import Pool

    class A(object):

        def square(self, x):

            return x * x

        def test(self):

            pool = Pool()

            result = pool.map(self.square, range(5))

            return result

But when I add an initialization of a Gurobi model in the class like this,

    from multiprocessing import Pool

    from gurobipy import *

    class A(object):

        def __init__(self):

            self.b = Model()

        def square(self, x):

            return x * x

        def test(self):

            pool = Pool()

            result = pool.map(self.square, range(5))

            return result

    A().test()

It returns the following error:

    File "model.pxi", line 290, in gurobipy.Model.__getattr__ (../../src/python/gurobipy.c:53411)

    KeyError: '__getstate__'

A serial version with Gurobi works fine:

    from gurobipy import *

    class A(object):

        def __init__(self):

            self.b = Model()

        

        def square(self, x):

            return x * x

    

        def test(self):

            res = [self.square(i) for i in range(5)]

            return res

    A().test()

[Daniel Espinoza]

Models are not thread safe in Python....

[Jason]

So how should I do parallelization in Python?

[Daniel Espinoza]

Hi Jason,

Once you build the model (which should take little time), Gurobi will work in parallel internally. If you want to solve several models in parallel, you first create (sequentially) empty models, and then you can work with them in different threads without any problem.
Is it the case that your model-building step is taking too long?

Best,

Daniel

[Jason]

Hi Daniel,

Thanks for your reply! In my case, I am solving several models in parallel. But even for the following simple case, I don't understand where comes the error. 

I build a model using the class constructor. After that, I do a parallelization of square function. The parallelization has nothing to do with the Gurobi model. 

    from multiprocessing import Pool

    from gurobipy import *

    class A(object):

        def __init__(self):

            self.b = Model()

        def square(self, x):

            return x * x

        def test(self):

            pool = Pool()

            result = pool.map(self.square, range(5))

            return result

    A().test()

It returns the following error:

    File "model.pxi", line 290, in gurobipy.Model.__getattr__ (../../src/python/gurobipy.c:53411)

    KeyError: '__getstate__'

Best,

Jason

[Daniel Espinoza]

Hi Jason,

Full disclosure first.... I've never used python's Pools.

I am guessing that since the __init__ creates the models, and that creation is run in parallel, and that call is accessing the `default` environment. This is also not thread safe, so model building and freeing should be done sequentially.

Now, the error points to querying `__getstate__` seems a little odd, as Gurobi does not use any such keyword....  may it be that there is some `pickling` involved? If so, for sure that is not supported for Models or Environments...

Best,

Daniel

[Mohamed Ibrahim]

You need to change your design and here is two issues with your code:

1 - The main method has to be used when using multiprocessing.  https://docs.python.org/3.7/library/multiprocessing.html#multiprocessing-programming

2- I think that the Pool in multiprocessing cannot store an object. I can tell that because if you declare the square() method as a static method the problem would be solved.

Here is how I use Gurobi with multiprocessing:

from multiprocessing import Pool
from gurobipy import *

class Square:
    def __init__(self, l):
        self.m = Model()
        self.l = l
        self.square(self.l)

    @staticmethod
    def square(l):
        return l * l


if __name__ == '__main__':
    with Pool() as pool:
        result = pool.map(Square.square, range(5))
        print(result)