How to append a Keras model into a multiprocessing.Manager().list? I got 'TypeError: can't pickle _thread.RLock objects' error

[zkx741...@gmail.com]

Here is the Error message:

Process Process-4:

Traceback (most recent call last):

  File "C:\Users\zkx74\Anaconda3\lib\multiprocessing\process.py", line 297, in _bootstrap

    self.run()

  File "C:\Users\zkx74\Anaconda3\lib\multiprocessing\process.py", line 99, in run

    self._target(*self._args, **self._kwargs)

  File "C:\Users\zkx74\PycharmProjects\DDPG_QuantTrade\MADDPG.py", line 162, in init_nn

    agent_list.append(ACModel(actor, critic, env, current_stock_state, current_agent_state))

  File "<string>", line 2, in append

  File "C:\Users\zkx74\Anaconda3\lib\multiprocessing\managers.py", line 795, in _callmethod

    conn.send((self._id, methodname, args, kwds))

  File "C:\Users\zkx74\Anaconda3\lib\multiprocessing\connection.py", line 206, in send

    self._send_bytes(_ForkingPickler.dumps(obj))

  File "C:\Users\zkx74\Anaconda3\lib\multiprocessing\reduction.py", line 51, in dumps

    cls(buf, protocol).dump(obj)

TypeError: can't pickle _thread.RLock objects

I want to init and run sereval Keras models with multiprocessing, so I have to save these model after init. I used  multiprocessing.Manager().list() to save them in the main process, but when I appending the model, this error message came out. 

Here is my code:https://paste.ubuntu.com/p/Z54XbRkk8h/  and the error is at line 33

[dese...@gmail.com]

did you figure this out? I am having the same problem.

[Sili]

Same here... Does someone have a solution?

[Blake Richey]

I figured out a solution to this a while back. 

What I do is I define a function that creates a network with the same architecture and make it visible to the subprocesses, then I pass a list of the weights to the subprocess, generate the model, and assign the weights. 

You can pickle a list of weights.

Here is some psuedocode:

```Python

import multiprocessing as mp

from multiprocessing import Process

def gen_network(params):

    #Generate model

    return model

def subprocess_function(weights):

    model = gen_network(params)

    model.set_weights(weights)

   #do stuff with new network

    

if __name__ == '__main__':

    num_processes = 4    

    processes = []

    for i in range(num_processes):

        model = gen_network(params)

        weights = model.get_weights()

        p = Process(target=subprocess_function, args = (weights,))

        processes.append(p)

        p.start()

    results = []

    for p in processes:

        res = p.join()

        results.append(res)

    print(results)

    

```