TypeError: can't pickle weakref objects

[Rachmad]

Hi, 

I'm new into Brian2 and Python. 

What I'm trying is to do both, testing mode of pretrained network and training mode, using code provided here https://github.com/zxzhijia/Brian2STDPMNIST with modifications for Brian2 compliance.

I did OK for testing mode since it used the existing pretrained parameters.

When I tried to do training mode, I got error when the simulator reached:

if j % save_connections_interval == 0 and j > 0 and not test_mode: 
   save_connections(str(j))
   save_tetha(str(j))

print ('save connections')
for connName in save_conns:
    conn = connections[connName]
    connListSparse = zip(conn.i, conn.j, conn.w)
    np.save(data_path + 'weights/' + connName + ending, connListSparse)

with error message in the end is: TypeError: can't pickle weakref objects

I think it is about error when trying to save to the *.npy files.

Since there are several tasks supposed to generate results in *.npy files, how can I solve this?

Thanks and best regards

[Rachmad]

Here is the error traceback:

0.35 s (100%) simulated in < 1s

save connections

ERROR      Brian 2 encountered an unexpected error. If you think this is bug in Brian 2, please report this issue either to the mailing list at <http://groups.google.com/group/brian-development/>, or to the issue tracker at <https://github.com/brian-team/brian2/issues>. Please include this file with debug information in your report: /tmp/brian_debug_5ex2qv3p.log  Additionally, you can also include a copy of the script that was run, available at: /tmp/brian_script_rwldyum3.py You can also include a copy of the redirected std stream outputs, available at /tmp/brian_stdout_4fukhg3_.log and /tmp/brian_stderr_31e8i8s8.log Thanks! [brian2]

Traceback (most recent call last):

  File "Diehl&Cook_spiking_MNIST_Brian2.py", line 462, in <module>

    save_connections(str(j))

  File "Diehl&Cook_spiking_MNIST_Brian2.py", line 89, in save_connections

    np.save(data_path + 'weights/' + connName + ending, connListSparse)

  File "/homes/rachmad/anaconda3/envs/brian2/lib/python3.6/site-packages/numpy/lib/npyio.py", line 509, in save

    pickle_kwargs=pickle_kwargs)

  File "/homes/rachmad/anaconda3/envs/brian2/lib/python3.6/site-packages/numpy/lib/format.py", line 576, in write_array

    pickle.dump(array, fp, protocol=2, **pickle_kwargs)

TypeError: can't pickle weakref objects

Thanks a lot and best regards

[Marcel Stimberg]

Hi,

I think this could be due to the fact that the code pickles Brian attributes like synaptic variables directly. These are `VariableView` objects that allow some fancy things like string indexing and therefore need to keep a reference to their parent group. For storing, you most likely don't need all that. The easiest way to get the simple array information is to use e.g. `conn.i[:]` instead of `conn.i`. For variables with units, this will still store the unit information (i.e. a `Quantity` object), to get rid of this as well use a trailing underscore (`conn.i_[:]`).

Best,

  Marcel

--

---
You received this message because you are subscribed to the Google Groups "Brian Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to brian-developm...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rachmad Vidya Achmad]

Hi,

It worked generating the *.npy file.

Unfortunately, I came to other problems.

It seems like the generated file (*.npy), can not be read when I tried to conduct the testing phase.

It gaves this traceback error:

() ./weights/XeAe.npy

ERROR      Brian 2 encountered an unexpected error. If you think this is bug in Brian 2, please report this issue either to the mailing list at <http://groups.google.com/group/brian-development/>, or to the issue tracker at <https://github.com/brian-team/brian2/issues>. Please include this file with debug information in your report: /tmp/brian_debug_f6wl9fa8.log  Additionally, you can also include a copy of the script that was run, available at: /tmp/brian_script_sam7uq__.py Thanks! [brian2]

Traceback (most recent call last):

  File "Diehl&Cook_spiking_MNIST_Brian2.py", line 401, in <module>

    weightMatrix = get_matrix_from_file(weight_path + connName + ending + '.npy')

  File "Diehl&Cook_spiking_MNIST_Brian2.py", line 80, in get_matrix_from_file

    value_arr[np.int32(readout[:,0]), np.int32(readout[:,1])] = readout[:,2]

IndexError: too many indices for array

It didn't happen when I used the pretrained network.

Typically, the generated file (*.npy) should be able to be read and give something like:

(313600, 3) ./weights/filename.npy

Starting simulation at t=0. s for a duration of 0.35 s

0.35 s (100%) simulated in < 1s

Starting simulation at t=0.5 s for a duration of 0.35 s

0.35 s (100%) simulated in < 1s

Starting simulation at t=1. s for a duration of 0.35 s

0.35 s (100%) simulated in < 1s

...

It seems something wrong with the indexing, am I correct?

How can I solve this?

To unsubscribe from this group and stop receiving emails from it, send an email to brian-development+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

---
You received this message because you are subscribed to the Google Groups "Brian Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to brian-development+unsubscribe@googlegroups.com.

[Rachmad]

Hi,

I'm thinking it might be something related to the format of how I generate and save the connection in *.npy file.

The original code for Brian (version 1) is: 

def save_connections(ending = ''):

    print 'save connections'

    for connName in save_conns:

        connMatrix = connections[connName][:]

        connListSparse = ([(i,j,connMatrix[i,j]) for i in xrange(connMatrix.shape[0]) for j in xrange(connMatrix.shape[1])])

        np.save(data_path + 'weights/' + connName + ending, connListSparse)

I changed it to the following code for Brian2:

def save_connections(ending = ''):

    print ('save connections')

    for connName in save_conns:

        conn = connections[connName]

        connListSparse = zip(conn.i[:], conn.j[:], conn.w[:])

        np.save(data_path + 'weights/' + connName + ending, connListSparse)

This modification gave me *.npy output which can not be read when I was conducting testing mode.

What should I do to solve this?

Thanks for the helps and best regards :)

[Marcel Stimberg]

Hi,

I don't have the time to dive into the code, but I think the code might simply be unnecessarily complicated for Brian 2. If you store synaptic indices and weights as flat vectors, then you do not have to reconstruct a 2d matrix to set them. You can do something along the lines of:

i, j, w = conn.i[:], conn.j[:]. conn.w[:]  # store connections and weights

...

conn = Synapses(...)

conn.connect(i=i, j=j)  # reconstruct connections

conn.w[:] = w  # reconstruct weights

Hope that helps, best

  Marcel

[karthi...@quantiphi.com]

[karthi...@quantiphi.com]

Hi,

  I have got this error , can you please help me to resolve it

     Traceback (most recent call last):

  File "vocabulary_expansion.py", line 205, in <module>

    tf.app.run()

  File "/home/quantiphi/.local/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 126, in run

    _sys.exit(main(argv))

  File "vocabulary_expansion.py", line 200, in main

    np.save(embeddings_file, embeddings[:])

[karthi...@quantiphi.com]

Hi,

  Traceback (most recent call last):

  File "vocabulary_expansion.py", line 205, in <module>

    tf.app.run()

  File "/home/quantiphi/.local/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 126, in run

    _sys.exit(main(argv))

  File "vocabulary_expansion.py", line 200, in main

    np.save(embeddings_file, embeddings)

  File "/home/quantiphi/.local/lib/python3.6/site-packages/numpy/lib/npyio.py", line 529, in save

    pickle_kwargs=pickle_kwargs)

  File "/home/quantiphi/.local/lib/python3.6/site-packages/numpy/lib/format.py", line 629, in write_array

    pickle.dump(array, fp, protocol=2, **pickle_kwargs)

TypeError: can't pickle odict_values objects

[Marcel Stimberg]

Hi,

this is a discussion forum for the Brian simulator, it seems your issues
do not have anything to do with this software. Please ask questions
about libraries like tensorflow in an appropriate forum (as a general
advice, please also have a look at
https://stackoverflow.com/help/how-to-ask).

Best,

  Marcel

[zhangxu...@gmail.com]

Hi, I am also with the same problem when running the training mode. I tried a lot of ways but didn't work. How did you solve this eventually? Thanks!

在 2018年6月27日星期三 UTC+8下午5:32:22，Rachmad写道：

[Marcel Stimberg]

Hi,

please have a look at the earlier answers in the thread (starting here: https://groups.google.com/d/msg/brian-development/tkHY0dBTZg8/9TdXM1MsCAAJ).

Best,

  Marcel

[zhangxu...@gmail.com]

Great. Thanks a lot!!!!

在 2020年6月30日星期二 UTC+8下午9:46:47，Marcel Stimberg写道：

[zhangxu...@gmail.com]

Great. Thanks a lot!!!!

在 2020年6月30日星期二 UTC+8下午9:46:47，Marcel Stimberg写道：

Hi,

[Aditi Bishnoi]

Hi, 

I am trying to run a Brian2 simulation for different values of poissonGroup inputs under a for loop, such that the output of the simulation (spikte trains from spikemonitors) is saved into numpy arrays in each loop. The function works perfectly but gives the following error when run using joblib.Parallel

[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.
---------------------------------------------------------------------------
_RemoteTraceback                          Traceback (most recent call last)
_RemoteTraceback:

"""
Traceback (most recent call last):

  File "C:\Users\ADITI\anaconda3\lib\site-packages\joblib\externals\loky\process_executor.py", line 356, in _sendback_result
    result_queue.put(_ResultItem(work_id, result=result,
  File "C:\Users\ADITI\anaconda3\lib\site-packages\joblib\externals\loky\backend\queues.py", line 241, in put
    obj = dumps(obj, reducers=self._reducers)
  File "C:\Users\ADITI\anaconda3\lib\site-packages\joblib\externals\loky\backend\reduction.py", line 271, in dumps
    dump(obj, buf, reducers=reducers, protocol=protocol)
  File "C:\Users\ADITI\anaconda3\lib\site-packages\joblib\externals\loky\backend\reduction.py", line 264, in dump
    _LokyPickler(file, reducers=reducers, protocol=protocol).dump(obj)
  File "C:\Users\ADITI\anaconda3\lib\site-packages\joblib\externals\cloudpickle\cloudpickle_fast.py", line 563, in dump
    return Pickler.dump(self, obj)
TypeError: cannot pickle 'weakref' object
"""

Am I writing the function in a way that Brian2 is unable to support parallel processing? I can share the function if needed.

[Aditi Bishnoi]

#build synapses using these weights and connect
        CE = Synapses(PC, In, on_pre='ge+=we')
        CE.connect(j='i') 

        CI = Synapses(In, PC, on_pre='gi+=wi')
        CI.connect(j='i')
   
   
        MPC = StateMonitor(PC, ['vme','Iexc','gi'], record=True)
        SPC = SpikeMonitor(PC)
   
        MIn = StateMonitor(In, ['vmi','Iinh','ge'], record=True)
        SIn = SpikeMonitor(In)
   
        run(Trun) 
        Network.t=0*second 
       
        spikets_in.extend(list(SIn.spike_trains()[0]))
        spikets_pc.extend(list(SPC.spike_trains()[0]))

[Marcel Stimberg]

Hi,

Brian supports multiprocessing, see e.g. this example https://brian2.readthedocs.io/en/stable/examples/multiprocessing.01_using_cython.html (the example is using multiprocessing.Pool, but it works in the same way with Parallel from joblib). It is hard to say what the problem is without having some minimal executable code that fails, though.

That said, this mailing list is not really used anymore – would you mind moving the discussion to https://brian.discourse.group/ ? Thanks!

Best,

  Marcel