ch 6 Memory Error MCS.fit(X_train, y_train)

17 views
Skip to first unread message

mike nesius

unread,
Sep 28, 2016, 11:06:18 AM9/28/16
to OpenCV with Python Blueprints
Hi Michael,

I'm getting a memory error when trying to run the chapter 6 code. I have 16gb of RAM...any suggestions on getting it to run?


Below is the traceback. Congrats on the recent post doc!

thanks,
Mike



>>>>
runfile('C:/Users/nesiusm/Documents/opencv-python-blueprints-master/chapter6/chapter6.py', wdir='C:/Users/nesiusm/Documents/opencv-python-blueprints-master/chapter6')
feature None
 - strategy one-vs-one
    - train
Traceback (most recent call last):

  File "<ipython-input-1-ddd8a3664d51>", line 1, in <module>
    runfile('C:/Users/nesiusm/Documents/opencv-python-blueprints-master/chapter6/chapter6.py', wdir='C:/Users/nesiusm/Documents/opencv-python-blueprints-master/chapter6')

  File "C:\Users\nesiusm\AppData\Local\Continuum\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 714, in runfile
    execfile(filename, namespace)

  File "C:\Users\nesiusm\AppData\Local\Continuum\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 74, in execfile
    exec(compile(scripttext, filename, 'exec'), glob, loc)

  File "C:/Users/nesiusm/Documents/opencv-python-blueprints-master/chapter6/chapter6.py", line 86, in <module>
    main()

  File "C:/Users/nesiusm/Documents/opencv-python-blueprints-master/chapter6/chapter6.py", line 53, in main
    MCS.fit(X_train, y_train)

  File "classifiers.py", line 250, in fit
    X_train_id = X_train[data_id, :]

MemoryError

Michael Beyeler

unread,
Sep 30, 2016, 1:45:48 PM9/30/16
to OpenCV with Python Blueprints
Hi Mike,

This is a tricky one... Here are some comments and ideas, hopefully one of them will work for your system.

First, is it possible that you're running 32-bit Python? I read that this puts a practical limit of about 2 GB of RAM on your application, no matter how much RAM the system has:
This would make much more sense, since I don't believe `X_train` is really that big. Of course with Python, it is often not straightforward to know how much memory is being used...

An easy fix could be to operate on a smaller image: set `small_size` to something smaller than (32, 32) on l.125 of gtsrb.py.

If you're using the code in a practical setting, you could just go with the HOG feature descriptor, which has the smallest number of features and also gives the best result. The other modes are merely there for educational purposes. :) After HOG, mode 'gray' is the next smallest. Avoid running 'SURF' mode, as it uses the most memory.

Then finally, I just pushed a quickfix to the master branch that improves the memory footprint of classifiers.py around lines 250 a bit.
I avoided copying parts of `X_train` to a new data structure `X_train_id`, and instead simply index into `X_train` when passing the data to the `fit` method.

Thanks for the kind words!

Best,
Michael
Reply all
Reply to author
Forward
0 new messages