ch 6 Memory Error MCS.fit(X_train, y_train)

[mike nesius]

Hi Michael,

I'm getting a memory error when trying to run the chapter 6 code. I have 16gb of RAM...any suggestions on getting it to run?

Below is the traceback. Congrats on the recent post doc!

thanks,

Mike

>>>>

runfile('C:/Users/nesiusm/Documents/opencv-python-blueprints-master/chapter6/chapter6.py', wdir='C:/Users/nesiusm/Documents/opencv-python-blueprints-master/chapter6')

feature None

 - strategy one-vs-one

    - train

Traceback (most recent call last):

  File "<ipython-input-1-ddd8a3664d51>", line 1, in <module>

    runfile('C:/Users/nesiusm/Documents/opencv-python-blueprints-master/chapter6/chapter6.py', wdir='C:/Users/nesiusm/Documents/opencv-python-blueprints-master/chapter6')

  File "C:\Users\nesiusm\AppData\Local\Continuum\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 714, in runfile

    execfile(filename, namespace)

  File "C:\Users\nesiusm\AppData\Local\Continuum\Anaconda2\lib\site-packages\spyderlib\widgets\externalshell\sitecustomize.py", line 74, in execfile

    exec(compile(scripttext, filename, 'exec'), glob, loc)

  File "C:/Users/nesiusm/Documents/opencv-python-blueprints-master/chapter6/chapter6.py", line 86, in <module>

    main()

  File "C:/Users/nesiusm/Documents/opencv-python-blueprints-master/chapter6/chapter6.py", line 53, in main

    MCS.fit(X_train, y_train)

  File "classifiers.py", line 250, in fit

    X_train_id = X_train[data_id, :]

MemoryError

[Michael Beyeler]

Hi Mike,

This is a tricky one... Here are some comments and ideas, hopefully one of them will work for your system.

First, is it possible that you're running 32-bit Python? I read that this puts a practical limit of about 2 GB of RAM on your application, no matter how much RAM the system has:

http://stackoverflow.com/a/19963430

This would make much more sense, since I don't believe `X_train` is really that big. Of course with Python, it is often not straightforward to know how much memory is being used...

An easy fix could be to operate on a smaller image: set `small_size` to something smaller than (32, 32) on l.125 of gtsrb.py.

If you're using the code in a practical setting, you could just go with the HOG feature descriptor, which has the smallest number of features and also gives the best result. The other modes are merely there for educational purposes. :) After HOG, mode 'gray' is the next smallest. Avoid running 'SURF' mode, as it uses the most memory.

Then finally, I just pushed a quickfix to the master branch that improves the memory footprint of classifiers.py around lines 250 a bit.

I avoided copying parts of `X_train` to a new data structure `X_train_id`, and instead simply index into `X_train` when passing the data to the `fit` method.

Thanks for the kind words!

Best,

Michael