Python real time image classification problems with Caffe CNN

[Python_Caffe_Guy]

I'm attempting use caffe and python to do real-time image classification. I'm using OpenCV to stream from my webcam in one process, and in a separate process, using caffe to perform image classification on the frames pulled from the webcam. Then I'm passing the result of the classification back to the main thread to caption the webcam stream.

The problem is that even though I have an NVIDIA GPU and am performing the caffe predictions on the GPU, the main thread gets slown down. Normally without doing any predictions, my webcam stream runs at 30 fps; however, with the predictions, my webcam stream gets at best 15 fps.

I've verified that caffe is indeed using the GPU when performing the predictions, and that my GPU or GPU memory is not maxing out. I've also verified that my CPU cores are not getting maxed out at any point during the program. I'm wondering if I am doing something wrong or if there is no way to keep these 2 processes truly separate. Any advice is appreciated. Here is my code for reference

   

class Consumer(multiprocessing.Process):
   
        def __init__(self, task_queue, result_queue):
            multiprocessing.Process.__init__(self)
            self.task_queue = task_queue
            self.result_queue = result_queue
            #other initialization stuff
   
        def run(self):
            caffe.set_mode_gpu()
            caffe.set_device(0)
            #Load caffe net -- code omitted
            while True:
                image = self.task_queue.get()
                #crop image -- code omitted
                text = net.predict(image)
                self.result_queue.put(text)
   
            return
   
    import cv2
    import caffe
    import multiprocessing
    import Queue

    tasks = multiprocessing.Queue()
    results = multiprocessing.Queue()
    consumer = Consumer(tasks,results)
    consumer.start()

    #Creating window and starting video capturer from camera
    cv2.namedWindow("preview")
    vc = cv2.VideoCapture(0)
    #Try to get the first frame
    if vc.isOpened():
        rval, frame = vc.read()
    else:
        rval = False
    frame_copy[:] = frame
    while rval:
        if tasks.empty():
           tasks.put(frame_copy)
        else:
           text = tasks.get()
           #Add text to frame
           cv2.putText(frame,text)

        #Showing the frame with all the applied modifications
        cv2.imshow("preview", frame)

        #Getting next frame from camera
        rval, frame = vc.read()
        frame_copy[:] = frame
        #Getting keyboard input
        key = cv2.waitKey(1)
        #exit on ESC
        if key == 27:
            break

I am pretty sure it is the caffe prediction slowing everything down, because when I comment out the prediction and pass dummy text back and forth between the processes, I get 30 fps again.

   

class Consumer(multiprocessing.Process):
   
        def __init__(self, task_queue, result_queue):
            multiprocessing.Process.__init__(self)
            self.task_queue = task_queue
            self.result_queue = result_queue
            #other initialization stuff
   
        def run(self):
            caffe.set_mode_gpu()
            caffe.set_device(0)
            #Load caffe net -- code omitted
            while True:
                image = self.task_queue.get()
                #crop image -- code omitted
                #text = net.predict(image)
                text = "dummy text"
                self.result_queue.put(text)
   
            return
   
    import cv2
    import caffe
    import multiprocessing
    import Queue

    tasks = multiprocessing.Queue()
    results = multiprocessing.Queue()
    consumer = Consumer(tasks,results)
    consumer.start()

    #Creating window and starting video capturer from camera
    cv2.namedWindow("preview")
    vc = cv2.VideoCapture(0)
    #Try to get the first frame
    if vc.isOpened():
        rval, frame = vc.read()
    else:
        rval = False
    frame_copy[:] = frame

    while rval:
        if tasks.empty():
           tasks.put(frame_copy)
        else:
           text = tasks.get()
           #Add text to frame
           cv2.putText(frame,text)

        #Showing the frame with all the applied modifications
        cv2.imshow("preview", frame)

        #Getting next frame from camera
        rval, frame = vc.read()
        frame_copy[:] = frame
        #Getting keyboard input
        key = cv2.waitKey(1)
        #exit on ESC
        if key == 27:
            break

[Prabindh Sundareson]

What is the CPU used here, and what resolution of images are being used for testing ?

[Python_Caffe_Guy]

The CPU is an Intel® Core™ i7-6500U CPU @ 2.50GHz × 4.

I'm passing images of size 256x256.

[Python_Caffe_Guy]

Just wanted to bump this. There's a 300 rep bounty I put on stackoverflow is anyone is interested in looking: http://stackoverflow.com/questions/39522693/python-real-time-image-classification-problems-with-neural-networks