libfreenect dropping depth frame when I block main

[Troy Boswell]

I am new to concurrent processing but I am wondering if someone can explain why libfreenect drops frames if I put a blocking function in main(). A good example is the opencv example, if cvwaitkey blocks for more than 5ms then libfreenect will drop frames. Thus you surely cannot use this function to block waiting for a key press (cvwaitkey(0)) or libfreenect will drops frames like crazy. I suspect it has something to do with usb timing. Maybe the callback blocks if main is block, any insight would be helpful. I am only concerned as I only need to periodically capture frames, if I could block main() for a period of time and let libfreenect do its thing it would reduce my cpu usage as I currently have to sit main in a while loop simply burning cpu time.

[drew.m...@gmail.com]

On Sun, Apr 22, 2012 at 07:12, Troy Boswell <troy.b...@gmail.com> wrote:
> I am new to concurrent processing but I am wondering if someone can explain why libfreenect drops frames if I put a blocking function in main(). A good example is the opencv example, if cvwaitkey blocks for more than 5ms then libfreenect will drop frames. Thus you surely cannot use this function to block waiting for a key press (cvwaitkey(0)) or libfreenect will drops frames like crazy. I suspect it has something to do with usb timing. Maybe the callback blocks if main is block, any insight would be helpful. I am only concerned as I only need to periodically capture frames, if I could block main() for a period of time and let libfreenect do its thing it would reduce my cpu usage as I currently have to sit main in a while loop simply burning cpu time.

libfreenect never runs or handles any packets unless you're actively
calling freenect_process_events(). It does not internally fork a
thread. So if you don't give it CPU time (by calling
freenect_process_events()) before the kernel fills up all the
isochronous buffers we've queued to store the data streaming from the
Kinect, then data will be lost, which will likely result in framedrop.

The usual solution is to spawn a thread which calls
freenect_process_events(), which is how the freenect_sync wrapper
works internally. Since you mentioned opencv, you might also be
interested in the libfreenect opencv wrapper, which wraps the
freenect_sync wrapper - take a look at wrappers/opencv/cvdemo.c for
example usage. With that API, you should be able to simply call
freenect_sync_get_depth_cv whenever you want to capture a frame - the
first time it is called, it will launch a separate thread running a
libfreenect mainloop.

Hope that helps explain things!

-Drew

[Troy Boswell]

Thank you very much, makes perfect sense.

Can I get access to freenect_process_event from my main() so I can thread from my program, our do I need to thread it within libfreenect.

Troy

[Troy Boswell]

Sorry drew

I should have looked in the header first. If I thread from main() is there any hidden issues?

Troy

[Troy Boswell]

I still have an issue with libfreenect dropping frames which results in sporadic response times of the kinect frames.

I have stopped using the C++ wrapper and I am using the glview example, I have tweeked it for opencv so that I had two separate threads, the main and freenect thread. However if I still block the main thread libfreenect drops frames like crazy. The problem is independent of opencv as I commented out all opencv calls and used getchar() to block main. I have also modified the freenect thread priority level to give it a much greater priority (i tried 20, 30 ,40 , 50) to call freenect_process_events(), however still no luck.

I know that my code works, because if main runs without any blocking frames aren't dropped and the response is very good, but it comes at the cost of running at about 80-90% cpu utilization which is not ideal.

What I wish to do is block main waiting for a ethernet packet and then process the received frames. To process these frames I have written my functionality using opencv.

I don't know the protocol for uploading code, but below are my functions if anyone is interested, I am thinking of giving the sync version a try, but I would ideally like to have full control:
The code is broken down as
 -- 2xCallbacks
 -- Kinect Thread
 -- Main
 -- 2xget frame functions
I haven't provided my global variables but I can, just thought the post would be long enough as it is.

Thankyou Troy

___________________________________________________________

// thread callbacks
void rgbCallback(freenect_device *dev, void *v_rgb, uint32_t timestamp){
              // cout << "RGB callback" << std::endl;
                pthread_mutex_lock(&rgbMutex);
                videoTimestamp = timestamp;
                rgb = (uint8_t*)v_rgb;
                new_rgb_frame = true;
                pthread_mutex_unlock(&rgbMutex);
}

void depthCallback(freenect_device *dev, void *v_depth, uint32_t timestamp){
             //  cout << "Depth callback" << std::endl;
                pthread_mutex_lock(&depthMutex);
                depthTimestamp = timestamp;
                depth = (uint16_t*)v_depth;
                new_depth_frame = true;
                pthread_mutex_unlock(&depthMutex);
}

void *kinectThreadFunction(void *arg){

        freenect_set_led(kinectDevice,LED_GREEN);//LED_RED);

        freenect_set_video_callback(kinectDevice,rgbCallback);
        freenect_set_depth_callback(kinectDevice,depthCallback);
        freenect_set_video_mode(kinectDevice,freenect_find_video_mode(FREENECT_RESOLUTION_MEDIUM,FREENECT_VIDEO_RGB));
        freenect_set_depth_mode(kinectDevice,freenect_find_depth_mode(FREENECT_RESOLUTION_MEDIUM,FREENECT_DEPTH_11BIT));

        freenect_start_video(kinectDevice);
        freenect_start_depth(kinectDevice);

        while(!die&& (freenect_process_events(kinectContext) >= 0)){
        }

        cout << "kinect shutting down" << endl;
        freenect_stop_video(kinectDevice);
        freenect_stop_depth(kinectDevice);
        freenect_close_device(kinectDevice);
        freenect_shutdown(kinectContext);
        cout << "kinect shutdown" << endl;
        die = true;
}

int main() {
        int RVAL;
        char key;

        depthImg = cvCreateImage(cvSize(640,480),16,1);
        rgbImg = cvCreateImage(cvSize(640,480),IPL_DEPTH_8U,3);
        IplImage *rbgFrame = cvCreateImage(cvSize(640,480),IPL_DEPTH_8U,3);
        IplImage *depthFrame = cvCreateImage(cvSize(640,480),16,1);

        rgb = (uint8_t*)malloc(640*480*3);
        depth = (uint16_t*)malloc(640*480);

        if(freenect_init(&kinectContext,NULL) < 0){
                cout<<"freenect initialisation failed!!"<<endl;
                return 1;
        }

       // freenect_set_log_level(kinectContext, FREENECT_LOG_DEBUG);
        freenect_select_subdevices(kinectContext,(freenect_device_flags)(FREENECT_DEVICE_CAMERA|FREENECT_DEVICE_MOTOR));

        if( freenect_num_devices(kinectContext) < 1){
                cout<<"No Device Present!!" << endl;
                return 1;
        }

        if(freenect_open_device(kinectContext,&kinectDevice,0) < 0){
                cout<<"Could Not Open Device!!"<< endl;
        }

        RVAL = pthread_create(&kinectThread,&kinectThreadAttr,kinectThreadFunction,NULL);
        if(RVAL){
                cout<<"pthread create failed"<<endl;
                freenect_shutdown(kinectContext);
                return 1;
        }
        uint32_t vid,dep;
        while(!die){
             cout<<"RGB ID = "<< vid <<"DEPTH ID = "<< dep<<endl;
                getVideo(rbgFrame,&vid);
                getDepth(depthFrame,&dep);

                cvShowImage("RGB", rbgFrame);
                cvShowImage("DEPTH", depthFrame);

                cvWaitKey(1);
                key = getchar();
                if(key == 27){
                        die = true;
                }

        }
        pthread_join(kinectThread,NULL); // wait for kinect thread to die
        cout << "Exiting"<< endl;
        return 0;
}



bool getVideo(IplImage *output,uint32_t *frameID) {
        pthread_mutex_lock(&rgbMutex);
        if(new_rgb_frame) {
                cvSetData(rgbImg, rgb, 640*3);
                cvCvtColor(rgbImg, output, CV_RGB2BGR);
                cvCopy(rgbImg,output);
                *frameID = videoTimestamp;
                new_rgb_frame = false;
                pthread_mutex_unlock(&rgbMutex);
                return true;
        } else {
                pthread_mutex_unlock(&rgbMutex);
                return false;
        }
}

bool getDepth(IplImage *output,uint32_t *frameID) {
        pthread_mutex_lock(&depthMutex);
        if(new_depth_frame) {
                cvSetData(depthImg, depth, depthImg->widthStep);
                cvCopy(depthImg,output);
                *frameID = depthTimestamp;
                new_depth_frame = false;
                pthread_mutex_unlock(&depthMutex);
                return true;
        } else {
                pthread_mutex_unlock(&depthMutex);
                return false;
        }
}

[ErikH2000]

Troy,

I would plan from the beginning to have the depth/rgb callbacks do nothing else except grab the frames and put them into a buffer read by a different thread that does the actual processing work. I wrote a thread-synchronized triple buffer for this, and now I never see dropped frames or "magic number" errors. The callbacks consistently grab frames at 30fps this way. Even though there is a little more CPU use in managing the triple buffer, the performance is substantially better. Basically, if you miss any USB packets, the whole message for frame data that contained that packet won't be sent to the callback. So your frame rate can drop drastically even when you are doing just a bit of processing in the callbacks.

-Erik

[Troy Boswell]

I don't believe I am doing to much work in the callback. I changed my callback to only read the kinect's depth and rgb frames. However the problem is identical. I have even rewritten the program so that the libfreenect process_events call sits in main spinning in a while loop and I create a thread to do the processing. Again if I block the thread which is independent of main and process_events libfreenect still drops frames. I will look into the triple buffering, is this similar to the buffering that occurs in glview.

[ErikH2000]

So you had one thread that calls process_events in a loop, and then you had another loop in the main thread? And when you put a call to *something that blocks*, then you start seeing frame drops?

If I have that right, then I can think of a few scenarios:

* The thing that blocks is locking a mutex or is involved with some other shared resource of the other thread.

* Your environment is CPU-starved.

What is the blocking action?  cvwaitkey?

I can't remember if glview was using triple buffering. The main idea is that one thread reads from a buffer and another can simultaneously write to a different buffer. The reading thread request a buffer to read and always gets the most recently written buffer, and locks it for the duration of the read. The writing thread requests a buffer to write, and gets a buffer that is not the one it previously wrote to and is not currently locked for reading. If you have three buffers, then there will always be a buffer available for writing thread. Thread synch is needed to lock, unlock, and check for locks.

-Erik

[Troy Boswell]

Erik

You have it correct, Except I believe that your scenarios do not correctly describe the problem. The libfreenect callbacks do not share resources at this point in time as I am just testing thus its not a mutex problem (sounds strange but I'm not outputting the images). I don't think the CPU is starved as system statistics says the CPU usage is at about 40% when I block the main thread (frames are dropped), however when main isn't blocked cpu usage is about 90% but libfreenect doesn't drop frames.

I use getchar() to block main.

I have uploaded a reduced version of glview that proves my problem, in main I have getchar() in a while loop, simple comment or uncomment getchar(). 

Thanks for the heads up about triple buffering, it is a pretty good technique. 
 
Troy

[ErikH2000]

Troy,

Well, I built the code and ran it. Yes, it gives me the same dropped frame errors. No quick answers for you, but if I comment out all the code in the two callbacks and also the getchar() statement, it will still show the dropped frames. So probably the problem has nothing to do with getchar().

That's probably as far as I want to dig into it. Maybe someone else will have more direct experience with the same problem. Or you might tshoot and narrow it down further.

-Erik

[Troy Boswell]

Erik

Thank you for your help, Its good to know its not just me.

I am going to ask my Real Time System Lecturer at uni about it, so hopefully he can give some answers.

Troy