Re: bob.rppg.base

[Guillaume Heusch]

Hi Javad,

Thanks for your interest in our work. First of all, this discussion group is the main entry point for getting help on bob, and there is no need to write to individuals in the hope of a faster reply.

Regarding your question, the code is indeed made to run on databases, but you can easily modify / reuse it to perform rPPG from a live feed. Assuming you have code to grab video frames from a camera, you should start by accumulating a reasonable amount of frames (let's say at least 2-3 seconds) and then you can apply any of the algorithms of the package to these frames. If you look carefully at the different scripts to compute the pulse signal (such as bob/rppg/ssr/script/spatial_subspace_rotation.py), you will see that they all have a for loop on frames, where the processing to retrieve the pulse signal is done.

Hope this helps,

Best regards

Guillaume

Le lundi 9 avril 2018 14:37:52 UTC+2, Sayyed Javad Ziaratnia a écrit :

Guys please help me!

[masakik...@gmail.com]

Dear Guillaume 

I have similar questions, which the first one is how can I feed it with the frames which I am getting form this opencv code?

My second question is that our goal is to get the heart rate-real time because we want to sync it with EEG, is it possible to make the heart rate real-time?

my third question is that there is a bob face detection and because I do not have a video database and i want to feed it with camera, i could not check it. so my questions is because our experiment condition is not fixed and there are possibility of moving the head during the experiment, is there any problem with moving? 

Thank you very much

Kikuchi

spatial_subspace_rotation.py

import os import sys import pkg_resources import logging __logging_format__='[%(levelname)s] %(message)s' logging.basicConfig(format=__logging_format__) logger = logging.getLogger("ssr_logger") from docopt import docopt version = pkg_resources.require('bob.rppg.base')[0].version import numpy import bob.io.video import bob.ip.color from ...base.utils import crop_face from ..ssr_utils import get_skin_pixels from ..ssr_utils import get_eigen from ..ssr_utils import plot_eigenvectors from ..ssr_utils import build_P def main(user_input=None): # Parse the command-line arguments if user_input is not None: arguments = user_input else: arguments = sys.argv[1:] prog = os.path.basename(sys.argv[0]) completions = dict( prog=prog, version=version, ) args = docopt( __doc__ % completions, argv=arguments, version='Spatial subspace rotation for videos (%s)' % version, ) # if the user wants more verbosity, lowers the logging level if args['--verbose'] == 1: logging.getLogger("ssr_logger").setLevel(logging.INFO) elif args['--verbose'] >= 2: logging.getLogger("ssr_logger").setLevel(logging.DEBUG) # chooses the database driver to use if args['cohface']: import bob.db.cohface if os.path.isdir(bob.db.cohface.DATABASE_LOCATION): logger.debug("Using Idiap default location for the DB") dbdir = bob.db.cohface.DATABASE_LOCATION elif args['--dbdir'] is not None: logger.debug("Using provided location for the DB") dbdir = args['--dbdir'] else: logger.warn("Could not find the database directory, please provide one") sys.exit() db = bob.db.cohface.Database(dbdir) if not((args['--protocol'] == 'all') or (args['--protocol'] == 'clean') or (args['--protocol'] == 'natural')): logger.warning("Protocol should be either 'clean', 'natural' or 'all' (and not {0})".format(args['--protocol'])) sys.exit() objects = db.objects(args['--protocol'], args['--subset']) elif args['hci']: import bob.db.hci_tagging import bob.db.hci_tagging.driver if os.path.isdir(bob.db.hci_tagging.driver.DATABASE_LOCATION): logger.debug("Using Idiap default location for the DB") dbdir = bob.db.hci_tagging.driver.DATABASE_LOCATION elif args['--dbdir'] is not None: logger.debug("Using provided location for the DB") dbdir = args['--dbdir'] else: logger.warn("Could not find the database directory, please provide one") sys.exit() db = bob.db.hci_tagging.Database() if not((args['--protocol'] == 'all') or (args['--protocol'] == 'cvpr14')): logger.warning("Protocol should be either 'all' or 'cvpr14' (and not {0})".format(args['--protocol'])) sys.exit() objects = db.objects(args['--protocol'], args['--subset']) # tells the number of grid objects, and exit if args['--gridcount']: print len(objects) sys.exit() # if we are on a grid environment, just find what I have to process. if os.environ.has_key('SGE_TASK_ID'): pos = int(os.environ['SGE_TASK_ID']) - 1 if pos >= len(objects): raise RuntimeError, "Grid request for job %d on a setup with %d jobs" % \ (pos, len(objects)) objects = [objects[pos]] # does the actual work for obj in objects: # expected output file output = obj.make_path(args['--outdir'], '.hdf5') # if output exists and not overwriting, skip this file if os.path.exists(output) and not args['--overwrite']: logger.info("Skipping output file `%s': already exists, use --overwrite to force an overwrite", output) continue # load the video sequence into a reader video = obj.load_video(dbdir) logger.info("Processing input video from `%s'...", video.filename) # indices where to start and to end the processing logger.debug("Sequence length = {0}".format(len(video))) start_index = int(args['--start']) end_index = int(args['--end']) if (end_index == 0): end_index = len(video) if end_index > len(video): logger.warn("Skipping Sequence {0} : not long enough ({1})".format(obj.path, len(video))) continue # truncate the signals if needed nb_final_frames = end_index - start_index logger.debug("Processing %d frames...", nb_final_frames) # load the result of face detection bounding_boxes = obj.load_face_detection() # the temporal stride temporal_stride = int(args['--stride']) # the result -> the pulse signal output_data = numpy.zeros(nb_final_frames, dtype='float64') # store the eigenvalues and the eigenvectors at each frame eigenvalues = numpy.zeros((3, nb_final_frames), dtype='float64') eigenvectors = numpy.zeros((3, 3, nb_final_frames), dtype='float64') ################ ### LET'S GO ### ################ counter = 0 for i, frame in enumerate(video): if i >= start_index and i < end_index: logger.debug("Processing frame %d/%d...", i, nb_final_frames) # get skin colored pixels try: if counter == 0: # init skin parameters in any cases if it's the first frame skin_pixels = get_skin_pixels(frame, i, True, float(args['--threshold']), bounding_boxes) else: skin_pixels = get_skin_pixels(frame, i, bool(args['--skininit']), float(args['--threshold']), bounding_boxes) except NameError: if counter == 0: skin_pixels = get_skin_pixels(frame, i, bool(args['--skininit']), float(args['--threshold'])) else: skin_pixels = get_skin_pixels(frame, i, bool(args['--skininit']), float(args['--threshold'])) logger.debug("There are {0} skin pixels in this frame".format(skin_pixels.shape[1])) # no skin pixels detected, generally due to no face detection # go back in time to find a face, and use this bbox to retrieve skin pixels in current frame if skin_pixels.shape[1] == 0: logger.warn("No skin pixels detected in frame {0}".format(i)) k = 1 while skin_pixels.shape[1] <= 0: try: skin_pixels = get_skin_pixels(video[i-k], (i-k), bool(args['--skininit']), float(args['--threshold']), bounding_boxes, skin_frame=frame) except NameError: skin_pixels = get_skin_pixels(video[i-k], (i-k), bool(args['--skininit']), float(args['--threshold']), skin_frame=frame) k += 1 logger.warn("got skin pixels in frame {0}".format(i-k)) # build c matrix and get eigenvectors and eigenvalues eigenvalues[:, counter], eigenvectors[:, :, counter] = get_eigen(skin_pixels) # plot the cluster of skin pixels and eigenvectors (see Figure 1) if bool(args['--plot']) and args['--verbose'] >= 2: plot_eigenvectors(skin_pixels, eigenvectors[:, :, counter]) # build P and add it to the pulse signal if counter >= temporal_stride: tau = counter - temporal_stride p = build_P(counter, int(args['--stride']), eigenvectors, eigenvalues) output_data[tau:counter] += (p - numpy.mean(p)) counter += 1 elif i > end_index : break # plot the pulse signal if bool(args['--plot']): import matplotlib.pyplot as plt fig = plt.figure() ax = fig.add_subplot(111) ax.plot(range(nb_final_frames), output_data) plt.show() # saves the data into an HDF5 file with a '.hdf5' extension outdir = os.path.dirname(output) if not os.path.exists(outdir): bob.io.base.create_directories_safe(outdir) bob.io.base.save(output_data, output) logger.info("Output file saved to `%s'...", output) return 0

OPENCV

import numpy as np import cv2 cap = cv2.VideoCapture(0) while(True): # Capture frame-by-frame ret, frame = cap.read() # Display the resulting frame cv2.imshow('frame',frame) if cv2.waitKey(1) & 0xFF == ord('q'): break # When everything done, release the capture cap.release() cv2.destroyAllWindows()

[Guillaume Heusch]

Dear Kikuchi,

Thanks for your interest in our work.

To answer your first question, you should first accumulate some frames (in a 4d numpy array for instance) before passing them to the processing done in the "for loop" in the script.  

Considering the real time issue, you cannot really get the instantaneous heart-rate, since again, you need several frames to retrieve the frequency of the pulse signal. What you can do however, is to accumulate frames, and recompute the heart rate every new x frames, and you will have an estimate of the heart rate in the previous x/fps seconds.

For face detection, it does not depend on whether the video comes from a database or not. Actually, it is done in each frame that you will process. Having some movements (to some reasonable extent) should not be a problem.

I hope I was clear in my answers, let me know if this is not the case.

Best,

Guillaume

[masakik...@gmail.com]

Dear Guillaume

As you said i tried to make it real time but there are two problems which the first one is that bob face detection is very slow so instead of it i used opencv to fix the speed problem and the second problem is that it is not accurate. So do you have any idea for making it more accurate?

Regards

Kikuchi

import bob.io.base
import bob.io.image
import bob.ip.facedetect
import bob.ip.draw
import bob.io.video
import bob.ip.color
from bob.rppg.base.utils import crop_face
from ssr_utils import get_skin_pixels
from ssr_utils import get_eigen
from ssr_utils import plot_eigenvectors
from ssr_utils import build_P
from docopt import docopt
import cv2
import numpy as np
import re
import sys
import threading
import time

import os
import sys
import pkg_resources

from scipy.signal import welch
import matplotlib
matplotlib.use("tkagg")
import matplotlib.pyplot as plt
buffer={}
timebuffer={}
writeindex=0
buffersize = 1000
def worker( ):
   faceDet = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')
   global writeindex
   global buffer
   global buffersize
   global timebuffer
   readindex = 0
   temporal_stride = 20
   nframeproc = 60
   skininit = False
   threshold = 0.5
   plot = True
   verbose = 1
   nfft = 128
   nsegments = 12
   segment_length = (2 * 256) // (nsegments + 1)
   count = 0
   counter = 0
   sbuffer={}
   readindex = writeindex
   XX = np.zeros(nframeproc, dtype='float64')
   output_data = np.zeros(nframeproc, dtype='float64')
   plt.ion()

   fig = plt.figure()
   ax = fig.add_subplot(111)

   line1, = ax.plot(XX, output_data, 'b-')
   fig.canvas.draw()
   while True:
       while not writeindex == readindex and count<nframeproc:
           sbuffer[count]=buffer[readindex].copy()#.transpose((2,0,1))
           XX[count]=timebuffer[readindex]
           readindex = (readindex + 1) % buffersize
           count+=1
       if count==nframeproc:
           count=0
           bounding_box={}
           for i in range(0,nframeproc):
               gray = cv2.cvtColor(sbuffer[i], cv2.COLOR_BGR2GRAY)
               face = faceDet.detectMultiScale(gray, scaleFactor=1.1, minNeighbors=10, minSize=(5, 5),flags=cv2.CASCADE_SCALE_IMAGE)
               if  len(face)>0:
                    x,y,w,h=face[0]
                    bounding_box[i]=bob.ip.facedetect.BoundingBox([y,x],[h,w])
                    print(i, "opencv face detector", bounding_box[i])
               else:
                    bounding_box[i], _ = bob.ip.facedetect.detect_single_face(sbuffer[i].transpose((2,0,1)))
                    print(i, " bob face detector", bounding_box[i])

           # the result -> the pulse signal

           # store the eigenvalues and the eigenvectors at each frame

           eigenvalues = np.zeros((3, nframeproc), dtype='float64')
           eigenvectors = np.zeros((3, 3, nframeproc), dtype='float64')
           for i in range(0,nframeproc):
               print(i, " th frame proceed")
               frame=sbuffer[i].transpose((2,0,1))
               try:
                  if i == 0:

                       # init skin parameters in any cases if it's the first frame

                       skin_pixels = get_skin_pixels(frame, i,True, threshold, bounding_box)
                  else:
                       skin_pixels = get_skin_pixels(frame, i,skininit, threshold,bounding_box)
               except NameError:
                  if i == 0:
                       skin_pixels = get_skin_pixels(frame, i,skininit, threshold)
                  else:
                       skin_pixels = get_skin_pixels(frame, i,skininit, threshold)

               # no skin pixels detected, generally due to no face detection
               # go back in time to find a face, and use this bbox to retrieve skin pixels in current frame
               if skin_pixels.shape[1] == 0:

                  #logger.warning("No skin pixels detected in frame {0}".format(i))

                  k = 1
                  while skin_pixels.shape[1] <= 0:
                     try:

                        skin_pixels = get_skin_pixels(sbuffer[i - k], (i - k),skininit, threshold,bounding_box, skin_frame=frame)
                     except NameError:
                        skin_pixels = get_skin_pixels(sbuffer[i - k], (i - k),skininit, threshold,skin_frame=frame)
                     k += 1
                  #logger.warning("got skin pixels in frame {0}".format(i - k))

               # build c matrix and get eigenvectors and eigenvalues

               eigenvalues[:, i], eigenvectors[:, :, i] = get_eigen(skin_pixels)

               # plot the cluster of skin pixels and eigenvectors (see Figure 1)

               if plot and verbose >= 2:
                   plot_eigenvectors(skin_pixels, eigenvectors[:, :, i])

               # build P and add it to the pulse signal

               if i >= temporal_stride:
                   tau = i - temporal_stride
                   p = build_P(i, temporal_stride, eigenvectors, eigenvalues)
                   output_data[tau:i] += (p - np.mean(p))
           frate=nframeproc*1.0/(XX.max()-XX.min())
           green_f, green_psd = welch(output_data, frate, nperseg=segment_length, nfft=nfft)
           first = np.where(green_f > 0.7)[0]
           last = np.where(green_f < 4)[0]
           first_index = first[0]
           last_index = last[-1]
           range_of_interest = range(first_index, last_index + 1, 1)
           max_idx = np.argmax(green_psd[range_of_interest])
           f_max = green_f[range_of_interest[max_idx]]
           hr = f_max * 60.0
           print("Heart Rate: ",hr)

           # plot the pulse signal

           if plot:
               line1.set_ydata(output_data)
               line1.set_xdata(XX)
               plt.axis([XX.min(), XX.max(), output_data.min() , output_data.max()])
               plt.title("Heart Rate: "+str(hr))
               #fig.canvas.draw()
               #plt.cla()
               #ax.plot(range(nframeproc), output_data)
               #plt.draw()
               readindex = writeindex
       else:
           time.sleep(0.1)
if __name__ == '__main__':
    #faceDet = cv2.CascadeClassifier('haarcascade_frontalface_default.xml')
    cap = cv2.VideoCapture(0)
    ret, frame = cap.read()
    t = threading.Thread(target=worker)
    t.start()
    start=time.time()
    while (cap.isOpened()):
        ret, frame = cap.read()
        buffer[writeindex]=frame.copy()
        timebuffer[writeindex]=time.time()-start
        writeindex=(writeindex+1)%buffersize
        cv2.imshow("hh",frame)

        if cv2.waitKey(1) & 0xFF == ord('q'):
            break
# When everything done, release the capture
cap.release()
cv2.destroyAllWindows()

[Guillaume Heusch]

Dear Kikuchi,

Thanks for your feedback. Indeed, the face detector from bob.ip.facedetect is not that fast. You can vary parameters to try and make it faster, just have a look at the documentation of the package (but I did not do it myself, so I don't have any advice here)

Alternatively, we have two other bob packages performing face detection (one of which is based on dlib). You may want to have a look here: 

https://gitlab.idiap.ch/bob/bob.ip.dlib

https://gitlab.idiap.ch/bob/bob.ip.mtcnn

Best,

Guillaume

[aric...@gmail.com]

Dear Kikuchi，

I run your code as you provided on the mailing-list, and the heart rate result is wrong(when I face to the camera and stay still, the HR is always higher than 200, and sometimes at 40 around). So I want to know whether the result you got is similar to mine or not,  and the other question is that I delete the thread, and make it run in one thread, because if it runs with two threads(main and the worker), the worker always occupies CPU, and stacks into the 'while not ' and 'if count == nframe',  even if I use the 'sleep', main thread can't get the CPU. So how can you get it running?  I know that is a program language basic problem, but I can't find the solution on the internet, hope you can help me out.

Regards,

Wang Yong

在 2018年4月24日星期二 UTC+8下午2:48:24，masakik...@gmail.com写道：