Latency in Web Cam preview using Javacv with JavaFX

[Rakesh Bhatt]

I am using Javacv to show user his preview in JavaFX. There is always delay in showing the preview i am confused either This latency is because of conversion of IplImage object to Image object of JavaFX. I am using below static class to initialize OpecvFrameGrabber object and grab the frame 

 public class InitializeCamera {
     
    /*
      Initilize the photo grabber 
    */
    static OpenCVFrameGrabber grabber;
    CvCapture capture;
    public static int CameraResolutionX;
    public static int CameraResolutionY;
    public static IplImage capturedStream=null;
    public static FrameRecorder recorder=null;
    public static int CameraId;
   
    
    static{
   
    CameraResolutionX=Integer.parseInt(ConfigXmlDTO1.getCameraResolutionX().trim());
    CameraResolutionY=Integer.parseInt(ConfigXmlDTO1.getCameraResolutiony().trim());
       
        CameraId=Integer.parseInt(ConfigXmlDTO1.getCameraId().trim());
    grabber=new OpenCVFrameGrabber(CameraId);
    grabber.setFrameRate(15);
   
           grabber.setImageHeight(CameraResolutionY);
           grabber.setImageWidth(CameraResolutionX);
           try {
    grabber.start();
   
    } catch (Exception e) {
   
    e.printStackTrace();
    }
    }
    public static int getGrabberImageWidth()
    {
    return grabber.getImageWidth();
    }
    public static int getGrabberImageHeight()
    {
    return grabber.getImageHeight();
    }
   
    public static IplImage getStream() {
   
    try {
    capturedStream=grabber.grab();
    } catch (Exception e) {
   
    e.printStackTrace();
    }
    return capturedStream.clone();
    }
   
    public static void flushGrbber()
    {
    try {
    grabber.flush();
   
    } catch (Exception e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
    }
    }
    public static void restartGrabber()
    {
    try {
    grabber.restart();
    } catch (Exception e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
    }
    }
    public static void stopGrabber(){
    try {
    
    grabber.flush();
    grabber.stop();
    grabber.release();
   
    } catch (Exception e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
    }
    }
    public static void releaseImage()
    {
    capturedStream.release();
    }
    public static void releaseGrabber()
    {
    try {
    grabber.release();
    } catch (Exception e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
    }
    }
    }

In UserPreviewFxmlController i am showing the preview of user in while loop 

    ObjectProperty<Image> imageProperty=new SimpleObjectProperty<Image>();
    @FXML ImageView camerapreview;
    
    public void showLiveStream() {
   
       carryImagePreview=true;
    Task<Void> task = new Task<Void>() {
   
    @Override
    protected Void call() throws Exception {
   
    while (carryImagePreview) {
    long start = System.nanoTime();
    try {
    if ((grabbedFrame = InitializeCamera.getStream()) != null) {
    final BufferedImage bimage = grabbedFrame
    .getBufferedImage();
    imgCnt++;
    long now = System.nanoTime();
    if (lastReportTime != -1
    && now - lastReportTime >= ONE_SECOND_IN_NANOS) {
    videoFps = ((double) imgCnt * ONE_SECOND_IN_NANOS)
    / (now - lastReportTime);
    imgCnt = 0;
    lastReportTime = now;
    } else if (lastReportTime == -1) {
    lastReportTime = now;
    }
    Platform.runLater(new Runnable() {
    @Override
    public void run() {
    final Image mainiamge = SwingFXUtils
    .toFXImage(bimage, null);
    imageProperty.set(mainiamge);
    }
    });
   
    bimage.flush();
   
    }
    } catch (Exception e) {
    } finally {
    long end = System.nanoTime();
    long waitTime = interval - (end - start);
    if (waitTime > 0) {
    LockSupport.parkNanos(waitTime);
    }
    }
    
    }
    
    InitializeCamera.releaseImage();
    grabbedFrame.release();
    grabbedFrame=null;
    return null; 
   
    }
    
    };
    Thread th = new Thread(task);
    th.setDaemon(true);
    th.start();
   
    camerapreview.imageProperty().bind(imageProperty);
    }

   

I had implemented same solution in Swing and achieved very nice performance with low CPU usage but in javaFx same procedure gives me huge delay in preview I am using Logitech 310 as external webcam 

Please help. Thanks in advance 

[Samuel Audet]

On 11/12/2013 02:32 PM, Rakesh Bhatt wrote:
> I had implemented same solution in Swing and achieved very nice
> performance with low CPU usage but in javaFx same procedure gives me
> huge delay in preview I am using Logitech 310 as external webcam
> Please help. Thanks in advance

I don't have a lot of experience with JavaFX myself, but have you tried
calling GraphicsContext.drawImage() on the Canvas?

Samuel

[Rakesh Bhatt]

Thanks for the answer, but using canvas the problem remains same still the web cam stream shown on canvas is very slow. I am still wondering for solution. I have Pentium Dual Core processor with 4 GB Ram. Using the same solution in Java Swing i have no problem of latency. 

[Samuel Audet]

On 11/27/2013 09:35 PM, Rakesh Bhatt wrote:
> Thanks for the answer, but using canvas the problem remains same still
> the web cam stream shown on canvas is very slow. I am still wondering
> for solution. I have Pentium Dual Core processor with 4 GB Ram. Using
> the same solution in Java Swing i have no problem of latency.

Well, let's see, this guy got it running fast on a Raspberry Pi, so I'm
pretty sure it should work fine on a Pentium:
http://www.vihrearobotti.com/other/javafx_raspberrypi
He has some code published there, so you could try it out and do the
same thing he does.

Samuel

[Rakesh Bhatt]

Thanks Samuel I had tried this solution but it is frustrating for me that it is  not working out. As you advice me to remove while loop and start animationTimer to do the same. I implemented it and  I checked its performance and i got fraction of lagging on my system itself but i am trying to run it in very lower configuration of Hardware  like Atom processor. It has become a challenge for me to show webcam preview with no latency which works very good on lower hardware configuration systems on JavaFX. I tried Timeline too to do the same but with no success. Till today JavaFX has become a nightmare for me. I am giving my sample code below which i had tried..

This is sample code of controller which is on clicking the start button shows the live stream from webcam : 

UpdateFrame method grab the current frame using InitializeCamera.getStream() method. I had shared code of InitializeCamera in my question. I am using OpencvFrameGrabber. 

@FXML StackPane webCamPreivewPanel;

ScreenControler controller;

Canvas _webCamCanvas;

AnimationTimer _cameraAnimation;

GraphicsContext _CanvasContext;

public void hanldeWebCamStartClick(MouseEvent e)

{

 

 Platform.runLater(new Runnable() {

@Override

public void run() {

createWebCamCanvas();

}

 });

 

}

private void createSwingComponet() {

_webCamCanvas=new Canvas(640,480);

_webCamCanvas.prefHeight(480);

_webCamCanvas.prefWidth(640);

_webCamCanvas.minHeight(480);

_webCamCanvas.minWidth(640);

_webCamCanvas.maxHeight(480);

_webCamCanvas.maxWidth(640);

_CanvasContext=_webCamCanvas.getGraphicsContext2D();

System.out.println("WEB CAM CANVAS UPDATED");

webCamPreivewPanel.getChildren().add(_webCamCanvas);

_cameraAnimation = new AnimationTimer() {

       @Override

           public void handle(long arg0) {

               updateFrame();

           }

       };

    _cameraAnimation.start();

}

private final void updateFrame(){

_CanvasContext.drawImage(SwingFXUtils.toFXImage(InitializeCamera

.getStream().getBufferedImage(), null), 0, 0);

}

public void hanldeWebCamStartClick(MouseEvent e)

{

 

_cameraAnimation.stop();

 

}

Thanks

On Tuesday, November 12, 2013 11:02:11 AM UTC+5:30, Rakesh Bhatt wrote:

[Samuel Audet]

On 12/18/2013 08:43 PM, Rakesh Bhatt wrote:
> Thanks Samuel I had tried this solution but it is frustrating for me
> that it is not working out. As you advice me to remove while loop and
> start animationTimer to do the same. I implemented it and I checked its
> performance and i got fraction of lagging on my system itself but i am
> trying to run it in very lower configuration of Hardware like Atom
> processor. It has become a challenge for me to show webcam preview with
> no latency which works very good on lower hardware configuration systems
> on JavaFX. I tried Timeline too to do the same but with no success. Till
> today JavaFX has become a nightmare for me. I am giving my sample code
> below which i had tried..

I'm afraid I do not have much experience with JavaFX, but do you need to
not use Swing? You could still use Swing for the "preview", and use
JavaFX for all the rest:
JavaFX for Swing Developers
http://docs.oracle.com/javafx/2/swing/jfxpub-swing.htm

Samuel

[Rakesh Bhatt]

Thank Samuel,

   I had tried it did some modification to my fxml file. The size of root Node(AnchorPane) is  very big : 1371(width) 758(height). I am displaying preview  in full screen and also resize controls at runtime to fit in full screen. This resizing is the major cause of delay in WebCam stream.

I reduced size of my parent node to 1091(width) and 634(height) and arranged other controls. This has significantly improved performance.

There are various ways to show live web cam feed.

using JavaFX ImageView with Animation Timer to show webcam preview rather than while loop 

using JavaFx Canvas with Animation Timer to show webcam preview rather than while loop 

using SwingNode contains JPanel having a JLabel on which you can show live stream in while loop

I found ImgeView is right option it provides you runtime resizing of WebCam preview.

I am still working on it found other options too but still the JavaFX seems to be new no direct solution are available 

Hope it helps some else 

Thanks Samuel