kivy with OpenCV and hopefully BlackMagic Design video input on Mac OSX

[Daan Hermans]

Hello all,

first of all, thanks for having Kivy around. I'm currently building an interactive installation on a large touch screen, and chose to go with Kivy since it's Python. Good to see there's this framework with so many platforms supported. 

What I'm building is going to be a "one-off" interactive installation, so I can afford to divert from supporting multiple hardware environments, and will probably need to for the following. 

Part of the installation is to get good (high quality) video input into the software, a bit better than a run of the mill webcam. From other video projects that we do, we're used to working with devices from Blackmagic Design.

Setup I currently have is an Apple MacMini with Mac OSX 10.10 (Yosemite) a bigger touch screen, and Kivy 1.8.0 . Attached to this is Blackmagic Design Ultrastudio Mini Recorder https://www.blackmagicdesign.com/products/ultrastudiothunderbolt , which provides HDMI or SDI video input through Thunderbolt. When I fire up the camera.py from examples/widgets I get. 

[DEBUG             ] [Camera      ] Ignored <avfoundation> (import error)
[INFO              ] [Camera      ] Provider: opencv(['camera_avfoundation'] ignored)
[INFO              ] [Text        ] Provider: pygame
QTKit didn't find any attached Video Input Devices!
Warning, camera failed to properly initialize!
Cleaned up camera.

So I tried the same on my MacBook Pro, and got the same result. Browsing this group I found I needed OpenCV, which I got in place, and hooked up to python/kivy via this helpful link https://jjyap.wordpress.com/2014/05/24/installing-opencv-2-4-9-on-mac-osx-with-python-support/ 

Now the Camera example works on my MacBook Pro, so that's one link in the chain. Also a "import cv" in the kivy environment now works without error.

Next thing I found this post about hooking up Blackmagic's SDK to OpenCV http://ndrewh.wordpress.com/2012/08/20/intensitypro-hdmi-input-to-opencv/ . 

The post mentions Qt. Is this the same framework as the error message above (QTKit didn't find any attache Video Input Devices!) ??

And if yes, how do I link this all up together? I think it should be something like:

- kivy uses OpenCV

- OpenCV talks to / embeds the BMD SDK input 

How do I test / start / try to do the second part. I have some C/C++ experience, but I'm not a blackbelt ninja in it. And how does kivy (Python) then link upto that part (C/C++)??

Any help or pointers appreciated.

[Daan Hermans]

Apologies for dubbel posting, I had missed the "your post will be published when somebody has a look at it" (or something similar) popup the first time. 

[Ben Hagen]

Hi Daan,

you can use GStreamer to get the frames from the Blackmagic Design Ultrastudio Mini Recorder.

You will need to:

brew install gstreamer gst-plugins-bad

you can find a sample of using GStreamer in opencv here:

https://gist.github.com/cbenhagen/76b24573fa63e7492fb6

Be sure to adjust the properties on lines 87 and 89. You can get all the available modes with "gst-inspect-1.0 decklinksrc".

This should get you started. OpenCV 3.0 will use GStreamer directly so things should get easier.

Best regards

Ben

[Ben Hagen]

Hi Daan,

actually things got a lot easier already :) GStreamer 1.0 support got backported into OpenCV 2.4.10 but is somehow blacklisted on OSX alltough it works. I am working on to get it in OpenCV and Homebrew. If you cant wait you can install it from my gist:

brew install --with-gstreamer https://gist.github.com/cbenhagen/0dc291b7c16c38806919/raw/d9adeaa422248e4f2164918b624c04a44bd49402/opencv.rb

and then

brew install gst-plugins-bad

You can find a small example of using input from a Blackmagic Decklink or Ultrastudio card in kivy here:

https://gist.github.com/cbenhagen/39a336e01e072617fa6f

Best regards

Ben

[Daan Hermans]

Hello Ben, thank you very much for these update's. I'm able to start messing around since today, back in the office et all. 

The "installation" that we build is now on the road with a webcam, but we're eager for improving the video result on it, so I'm at a test machine for the BMD input. 

If I'm not mistaking, I also see that OpenCV 2.4.10.1 is now the stable release via homebrew , so I should need to get your custom .rb version, do I? 

And then run the 

brew install gst-plugins-bad

after that. 

I suppose I should just try both, and see where it gets me. 

Thanks again.

[Ben Hagen]

Hi Daan,

my changes are already in homebrew so yes you can install opencv via:

brew install --with-gstreamer opencv

Id love to know more about your project. Is it Documented somewhere?

Best regards

Ben

[Daan Hermans]

Hi Ben,

thanks again, I've got it working pretty quick, and have working input from SDI now into Kivy. Really great!

The project isn't documented somewhere yet, and I probably should. It's a job where doing for a design agency, so I should probably also get there approval for sharing the source etc., but I'll convince them. 

The setup is a big touch screen (75") which is mounted vertically so it's really "man high" when you'r standing opposite it. It's sort of a photobooth experience where you select a custom background, a speech bubble, and then shoot a picture of yourself. Since where space limited (it's all installed in a driving truck ;-) ), I've build a LED light panel back wall that's controlled from an Arduino. The arduino is then controlled via Telnet from the kivy app to light up when the photo is shot. I then use imagemagick to key the person from that back wall, and use kivy again to composite it all together. 

The project is out there and running, but two things are needed to get better quality

√ BMD video input so we can use a better cam then the logitech.

- A better back wall so we don't get the bezels from the 60x60cm led panels that we're using now. 

So, next challenge, building our own bigger custom LED panel....

[Aesop Wolf]

Is this still blacklisted on OSX?

I'm on Mac 10.10.2 and ran:

$ brew install --with-gstreamer opencv

$ brew install gst-plugins-bad

Also, what am I supposed to pass into VideoCapture? I tried cv2.VideoCapture('decklinksrc mode=7 connection=0 ! videoconvert ! appsink') without any luck.

I'm using an UltraStudio Mini Recorder. As a side note, when I pass in just the number 0 it gives me my webcam.

Thanks!

[Daan Hermans]

I don't think it's still blacklisted anymore, since I've just installed it on another machine as well. I am running into a whole set of other stuff though. Like you, when I run it via the webcam it works with the line. 

self.capture = cv2.VideoCapture(0)

When I run it with the Decklink in their, also a Ultrastudio Mini Recorder (tested input in BMD Media Express, get a 1080p25 picture), it seems to crash on the point of 

 def update(self, dt):
     ret, frame = self.capture.read()

Fatal Python error: (pygame parachute) Segmentation Fault

Abort trap: 6

I've pinpointed it to that point by doing a schedule_once call before the schedule_interval that calls the update function. That way I see that I'm drawing a kivy window, all is setup and up for 5 seconds, and then crashes on the Segmentation Fault. As for the pointers in the kivy FAQ http://kivy.org/docs/faq.html , these don't fix it.

I've tried taking this apart from kivy into python, so I'm running 

$ python cvtest.py

import numpy as np
import cv2


cap = cv2.VideoCapture('decklinksrc mode=7 connection=0 ! videoconvert ! appsink')


while(True):
    # Capture frame-by-frame
    ret, frame = cap.read()


    # Our operations on the frame come here
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)


    # Display the resulting frame
    cv2.imshow('frame',gray)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break


# When everything done, release the capture
cap.release()
cv2.destroyAllWindows()

That works! I'm then getting the BMD input, and it's been made gray values. So somehow pygame isn't happy with opencv or gstreamer anymore? 

Anybody any pointers for this? 

Cheers -- Daan

[Daan Hermans]

OK, so I think I got things working again. I started experimenting with my little cvtest.py script from kivy. If i ran $ kivy cvtest.py it would also give a segmentation error. So I looked at what exectly was in the kivy script in MacOS. Apparently it contains some internal gstreamer binaries as well, which either conflict with the homebrew ones, or just aren't as complete as the ones homebrew installs. I suppose the Kivy.app contained ones don't reference to the gst-pluginns-bad ones, that take the BMD input. So my script file in /Applications/Kivy.app/Contents/Resources now has the GST stuff in their commented out: 

#!/bin/bash
SCRIPT_PATH="${BASH_SOURCE[0]}";
if([ -h "${SCRIPT_PATH}" ]) then
  while([ -h "${SCRIPT_PATH}" ]) do SCRIPT_PATH=`readlink "${SCRIPT_PATH}"`; done
fi
SCRIPT_PATH=`dirname ${SCRIPT_PATH}`


export PYTHONPATH=${SCRIPT_PATH}/kivy:${SCRIPT_PATH}/lib/sitepackages:$PYTHONPATH
export DYLD_FALLBACK_LIBRARY_PATH=${SCRIPT_PATH}/lib:$DYLD_FALLBACK_LIBRARY_PATH
export LD_PRELOAD_PATH=${SCRIPT_PATH}/lib:$LD_PRELOAD_PATH
##export GST_PLUGIN_PATH=${SCRIPT_PATH}/lib/gst-plugins:$GST_PLUGIN_PATH
##export GST_PLUGIN_SCANNER=${SCRIPT_PATH}/lib/bin/gst-plugin-scanner
##export GST_REGISTRY_FORK="no"


exec $(python -c "import os, sys; print os.path.normpath(sys.prefix)")/bin/python2.7 "$@"

I can now both run $ kivy cvtest.py as shown above, and also have the camapp.py example from Ben running again. 

[Human#68]

Hello Everyone,

I'm tried to run cvtest.py and it could not open my decklink card. Because I'm new are OpenCV2, I have a few easy questions that I am hoping that someone can help me with. I'm running Ubuntu on an Intel system with one Decklink Duo 2 card.

I can see the devices: /dev/blackmagic/iox and Blackmagic's MediaExpress can display the video from them. When I run $ python3 cvtest.py, VideoCapture fails to open it.

import cv2
import numpy as np

cap = cv2.VideoCapture('decklinksrc mode=7 connection=0 ! videoconvert ! appsink')

if cap.isOpened() == 0:
    print ("No camera found ")

while True:
    ret, frame = cap.read()

    cv2.namedWindow('my window', cv2.WINDOW_NORMAL)
    cv2.resizeWindow('my window', 960,540)

    cv2.imshow("my window", frame)    
    keyCode = cv2.waitKey(30) & 0xff
    if keyCode ==27:
        break

cap.release()
cv2.destroyAllWindows()

Here is the result:

$ python3 cvtest.py
No camera found
Traceback (most recent call last):
  File "cvtest.py", line 24, in <module>
    cv2.imshow("my window", frame)    
cv2.error: OpenCV(4.1.2) /io/opencv/modules/highgui/src/window.cpp:376: error: (-215:Assertion failed) size.width>0 && size.height>0 in function 'imshow'

- I install the kivy libraries, but in this case I don't see how cvtest.py using them. How does cv2.VideoCapture use decklinksrc?

- Where do I get decklinksrc?

- Where can I find the attributes that decklinksrc needs? Can't seem to find where opencv.org defines VideoCapture() using this structure.

Thanks for the help.