making streaming anaglyphs using OpenCV for Python

[pezman]

I was playing around making some image processing sample code for folks in the Drexel DesignFutures program and decided to make anaglyphs that are viewable using red/blue glasses.

This code was running on a PCDuino, so it was a little tough figuring out how to make it work fast enough for real-time video, and most examples that I found on the net were fairly slow. At any rate, since it took so much tinkering to figure out how to do this "right" in Python, I figured I would share.

To make the code run, you need a system with two cameras (I used some cameras that I bought at RadioShack about two years ago .. $10 each, and I cleaned out every local store I could find).

Then you need to install opencv-python and scipy (easy enough to look that one up).

You need to set the cameras up properly for this to work. They should be level with each other, eye-width apart and adjusted so that vertical objects run pretty much straight up the screen.

The following code will grab video and show it on the screen such that red/blue glasses will see the scene in 3d. The effect is surprisingly good, and the scene even has muted colors, so it is not half-bad, given the ease of implementation.


import numpy as np
import cv2.cv as cv
import cv2 as cv2


# declare various color blending algorithms to mix te pixels
# from different perspectives so that red/blue lens glasses
# make the image look 3D
mixMatrices = {
'true': [ [ 0.299, 0.587, 0.114, 0, 0, 0, 0, 0, 0 ], [ 0, 0, 0, 0, 0, 0, 0.299, 0.587, 0.114 ] ],
'mono': [ [ 0.299, 0.587, 0.114, 0, 0, 0, 0, 0, 0 ], [ 0, 0, 0, 0.299, 0.587, 0.114, 0.299, 0.587, 0.114 ] ],
'color': [ [ 1, 0, 0, 0, 0, 0, 0, 0, 0 ], [ 0, 0, 0, 0, 1, 0, 0, 0, 1 ] ],
'halfcolor': [ [ 0.299, 0.587, 0.114, 0, 0, 0, 0, 0, 0 ], [ 0, 0, 0, 0, 1, 0, 0, 0, 1 ] ],
'optimized': [ [ 0, 0.7, 0.3, 0, 0, 0, 0, 0, 0 ], [ 0, 0, 0, 0, 1, 0, 0, 0, 1 ] ],
}

# make_anaglyphBGR is a routine that blends two
# perspectives into a single image that will be perceived as
# 3d when using red/blue glasses
# it accepts:
# leftImage -- an image that corresponds to the left eye
# rightImage -- an image that corresponds to the right eye
# color -- a string that specifies a blending strategy by indexing into mixMatrices
# result -- an image buffer that will hold the resulting image
# Note that make_anaglyphBGR presumes that the image is in BGR format
def make_anaglyphBGR(leftImage, rightImage, color, result):
# use the color argument to select a color separation formula from mixMatrices
m = mixMatrices[color]

# split the left and right images into separate blue, green and red images
lb,lg,lr = cv2.split(np.asarray(leftImage[:,:]))
rb,rg,rr = cv2.split(np.asarray(rightImage[:,:]))
resultArray = np.asarray(result[:,:])
resultArray[:,:,0] = lb*m[0][6] + lg*m[0][7] + lr*m[0][8] + rb*m[1][6] + rg*m[1][7] + rr*m[1][8]
resultArray[:,:,1] = lb*m[0][3] + lg*m[0][4] + lr*m[0][5] + rb*m[1][3] + rg*m[1][4] + rr*m[1][5]
resultArray[:,:,2] = lb*m[0][0] + lg*m[0][1] + lr*m[0][2] + rb*m[1][0] + rg*m[1][1] + rr*m[1][2]


def main():
# create to camera capture objects
camRight = cv.CaptureFromCAM(0)
camLeft = cv.CaptureFromCAM(1)

# Set capture properties for the two cameras
cv.SetCaptureProperty(camLeft, cv.CV_CAP_PROP_FRAME_WIDTH, 320)
cv.SetCaptureProperty(camLeft, cv.CV_CAP_PROP_FRAME_HEIGHT, 240)
cv.SetCaptureProperty(camRight, cv.CV_CAP_PROP_FRAME_WIDTH, 320)
cv.SetCaptureProperty(camRight, cv.CV_CAP_PROP_FRAME_HEIGHT, 240)

# make a named window to hold the resulting anaglyphic image
cv.NamedWindow ("anaglyph")

# call cv.QueryFrame(camLeft) to grab an image
# then use cv.CreateImage to create anaglyphImage so that it's properly sized to hold the result
leftImage=cv.QueryFrame(camLeft)
anaglyphImage=cv.CreateImage(cv.GetSize(leftImage),8,3)

# select an anaglyph color separation strategy
colorMatrix = 'optimized'

loop = True
while loop :
# grab left and right image
leftImage=cv.QueryFrame(camLeft);
rightImage=cv.QueryFrame(camRight);

# make an anaglyph (note that we presume the image is in BGR format)
make_anaglyphBGR(leftImage, rightImage, colorMatrix, anaglyphImage)

# display the anaglyph image
cv.ShowImage("anaglyph",anaglyphImage);

char = cv.WaitKey(10)
loop = (char == -1)

cv2.imwrite('anaglyph_sample.png', np.asarray(anaglyphImage[:,:]))

if __name__=="__main__":
main()

[Sean McBeth]

Speaking of stereo imagery, I recently purchased one of these: http://www.amazon.com/gp/product/B001NXDGFY/ref=oh_aui_detailpage_o03_s00?ie=UTF8&psc=1

They are probably the cheapest stereo webcam you'll be able to find. It isn't great, but it isn't the worst, either, especially for only $30. The device just shows up as two webcams, so connect to both however you do.

I'm going to strap mine to the back of my Oculus Rift.

--
To post to this group, send email to hive76-d...@googlegroups.com
To unsubscribe send email to hive76-discuss...@googlegroups.com
For more awesome goto http://groups.google.com/group/hive76-discussion?hl=en
---
You received this message because you are subscribed to the Google Groups "Hive76 Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hive76-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sean McBeth]

Oh, BTW, also using OpenCV, though in C#, not Python. For anyone who has never done anything involving programming of a webcam, OpenCV makes it incredibly easy.

[pezman]

OpenCV and PIL (which is probably exclusively python) make a lot of image and video stuff easy.  

The only issue I had with OpenCV is that the datatypes involved are not that intuitive, and the anaglyph routine took bit more research and trial/error than usual.  The fact that numpy.asarray() returned an array wrapper that accessed the contents of the images by reference was not at all apparent .. in fact, I was actually just tinkering with the routine, taking baby steps and suddenly it worked (because of the "array wrapper around a reference" behavior of numpy.asarray().

Now that I know this, I'm sure I could find documentation that says as much, but none of the code samples that I found hinted at it.

@Sean -- Any easy way to zing the video to web page so that it can be viewed as a live stream?  Also, I am kind of interested in doing something similar to stream stereo images so that they can be seen using Google Cardboard.  Something like your old app that switched between various viewing modes for stereo pictures, only adapted for video, would be awesome.

[Sean McBeth]

It's possible to do live video streams directly in JavaScript. You need the MediaStream API. That's actually how I built the still-image thinger. I created a <video> element that uses a media stream as its source, then rendered the video element to a <canvas> element with the 2D graphics context. Then, I was able to get at the raw pixel data through the graphics context.

https://developer.mozilla.org/en-US/docs/Web/API/MediaStream_API

But, you're not going to have access to OpenCV from JS. However, you might be able do the processing server-side, in Python, and stream it to the client side over WebRTC, which is one of the use-cases of the MediaStream API.

Or you could maybe transpile OpenCV to JS using Emscripten. That is probably a lot more work.