[osg-users] Video capture with 3D augmented reality

[benedikt naessens]

I want to record a video and put 3D augmented reality on top of each frame of the AVI. Due to the fact that I don't want to skip frames, I store all AVI frames in memory during recording and after each video frame has been captured, I also store the view and projection matrices of the 3D view that is currently applicable on this video frame.

The input of the camera is 640 x 480 and my application usually renders in 1280 x 1024 windows. The video frames can be retrieved from an array of unsigned char arrays (stored in m_RingBuffer)

I want to achieve the rendering of the 3D data on top of the video frames in a post-processing step (which is part of the work done in a thread called VideoPostRecThread).

I follow this strategy:

1) I set up a pre-rendering HUD camera (m_pHudCamera) which looks at a quad geometry with one of the video frames as a texture (m_RenderTexture)
2) The HUD camera is the child of a snapshot camera (m_pSnapshotcamera) . The snapshot renders the 3D data. The output of the snapshot camera should go back to the video frame, but I also store it temporarily in an image (m_SnapshotImage). For this, I disabled the GL_COLOR_BUFFER_BIT clear mask of the snapshot camera, to make sure the rendered outputs of the HUD camera are not cleared.

I also use two callbacks:
1) a pre-draw callback applied on the HUD camera: the texture of the quad geometry (m_RenderTexture) is updated each time with a new frame. Name: m_UpdateCallback (instance of the TexturePreDrawCallback struct).
2) a post-draw callback: my thread is blocked until all the 3D data is rendered on top of the HUD contents; the callback unblocks my thread (using a mutex called m_SnapshotMutex). After this, I can do some post-processing (like for example saving the AVI or updating my GUI that another frame has been post-processed). Name: m_VideoCallback (instance of the VideoPostDrawCallback struct)

Here are my callback definitions:

Code:

struct VideoPostDrawCallback : public osg::Camera::DrawCallback
{
VideoPostDrawCallback() {}

virtual void operator() ( osg::RenderInfo& renderInfo) const
{
renderingCompleted();
}

boost::signals2::signal<void(void)> renderingCompleted;
};

struct TexturePreDrawCallback : public osg::Camera::DrawCallback
{
TexturePreDrawCallback() {}

virtual void operator() ( osg::RenderInfo& renderInfo) const
{
updateCamera();
}

boost::signals2::signal<void(void)> updateCamera;
};

And here is the definition of my thread class

Code:

class VideoPostRecThread : public VideoRecWithArThread
{
Q_OBJECT

friend class VideoPostDrawCallback;

public:
VideoPostRecThread (boost::shared_ptr<IDSCameraManager> pCamMgr, unsigned int maxFrames, QObject *parent = NULL);
~VideoPostRecThread ();

void renderingCompleted();
void updateTextureCamera();
private:
virtual void postProcess();
void setupImages();
void setupHudCamera();
void setupSnapshotCamera();

osg::ref_ptr<osg::Camera> m_pSnapshotcamera;
osg::ref_ptr<osg::Camera> m_pHudCamera;
osg::ref_ptr<osg::Image> m_TextureImage;
osg::ref_ptr<osg::Image> m_SnapshotImage;
osg::ref_ptr<osg::Texture2D> m_RenderTexture;
osg::ref_ptr<osg::Geode> m_QuadGeode;
osg::ref_ptr<VideoPostDrawCallback> m_VideoCallback;
osg::ref_ptr<TexturePreDrawCallback> m_UpdateCallback;

QWaitCondition m_SnapshotCondition;
QMutex m_SnapshotMutex;

unsigned int m_CurrentArFrameIndex;
};

Here is the implementation of the VideoPostRecThread class.

Code:

void VideoPostRecThread ::setupImages()
{
m_SnapshotImage = new Image();
m_SnapshotImage->allocateImage(1280,1024,1, GL_RGBA, GL_UNSIGNED_BYTE);
m_TextureImage = new Image();
}

void VideoPostRecThread ::setupHudCamera()
{
// Create the texture to render to
m_RenderTexture = new osg::Texture2D;
m_RenderTexture->setDataVariance(osg::Object::DYNAMIC);
m_RenderTexture->setInternalFormat(GL_RGBA);

osg::ref_ptr<osg::Geometry> screenQuad;
screenQuad = osg::createTexturedQuadGeometry(osg::Vec3(),
osg::Vec3(1280, 0.0, 0.0),
osg::Vec3(0.0, 1024, 0.0));
m_QuadGeode = new osg::Geode;
m_QuadGeode->addDrawable(screenQuad.get());

m_pHudCamera = new osg::Camera;
m_pHudCamera->setReferenceFrame(osg::Transform::ABSOLUTE_RF);
m_pHudCamera->setRenderOrder(osg::Camera::PRE_RENDER);
m_pHudCamera->setClearMask(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
m_pHudCamera->setProjectionMatrix(osg::Matrix::ortho2D(0, 1280, 0, 1024));
m_pHudCamera->setViewMatrix(osg::Matrix::identity());
m_pHudCamera->setViewport(0,0,1280,1024);
m_pHudCamera->addChild(m_QuadGeode);

m_UpdateCallback = new TexturePreDrawCallback();
m_UpdateCallback->updateCamera.connect(boost::bind(&PvaitvVideoRecThread::updateTextureCamera,this));
m_pHudCamera->setPreDrawCallback(m_UpdateCallback);
}

void VideoPostRecThread ::setupSnapshotCamera()
{
m_pSnapshotcamera = new osg::Camera();
m_pSnapshotcamera->setReferenceFrame(osg::Transform::ABSOLUTE_RF);
//m_pSnapshotcamera->setRenderOrder(osg::Camera::POST_RENDER);
m_pSnapshotcamera->setClearMask(GL_DEPTH_BUFFER_BIT);
//m_pSnapshotcamera->setDrawBuffer(GL_BACK);
//m_pSnapshotcamera->setReadBuffer(GL_BACK);
//m_pSnapshotcamera->setRenderTargetImplementation(osg::Camera::FRAME_BUFFER_OBJECT);
m_pSnapshotcamera->attach(osg::Camera::COLOR_BUFFER, m_SnapshotImage,0,0);
osg::ref_ptr<osg::Node> pScene = getSceneManager()->getSceneData();
m_pSnapshotcamera->addChild(pScene);
getSceneManager()->getSceneRoot()->addChild(m_pSnapshotcamera.get());
m_pSnapshotcamera->addChild(m_pHudCamera);

m_VideoCallback = new VideoPostDrawCallback();
m_VideoCallback->renderingCompleted.connect(boost::bind(&PvaitvVideoRecThread::renderingCompleted,this));
m_pSnapshotcamera->setPostDrawCallback(m_VideoCallback);
}

void VideoPostRecThread ::postProcess()
{
setupImages();

setupHudCamera();
setupSnapshotCamera();

m_CurrentArFrameIndex = 0;

for (unsigned int i = 0; i < m_RecordedFrames; ++i)
{
m_SnapshotMutex.lock();
m_SnapshotCondition.wait(&m_SnapshotMutex);
m_SnapshotMutex.unlock();

osgDB::writeImageFile(*m_SnapshotImage, "file2.bmp" );

// Do some post-processing code
}

m_VideoCallback->renderingCompleted.disconnect_all_slots();
m_UpdateCallback->updateCamera.disconnect_all_slots();
m_pSnapshotcamera->setPostDrawCallback(NULL);
m_pHudCamera->setPreDrawCallback(NULL);

getSceneManager()->getSceneRoot()->removeChild(m_pHudCamera);
getSceneManager()->getSceneRoot()->removeChild(m_pSnapshotcamera);

// End step of the post processing
}

void VideoPostRecThread ::renderingCompleted()
{
m_SnapshotMutex.lock();
m_SnapshotCondition.wakeAll();
m_SnapshotMutex.unlock();
}

void VideoPostRecThread ::updateTextureCamera()
{
if (m_CurrentArFrameIndex < m_RecordedFrames)
{
m_TextureImage->setImage(m_VideoWidth,m_VideoHeight,1, 3, GL_RGBA, GL_UNSIGNED_BYTE,
(unsigned char*) m_RingBuffer[m_CurrentArFrameIndex], Image::NO_DELETE);
m_RenderTexture->setImage(m_TextureImage);

osg::StateSet* quadState = m_QuadGeode->getOrCreateStateSet();
quadState->setTextureAttributeAndModes(0, m_RenderTexture, osg::StateAttribute::ON);

float snapHorFov = 60.0, snapVerFov = 60.0;
float snapAspect = snapHorFov/snapVerFov;
m_pSnapshotcamera->setProjectionMatrixAsPerspective(snapVerFov, snapAspect,0.1, 40000);
m_pSnapshotcamera->setViewMatrix(m_ViewMatrices[m_CurrentArFrameIndex]);
//m_pSnapshotcamera->attach(osg::Camera::COLOR_BUFFER, m_SnapshotImage,0,0);

++m_CurrentArFrameIndex;
}
}

You can see a do a quick test each time the post draw callback has been called, by writing the contents of m_SnapshotImage to a BMP (file2.bmp).

Now, when I look at this file, I see that indeed the 3D data is on top of my video frame. Here are my questions now:

1) My GUI has a sidebar on the right hand side. The image has just gray pixels in the same area as my sidebar is.
2) Suppose I want to render to a 640 x 480 image instead of a 1280 x 1024 image, how do I do that ? I want to preserve the same contents, so just a resized format basically.
3) How can I avoid that also the images that are written to the video stream (and file2.bmp) appear on the screen ?
4) I see my video frame is flipped horizontally. This is probably due to the fact that OSG has the (0,0) coordinate in the left bottom of the screen. Since the end result in stored in the same format as the video stream, I would optimally flip the rendering of the 3D data (not the video frames/ HUD contents). How can I flip the rendering of my 3D data ?

Thank you!

Cheers,
Benedikt

------------------
Read this topic online here:
http://forum.openscenegraph.org/viewtopic.php?p=33172#33172

_______________________________________________
osg-users mailing list
osg-...@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

[benedikt naessens]

The weird thing is that it is not working with the frame buffer object render target (black image).

Should I consider a pbuffer ?

Thank you!

Cheers,
benedikt

------------------
Read this topic online here:

http://forum.openscenegraph.org/viewtopic.php?p=33190#33190