[osg-users] Stencil and render to texture

[Nicolas Baillard]

Hello everyone.

I'm having trouble using the stencil buffer while doing a render to texture. My purpose is to apply a black mask in front of my scene, as if I was looking at the world through a crack in a wall. I managed to achieve this effect using the stencil before. But back then I was doing direct rendering on the screen. Now I'm trying to render to a texture and it doesn't work any more.

Here is what I do:
First I create the mask I want to draw into the stencil buffer. Here it's a simple quad. I set a stencil function on it so it writes 1 into the stencil buffer for each pixel.

Code:

osg::Vec3Array *vertices = new osg::Vec3Array();
vertices->push_back(osg::Vec3(0.25, 0.25, 0.0));
vertices->push_back(osg::Vec3(0.75, 0.25, 0.0));
vertices->push_back(osg::Vec3(0.75, 0.75, 0.0));
vertices->push_back(osg::Vec3(0.25, 0.75, 0.0));
_stencil_geometry = new osg::Geometry();
_stencil_geometry->setVertexArray(vertices);
_stencil_geometry->addPrimitiveSet(new osg::DrawArrays(GL_QUADS, 0, 4));
_stencil_node = new osg::Geode();
_stencil_node->addDrawable(_stencil_geometry);
_stencil_node->getOrCreateStateSet()->setMode(GL_LIGHTING, osg::StateAttribute::OFF);
_stencil_node->getOrCreateStateSet()->setMode(GL_DEPTH_TEST, osg::StateAttribute::OFF);
osg::Stencil* stencilSet = new osg::Stencil();
stencilSet->setFunction(osg::Stencil::ALWAYS, 1, ~0u);
stencilSet->setOperation(osg::Stencil::REPLACE, osg::Stencil::REPLACE, osg::Stencil::REPLACE);
_stencil_node->getOrCreateStateSet()->setAttributeAndModes(stencilSet, osg::StateAttribute::ON | osg::StateAttribute::OVERRIDE);




Then I create a camera to render my mask into the stencil buffer. I make it so that this camera clears the whole stencil to 0 before rendering.

Code:

_stencil_camera = new osg::Camera();
_stencil_camera->setViewport(0, 0, _w, _h);
_stencil_camera->setRenderOrder(osg::Camera::PRE_RENDER, 1);
_stencil_camera->setClearMask(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);
_stencil_camera->setClearColor(Color(0., 0., 0., 1.));
_stencil_camera->setClearStencil(0);
_stencil_camera->setReferenceFrame(osg::Transform::ABSOLUTE_RF);
_stencil_camera->setProjectionMatrixAsOrtho2D(0., 1., 0., 1.);
_stencil_camera->setRenderTargetImplementation(osg::Camera::FRAME_BUFFER_OBJECT);
_stencil_camera->attach(osg::Camera::PACKED_DEPTH_STENCIL_BUFFER, GL_DEPTH_STENCIL_EXT);
_stencil_camera->addChild(_stencil_node);




Finally I create the camera that will render my scene and I attach the stencil camera to it as a child. I set the render order so that this camera is rendered after the stencil camera. I configure this camera to render into a texture and I add it to the view.

Code:

_display_buffer = new osg::Texture2D();
_display_buffer->setTextureSize(_w, _h);
_display_buffer->setInternalFormat(GL_RGB);
_display_buffer->setFilter(osg::Texture2D::MIN_FILTER, _min_filter_mode);
_display_buffer->setFilter(osg::Texture2D::MAG_FILTER, _mag_filter_mode);

_display_camera = new osg::Camera();
_display_camera->setGraphicsContext(context);
_display_camera->setViewport(0, 0, _w, _h);
_display_camera->setRenderOrder(osg::Camera::PRE_RENDER, 2);
_display_camera->setClearColor(Color(0., 0., 0., 1.));
_display_camera->setClearMask(0);
_display_camera->setReferenceFrame(osg::Transform::ABSOLUTE_RF);
_display_camera->addChild(_stencil_camera);
_display_camera->addChild(_my_scene_root_node);
_display_camera->setProjectionMatrixAsOrtho2D(0., 1., 0., 1.);
_display_camera->setRenderTargetImplementation(osg::Camera::FRAME_BUFFER_OBJECT);
_display_camera->attach(osg::Camera::COLOR_BUFFER, _display_buffer);
_display_camera->attach(osg::Camera::PACKED_DEPTH_STENCIL_BUFFER, GL_DEPTH_STENCIL_EXT);
_view->addSlave(_display_camera, false);




Finally I set a stencil test on the whole scene. The scene shall only be drawn on pixels where the stencil is not 0 (so whereever my mask was drawn).

Code:

osg::Stencil* stencilTest = new osg::Stencil();
stencilTest->setFunction(osg::Stencil::NOTEQUAL, 0, ~0u);
stencilTest->setOperation(osg::Stencil::KEEP, osg::Stencil::KEEP, osg::Stencil::KEEP);
_my_scene_root_node->getOrCreateStateSet()->setAttributeAndModes(stencilTest, osg::StateAttribute::ON);




Well, this doesn't work. All I get is a black texture (or whatever clear color I set on my display camera). When I do the very same thing without the render to texture part it works. But with the render to texture, it's as if my mask was never drawn into the stencil buffer. What's weird is that if I configure my stencil camera to clear the stencil buffer to a value different that 0 then my scene is rendered to the texture. So it proves that the stencil works. But for some reasons I don't understand my mask seems to never be drawn into it.

Could someone please tell me what I did wrong ?

Regards,
Nicolas

------------------
Read this topic online here:
http://forum.openscenegraph.org/viewtopic.php?p=65335#65335





_______________________________________________
osg-users mailing list
osg-...@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

[Sebastian Messerschmidt]

Hello Nicolas,

From what I see, you are simply using two different depth- and stencil
buffers.
Try to share the depth buffer between the cameras by explicitly ataching
a depth texture. This should also share the stencil.
Is the depth buffer working as expected? Could you try if writing to the
depth buffer in the "stencil" pass actually is visible in the second pass.

cheers
Sebastian

[Nicolas Baillard]

Hello Sebastian. Thank you very much for your answer.

I can create a depth texture and share it between the two cameras. It works. However it doesn't share the stencil buffer.

I tried creating a texture for the stencil and attaching it to both camera, just like I did for the depth buffer. Like this :

Code:

osg::Texture2D* _display_stencil = new osg::Texture2D();
_display_stencil->setInternalFormat(GL_STENCIL_INDEX8_EXT);
_display_stencil->setFilter(osg::Texture2D::MIN_FILTER, osg::Texture2D::NEAREST);
_display_stencil->setFilter(osg::Texture2D::MAG_FILTER, osg::Texture2D::NEAREST);
_stencil_camera->attach(osg::Camera::STENCIL_BUFFER, _display_stencil, 0, 0, false, 0, 0);
_display_camera->attach(osg::Camera::STENCIL_BUFFER, _display_stencil, 0, 0, false, 0, 0);



But it gave me an error message "RenderStage::runCameraSetUp(), FBO setup failed, FBO status= 0x8cd6", and it doesn't work of course.

I also tried creating a common texture for both the stencil and the depth buffer, using GL_DEPTH_STENCIL_EXT or GL_DEPTH24_STENCIL8_EXT format, and attaching it to both cameras, but it gave me the same error.

------------------
Read this topic online here:

http://forum.openscenegraph.org/viewtopic.php?p=65346#65346

[Sebastian Messerschmidt]

Hi Nicolas,

The error indicates an incomplete frame buffer setup.
As far as I remember You need to use the PACKED_DEPTH_STENCIL_BUFFER, as
stencil buffers are not attachable themselves (you cannot directly load
back a stencil buffer) but only in combination with the depth buffer.
Take a look at the osgpackeddepthstencil example, it might contain some
additional help.
My own MRT/FBO code is to convoluted to be of any use directly, but it
seems I use PACKED_DEPTH_STENCIL_BUFFER as well.

Cheers
Sebastian