[osg-users] Proper way to display a HUD

[Chris Hidden]

Hello OSGers.

I have been trying to implement a HUD in my unorthodox application. I am currently displaying 3 models to the screen where I have everything else transparent. Thus the application works as an overlay on top of whatever other application the user wishes to run. To do this I am using the pbuffer in OSG, where I copy all the pixel info over into a struct and then pre multiply the alpha values of everything and then render it back to the screen.

Because of this I have a very specific way in which I set up my scene and camera:


Code:

osg::ref_ptr<osg::Referenced> windata = new osgViewer::GraphicsWindowWin32::WindowData(hWnd);

// Establish Viewer and Camera;
g_viewer = new osgViewer::Viewer();
g_camera = new osg::Camera;
g_camera = g_viewer->getCamera();

// Create and assignTraits for the Graphics Context
osg::ref_ptr<osg::GraphicsContext::Traits> traits = new osg::GraphicsContext::Traits;
initOSGTraits (*traits, windata);
osg::ref_ptr<osg::GraphicsContext> gc = osg::GraphicsContext::createGraphicsContext(traits.get());

//Set camera state
g_camera->setGraphicsContext (gc);
g_camera->setDrawBuffer (GL_NONE);
g_camera->setReadBuffer (GL_NONE);
g_camera->setProjectionMatrixAsPerspective (30.0f, (double)sWidth / (double)sHeight, 1.0, 1000.0);
g_camera->setViewport (new osg::Viewport(0, 0, sWidth, sHeight));
g_camera->setClearColor (osg::Vec4f(0.0f, 0.0f, 0.0f, 0.0f));
g_camera->setRenderOrder (osg::Camera::PRE_RENDER);
g_camera->setRenderTargetImplementation (osg::Camera::FRAME_BUFFER_OBJECT);

// Set camera direction and perspective
osg::Vec3d eye = osg::Vec3d(0, -100, 0); // position of the camera
osg::Vec3d center = osg::Vec3d(0, 0, 0); // where does the camera look.
osg::Vec3d up = osg::Vec3d(0, 0, 1); // the up vector of the camera
g_camera->setViewMatrixAsLookAt(eye, center, up);

// Establish PBuffer using OSG's image object. Attach it to the camera.
g_image = new osg::Image();
g_image->allocateImage(sWidth, sHeight, 1, GL_RGBA, GL_UNSIGNED_BYTE);
g_camera->attach(osg::Camera::COLOR_BUFFER, g_image);

// Set Callback for adding textures. Makes sharing textures possible.
osgDB::Registry::instance()->setReadFileCallback(new ReadAndShareImageCallback);
osgDB::Registry::instance()->getOrCreateSharedStateManager();

osg::ref_ptr<osg::Group> models = InitModels();
osg::ref_ptr<osg::Group> root = new osg::Group;
root->addChild(models);

// Run OSG optimizer for efficiency.
osgUtil::Optimizer optimizer;
optimizer.optimize(root.get());

//Set up threads to windows, getScene data and set GL mode for rescaling normals.
g_viewer->realize();
g_viewer->setSceneData(root.get());
root->getOrCreateStateSet()->setMode(GL_NORMALIZE, osg::StateAttribute::ON);






I tried using the example from the OSG for beginners book in chapter 7: Creating an HUD camera. I also tried to follow the example here (http://trac.openscenegraph.org/projects/osg//browser/OpenSceneGraph/trunk/examples/osghud/osghud.cpp). Unfortunately none of these seemed to work for me. In fact nothing showed up on my screen at all. I was thinking it may be because of the way I define my camera matrix. Maybe not, Im a bit too much of a newbie to know.

Anyways, I figured a HUD can just as well be a quad that is right up close to the camera. So I used this code :


Code:

osg::ref_ptr<osg::Geode> geod = new osg::Geode;
osg::ref_ptr< osg::Geometry> geom = new osg::Geometry;
osg::ref_ptr<osg::Vec3Array> verts = new osg::Vec3Array;
osg::ref_ptr<osg::Vec3Array> norms = new osg::Vec3Array;
osg::ref_ptr<osg::Vec4Array> colors = new osg::Vec4Array;

float depth = -99.0f;

verts->push_back(osg::Vec3((1.0f / 3.0f), depth, (1)));
verts->push_back(osg::Vec3((1), depth, (1)));
verts->push_back(osg::Vec3((1), depth, -(1)));
verts->push_back(osg::Vec3((1.0f / 3.0f), depth, -(1)));
norms->push_back(osg::Vec3(0.0, 0.0f, 0.0f));
colors->push_back(osg::Vec4(1.0f, 1.0f, 1.0f, 0.8f));
geom->setVertexArray(verts);
geom->setNormalArray(norms, osg::Array::BIND_OVERALL);
geom->setColorArray(colors, osg::Array::BIND_OVERALL);
geom->addPrimitiveSet(new osg::DrawArrays(GL_QUADS, 0, 4));
osg::ref_ptr<osg::StateSet> stateset = geom->getOrCreateStateSet();
stateset->setMode(GL_BLEND, osg::StateAttribute::ON);
stateset->setRenderingHint(osg::StateSet::TRANSPARENT_BIN);
geod->addDrawable(geom);




After playing around I realized that the 0,0 point of the quad was in fact right in the middle of my scene. But by using the z depth of -99 which is 1 point off my camera z depth I can use the [0,1] range to make it fit where I want. Thus I get a nice horizontal bar along the left side of the screen.

Now this feels like a somewhat unusual way of doing things. How would I do this in a more professional way or standardized way, I guess I should say? Also on a side note, I only get about 36 fps with this application at the moment. If anyone feels inspired to give me some tips on how to make this more efficient that would be most appreciated.

------------------
Read this topic online here:
http://forum.openscenegraph.org/viewtopic.php?p=60885#60885





_______________________________________________
osg-users mailing list
osg-...@lists.openscenegraph.org
http://lists.openscenegraph.org/listinfo.cgi/osg-users-openscenegraph.org

[Robert Osfield]

Hi Chris,

I have read your post but didn't spend time looking at the specific code implementation as it's hard to know what you are actually attempting to do - knowing exactly what you are trying to achieve is precursor to knowing whether the approach you are taking is appropriate or not.

My first question would be, when you say "whatever other application the user wishes to run" what do you actually mean?  It's so open ended to what this might be.  Are you talking other things on the desktop such as other programs that might be running?  Or is it you want a 3D overlap ontop of 2D UI elements?  Or just a 3D scene with various overlays onto of it all drawn in OpenGL/OSG.

Also when you say the osghud example doesn't work for you, what do you mean?  What you should see when you run osghud and have OpenSceneGraph-Data on your path so it can find the 3D dumptruck model is a 3D truck with a text overlay and transparent quad that form a HUD.

Robert.

[Chris Hidden]

Hi Robert

Thanks for your answer!

Ill try and refine what it is I'm doing. When our application is running we want OSG to display 3 models on the screen and nothing else. We want a user to be able to run any program they wish underneath it. This means that when our application is on a user could run, MS word, Google Chrome, a Game or any other application. Our program will always display its 3 models above everything. So far this works.

We also want to have a HUD or basically a side bar that runs the height of the screen on the right hand side of the screen. Maybe about 1/5 the width of the screen:

0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 # # # # #
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 # # # # #
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 # # # # #
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 # # # # #
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 # # # # #
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 # # # # #
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 # # # # #

If the above was the screen the # represent the side bar. Just a black ordinary side bar. Then it will be able to be hidden so it kind of slides away and then reappears again when needed (a specific key is hit, or event is fulfilled or whatever). On this side bar there will be text and maybe a looping animation. Not sure yet, its a place to display information.

I figured with my approach that I would just use a quad to represent that space as the HUD. When I tried to run the other examples I linked and or mentioned I was not able to make ANYTHING display. If the HUD was indeed transparent, I was never able to figure out how to make it non transparent. I could try again and play around with things but for the moment my current approach seems to give what I want. I don't see why I wound't be able to append text to it later, but appending another animation for example might be tricky. Im not sure. My approach however feels like a weird work around to a problem that may otherwise be solved in a more standard and efficient manner, hence my question :D

I hope this helps clarify my question :D

Thanks!

------------------
Read this topic online here:

http://forum.openscenegraph.org/viewtopic.php?p=60887#60887

[Robert Osfield]

Hi Chris,

Thanks for the more detailed explanation.  From what I can make out you are able to render the 3 models to a pbuffer than render this with transparency over the desktop OK, this part if functioning as intended.  Is this correct?

There isn't any way for me to know what is specifically wrong at your end with the next step. What I would recommend is taking a step back and just get your 3 models and your HUD rendering to conventional GraphicsWindow without the additional pbuffer stages.  Once you have got this rendering as you want then you can re-introduce the rendering to a pbuffer. 

The only thing I add to the general pbuffer/final desktop composition side is that copying data from the GPU to an CPU base osg::Image will be slow, I don't know what trick you are using to desktop composition but you'd get much better performance if you could avoid the round trip of retreaving image data from the GPU to CPU only to re-render it in some manner back to the GPU.  This round trip will stall the GPU and CPU and force them to run in series for much of the frame.  The best solution would be one where you can render directly to a buffer than can be reused directly on the GPU.  I can't comment on how to do this as I know nothing about Windows having not touched the OS for over a decade.

As for the HUD examples working.  I really can't comment much as the only information we have to go on is that they don't display anything.  If the examples aren't working then it'll either be an issue with the examples finding the data to display or a problem with the OpenGL driver.

Try running the OSG examples on another system.

Robert.

[Chris Hidden]

Hey Robert

>
> Thanks for the more detailed explanation. From what I can make out you are able to render the 3 models to a pbuffer than render this with transparency over the desktop OK, this part if functioning as intended. Is this correct?
>

Yes this is correct!

As for the HUD Ill try getting it working in a standard OSG graphics context first as you said. Probably the best idea.

As far as your suggestions for optimization. I think I understand. However I was under the impression that the osg::Image was the pbuffer.

In order to to my transparent screen trick I have a struct:


Code:

typedef struct
{
int width;
int height;
int pitch;
HDC hdc;
HBITMAP hBitmap;
BITMAPINFO info;
BYTE *pPixels;
} Scene;




Then I initiate the struct with some init values. Then the I do a memcpy from the pbuffer/image to the struct:


Code:

for (int i = 0; i < g_image->t(); ++i)
{
memcpy(&g_scene.pPixels[g_scene.pitch * i],
&g_image->data()[((sHeight - 1) - i) * (sWidth * 4)],
sWidth * 4);
}




Here g_image is my osg::image and g_scene is my struct. Once I have copied the data I redraw to my "layered window" . This method takes care of applying the necessary actions using BLENDFUNCTION and UpdateLayeredWindow.


Code:

void RedrawLayeredWindow()
{
// The call to UpdateLayeredWindow() is what makes a non-rectangular
// window possible. To enable per pixel alpha blending we pass in the
// argument ULW_ALPHA, and provide a BLENDFUNCTION structure filled in
// to do per pixel alpha blending.

HDC hdc = GetDC(g_hWnd);

if (hdc)
{
HGDIOBJ hPrevObj = 0;
POINT ptDest = { 0, 0 };
POINT ptSrc = { 0, 0 };
SIZE client = { g_scene.width, g_scene.height };
BLENDFUNCTION blendFunc = { AC_SRC_OVER, 0, 255, AC_SRC_ALPHA };
hPrevObj = SelectObject(g_scene.hdc, g_scene.hBitmap);

ClientToScreen(g_hWnd, &ptDest);
UpdateLayeredWindow(
g_hWnd,
hdc,
&ptDest,
&client,
g_scene.hdc,
&ptSrc,
0,
&blendFunc,
ULW_ALPHA);

SelectObject(g_scene.hdc, hPrevObj);
ReleaseDC(g_hWnd, hdc);
}
}




I suppose I could use something other than an OSG::Image to bind to the BYTE property of my struct. The thing is Im still quite new to a lot of this so I don't know how I would do that.

Let me know if Ive been unclear. Its also a challenge explaining this stuff for me as well :S

------------------
Read this topic online here:

http://forum.openscenegraph.org/viewtopic.php?p=60891#60891

[Sergey Kurdakov]

Hi Chris,

>I suppose I could use something other than an OSG::Image to bind to the BYTE property of my struct.

whatever you bind, the way you copy rendered results to layered window ( see your
memcpy which is exactly copying via system memory  )  will move video memory to system memory and this is a bottleneck

the suggestion was to avoid this step and perform whatever you want in video memory ( even if a copy will occur- the copy between video memory is much faster than a copy to system memory.

You might remember, I already sent you three approaches to avoid working via system memory ( maybe others will find this tip useful too in case of such an attempt ).

1) use windows DWM trick as in http://www.youtube.com/watch?v=4qWKSYWIqdk

the code  to look at how it is achived is here https://github.com/sphair/ClanLib/tree/master/Examples/GUI/GUI_Layered

the idea is here is that you render using DWM system calls  and then compose layered windows without old layered windows api which you try to use to achive result. Here  layering is achieved by exploiting  the fact that in new windows ( since Vista ) all windows first are rendered to offscreen surface, but then are composed to produce final screen result  - you will be limited here to any windows operation system above windows xp ( but it is already retired and Vista is also not widely used - so limiting yourself just to Win7+ will make all coding much easier ). BTW on linux the same result possibly could be achieved via wayland compositor.

You might compile ClanLib and see how that achieved - it is fairly easy and takes just few lines of code during creation of window and  the complete code to achive what you need is provided, so you will need just add the same code to osg window creation function ( and use appropriate flags elsewhere to use DWM hack, but still, as in this example it is fairly easy to achive ).

2) use http://msdn.microsoft.com/en-us/magazine/ee819134.aspx approach and then https://sites.google.com/site/snippetsanddriblits/OpenglDxInterop  to copy opengl rendered result  to Direct2D layered window in video memory  ( currently NV_DX_interop and NV_DX_interop2 are supported on almost all video cards ), that might not work, due to possible problems with very difficult path via Direct3D/Direct2D texture, but still it is possible that it will work and will give you better results than now

3) you might use rendering with OpenGL ES with osg ( though you would need to have your own model loader ) via  ANGLE to use http://msdn.microsoft.com/en-us/magazine/ee819134.aspx trick, as final rendered result in ANGLE might be a direct3d texture, which you could apply to Direct2D layered window.

all approaches are not easy to implement to novice, but still - in case you want to significantly improve fps - you should avoid working via system memory, maybe someone else can offer some other insights - but I think that there are hardly any more, so  you still will have to implement one of suggested approaches. And looks like the first suggested approach is most simple one and also can be possibly ported to linux, at least ClanLib team tries to have that example to run on linux too ,to significantly improve fps for the task you wish to solve.

Regards
Sergey