glEnable(GL_POLYGON_SMOOTH)

[Jonas Geduldig]

I am animating a simple textured rectangle. The rectangle is
being drawn over a completely transparent background for the
purpose of compositing over video.

My problem arises when I try to antialias the edges of the
rectangle. If I call glEnable(GL_POLYGON_SMOOTH) I get what
appear to be tessalation lines drawn accross the textture. The
lines change depending on how the viewport is clipping the
rectangle. If the backgroun is red, the lines are red.

Any suggestions? Here is my init() and display()...

void init()
{
glClearColor(1, 0, 0, 0);
glShadeModel(GL_FLAT);
glDisable(GL_DITHER);
glEnable(GL_TEXTURE_2D);
glPolygonMode(GL_FRONT_AND_BACK, GL_FILL);

// for antialiasing
glEnable(GL_DEPTH_TEST);
glHint(GL_POLYGON_SMOOTH_HINT, GL_NICEST);

// for transparent texture
glEnable(GL_BLEND);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
}

void display()
{
glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

glPushMatrix();
glTranslatef(x, y, z);
glRotatef(xangle, 1., 0., 0.);
glRotatef(yangle, 0., 1., 0.);
glRotatef(zangle, 0., 0., 1.);

// draw antialiased textured rectangle
glEnable(GL_POLYGON_SMOOTH);
glCallList(MY_RECTANGLE);
glDisable(GL_POLYGON_SMOOTH);

glPopMatrix();
}

[Julien]

In case of multiple polygons with some on others, you have to order
your polygons, and use the appropriate BlendFunc.

But if you want full scene anti-aliasing, it's easier to do
it with the accumulation buffer.

Can you put your video background before the polygon rendering ?

julien.

--
______________________________ _______________________________
Julien ROGER | Institut Image / ENSAM
Tel: +33 (0) 385.424.350 | http://www.ai.cluny.ensam.fr

[Jonas Geduldig]

Julien <ro...@ai.cluny.ensam.fr> wrote:
: In case of multiple polygons with some on others, you have to order

: your polygons, and use the appropriate BlendFunc.

I have only one polygon: it's a rectangle.

: But if you want full scene anti-aliasing, it's easier to do

: it with the accumulation buffer.

All I want is the edge of my textured rectangle antialiased. As a
last resort I'll try antialiasing the full scene.

: Can you put your video background before the polygon rendering ?

I'm not using opengl to render the video background. I'm using opengl to
create a series of targas with transparent background. Then I use
a special card to 'play' the targas over live video.

I've seen plenty of examples using GL_POLYGON_SMOOTH. They look
fairly simply. Does it just not work with a transparent background?

[Julien]

Have you got an alpha layer in your opengl window (glutInitDisplayMode)
?

julien.

[Jonas Geduldig]

Julien <ro...@ai.cluny.ensam.fr> wrote:
: Have you got an alpha layer in your opengl window (glutInitDisplayMode)
: ?

I'm not using glut. But, yes, I do have an alpha layer.