This constraint might actually define a manifold (to be checked), in which case it could also be implemented with its own retraction etc. as a factory.
From what I understand, you are considering 3x3 orthogonal matrices (let's say with determinant +1), with a single entry forced to zero (without loss of generality, let's say it is the bottom-left entry of the matrix that is zero).
Then:
* the first column is really just a unit-norm vector in the XY plane (a point on the unit circle in that plane).
* the second column is a unit norm vector orthogonal to the first column.
* the third column is entirely determined by the first two, simply as their cross product.
Thus, your set is in direct correspondence with { (u, v) : |u|^2 = 1, |v|^2 = 1, <u, v> = 0 } where u is in the XY plane of R^3 and v is in R^3.
The defining equation is:
0 = h(u, v) = (|u|^2 - 1, |v|^2 - 1, <u, v>)
The differential is:
Dh(u, v)[a, b] = (2<u, a>, 2<v, b>, <a, v> + <u, b>)
To show that the set under consideration is a smooth manifold, it is sufficient to show that for all (u, v) in the set the map Dh(u, v) is surjective, that is: for all z in R³, is it possible to choose a and b (where a in the in XY plane and b is free) in order to have Dh(u, v)[a, b] = z ?
If so, then it should be possible to write a factory file to optimize over that set directly, without constraints (other than being on the smooth manifold). This won't be "flexible" though: if you later need to be able to have more than one entry equal to zero, on larger orthogonal groups, etc., you mind find it easier to just proceed with a constrained-optimization approach.
It could be that there is some trouble specifically when v = (0, 0, 1) or v = (0, 0, -1), but it would require a check.
Best,
Nicolas