It looks like the spiral staircase gets only one region which overlaps
itself. This in turn makes the triangulation fail, because it assumes
2D input.
Is it possible to get this level geometry and the exact generation
parameters which make this problem appear?
Issue 2 should not happen. Someone else is having similar issue:
http://code.google.com/p/recastnavigation/issues/detail?id=101
--mikko
I think was able to reproduce the spiral stair case on a simple test level.
Unfortunately, I don't have a quick fix for it. In the meanwhile you
can call rcBuildRegionsMonotone() instead of rcBuildRegions() to keep
going. It uses slightly different algorithm to create the areas and
does not suffer from the spiral staricase problem. When using the
monotone partitioning the resulting polygons on large open areas areas
are not as nice as with watershed partitioning, which can affect the
resulting path (more detours). If you are using small tile size, the
difference is really small.
I added bug http://code.google.com/p/recastnavigation/issues/detail?id=102
so you can follow the progress. I cannot promise any quick fix as
quite a bit of the code assumes that that case should never happen.
Luckily, the case is simple to catch and should not happen often, so I
might be able to craft some special case code for that.
--mikko