bug in dtNavMesh raycast?

[Cam]

I'm getting some unexpected results from dtNavMesh::raycast. I'm using
r143.

When I perform a raycast from a poly that is on the edge of the mesh
so the ray goes outside the nav mesh, t is always returned as 1,
rather than the distance along the ray at which the edge was hit.

The following code is run before returning:

if (!nextRef || !passFilter(filter, getPolyFlags(nextRef)))
{
// No neighbour, we hit a wall.

// Calculate hit normal.
const int a = segMax;
const int b = segMax+1 < nv ? segMax+1 : 0;
const float* va = &verts[a*3];
const float* vb = &verts[b*3];
const float dx = vb[0] - va[0];
const float dz = vb[2] - va[2];
hitNormal[0] = dz;
hitNormal[1] = 0;
hitNormal[2] = -dx;
vnormalize(hitNormal);

return n;
}

At this point nextRef is 0, segMax is -1 which is clearly wrong. t at
this point is 1.0.

I haven't read the code thoroughly enough to see why this is incorrect
other than getting weird results and noticing the internal state of
raycast seemed incorrect.

[Mikko Mononen]

Hi,

If you have repro case, can you print out the following data:

- nv
- contents of verts
- tmin, tmax
- segMin, segMax;

If possible, floats as decimal numbers as well as their actual hex
data* so that I can test the right data.

--mikko

*) Something like: float p = 1.0f; printf("0x%08x", *(unsigned int*)&p);

[Cam]

I've worked out the cause. It happens if the line segment doesn't hit
anything, that is if both ends are in the same poly.

My steps are:
1. Start Recast Demo
2. Open nav_test.obj and build as tile mesh with default settings
3. Select test nav mesh and Ray cast
4. Place both ends in the same poly

This returns 1 with a t value of 1.0. Also in the code I mention above
maxStep is -1 and used to index an array for calculating the hit
normal.

I guess I would expect raycast to return 0 in this case since it
didn't hit anything. Should intersectSegmentPoly2D be returning false
in this case?

On Mar 11, 9:09 pm, Mikko Mononen <memono...@gmail.com> wrote:
> Hi,
>
> If you have repro case, can you print out the following data:
>
> - nv
> - contents of verts
> - tmin, tmax
> - segMin, segMax;
>
> If possible, floats as decimal numbers as well as their actual hex
> data* so that I can test the right data.
>
> --mikko
>
> *) Something like: float p = 1.0f;  printf("0x%08x", *(unsigned int*)&p);
>

[Mikko Mononen]

Hi,

This is actually pretty much correct behavior of the function. The
return values are not super intuitive, though.

Using the nagative index is bug, though.

The API states the function values as follows. The only inconsistency
is the T value, the statement "0 if no hit" is not correct. T varies
from 0..1 along the ray.

// Finds intersection againts walls starting from start pos.
// Params:
// startRef - (in) ref to the polygon where the start lies.
// startPos[3] - (in) start position of the query.
// endPos[3] - (in) end position of the query.
// t - (out) hit parameter along the segment, 0 if no hit.
// hitNormal[3] - (out) normal of the nearest hit.
// filter - (in) path polygon filter.
// path - (out) visited path polygons.
// pathSize - (in) max number of polygons in the path array.
// Returns: Number of polygons visited or 0 if failed.
int raycast(dtPolyRef startRef, const float* startPos, const float*
endPos, dtQueryFilter* filter,
float& t, float* hitNormal, dtPolyRef*
path, const int pathSize);

The reason the return values are laid out that way is that it allows
the raycast to be used to create a path. For example in some cases it
can be beneficial to use raycast instead of findPath() to find short
pathts or to adjust the end of path when your target moves, or to
check of the path corridor can be straightened, etc.

Currently the way to test if ray hit at all or not is to test if the T
is close to 1.0, which is a bit silly I know.

Please add issue about the normal calculation, that definitely needs
fixing. The documentation could be updated too

--mikko

[Cam]

Ah OK, between the return value and the t value I was a bit confused
about what was going on. In my code it wasn't obvious that the line
segment was within a single poly which also threw me.

It would be good if a t of 0 meant no hit as then I only have to check
for that instead of t > 0 && t < 1 for a hit. Also, what does the hit
normal mean if there was no hit?

I've opened issue 55 about the negative array access and associated
raycast issues.

On Mar 11, 9:31 pm, Mikko Mononen <memono...@gmail.com> wrote:
> Hi,
>

[Mikko Mononen]

Hi,

0 should mean that the start location is "hit" or obscured, that is
the ray did not travel, some cases this happens too. The documentation
is kinda wrong in that case.

Maybe a clean solution would be to return FLT_MAX in case there was no
hit, so that t > 1.0f would mean no hit, the ray travelled as far as
it could. In case of no hit, the normal probably should be zero or
undefined.

Thanks for the issue, I will process it once we get past the crunch :)

--mikko