The point is that these characters gravity will always be the opposite to the normal of the static mesh they sit on.
If I specify from Engine the new coordinates system and the transformations so Recast interpret it as XZ plane with gravity being the Y vector, wont it work?
In Unreal Engine, they are transforming the plane XY (the plane they consider as ground) and their Z vector (gravity vector) to Recast/Detour coordinates system in RecastHelpers.cpp (functions as Unreal2RecastPoint, Unreal2RecastMatrix, Recast2UnrealPoint, Recast2UnrealMatrix...).
My proposition is to change this file to transform the plane XZ and the Y vector to your coordinates system, so UE generates the NavMesh on that plane instead of on the ground as they will consider their ground in the new coordinates I am specifying to your library. I might do this 6 times:
- Ground: Plane XY with Z vector up.
- Walls: Plane XZ with Y vector up, plane XZ with Y vector down, plane YZ with X vector up, plane YZ with X vector down.
- Ceilings: Plane XY with Z vector down.
The gravity system of the units is something I have managed myself in the movement component of these characters.