hand landmarks documentation

[laurent bouis]

Hi,

I am working on hands detection, I need to translate the hand coordinates in unity 3d coordinates (to show the hands using Unity ARfoundation).

Where can I find some information about what the hand landmark coordinates (x,y,z) represent? I haven't found information about this in the documentation (only that x,y are normalized), some in slack, but nothing very clear, especially about the z (depth).

Best info I could find so far was at https://translate.google.co.jp/translate?hl=ja&sl=auto&tl=en&u=https%3A%2F%2Fdev.classmethod.jp%2Farticles%2Fmediapipe-analyze-z-cood-in-landmark%2F

Thanks.

Laurent

[Brian Miller]

For starters, read through the comments on the following page, although it is about the Java Android version of MediaPipe

instead of Unity, it may still be relevant:

https://github.com/google/mediapipe/issues/742#issuecomment-639104199

[laurent bouis]

Thanks Brian for the help. Very useful link. But unfortunately at the end I am still not clear on whether I can get a value for the Z coordinate that is useful to know the location of the hand and fingers (something that I can translate into a 3d coordinate in unity basically; I got a good understand of how to get x and y actual coordinates but not z).

Few questions:

a- Looks like z is relative to the wrist. Is the wrist location the hand landmark with index 0?

b- Not sure what this means mathematically (even after reading a bit about weak projections online), what mathematical formula I can derive from this description:

"it is scaled proportionally along with x and y (via weak projection), and expressed in the same units as x & y.

There is a root landmark point (wrist) that all the other landmark depths are relative to (again normalized via weak projection w.r.t. x & y)."

c- Based on the comment at https://github.com/google/mediapipe/issues/99 and https://github.com/google/mediapipe/issues/156, I get the feeling that if it's just not possible to calculate the actual Z coordinates. Is my understanding correct that it has no unit basically?

d- Is there a way to get the value of the Zavg mentioned in those tickets so that I can roughly know the position of the hand at least in 3d coordinates?

The whole thing is still quite confusing to me, I haven't found any definitive answers after reading those tickets so far; lots of assumptions not confirmed or questions not answered yet.

Laurent