Understanding gaze data output by pupil labs

[benjami...@gmail.com]

Dear community,

I am having a difficult time trying to interpret the output data from pupil, and can't seem to find documentation or a description of how this is all working.

I want to export world-space gaze data using zeroMQ for both eyes separately (so that I can further process to extract depth-of-gaze in real time). The output data from zeroMQ looks like this:

"
Gaze

norm_pos:(0.1941707819353955, 0.6768313420701656)

confidence:0.92664009946

base:[{'diameter': 73.97959297850755, 'confidence': 0.9984733577891617, 'circle3D': {'radius': 2.2543469285733115, 'center': (-5.5309673942636035, 0.1646849204894858, 37.92201606732658), 'normal': (-0.6342652365055154, -0.38617996229185925, -0.6697556617788765)}, 'norm_pos': (0.3598880310422482, 0.4931967542958361), 'modelConfidence': 0.998958043316315, 'phi': -2.3289850594191694, 'id': 0, 'sphere': {'radius': 12.0, 'center': (2.0802154438025817, 4.798844467991796, 45.959084008673095)}, 'diameter_3D': 4.508693857146623, 'timestamp': 186472.401110648, 'theta': 1.1743096481916662, 'ellipse': {'axes': (42.64643259314175, 73.97959297850755), 'angle': 28.405199807645793, 'center': (230.32833986703884, 243.2655579379987)}, 'method': '3D c++', 'modelID': 592}, {'diameter': 104.67934401649987, 'confidence': 0.8548068411309877, 'circle3D': {'radius': 2.156813389149024, 'center': (-1.4389641595129572, 4.136601017490061, 25.584886122469758), 'normal': (-0.22770812502176163, 0.27977080982234676, -0.9326721309069024)}, 'norm_pos': (0.4469278674419142, 0.29336559495620473), 'modelConfidence': 0.9984700894778902, 'phi': -1.8102577537377196, 'id': 1, 'sphere': {'radius': 12.0, 'center': (1.2935333407481824, 0.7793512996218998, 36.776951693352586)}, 'diameter_3D': 4.313626778298048, 'timestamp': 186472.41006865902, 'theta': 1.8543517045440048, 'ellipse': {'axes': (91.42570484136908, 104.67934401649987), 'angle': -55.32522887505306, 'center': (286.0338351628251, 339.18451442102173)}, 'method': '3D c++', 'modelID': 446}]
timestamp:186472.40559
"

In the "base" there are two sets of data, one for each eye or "id". The 'norm_pos' in the "base" data seems to be raw pupil position from the eye-facing cameras as far as I can tell from looking at the data. The 'norm_pos' that is listed outside and above the "base" seems to be the actual gaze data in world space, but is it somehow calculated from both eyes?

How can I get the world space gaze data for each eye individually?

I would also like to know what all the other parameters listed actually refer to. Is there a page that documents and describes all these? Such as 'angle', 'circle3D', 'sphere' etc.

I apologise if there is a good page of documentation out there that I have missed. If so please point me in the right direction.

[Rafael Picanço]

Hi Benjamin,

Pupil already implemented what you are describing as "world space calculated from both eyes". So, yes, the outmost norm_pos refers to the world space after what pupil calls "binocular gaze mapping". Please, let me know if your job is to make a custom implementation. I did not investigate the 3d tracking in depth, not yet, so I could't help you on this.

base:
[

# normalized gaze position at the world 2d space
norm_pos:(0.1941707819353955, 0.6768313420701656)

# overall quality of the pupil detection (mean?)
confidence:0.92664009946

  # eye camera 0
  {
    # pupil diamenter in pixels
    'diameter': 73.97959297850755,

    # quality of the pupil detection
    'confidence': 0.9984733577891617,

    # ?

    'circle3D': {
      'radius': 2.2543469285733115,
      'center': (-5.5309673942636035, 0.1646849204894858, 37.92201606732658),
      'normal': (-0.6342652365055154, -0.38617996229185925, -0.6697556617788765)
    },

    # normalized pupil position at the eye 0 2d space
    'norm_pos': (0.3598880310422482, 0.4931967542958361),

    # quality of the 3d tracking
    'modelConfidence': 0.998958043316315,

    # ?
    'phi': -2.3289850594191694,

    # eye camera id
    'id': 0,

    # parameters of the 3d sphere of the eye 0

    'sphere': {
      'radius': 12.0,
      'center': (2.0802154438025817, 4.798844467991796, 45.959084008673095)
    },

    # ?
    'diameter_3D': 4.508693857146623,

    # timestamp 0
    'timestamp': 186472.401110648,

    # ?
    'theta': 1.1743096481916662,

    # parameters of the detected pupil ellipse (https://en.wikipedia.org/wiki/Ellipse#/media/File:Ellipse_Polar_center.svg)

    'ellipse': {
      'axes': (42.64643259314175, 73.97959297850755),
      'angle': 28.405199807645793,
      'center': (230.32833986703884, 243.2655579379987)
    },

    # detection method
    'method': '3D c++',

    # 3d eye model id
    'modelID': 592
  },

  # eye camera 1
  {

    'diameter': 104.67934401649987,
    'confidence': 0.8548068411309877,
    'circle3D': {
      'radius': 2.156813389149024,
      'center': (-1.4389641595129572, 4.136601017490061, 25.584886122469758),
      'normal': (-0.22770812502176163, 0.27977080982234676, -0.9326721309069024)
    },
    'norm_pos': (0.4469278674419142, 0.29336559495620473),
    'modelConfidence': 0.9984700894778902,
    'phi': -1.8102577537377196,
    'id': 1,
    'sphere': {
      'radius': 12.0, 'center': (1.2935333407481824, 0.7793512996218998, 36.776951693352586)
    },
    'diameter_3D': 4.313626778298048,
    'timestamp': 186472.41006865902,
    'theta': 1.8543517045440048,
    'ellipse': {
      'axes': (91.42570484136908, 104.67934401649987),
      'angle': -55.32522887505306,
      'center': (286.0338351628251, 339.18451442102173)
    },
    'method': '3D c++',
    'modelID': 446
  }
]

Hope it helps,
Rafael

[benjami...@gmail.com]

Thank you for your reply.

That clears things up somewhat, thank you. However, it seems that the information I need is not there. Namely, the world-space tracked positions of each eye individually.

The outmost "norm_pos", is the world space positions using both eyes, of 'binocular gaze mapping', as you say, and the other "norm_pos" values are in the space of the eye cameras. There is no information about the calibration itself, so it seems it is not currently possible to get each eye's world space positions individually, which I need to be able to calculate the gaze depth.

Does anyone know how I can get this data out of the pupil tracker?

One solution would be to run two instances of the pupil capture software with one eye camera each, and they would then output each eye's world space separately, but this seems like a very inelegant solution.

Ben

[Rafael Picanço]

Hi Benjamin,

Calibration itself can be found at the [cal_pt_cloud.npy] file.

As you noted, Pupil already implemented a monocular gaze mapper. Well, your idea seems perfect to me, it does not require any new functionality and, as far as I know, is ready to go for (offline) post facto analysis. Take a look here:

https://github.com/pupil-labs/pupil/blob/v0.7.3/pupil_src/capture/main.py#L53

All you need to do is to open each camera separately and export the data you need.

I think you could implement a custom gaze mapper inheriting some code for online tracking though...

Hope it helps,
Rafael

[Ben Outram]

Dear Rafael,

I am hoping to get the data in real time, rather than analysing it post recording, as I want to use the depth gaze as input to control something else in real time.

I will let you know how my solution works out.

Ben